Decentralized Erasure Coding for Efficient Data Archival in Distributed Storage Systems

Abstract

Distributed storage systems usually achieve fault tolerance by replicating data across different nodes. However, redundancy schemes based on erasure codes can provide a storage-efficient alternative to replication. This is particularly suited for data archival since archived data is rarely accessed. Typically, the migration to erasure-encoded storage does not leverage on the existing replication based redundancy, and simply discards (garbage collects) the excessive replicas. In this paper we propose a new decentralized erasure coding process that achieves the migration in a network-efficient manner in contrast to the traditional coding processes. The proposed approach exploits the presence of data that is already replicated across the system and distributes the redundancy generation among those nodes that store part of this replicated data, which in turn reduces the overall amount of data transferred during the encoding process. By storing additional replicated blocks at nodes executing the distributed encoding tasks, the necessary network traffic for archiving can be further reduced. We analyze the problem using symbolic computation and show that the proposed decentralized encoding process can reduce the traffic by up to 56% for typical system configurations.