Design Considerations and Analysis of Multi-Level Erasure Coding in Large-Scale Data Centers
Multi-level erasure coding (MLEC) has seen large deployments in the field, but there is no in-depth study of design considerations for MLEC at scale. In this paper, we provide comprehensive design considerations and analysis of MLEC at scale. We introduce the design space of MLEC in multiple dimensions, including various code parameter selections, chunk placement schemes, and various repair methods. We quantify their performance and durability, and show which MLEC schemes and repair methods can provide the best tolerance against independent/correlated failures and reduce repair network traffic by orders of magnitude. To achieve this, we use various evaluation strategies including simulation, splitting, dynamic programming, and mathematical modeling. We also compare the performance and durability of MLEC with other EC schemes such as SLEC and LRC and show that MLEC can provide high durability with higher encoding throughput and less repair network traffic over both SLEC and LRC.
- Research Organization:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 2224154
- Resource Relation:
- Conference: International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 23) - Denver, Colorado, United States of America - 11/12/2023 10:00:00 AM-11/17/2023 10:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
ACStor: Optimizing Access Performance of Virtual Disk Images in Clouds
Authentication Protocol for ICS without Encryption