Abstract
Replication has been widely used to ensure the data availability in a distributed file system. In recent years, erasure coding (EC) has been adopted to overcome the problem of space efficiency in Replication. However, EC has various performance degrading factors such as parity calculation and degraded input/output. In particular, the recovery performance of EC is degraded because of various factors when the distributed file systems become large. Nonetheless, few studies have been conducted to improve the recovery performance. Thus, this paper proposes an efficient parallel recovery technique in an EC-based distributed file system. We describe the contention avoidance method, chunk allocation method, and asynchronous recovery method, to improve the parallel recovery performance. The contention avoidance method can minimize the contention for resources. The chunk allocation method and asynchronous recovery method can increase the efficiency of the parallel recovery. Finally, we verify that when the proposed parallel recovery technique in this paper is applied to actual distributed file systems, its recovery performance is improved by 263% compared to that of existing methods in the performance evaluation.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Nicloe H (2017) An Exascale timeline for storage and I/O systems. The Next Platform. https://www.nextplatform.com/2017/08/16/exascale-timeline-storage-io-systems/. Accessed 21 Mar 2019
Kunkel JM, Kuhn M, Ludwig T (2014) Exascale storage systems—an analytical study of expenses. Supercomput Front Innov 1(1):116–134. https://doi.org/10.14529/jsfi140106
Bergman K, Borkar S, Campbell D, Carlson W, Dally W, Denneau M, Franzon P, Harrod W, Hill K, Hiller J, Karp S, Keckler S, Klein D, Lucas R, Richards M, Scarpelli A, Scott S, Snavely A, Sterling T, Williams RS, Yelick K (2018) ExaScale computing study: technology challenges in achieving exascale systems, Defense Advanced Research Projects Agency Information Processing Techniques Office (DARPA IPTO). Tech., vol 15. http://www.cse.nd.edu/Reports/2008/TR-2008-13.pdf. Accessed 21 Mar 2019
Chun BG, Dabek F, Haeberlen A, Sit E, Weatherspoon H, Kaashoek MF, Kubiatowicz J, Morris R (2006) Efficient replica maintenance for distributed storage systems. In: NSDI’06 Proceedings of the 3rd conference on networked systems design & implementation, vol 6, pp 45–58. http://usenix.org/events/nsdi06/tech/full_papers/chun/chun.pdf. Accessed 21 Mar 2019
Cook JD, Primmer R, de Kwant A (2014) Compare cost and performance of replication and erasure coding. Hitachi Rev 63:304
Sathiamoorthy M, Asteris M, Papailiopoulos D, Dimakis AG, Vadali R, Chen S, Borthakur D (2013) Xoring elephants: novel erasure codes for big data. Proc VLDB Endow 6(5):325–336. https://doi.org/10.14778/2535573.2488339
Rodrigues R, Liskov B (2005) High availability in DHTs: erasure coding vs. replication. In: International workshop on peer-to-peer systems. Springer, pp 226–239. http://dx.doi.org/10.1007/11558989_21
Shenoy A (2015) The Pros and Cons of erasure coding & replication vs. RAID in. Next-Gen Storage Platforms, HEDVIG. http://www.snia.org/sites/default/files/SDC15_presentations/datacenter_infra/Shenoy_The_Pros_and_Cons_of_Erasure_v3-rev.pdf. Accessed 21 Mar 2019
Gligoroski D, Kralevska K, Jensen RE, Simonsen P (2017) Repair duality with locally repairable and locally regenerating codes. In: 3rd International conference on big data intelligence and computing and cyber science and technology congress, pp 979–984. http://dx.doi.org/10.1109/DASC-PICom-DataCom-CyberSciTec.2017.162
Li J, Li B (2013) Erasure coding for cloud storage systems: a survey. Tsinghua Sci Technol 18(3):259–272. https://doi.org/10.1109/TST.2013.6522585
Kralevska K, Gligoroski D, Jensen RE, Øverby H (2018) HashTag erasure codes: from theory to practice. IEEE Trans Big Data 4(4):516–529. https://doi.org/10.1109/TBDATA.2017.2749255
Pamies-Juarez L, Blagojevic F, Mateescu R, Gyuot C, Gad EE, Bandic Z (2016) Opening the chrysalis: On the real repair performance of MSR codes, In 14th USENIX conference on file and storage technologies (FAST 16). USENIX Association, pp 81–94
Apache Hadoop 3.0.0. Apache http://hadoop.apache.org/docs/r3.0.0-alpha4
Erasure Coding Support inside HDFS, Apache. https://issues.apache.org/jira/browse/HDFS-7285. Accessed 21 Mar 2019
Li R, Zhang Z, Zheng K, Wang A (2016) Progress report: bringing erasure coding to Apache Hadoop. Cloudera Engineering Blog. http://blog.cloudera.com/blog/2016/02/progress-report-bringing-erasure-coding-to-apache-hadoop/. Accessed 21 Mar 2019
Ceph Storage Cluster, ceph. http://docs.ceph.com/docs/master/rados/. Accessed 21 Mar 2019
Red Hat Gluster Storage, redhat. https://access.redhat.com/products/red-hat-storage/. Accessed 21 Mar 2019
Huang C, Simitci H, Xu Y, Ogus A, Calder B, Gopalan P, Li J, Yekhanin S (2012) Erasure coding in windows Azure storage. In: 2012 USENIX annual technical conference, pp 15–26. https://www.usenix.org/system/files/conference/atc12/atc12-final181_0.pdf. Accessed 21 Mar 2019
Plank JS, Simmerman S, Schuman CD (2008) Jerasure: a library in C/C++ facilitating erasure coding for storage applications—version 1.2, University of Tennessee, Technical Report CS-08-627, vol 23. https://web.eecs.utk.edu/~plank/plank/papers/CS-08-627.pdf. Accessed 21 Mar 2019
Esmaili KS, Pamies-Juarez L, Datta A (2013) The CORE storage primitive: cross-object redundancy for efficient data repair & access in erasure coded storage. arXiv preprint arXiv:1302.5192. https://arxiv.org/pdf/1302.5192.pdf
Dimakis AG, Godfrey PB, Wu Y, Wainwright MJ, Ramchandran K (2010) Network coding for distributed storage systems. IEEE Trans Inf Theory 56(9):4539–4551. https://doi.org/10.1109/TIT.2010.2054295
Sun D, Xu Y, Li Y, Wu S, Tian C (2016) Efficient parity update for scaling RAID-like storage systems. In: 2016 IEEE international conference on networking, architecture and storage (NAS), pp 1–10. http://dx.doi.org/10.1109/NAS.2016.7549400
Rashmi KV, Shah NB, Gu D, Kuang H, Borthakur D, Ramchandran K (2014) A “hitchhiker’s” guide to fast and efficient data reconstruction in erasure-coded data centers. In: Proceedings of the 2014 ACM conference on SIGCOMM, vol 44, no 4, pp 331–342. http://dx.doi.org/10.1145/2619239.2626325
Rashmi KV, Shah NB, Gu D, Kuang H, Borthakur D, Ramchandran K (2013) A solution to the network challenges of data recovery in erasure-coded distributed storage systems: a study on the Facebook Warehouse cluster. In: 5th USENIX workshop on Hot Topics in Storage and File Systems (HotStorage). https://www.usenix.org/system/files/conference/hotstorage13/hotstorage13-rashmi.pdf. Accessed 21 Mar 2019
Mohan LJ, Harold RL, Caneleo PIS, Parampalli U, Harwood A (2015) Benchmarking the performance of Hadoop triple replication and erasure coding on a nation-wide distributed cloud. In: 2015 international symposium on Network Coding (NetCod), pp 61–65. https://doi.org/10.1109/netcod.2015.7176790
Dustin B (2017) Red Hat Gluster storage on supermicro storage servers powered by Intel Xeon processors. Super Micro Computer, San Jose
Miyamae T, Nakao T, Shiozawa K (2014) Erasure code with shingled local parity groups for efficient recovery from multiple disk failures. In: 10th workshop on hot topics in system dependability. https://www.usenix.org/system/files/conference/hotdep14/hotdep14-miyamae.pdf. Accessed 21 Mar 2019
Luo X, Shu J (2013) Load-balanced recovery schemes for single-disk failure in storage systems with any erasure code. In: 2013 42nd international conference on parallel processing, pp 552–561. http://dx.doi.org/10.1109/ICPP.2013.69
Considerations for RAID-6 Availability and Format/Rebuild Performance on the DS5000, Redpaper. http://www.redbooks.ibm.com/redpapers/pdfs/redp4484.pdf. Accessed 21 Mar 2019
Miranda A, Effert S, Kang Y, Miller EL, Popov I, Brinkmann A, Friedetzky T, Cortes T (2014) Random slicing: efficient and scalable data placement for large-scale storage systems. ACM Trans Storage (TOS) 10(3):9
Liu C, Chu X, Liu H, Leung YW (2016) ESet: placing data towards efficient recovery for large-scale erasure-coded storage systems. In: 2016 25th international conference on computer communication and networks (ICCCN), pp 1–9. http://dx.doi.org/10.1109/ICCCN.2016.7568521
Chu X, Liu C, Ouyang K, Yung LS, Liu H, Leung Y-W (2015) PErasure: a parallel Cauchy Reed-Solomon coding library for GPUs. In: 2015 IEEE international conference on communications (ICC), pp 436–441. http://dx.doi.org/10.1109/ICC.2015.7248360
Mitra S, Panta R, Ra MR, Bagchi S (2016) Partial-parallel-repair (PPR): a distributed technique for repairing erasure coded storage. In: Proceedings of the eleventh European conference on computer systems, p 30. http://dx.doi.org/10.1145/2901318.2901328
Erasure Code Support, Sheepdog Wiki. https://github.com/sheepdog/sheepdog/wiki/Erasure-Code-Support. Accessed 21 Mar 2019
Plank JS (2013) Erasure codes for storage systems, login The Usenix Magazine Sysadmin Section, vol 38, no 6. https://www.usenix.org/system/files/login/articles/10_plank-online.pdf. Accessed 21 Mar 2019
Filesystem in Userspace, SourceForge. https://sourceforge.net/projects/fuse/. Accessed 21 Mar 2019
Kim YC, Kim DO, Kim HY, Kim YK, Choi W (2013) MAHA-FS: a distributed file system for high performance metadata processing and random IO. KIPS Trans Softw Data Eng 2(2):91–96. https://doi.org/10.3745/KTSDE.2013.2.2.091
Kim CY, Kim DO, Kim HY, Kim YK, Seo DW (2016) Torus network based distributed storage system for massive multimedia contents. J Korea Multimed Soc 19(8):1487–1497. https://doi.org/10.9717/kmms.2016.19.8.1487
Plank JS (1997) A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems. Softw Pract Exp 27(9):995–1012. https://doi.org/10.1002/(SICI)1097-024X(199709)27:93.0.CO;2-6
Plank JS, Xu L (2016) Optimizing Cauchy Reed-solomon codes for fault-tolerant network storage applications. In: Fifth IEEE international symposium on network computing and applications (NCA’06), pp 173–180. https://doi.org/10.1109/nca.2006.43
Plank JS (2009) The RAID-6 liberation codes. Int J High Perform Comput Appl 23(3):242–251. https://doi.org/10.1177/1094342009106191
Patterson DA, Hennessy JL (1998) Computer organization and design. In: The hardware/software interface, 2nd edn. Elsevier Science, Waltham, MA, USA. ISBN:978-0-12-407726-3
Intel® Intelligent Storage Acceleration Library (Intel® ISA-L) Open Source Version, API Reference Manual—Version 2.10, April 8, 2014. https://01.org/sites/default/files/documentation/isa-l_open_src_2.10.pdf. Accessed 21 Mar 2019
Kim DO, Kim HY, Kim YK, Kim JJ (2018) Cost analysis of erasure coding for exa-scale storage. J Supercomput. https://doi.org/10.1007/s11227-018-2663-4
Acknowledgements
This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No. 2015-0-00262, Management of Developing ICBMS (IoT, Cloud, Bigdata, Mobile, Security) Core Technologies and Development of Exascale Cloud Storage Technology).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kim, DO., Kim, HY., Kim, YK. et al. Efficient techniques of parallel recovery for erasure-coding-based distributed file systems. Computing 101, 1861–1884 (2019). https://doi.org/10.1007/s00607-019-00714-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00607-019-00714-7