Skip to main content
SpringerLink
Log in

Efficient techniques of parallel recovery for erasure-coding-based distributed file systems

  • Published:
Computing Aims and scope Submit manuscript

Abstract

Replication has been widely used to ensure the data availability in a distributed file system. In recent years, erasure coding (EC) has been adopted to overcome the problem of space efficiency in Replication. However, EC has various performance degrading factors such as parity calculation and degraded input/output. In particular, the recovery performance of EC is degraded because of various factors when the distributed file systems become large. Nonetheless, few studies have been conducted to improve the recovery performance. Thus, this paper proposes an efficient parallel recovery technique in an EC-based distributed file system. We describe the contention avoidance method, chunk allocation method, and asynchronous recovery method, to improve the parallel recovery performance. The contention avoidance method can minimize the contention for resources. The chunk allocation method and asynchronous recovery method can increase the efficiency of the parallel recovery. Finally, we verify that when the proposed parallel recovery technique in this paper is applied to actual distributed file systems, its recovery performance is improved by 263% compared to that of existing methods in the performance evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

References

  1. Nicloe H (2017) An Exascale timeline for storage and I/O systems. The Next Platform. https://www.nextplatform.com/2017/08/16/exascale-timeline-storage-io-systems/. Accessed 21 Mar 2019

  2. Kunkel JM, Kuhn M, Ludwig T (2014) Exascale storage systems—an analytical study of expenses. Supercomput Front Innov 1(1):116–134. https://doi.org/10.14529/jsfi140106

    Article  Google Scholar 

  3. Bergman K, Borkar S, Campbell D, Carlson W, Dally W, Denneau M, Franzon P, Harrod W, Hill K, Hiller J, Karp S, Keckler S, Klein D, Lucas R, Richards M, Scarpelli A, Scott S, Snavely A, Sterling T, Williams RS, Yelick K (2018) ExaScale computing study: technology challenges in achieving exascale systems, Defense Advanced Research Projects Agency Information Processing Techniques Office (DARPA IPTO). Tech., vol 15. http://www.cse.nd.edu/Reports/2008/TR-2008-13.pdf. Accessed 21 Mar 2019

  4. Chun BG, Dabek F, Haeberlen A, Sit E, Weatherspoon H, Kaashoek MF, Kubiatowicz J, Morris R (2006) Efficient replica maintenance for distributed storage systems. In: NSDI’06 Proceedings of the 3rd conference on networked systems design & implementation, vol 6, pp 45–58. http://usenix.org/events/nsdi06/tech/full_papers/chun/chun.pdf. Accessed 21 Mar 2019

  5. Cook JD, Primmer R, de Kwant A (2014) Compare cost and performance of replication and erasure coding. Hitachi Rev 63:304

    Google Scholar 

  6. Sathiamoorthy M, Asteris M, Papailiopoulos D, Dimakis AG, Vadali R, Chen S, Borthakur D (2013) Xoring elephants: novel erasure codes for big data. Proc VLDB Endow 6(5):325–336. https://doi.org/10.14778/2535573.2488339

    Article  Google Scholar 

  7. Rodrigues R, Liskov B (2005) High availability in DHTs: erasure coding vs. replication. In: International workshop on peer-to-peer systems. Springer, pp 226–239. http://dx.doi.org/10.1007/11558989_21

  8. Shenoy A (2015) The Pros and Cons of erasure coding & replication vs. RAID in. Next-Gen Storage Platforms, HEDVIG. http://www.snia.org/sites/default/files/SDC15_presentations/datacenter_infra/Shenoy_The_Pros_and_Cons_of_Erasure_v3-rev.pdf. Accessed 21 Mar 2019

  9. Gligoroski D, Kralevska K, Jensen RE, Simonsen P (2017) Repair duality with locally repairable and locally regenerating codes. In: 3rd International conference on big data intelligence and computing and cyber science and technology congress, pp 979–984. http://dx.doi.org/10.1109/DASC-PICom-DataCom-CyberSciTec.2017.162

  10. Li J, Li B (2013) Erasure coding for cloud storage systems: a survey. Tsinghua Sci Technol 18(3):259–272. https://doi.org/10.1109/TST.2013.6522585

    Article  Google Scholar 

  11. Kralevska K, Gligoroski D, Jensen RE, Øverby H (2018) HashTag erasure codes: from theory to practice. IEEE Trans Big Data 4(4):516–529. https://doi.org/10.1109/TBDATA.2017.2749255

    Article  Google Scholar 

  12. Pamies-Juarez L, Blagojevic F, Mateescu R, Gyuot C, Gad EE, Bandic Z (2016) Opening the chrysalis: On the real repair performance of MSR codes, In 14th USENIX conference on file and storage technologies (FAST 16). USENIX Association, pp 81–94

  13. Apache Hadoop 3.0.0. Apache http://hadoop.apache.org/docs/r3.0.0-alpha4

  14. Erasure Coding Support inside HDFS, Apache. https://issues.apache.org/jira/browse/HDFS-7285. Accessed 21 Mar 2019

  15. Li R, Zhang Z, Zheng K, Wang A (2016) Progress report: bringing erasure coding to Apache Hadoop. Cloudera Engineering Blog. http://blog.cloudera.com/blog/2016/02/progress-report-bringing-erasure-coding-to-apache-hadoop/. Accessed 21 Mar 2019

  16. Ceph Storage Cluster, ceph. http://docs.ceph.com/docs/master/rados/. Accessed 21 Mar 2019

  17. Red Hat Gluster Storage, redhat. https://access.redhat.com/products/red-hat-storage/. Accessed 21 Mar 2019

  18. Huang C, Simitci H, Xu Y, Ogus A, Calder B, Gopalan P, Li J, Yekhanin S (2012) Erasure coding in windows Azure storage. In: 2012 USENIX annual technical conference, pp 15–26. https://www.usenix.org/system/files/conference/atc12/atc12-final181_0.pdf. Accessed 21 Mar 2019

  19. Plank JS, Simmerman S, Schuman CD (2008) Jerasure: a library in C/C++ facilitating erasure coding for storage applications—version 1.2, University of Tennessee, Technical Report CS-08-627, vol 23. https://web.eecs.utk.edu/~plank/plank/papers/CS-08-627.pdf. Accessed 21 Mar 2019

  20. Esmaili KS, Pamies-Juarez L, Datta A (2013) The CORE storage primitive: cross-object redundancy for efficient data repair & access in erasure coded storage. arXiv preprint arXiv:1302.5192. https://arxiv.org/pdf/1302.5192.pdf

  21. Dimakis AG, Godfrey PB, Wu Y, Wainwright MJ, Ramchandran K (2010) Network coding for distributed storage systems. IEEE Trans Inf Theory 56(9):4539–4551. https://doi.org/10.1109/TIT.2010.2054295

    Article  MATH  Google Scholar 

  22. Sun D, Xu Y, Li Y, Wu S, Tian C (2016) Efficient parity update for scaling RAID-like storage systems. In: 2016 IEEE international conference on networking, architecture and storage (NAS), pp 1–10. http://dx.doi.org/10.1109/NAS.2016.7549400

  23. Rashmi KV, Shah NB, Gu D, Kuang H, Borthakur D, Ramchandran K (2014) A “hitchhiker’s” guide to fast and efficient data reconstruction in erasure-coded data centers. In: Proceedings of the 2014 ACM conference on SIGCOMM, vol 44, no 4, pp 331–342. http://dx.doi.org/10.1145/2619239.2626325

  24. Rashmi KV, Shah NB, Gu D, Kuang H, Borthakur D, Ramchandran K (2013) A solution to the network challenges of data recovery in erasure-coded distributed storage systems: a study on the Facebook Warehouse cluster. In: 5th USENIX workshop on Hot Topics in Storage and File Systems (HotStorage). https://www.usenix.org/system/files/conference/hotstorage13/hotstorage13-rashmi.pdf. Accessed 21 Mar 2019

  25. Mohan LJ, Harold RL, Caneleo PIS, Parampalli U, Harwood A (2015) Benchmarking the performance of Hadoop triple replication and erasure coding on a nation-wide distributed cloud. In: 2015 international symposium on Network Coding (NetCod), pp 61–65. https://doi.org/10.1109/netcod.2015.7176790

  26. Dustin B (2017) Red Hat Gluster storage on supermicro storage servers powered by Intel Xeon processors. Super Micro Computer, San Jose

    Google Scholar 

  27. Miyamae T, Nakao T, Shiozawa K (2014) Erasure code with shingled local parity groups for efficient recovery from multiple disk failures. In: 10th workshop on hot topics in system dependability. https://www.usenix.org/system/files/conference/hotdep14/hotdep14-miyamae.pdf. Accessed 21 Mar 2019

  28. Luo X, Shu J (2013) Load-balanced recovery schemes for single-disk failure in storage systems with any erasure code. In: 2013 42nd international conference on parallel processing, pp 552–561. http://dx.doi.org/10.1109/ICPP.2013.69

  29. Considerations for RAID-6 Availability and Format/Rebuild Performance on the DS5000, Redpaper. http://www.redbooks.ibm.com/redpapers/pdfs/redp4484.pdf. Accessed 21 Mar 2019

  30. Miranda A, Effert S, Kang Y, Miller EL, Popov I, Brinkmann A, Friedetzky T, Cortes T (2014) Random slicing: efficient and scalable data placement for large-scale storage systems. ACM Trans Storage (TOS) 10(3):9

    Google Scholar 

  31. Liu C, Chu X, Liu H, Leung YW (2016) ESet: placing data towards efficient recovery for large-scale erasure-coded storage systems. In: 2016 25th international conference on computer communication and networks (ICCCN), pp 1–9. http://dx.doi.org/10.1109/ICCCN.2016.7568521

  32. Chu X, Liu C, Ouyang K, Yung LS, Liu H, Leung Y-W (2015) PErasure: a parallel Cauchy Reed-Solomon coding library for GPUs. In: 2015 IEEE international conference on communications (ICC), pp 436–441. http://dx.doi.org/10.1109/ICC.2015.7248360

  33. Mitra S, Panta R, Ra MR, Bagchi S (2016) Partial-parallel-repair (PPR): a distributed technique for repairing erasure coded storage. In: Proceedings of the eleventh European conference on computer systems, p 30. http://dx.doi.org/10.1145/2901318.2901328

  34. Erasure Code Support, Sheepdog Wiki. https://github.com/sheepdog/sheepdog/wiki/Erasure-Code-Support. Accessed 21 Mar 2019

  35. Plank JS (2013) Erasure codes for storage systems, login The Usenix Magazine Sysadmin Section, vol 38, no 6. https://www.usenix.org/system/files/login/articles/10_plank-online.pdf. Accessed 21 Mar 2019

  36. Filesystem in Userspace, SourceForge. https://sourceforge.net/projects/fuse/. Accessed 21 Mar 2019

  37. Kim YC, Kim DO, Kim HY, Kim YK, Choi W (2013) MAHA-FS: a distributed file system for high performance metadata processing and random IO. KIPS Trans Softw Data Eng 2(2):91–96. https://doi.org/10.3745/KTSDE.2013.2.2.091

    Article  Google Scholar 

  38. Kim CY, Kim DO, Kim HY, Kim YK, Seo DW (2016) Torus network based distributed storage system for massive multimedia contents. J Korea Multimed Soc 19(8):1487–1497. https://doi.org/10.9717/kmms.2016.19.8.1487

    Article  Google Scholar 

  39. Plank JS (1997) A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems. Softw Pract Exp 27(9):995–1012. https://doi.org/10.1002/(SICI)1097-024X(199709)27:93.0.CO;2-6

    Article  MathSciNet  Google Scholar 

  40. Plank JS, Xu L (2016) Optimizing Cauchy Reed-solomon codes for fault-tolerant network storage applications. In: Fifth IEEE international symposium on network computing and applications (NCA’06), pp 173–180. https://doi.org/10.1109/nca.2006.43

  41. Plank JS (2009) The RAID-6 liberation codes. Int J High Perform Comput Appl 23(3):242–251. https://doi.org/10.1177/1094342009106191

    Article  Google Scholar 

  42. Patterson DA, Hennessy JL (1998) Computer organization and design. In: The hardware/software interface, 2nd edn. Elsevier Science, Waltham, MA, USA. ISBN:978-0-12-407726-3

  43. Intel® Intelligent Storage Acceleration Library (Intel® ISA-L) Open Source Version, API Reference Manual—Version 2.10, April 8, 2014. https://01.org/sites/default/files/documentation/isa-l_open_src_2.10.pdf. Accessed 21 Mar 2019

  44. Kim DO, Kim HY, Kim YK, Kim JJ (2018) Cost analysis of erasure coding for exa-scale storage. J Supercomput. https://doi.org/10.1007/s11227-018-2663-4

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No. 2015-0-00262, Management of Developing ICBMS (IoT, Cloud, Bigdata, Mobile, Security) Core Technologies and Development of Exascale Cloud Storage Technology).

Author information

Authors and Affiliations

  1. SW·Contents Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Korea

    Dong-Oh Kim, Hong-Yeon Kim & Young-Kyun Kim

  2. Department of Computer Engineering, Korea Polytechnic University, Siheung, Gyeonggi-do, Korea

    Jeong-Joon Kim

Authors
  1. Dong-Oh Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong-Yeon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Young-Kyun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeong-Joon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeong-Joon Kim.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, DO., Kim, HY., Kim, YK. et al. Efficient techniques of parallel recovery for erasure-coding-based distributed file systems. Computing 101, 1861–1884 (2019). https://doi.org/10.1007/s00607-019-00714-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-019-00714-7

Keywords

Search

Navigation