Skip to main content
Springer Nature Link
Log in

FastDRC: Fast and Scalable Genome Compression Based on Distributed and Parallel Processing

  • Conference paper
  • First Online:
Algorithms and Architectures for Parallel Processing (ICA3PP 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11945))

  • 1986 Accesses

Abstract

With the advent of next-generation sequencing technology, sequencing costs have fallen sharply compared to the previous sequencing technologies. Genomic big data has become the significant big data application. In the face of growing genomic data, its storage and migration face enormous challenges. Therefore, researchers have proposed a variety of genome compression algorithms, but these algorithms cannot meet the processing requirements for large amount of biological data and high processing speed. This manuscript proposes a parallel and distributed referential genome compression algorithm-Fast Distributed Referential Compression (FastDRC). This algorithm compresses a large number of genomic sequences in parallel under the Apache Hadoop distributed computing framework. Experiments show that the compression efficiency of the FastDRC is greatly improved when it compresses large quantities of genomic data. Moreover, FastDRC leads to the only distributed computing method known to us in the field of genome compression. The source code for FastDRC can be obtained from this link: https://github.com/GhostCCCatHenry/FastDRC.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Kahn, S.D.: On the future of genomic data. Science 331(6018), 728–729 (2011)

    Article  Google Scholar 

  2. Pearson, W.R.: Rapid and sensitive sequence comparison with FASTP and FASTA. Methods Enzymol. 183(1), 63–98 (1990)

    Article  Google Scholar 

  3. Xie, X., Zhou, S., Guan, J.: CoGI: towards compressing genomes as an image. IEEE/ACM Trans. Comput. Biol. Bioinform. 12(6), 1275–1285 (2015)

    Article  Google Scholar 

  4. Deorowicz, S., Grabowski, S., Ochoa, I., et al.: ERGC: an efficient referential genome compression algorithm. Bioinformatics 31(21), 3468–3475 (2015)

    Article  Google Scholar 

  5. Wandelt, S., Leser, U.: FRESCO: referential compression of highly similar sequences. IEEE/ACM Trans. Comput. Biol. Bioinform. 10(5), 1275–1288 (2014)

    Article  Google Scholar 

  6. Wu, X.-D., Ji, S.-W.: Comparative study on MapReduce and spark for big data analytics. J. Softw. 29(6), 1770–1791 (2018)

    MathSciNet  Google Scholar 

  7. Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), MSST 2010, pp. 1–10. IEEE Computer Society, Washington, DC (2010)

    Google Scholar 

  8. Abecasis, G.: The 1000 genomes project consortium. An integrated map of genetic variation from 1,092 human genomes. Nature 491, 56–65 (2012)

    Google Scholar 

  9. Vavilapalli, V.K,, Murthy, A.C., Douglas, C., Agarwal, S., Konar, M., Evans, R., et al.: Apache hadoop YARN: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing, p. 5. ACM, New York (2013)

    Google Scholar 

  10. Liu, Y.S., et al.: High-speed and high-ratio referential genome compression. Bioinformatics 33(21), 3364–3372 (2017)

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank all reviewers for their valuable comments and suggestions to improve the quality of our manuscript.

Funding

This work was supported by the National Key R&D Program of China [2017YFB1401302, 2017YFB0202200], the National Natural Science Foundation of P. R. China [No. 61572260, 61872196], Outstanding Youth of Jiangsu Natural Science Foundation [BK20170100], Key R&D Program of Jiangsu [BE2017166], Postgraduate Research & Practice Innovation Program of Jiangsu Province [KYCX19_0906, KYCX19_0921], The Natural Science Foundation of the Jiangsu Higher Education Institutions of China [19KJD520006] and Modern Educational Technology Research Program of Jiangsu Province in 2019 [2019-R-67748].

Author information

Authors and Affiliations

  1. School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023, China

    Yimu Ji, Houzhi Fang, Haichang Yao, Shuai Chen, Kui Li & Shangdong Liu

  2. School of Computer and Software, Nanjing Institute of Industry Technology, Nanjing, 210023, China

    Haichang Yao

  3. School of Software and Electrical Engineering, Swinburne University of Technology, Melbourne, 3122, Australia

    Jing He

  4. Institute of High Performance Computing and Big Data, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China

    Yimu Ji

  5. Nanjing Center of HPC China, Nanjing, 210003, China

    Yimu Ji

  6. Jiangsu HPC and Intelligent Processing Engineer Research Center, Nanjing, 210003, China

    Yimu Ji

Authors
  1. Yimu Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Houzhi Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haichang Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shuai Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shangdong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haichang Yao .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Software Engineering, Swinburne University of Technology, Hawthorn, Melbourne, VIC, Australia

    Sheng Wen

  2. School of Computer Science, The University of Sydney, Camperdown, NSW, Australia

    Albert Zomaya

  3. Department of Computer Science, St. Francis Xavier University, Antigonish, NS, Canada

    Laurence T. Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ji, Y. et al. (2020). FastDRC: Fast and Scalable Genome Compression Based on Distributed and Parallel Processing. In: Wen, S., Zomaya, A., Yang, L.T. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2019. Lecture Notes in Computer Science(), vol 11945. Springer, Cham. https://doi.org/10.1007/978-3-030-38961-1_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-38961-1_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-38960-4

  • Online ISBN: 978-3-030-38961-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics