Skip to main content
Springer Nature Link
Log in

A Memory Efficient Parallel All-Pairs Computation Framework: Computation – Communication Overlap

  • Conference paper
  • First Online:
Parallel Processing and Applied Mathematics (PPAM 2017)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10777))

  • 1550 Accesses

Abstract

All-Pairs problems require each data element in a set of N data elements to be paired with every other data element for specific computation using the two data elements. Our framework aims to address recurring problems of scalability, distributing equal work load to all nodes and by reducing memory footprint. We reduce memory footprint of All-Pairs problems, by reducing memory requirement from \(N/\sqrt{P}\) to 3N/P. A bio-informatics application is implemented to demonstrate the scalability ranging up to 512 cores for the data set we experimented, redundancy management, and speed up performance of the framework.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Hedegaard, R.: Handshake problem, January 2016. http://mathworld.wolfram.com/HandshakeProblem.html

  2. Watson-Haigh, N.S., Kadarmideen, H.N., Reverter, A.: PCIT: an R package for weighted gene co-expression networks based on partial correlation and information theory approaches. Bioinformatics 26(3), 411–413 (2010)

    Article  Google Scholar 

  3. Chapman, T., Kalyanaraman, A.: An OpenMP algorithm and implementation for clustering biological graphs. In: Proceedings of the First Workshop on Irregular Applications: Architectures and Algorithm, p. 310. ACM (2011)

    Google Scholar 

  4. Phillips, P., et al.: Overview of the face recognition grand challenge. In: IEEE Computer Vision and Pattern Recognition (2005)

    Google Scholar 

  5. Moretti, C., Bui, H., Hollingsworth, K., Rich, B., Flynn, P., Thain, D.: All-Pairs: an abstraction for data-intensive computing on campus grids. IEEE Trans. Parallel Distrib. Syst. 21(1), 33–46 (2010)

    Article  Google Scholar 

  6. Chae, H., Jung, I., Lee, H., Marru, S., Lee, S.-W., Kim, S.: Bio and health informatics meets cloud: BioVLab as an example. Health Inf. Sci. Syst. 1(1), 6 (2013)

    Article  Google Scholar 

  7. Kleinheksel, C.J., Somani, A.K.: Scaling distributed all-pairs algorithms. Information Science and Applications (ICISA) 2016. LNEE, vol. 376, pp. 247–257. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-0557-2_25

    Chapter  Google Scholar 

  8. Plimpton, S.: Fast parallel algorithms for short-range molecular dynamics. J. Comput. Phys. 117(1), 119 (1995)

    Article  MATH  Google Scholar 

  9. Driscoll, M., Georganas, E., Koanantakool, P., Solomonik, E., Yelick, K.: A communication-optimal n-body algorithm for direct interactions. In: Proceedings of the IEEE 27th International Symposium on in Parallel and Distributed Processing (IPDPS), pp. 1075–1084. IEEE (2013)

    Google Scholar 

  10. Doerfler, D., Brightwell, R.: Measuring MPI send and receive overhead and application availability in high performance network interfaces. In: Mohr, B., Träff, J.L., Worringen, J., Dongarra, J. (eds.) EuroPVM/MPI 2006. LNCS, vol. 4192, pp. 331–338. Springer, Heidelberg (2006). https://doi.org/10.1007/11846802_46

    Chapter  Google Scholar 

  11. Mishra, P., Somani, A.K.: Host managed contention avoidance storage solutions for Big Data. J. Big Data 4(1), 18 (2017)

    Article  Google Scholar 

  12. Ozkural, E., Aykanat, C: 1-D and 2-D parallel algorithms for all-pairs similarity problem. CoRR abs/1402.3010 (2014)

    Google Scholar 

Download references

Acknowledgements

The research reported in this paper is partially supported by the Philip and Virginia Sproul Professor Endowment and HPC@ISU equipment at Iowa State University, some of which has been purchased through funding provided by NSF under MRI grant number NSF CNS grant number 1229081 and NSF CRI grant number 1205413. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Iowa State University, Ames, IA, 50010, USA

    Venkata Kasi Viswanath Yeleswarapu & Arun K. Somani

Authors
  1. Venkata Kasi Viswanath Yeleswarapu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arun K. Somani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Venkata Kasi Viswanath Yeleswarapu .

Editor information

Editors and Affiliations

  1. Czestochowa University of Technology, Czestochowa, Poland

    Roman Wyrzykowski

  2. University of Tennessee, Knoxville, Tennessee, USA

    Jack Dongarra

  3. University of Southern California, Marina Del Rey, California, USA

    Ewa Deelman

  4. Czestochowa University of Technology, Czestochowa, Poland

    Konrad Karczewski

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yeleswarapu, V.K.V., Somani, A.K. (2018). A Memory Efficient Parallel All-Pairs Computation Framework: Computation – Communication Overlap. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds) Parallel Processing and Applied Mathematics. PPAM 2017. Lecture Notes in Computer Science(), vol 10777. Springer, Cham. https://doi.org/10.1007/978-3-319-78024-5_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-78024-5_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-78023-8

  • Online ISBN: 978-3-319-78024-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics