Skip to main content
Springer Nature Link
Log in

Exploiting Efficient Transpacking for One-Sided Communication and MPI-IO

  • Conference paper
Recent Advances in Parallel Virtual Machine and Message Passing Interface (EuroPVM/MPI 2009)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 5759))

  • 1158 Accesses

Abstract

Based on a construction of socalled input-output datatypes that define a mapping between non-consecutive input and output buffers, we outline an efficient method for copying of structured data. We term this operation transpacking, and show how transpacking can be applied for the MPI implementation of one-sided communication and MPI-IO. For one-sided communication via shared-memory, we demonstrate the expected performance improvements by up to a factor of two. For individual MPI-IO, the time to read or write from file dominates the overall time, but even here efficient transpacking can in some scenarios reduce file I/O time considerably. The reported results have been achieved on a single NEC SX-8 vector node.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Augustin, W., Straub, M.-O., Worsch, T.: Benchmarking one-sided communication with SKaMPI 5. In: Di Martino, B., Kranzlmüller, D., Dongarra, J. (eds.) EuroPVM/MPI 2005. LNCS, vol. 3666, pp. 301–308. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  2. Byna, S., Gropp, W.D., Sun, X.-H., Thakur, R.: Improving the performance of MPI derived datatypes by optimizing memory-access cost. In: IEEE International Conference on Cluster Computing (CLUSTER 2003), pp. 412–419 (2003)

    Google Scholar 

  3. Byna, S., Sun, X.-H., Thakur, R., Gropp, W.D.: Automatic memory optimizations for improving MPI derived datatype performance. In: Mohr, B., Träff, J.L., Worringen, J., Dongarra, J. (eds.) PVM/MPI 2006. LNCS, vol. 4192, pp. 238–246. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  4. Isaila, F., Tichy, W.F.: Mapping functions and data redistribution for parallel files. The Journal of Supercomputing 46(3), 213–236 (2008)

    Article  Google Scholar 

  5. Mir, F.G., Träff, J.L.: Constructing MPI input-output datatypes for efficient transpacking. In: Lastovetsky, A., Kechadi, T., Dongarra, J. (eds.) EuroPVM/MPI 2008. LNCS, vol. 5205, pp. 141–150. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  6. MPI Forum.MPI: A Message-Passing Interface Standard. Version 2.1, September 4 (2008), http://www.mpi-forum.org

  7. Ross, R., Miller, N., Gropp, W.D.: Implementing fast and reusable datatype processing. In: Dongarra, J., Laforenza, D., Orlando, S. (eds.) EuroPVM/MPI 2003. LNCS, vol. 2840, pp. 404–413. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  8. Santhanaraman, G., Wu, D., Panda, D.K.: Zero-copy MPI derived datatype communication over InfiniBand. In: Kranzlmüller, D., Kacsuk, P., Dongarra, J. (eds.) EuroPVM/MPI 2004. LNCS, vol. 3241, pp. 47–56. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  9. Thakur, R., Gropp, W., Lusk, E.: Optimizing noncontiguous accesses in MPI-IO. Parallel Computing 28, 83–105 (2002)

    Article  MATH  Google Scholar 

  10. Träff, J.L., Hempel, R., Ritzdorf, H., Zimmermann, F.: Flattening on the fly: efficient handling of MPI derived datatypes. In: Margalef, T., Dongarra, J., Luque, E. (eds.) PVM/MPI 1999. LNCS, vol. 1697, pp. 109–116. Springer, Heidelberg (1999)

    Chapter  Google Scholar 

  11. Träff, J.L., Ritzdorf, H., Hempel, R.: The implementation of MPI-2 one-sided communication for the NEC SX-5. In: Supercomputing (2000), http://www.sc2000.org/proceedings/techpapr/index.htm#01

  12. Worringen, J., Träff, J.L., Ritzdorf, H.: Fast parallel non-contiguous file access. In: Supercomputing (2003), http://www.sc-conference.org/sc2003/tech_papers.php

  13. Wu, J., Wyckoff, P., Panda, D.K.: High performance implementation of MPI derived datatype communication over InfiniBand. In: 18th International Parallel and Distributed Processing Symposium (IPDPS 2004), p. 14 (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. NEC Laboratories Europe, NEC Europe Ltd., Rathausallee 10, D-53757, Sankt Augustin, Germany

    Faisal Ghias Mir & Jesper Larsson Träff

Authors
  1. Faisal Ghias Mir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jesper Larsson Träff
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Information Technology, Åbo Akademi, 20500, Turku, Finland

    Matti Ropo  & Jan Westerholm  & 

  2. Department of Electrical Engineering and Computer Science, University of Tennessee, 37996-3450, Knoxville, TN, USA

    Jack Dongarra

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Mir, F.G., Träff, J.L. (2009). Exploiting Efficient Transpacking for One-Sided Communication and MPI-IO. In: Ropo, M., Westerholm, J., Dongarra, J. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2009. Lecture Notes in Computer Science, vol 5759. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03770-2_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-03770-2_22

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-03769-6

  • Online ISBN: 978-3-642-03770-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics