Skip to main content
SpringerLink
Log in

One-Sided Communication on Clusters with Myrinet

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

This paper describes an efficient implementation of one-sided communication on top of the GM low-level message-passing library for clusters with Myrinet. This approach is compatible with shared memory, exploits pipelining, nonblocking communication, and overlapping memory registration with memory copy to maximize the transfer rate. The paper addresses critical design issues faced on the commodity clusters and then describes possible solutions for matching the low-level network protocol with user-level interfaces. The performance implications of the design decisions are presented and discussed in context of a standalone communication benchmark as well as two applications. Finally, the paper offers some indications on what additional features would be desirable in a communication library like GM to better support one-sided communication.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. S. Lumetta, A. Mainwaring and D. Culler, Multi-protocol active messages on a cluster of SMP's, in: Proceedings of Supercomputing '97 (1997).

  2. H. Tezuka, A. Hori, Y. Ishikawa and M. Sato, PM: An operating system coordinated high performance communication library, in: High Performance Computing and Networking, Lecture Notes in Computer Science, Vol. 1225 (Springer, 1997).

  3. L. Prylli and B. Tourancheau, BIP: A new protocol designed for high performance networking on Myrinet, in: Workshop PC-NOW, IPPS/SPDP98 (Orlando, USA, 1998).

  4. S. Parkin, M. Lauria, A. Chien et al., High Performance Virtual Machines (HPVM): Clusters with supercomputing APIs and performance, in: 8th SIAM Conference on Parallel Processing for Scientific Computing (PP97) (1997).

  5. Myricom, The GM Message Passing System (10/16/1999).

  6. J. Nieplocha and B. Carpenter, ARMCI: A portable remote memory copy library for distributed array libraries and compiler run-time systems, in: Proc. RTSPP IPPS/SDP'99 (1999).

  7. K. Parzyszek, J. Nieplocha and R. Kendall, A generalized portable SHMEM library for high performance computing, in: Proceedings Int. Conf. on Parallel and Distributed Computing and Systems, PDCS-2000 (2000).

  8. R. Bariuso and A. Knies, SHMEM's User's Guide, SN-2516, rev. 2.2 (Cray Research, Inc., Eagan, MN, 1994).

    Google Scholar 

  9. J. Nieplocha, J. Ju and T.P. Straatsma, A multiprotocol communication support for the global address space programming model on the IBM SP, in: Proc. EuroPar-2000, Lecture Notes in Computer Science, Vol. 1900 (Springer, 2000).

  10. K. Langendoen, J. Romein, R. Bhoedjang and H. Bal, Integrating polling, interrupts, and thread management, in: Proc. Frontiers'96 (1996).

  11. J. Hsieh, T. Leng, V. Mashayekhi and R. Rooholamini, Architectural and performance evaluation of GigaNet and Myrinet interconnects on clusters of small-scale SMP servers, in: Proceedings SC2000, http://www.sc2000.org/techpapr/papers/pap.pap294.pdf

  12. M. Banikazemi, R. Govindaraju, R. Blackmore and D.K. Panda, Implementing efficient MPI on LAPI for the IBM RS/600 SP systems: Experiences and performance evaluation, in: Proc. IPPS99 (1999).

  13. A. Gallatin, J. Chase and K. Yocum, Trapeze/IP: TCP/IP at near-gigabit speeds, in: 1999 USENIX Technical Conference, June 1999.

  14. K. Moldeklev and P. Gunningberg, How a large ATM MTU causes deadlocks in TCP data transfers, IEEE/ACMTransactions on Networking 3(4) (August 1995).

  15. J. Nieplocha, J. Ju and E. Apra, One-sided communication on SMP clusters with Myrinet using the GM message-passing library, in: Proc. CAC'01/IPDPS'01 (2001).

  16. Myricom, Portable MPI Model Implementation over GM, version 1.2.1 (file README-GM) (23 June, 2000).

  17. L.A. Giannini and A. Chien, A software architecture for Global address space on communication on clusters: Put/Get on Fast Messages, in: 7th Int. IEEE Symposium on High Performance Distributed Computing, HPDC-7 (1998).

  18. S.C. Woo, M.O. Ohara, E. Torrie, J.P. Singh and A. Gupta, The SPLASH-2 programs: Characterization and methodological considerations, in: Proc. 22nd International Symposium on Computer Architecture (1995).

  19. High Performance Computational Chemistry Group, NWChem, A Computational Chemistry Package for Parallel Computers, Version 4.0, Pacific Northwest National Laboratory, Richland, WA (2000), http://www.emsl.pnl.gov:2080/docs/nwchem

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Pacific Northwest National Laboratory, USA

    Jarek Nieplocha & Edoardo Apra

  2. Tivoli Systems Inc, USA

    Jialin Ju

  3. Ohio State University, USA

    Vinod Tipparaju

Authors
  1. Jarek Nieplocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edoardo Apra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jialin Ju
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vinod Tipparaju
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jarek Nieplocha.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nieplocha, J., Apra, E., Ju, J. et al. One-Sided Communication on Clusters with Myrinet. Cluster Computing 6, 115–124 (2003). https://doi.org/10.1023/A:1022800521563

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1022800521563

Search

Navigation