Abstract
Modern network communication libraries that leverage Remote Directory Memory Access (RDMA) and OS bypass protocols, such as Infiniband [2] and Myrinet [10] can offer significant performance advantages over conventional send/receive protocols. However, this performance often comes with hidden per buffer setup costs [4]. This paper describes a unique long-message MPI [9] library ‘pipeline’ protocol that addresses these constraints while avoiding some of the pitfalls of existing techniques. By using portable send/receive semantics to hide the cost of initializing the pipeline algorithm, and then effectively overlapping the cost of memory registration with RDMA operations, this protocol provides very good performance for any large-memory usage pattern. This approach avoids the use of non-portable memory hooks or keeping registered memory from being returned to the OS. Through this approach, bandwidth may be increased up to 67% when memory buffers are not effectively reused while providing superior performance in the effective bandwidth benchmark. Several user level protocols are explored using Open MPI’s PML (Point to point messaging layer) and compared/contrasted to this ‘pipeline’ protocol.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Performance of mpich-gm, http://www.myri.com/myrinet/performance/MPICH-GM/index.html
I. T. Association. Infiniband architecture specification. release 1.2, vol. 1 (2004), www.infinibandta.org
Beecroft, J., Addison, D., Petrini, F., McLaren, M.: QsNetII: An interconnect for supercomputing applications (2003)
Brightwell, R., Maccabe, A.: Scalability limitations of VIA-based technologies in supporting MPI. In: Proceedings of the Fourth MPI Developer’s and User’s Conference (March 2000)
Chadalapaka, M., Shah, H., Elzur, U., Thaler, P., Ko, M.: A study of iscsi extensions for rdma (iser). In: NICELI 2003: Proceedings of the ACM SIGCOMM workshop on Network-I/O convergence, pp. 209–219. ACM Press, New York (2003)
Garbriel, E., Fagg, G., Bosilica, G., Angskun, T., Squyres, J.J.D.J., Sahay, V., Kambadur, P., Barrett, B., Lumsdaine, A., Castain, R., Daniel, D., Graham, R., Woodall, T.: Open MPI: goals, concept, and design of a next generation MPI implementation. In: Proceedings, 11th European PVM/MPI Users’ Group Meeting (2004)
Geist, A., Gropp, W., Huss-Lederman, S., Lumsdaine, A., Lusk, E., Saphir, W., Skjellum, T., Snir, M.: MPI-2: Extending the Message-Passing Interface. In: Euro-Par 1996 Parallel Processing, pp. 128–135. Springer, Heidelberg (1996)
Liu, J., Wu, J., Kini, S.P., Wyckoff, P., Panda, D.K.: High performance RDMA-based MPI implementation over infiniband. In: ICS 2003: Proceedings of the 17th annual international conference on Supercomputing, pp. 295–304. ACM Press, New York (2003)
Message Passing Interface Forum. MPI: A Message Passing Interface. In: Proc. of Supercomputing 1993, pp. 878–883. IEEE Computer Society Press, Los Alamitos (1993)
Myricom. Myrinet-on-VME protocol specification, http://www.myri.com/open-specs/
Rabenseifner, R., Koniges, A.: The parallel communication and i/o bandwidth benchmarks: b_eff and b_eff_io (2001)
Tipparaju, V., Santhanaraman, G., Nieplocha, J., Panda, D.K.: Host-assisted zero-copy remote memory access communication on infiniband. ipdps 01, 31a (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Woodall, T.S., Shipman, G.M., Bosilca, G., Graham, R.L., Maccabe, A.B. (2006). High Performance RDMA Protocols in HPC. In: Mohr, B., Träff, J.L., Worringen, J., Dongarra, J. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2006. Lecture Notes in Computer Science, vol 4192. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11846802_18
Download citation
DOI: https://doi.org/10.1007/11846802_18
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-39110-4
Online ISBN: 978-3-540-39112-8
eBook Packages: Computer ScienceComputer Science (R0)