Abstract
SHMEM has a long history as a parallel programming model. It is extensively used since 1993, starting from Cray T3D systems. For the past two decades SHMEM library implementation in Cray systems evolved through different generations. The current generation of the SHMEM implementation for Cray XC and XK systems is called Cray SHMEM. It is a proprietary SHMEM implementation from Cray Inc. In this work, we provide an in-depth analysis of need for a new SHMEM implementation and then introduce the next evolution of Cray SHMEM implementation for current and future generation Cray systems. We call this new implementation Cray OpenSHMEMX. We provide brief design overview, along with a review of functional and performance differences in Cray OpenSHMEMX comparing against the existing Cray SHMEM implementation.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Though SMAT layer supports different communication layers, we use DMAPP and XPMEM in this work as others are experimental and not available in the current released Cray OpenSHMEMX version 8.0.1.
- 2.
Number of Processing Elements (PEs) Per Node.
References
GASNet: a portable high-performance communication layer for global address-space languages. http://gasnet.lbl.gov/pubs/258-paper.pdf
OpenSHMEM standard version-1.3. http://openshmem.org/site/sites/default/site_files/OpenSHMEM-1.3.pdf, a
OpenSHMEM standard version-1.4. http://openshmem.org/site/sites/default/site_files/OpenSHMEM-1.4.pdf, b
OSU Micro-benchmarks. http://mvapich.cse.ohio-state.edu/benchmarks/
Cavium Thunder X2 ARM Processors. https://www.cavium.com/product-thunderx2-arm-processors.html
Global arrays: a portable shared memory model for distributed memory computers (1994)
Open MPI: a flexible high performance MPI (2005)
Almasi, G.: In: Padua, D.A. (ed.) Encyclopedia of Parallel Computing (2011)
Balaji, P., et al.: PMI: a scalable parallel process-management interface for extreme-scale systems. In: Keller, R., Gabriel, E., Resch, M., Dongarra, J. (eds.) EuroMPI 2010. LNCS, vol. 6305, pp. 31–41. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15646-5_4
Chapman, B., et al.: Introducing OpenSHMEM: SHMEM for the PGAS Community. In: Proceedings of the Fourth Conference on Partitioned Global Address Space Programming Model, PGAS 2010 (2010)
Doweck, J., et al.: Inside 6th-generation Intel Core: new microarchitecture code-named Skylake. IEEE Micro (2017)
Message Passing Forum. MPI: a message-passing interface standard. Technical report (1994)
Gropp, W.: MPICH2: a new start for MPI implementations. In: Kranzlmüller, D., Volkert, J., Kacsuk, P., Dongarra, J. (eds.) EuroPVM/MPI 2002. LNCS, vol. 2474, pp. 7–7. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45825-5_5
Hammarlund, P., et al.: Haswell: the fourth-generation intel core processor. IEEE Micro 34, 6–20 (2014)
Kandalla, K., et al.: Current state of the cray MPT software stacks on the cray XC series supercomputers. In: Cray User Group (CUG) meeting 2017 (2017)
Knaak, D., Namashivayam, N.: Proposing OpenSHMEM extensions towards a future for hybrid programming and heterogeneous computing. In: Gorentla Venkata, M., Shamis, P., Imam, N., Lopez, M.G. (eds.) OpenSHMEM 2014. LNCS, vol. 9397, pp. 53–68. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-26428-8_4
Nalamalpu, A., et al.: Broadwell: a family of IA 14nm processors. In: Symposium on VLSI Circuits (VLSI Circuits) (2015)
Namashivayam, N., Knaak, D., Cernohous, B., Radcliffe, N., Pagel, M.: An evaluation of thread-safe and contexts-domains features in cray SHMEM. In: Gorentla Venkata, M., Imam, N., Pophale, S., Mintz, T.M. (eds.) OpenSHMEM 2016. LNCS, vol. 10007, pp. 163–180. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-50995-2_11
OpenMP Architecture Review Board. OpenMP application program interface version 4.5, November 2015. https://www.openmp.org/wp-content/uploads/openmp-4.5.pdf
Sodani, S.A., et al.: Knights landing: second-generation Intel Xeon Phi product (2016)
ten Bruggencate, M., Roweth, D.: DMAPP: an API for one-sided programming model on baker systems. Technical report, Cray Users Group (CUG), August 2010
ten Bruggencate, M., Roweth, D., Oyanagi, S.: Thread-safe SHMEM extensions. In: Poole, S., Hernandez, O., Shamis, P. (eds.) OpenSHMEM 2014. LNCS, vol. 8356, pp. 178–185. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-05215-1_13
Acknowledgment and Disclaimer
The authors would like to thank all the members of Cray Message Passing Toolkit for their involvement in the design and implementation of the Cray OpenSHMEMX software stack. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of associated organizations.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Namashivayam, N., Cernohous, B., Pou, D., Pagel, M. (2019). Introducing Cray OpenSHMEMX - A Modular Multi-communication Layer OpenSHMEM Implementation. In: Pophale, S., Imam, N., Aderholdt, F., Gorentla Venkata, M. (eds) OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Extreme Heterogeneity. OpenSHMEM 2018. Lecture Notes in Computer Science(), vol 11283. Springer, Cham. https://doi.org/10.1007/978-3-030-04918-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-04918-8_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04917-1
Online ISBN: 978-3-030-04918-8
eBook Packages: Computer ScienceComputer Science (R0)