Abstract
The OpenSHMEM programming model encourages application developers to partition memory into local and symmetric segments through the use of the SHMEM_SYMMETRIC_SIZE environment variable. While this can lead to improved communication efficiency, it requires applications to partition the available memory. Setting this value requires that users calculate the amount of memory an application requires for a given dataset or problem. It also presents challenges to applications that progress through phases where OpenSHMEM is not used in every phase and the full memory capacity is needed when OpenSHMEM is not in use. This work presents a dynamic mapping approach to establishing the symmetric heap in NVSHMEM, an OpenSHMEM library for clusters of NVIDIA GPUs. Results indicate that this approach obviates the need for static partitioning of memory with low overheads, significantly improving the usability and flexibility of the NVSHMEM library.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
perftest/device/pt-to-pt/shmem_p_bw.cu in the NVSHMEM distribution [11].
References
Bachan, J., et al.: UPC++ programmer’s guide, v1.0-2018.3.0, Technical report, LBNL-2001136, Lawrence Berkeley National Laboratory (2018)
Barriuso, R., Knies, A.: SHMEM user’s guide (1994)
Chen, W.-Y., Bonachea, D., Duell, J., Husbands, P., Iancu, C., Yelick, K.: A performance analysis of the Berkeley UPC compiler. In: Proceedings of the 17th Annual International Conference on Supercomputing, pp. 63–73 (2003)
El-Ghazawi, T., Carlson, W., Sterling, T., Yelick, K.: UPC: Distributed Shared Memory Programming, vol. 40. Wiley (2005)
El-Ghazawi, T.A., Cantonnet, F., Yao, Y., Vetter, J.: Evaluation of UPC on the Cray X1. In: Cray User Group Proceedings. Citeseer (2005)
GPUDirect RDMA. https://docs.nvidia.com/cuda/gpudirect-rdma/index.html
Ionkov, L., Young, G.: Asymmetric memory extension for Openshmem. In: Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models, PGAS 2014. Association for Computing Machinery, New York (2014)
Linux RDMA core, 8 June 2020. https://github.com/linux-rdma/rdma-core
MPI Forum: A message-passing interface standard, version 4.0. Technical report, University of Tennessee, Knoxville, June 2021
Namashivayam, N., et al.: Symmetric memory partitions in OpenSHMEM: a case study with Intel KNL. In: Gorentla Venkata, M., Imam, N., Pophale, S. (eds.) OpenSHMEM 2017. LNCS, vol. 10679, pp. 3–18. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73814-7_1
Nvshmem repository. https://developer.download.nvidia.com/compute/redist/nvshmem/2.2.1/. Accessed 9 Sept 2021
Openshmem 1.5 specification. http://www.openshmem.org/site/sites/default/site_files/OpenSHMEM-1.5.pdf. Accessed 8 June 2020
Perry, C., Sakharnykh, N.: Introducing low-level GPU virtual memory management, 15 April 2020. https://developer.nvidia.com/blog/introducing-low-level-gpu-virtual-memory-management/
Welch, A., Pophale, S., Shamis, P., Hernandez, O., Poole, S., Chapman, B.: Extending the Openshmem memory model to support user-defined spaces. In: Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models, pp. 1–10 (2014)
Zheng, Y., Kamil, A., Driscoll, M.B., Shan, H., Yelick, K.: UPC++: A PGAS extension for C++. In: 2014 IEEE 28th International Parallel and Distributed Processing Symposium, pp. 1105–1114. IEEE (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Langer, A., Howell, S., Potluri, S., Dinan, J., Kraus, J. (2022). Dynamic Symmetric Heap Allocation in NVSHMEM. In: Poole, S., Hernandez, O., Baker, M., Curtis, T. (eds) OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Exascale and Smart Networks. OpenSHMEM 2021. Lecture Notes in Computer Science, vol 13159. Springer, Cham. https://doi.org/10.1007/978-3-031-04888-3_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-04888-3_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04887-6
Online ISBN: 978-3-031-04888-3
eBook Packages: Computer ScienceComputer Science (R0)