Abstract
HPC architectures are increasingly handling workloads such as AI/ML or high performance data analytics where the working data set cannot be easily partitioned, or does not fit into node local memory. This poses challenges for programming models such as OpenSHMEM, which require data in the working set to fit in the symmetric heap. Emerging fabric-attached memory (FAM) architectures enable data to be held in external memory accessible to all compute nodes, thus providing a new approach to handling large data sets. Unfortunately, most HPC libraries do not currently support FAM, and programmers use file system or key-value store abstractions to access data that is resident off-node, resulting in lower application performance because of the deep software stack necessary in the data path.
The OpenFAM API treats data in FAM as memory-resident, and provides memory management and data operation APIs patterned after OpenSHMEM. In this paper, we discuss the design of an open-source reference implementation of the API, and demonstrate its efficiency using micro-benchmarks on a 32-node EDR InfiniBand cluster. We conclude with a discussion of future work and relation to OpenSHMEM.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Wasi-ur-Rahman, M., Lu, X., Islam, N.S., Rajachandrasekar, R., Panda, D.K.: High-Performance design of YARN mapreduce on modern HPC clusters with lustre and RDMA. In: 2015 IEEE International Parallel and Distributed Processing Symposium, May 2015, pp. 291–300 (2015). https://doi.org/10.1109/IPDPS.2015.83
Ramirez-Gargallo, G., Garcia-Gasulla, M., Mantovani, F.: TensorFlow on state-of-the-art HPC clusters: a machine learning use case. In: 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), May 2019, pp. 526–533 (2019). https://doi.org/10.1109/CCGRID.2019.00067
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., Rellermeyer, J.S.: A survey on distributed machine learning. ACM Comput. Surv. 53(2), 30:1–30:33 (2020). https://doi.org/10.1145/3377454
Memory Bandwidth and System Balance in HPC Systems Archives. SC16. http://sc16.supercomputing.org/tag/memory-bandwidth-and-system-balance-in-hpc-systems. Accessed 21 June 2021
Peng, I., Pearce, R., Gokhale, M.: On the memory underutilization: exploring disaggregated memory on HPC systems. In: 2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), September 2020, pp. 183–190 (2020). https://doi.org/10.1109/SBAC-PAD49847.2020.00034
De Sensi, D., Di Girolamo, S., McMahon, K.H., Roweth, D., Hoefler, T.: An in-depth analysis of the slingshot interconnect. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, November 2020, pp. 1–14 (2020). https://doi.org/10.1109/SC41405.2020.00039
Weiland, M., et al.: An early evaluation of Intel’s optane DC persistent memory module and its impact on high-performance scientific applications. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, New York, NY, USA, November 2019, pp. 1–19 (2019). https://doi.org/10.1145/3295500.3356159
Keeton, K., et al.: MODC: resilience for disaggregated memory architectures using task-based programming. In: Workshop on Resources Disaggregation and Serverless, April 2021. https://wuklab.github.io/words/
Grodowitz, M., Shamis, P., Poole, S.: OpenSHMEM I/O extensions for fine-grained access to persistent memory storage. In: Nichols, J., et al. (eds.) SMC 2020. CCIS, vol. 1315, pp. 318–333. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63393-6_21
DAOS and Intel® OptaneTM Technology for High-Performance Storage. Intel. https://www.intel.com/content/www/us/en/high-performance-computing/daos-high-performance-storage-brief.html. Accessed 27 Aug 2020
Keeton, K., Singhal, S., Raymond, M.: The OpenFAM API: a programming model for disaggregated persistent memory. In: Pophale, S., Imam, N., Aderholdt, F., GorentlaVenkata, M. (eds.) OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Extreme Heterogeneity. LNCS, vol. 11283, pp. 70–89. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-04918-8_5
OpenSHMEM Specification 1.5. http://openshmem.org/site/Specification. Accessed 05 Sep 2020
Becker, M., et al.: Accelerated genomics data processing using memory-driven computing. In: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), November 2019, pp. 1850–1855. https://doi.org/10.1109/BIBM47256.2019.8983296
Becker, M., et al.: Scaling genomics data processing with memory-driven computing to accelerate computational biology. In: Sadayappan, P., Chamberlain, B.L., Juckeland, G., Ltaief, H. (eds.) High Performance Computing. Lecture Notes in Computer Science, vol. 12151, pp. 328–344. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50743-5_17
OpenFAM. https://github.com/OpenFAM. Accessed 06 Sep 2020
Yoo, A., Jette, M., Grondona, M.: SLURM: simple linux utility for resource management. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 44–60. Springer, Heidelberg (2003). https://doi.org/10.1007/10968987_3
Libfabric Programmer’s Manual. https://ofiwg.github.io/libfabric/. Accessed 05 Sep 2020
“PMI v2 API - Mpich. https://wiki.mpich.org/mpich/index.php/PMI_v2_API. Accessed 21 June 2021
“gRPC: A high performance, open source universal RPC framework. gRPC. https://grpc.io/. Accessed 05 Sep 2020
HewlettPackard/gull. Hewlett Packard Enterprise (2021). https://github.com/HewlettPackard/gull. Accessed 21 June 2021
HewlettPackard/meadowlark. Hewlett Packard Enterprise (2021). https://github.com/HewlettPackard/meadowlark. Accessed 21 June 2021
OpenFAM API Examples. OpenFAM (2021). https://github.com/OpenFAM/OpenFAM/tree/master/examples/api. Accessed 03 Sep 2021
OpenFAM Example Applications. OpenFAM (2021). https://github.com/OpenFAM/OpenFAM/tree/master/test/apps. Accessed 03 Sep 2021
“ibv_rc_pingpong(1) - Linux manual page. https://man7.org/linux/man-pages/man1/ibv_rc_pingpong.1.html. Accessed 21 June 2021
Large Atomic Transfer Library for OpenFAM. OpenFAM (2021). https://github.com/OpenFAM/OpenFAM_ATL. Accessed 06 July 2021
OpenSHMEM Memory Spaces. GitHub: OpenSHMEM (2021). https://github.com/openshmem-org/specification/wiki/Memory-Spaces. Accessed 21 June 2021
Ma, T., Zhang, M., Chen, K., Song, Z., Wu, Y., Qian, X.: AsymNVM: an efficient framework for implementing persistent data structures on asymmetric NVM architecture. In: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, New York, NY, USA, March 2020, pp. 757–773 (2020). https://doi.org/10.1145/3373376.3378511.
Shan, Y., Tsai, S.-Y., Zhang, Y.: Distributed shared persistent memory. In: Proceedings of the 2017 Symposium on Cloud Computing, Santa Clara, California, September 2017, pp. 323–337 (2017). https://doi.org/10.1145/3127479.3128610
Calciu, I., et al.: Rethinking software runtimes for disaggregated memory. In: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, New York, NY, USA, pp. 79–92. Association for Computing Machinery (2021). https://doi.org/10.1145/3445814.3446713. Accessed 14 May 2021
Kommareddy, V.R., Hughes, C, Hammond, S.D., Awad, A.: DeACT: Architecture-Aware Virtual Memory Support for Fabric Attached Memory Systems, August 2020. http://arxiv.org/abs/2008.00171. Accessed 09 Nov 2020
Aguilera, M.K., et al.: Remote regions: a simple abstraction for remote memory, pp. 775–787 (2018). https://www.usenix.org/conference/atc18/presentation/aguilera. Accessed 02 May 2020
Yu, S., Xiao, N., Deng, M., Xing, Y., Liu, F., Chen, W.: Megalloc: fast distributed memory allocator for NVM-based cluster. In: 2017 International Conference on Networking, Architecture, and Storage (NAS), August 2017, pp. 1–9 (2017). https://doi.org/10.1109/NAS.2017.8026865
Giles, E., Doshi, K., Varman, P.: Bridging the programming gap between persistent and volatile memory using WrAP. In: Proceedings of the ACM International Conference on Computing Frontiers, New York, NY, USA, May 2013, pp. 1–10. https://doi.org/10.1145/2482767.2482806.
Tsai, S.-Y., Zhang, Y.: Building Atomic, Crash-Consistent Data Stores with Disaggregated Persistent Memory, January 2019. https://arxiv.org/abs/1901.01628v1. Accessed 12 May 2020
Ruan, Z., Schwarzkopf, M., Aguilera, M.K., Belay, A.: AIFM: high-performance, application-integrated far memory, pp. 315–332 (2020). https://www.usenix.org/conference/osdi20/presentation/ruan. Accessed 22 June 2021
Zamanian, E., Yu, X., Stonebraker, M., Kraska, T.: Rethinking database high availability with RDMA networks. Proc. VLDB Endow. 12(11), 1637–1650 (2019). https://doi.org/10.14778/3342263.3342639
Tsai, S.-Y., Shan, Y., Zhang, Y.: Disaggregating Persistent Memory and Controlling Them Remotely: An Exploration of Passive Disaggregated Key-Value Stores, pp. 33–48 (2020). https://www.usenix.org/conference/atc20/presentation/tsai. Accessed 30 Oct 2020
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Singhal, S. et al. (2022). OpenFAM: A Library for Programming Disaggregated Memory. In: Poole, S., Hernandez, O., Baker, M., Curtis, T. (eds) OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Exascale and Smart Networks. OpenSHMEM 2021. Lecture Notes in Computer Science, vol 13159. Springer, Cham. https://doi.org/10.1007/978-3-031-04888-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-04888-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04887-6
Online ISBN: 978-3-031-04888-3
eBook Packages: Computer ScienceComputer Science (R0)