Skip to main content
SpringerLink
Log in

OpenFAM: A Library for Programming Disaggregated Memory

  • Conference paper
  • First Online:
OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Exascale and Smart Networks (OpenSHMEM 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13159))

Included in the following conference series:

  • 333 Accesses

Abstract

HPC architectures are increasingly handling workloads such as AI/ML or high performance data analytics where the working data set cannot be easily partitioned, or does not fit into node local memory. This poses challenges for programming models such as OpenSHMEM, which require data in the working set to fit in the symmetric heap. Emerging fabric-attached memory (FAM) architectures enable data to be held in external memory accessible to all compute nodes, thus providing a new approach to handling large data sets. Unfortunately, most HPC libraries do not currently support FAM, and programmers use file system or key-value store abstractions to access data that is resident off-node, resulting in lower application performance because of the deep software stack necessary in the data path.

The OpenFAM API treats data in FAM as memory-resident, and provides memory management and data operation APIs patterned after OpenSHMEM. In this paper, we discuss the design of an open-source reference implementation of the API, and demonstrate its efficiency using micro-benchmarks on a 32-node EDR InfiniBand cluster. We conclude with a discussion of future work and relation to OpenSHMEM.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Wasi-ur-Rahman, M., Lu, X., Islam, N.S., Rajachandrasekar, R., Panda, D.K.: High-Performance design of YARN mapreduce on modern HPC clusters with lustre and RDMA. In: 2015 IEEE International Parallel and Distributed Processing Symposium, May 2015, pp. 291–300 (2015). https://doi.org/10.1109/IPDPS.2015.83

  2. Ramirez-Gargallo, G., Garcia-Gasulla, M., Mantovani, F.: TensorFlow on state-of-the-art HPC clusters: a machine learning use case. In: 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), May 2019, pp. 526–533 (2019). https://doi.org/10.1109/CCGRID.2019.00067

  3. Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., Rellermeyer, J.S.: A survey on distributed machine learning. ACM Comput. Surv. 53(2), 30:1–30:33 (2020). https://doi.org/10.1145/3377454

  4. Memory Bandwidth and System Balance in HPC Systems Archives. SC16. http://sc16.supercomputing.org/tag/memory-bandwidth-and-system-balance-in-hpc-systems. Accessed 21 June 2021

  5. Peng, I., Pearce, R., Gokhale, M.: On the memory underutilization: exploring disaggregated memory on HPC systems. In: 2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), September 2020, pp. 183–190 (2020). https://doi.org/10.1109/SBAC-PAD49847.2020.00034

  6. De Sensi, D., Di Girolamo, S., McMahon, K.H., Roweth, D., Hoefler, T.: An in-depth analysis of the slingshot interconnect. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, November 2020, pp. 1–14 (2020). https://doi.org/10.1109/SC41405.2020.00039

  7. Weiland, M., et al.: An early evaluation of Intel’s optane DC persistent memory module and its impact on high-performance scientific applications. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, New York, NY, USA, November 2019, pp. 1–19 (2019). https://doi.org/10.1145/3295500.3356159

  8. Keeton, K., et al.: MODC: resilience for disaggregated memory architectures using task-based programming. In: Workshop on Resources Disaggregation and Serverless, April 2021. https://wuklab.github.io/words/

  9. Grodowitz, M., Shamis, P., Poole, S.: OpenSHMEM I/O extensions for fine-grained access to persistent memory storage. In: Nichols, J., et al. (eds.) SMC 2020. CCIS, vol. 1315, pp. 318–333. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63393-6_21

    Chapter  Google Scholar 

  10. DAOS and Intel® OptaneTM Technology for High-Performance Storage. Intel. https://www.intel.com/content/www/us/en/high-performance-computing/daos-high-performance-storage-brief.html. Accessed 27 Aug 2020

  11. Keeton, K., Singhal, S., Raymond, M.: The OpenFAM API: a programming model for disaggregated persistent memory. In: Pophale, S., Imam, N., Aderholdt, F., GorentlaVenkata, M. (eds.) OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Extreme Heterogeneity. LNCS, vol. 11283, pp. 70–89. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-04918-8_5

    Chapter  Google Scholar 

  12. OpenSHMEM Specification 1.5. http://openshmem.org/site/Specification. Accessed 05 Sep 2020

  13. Becker, M., et al.: Accelerated genomics data processing using memory-driven computing. In: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), November 2019, pp. 1850–1855. https://doi.org/10.1109/BIBM47256.2019.8983296

  14. Becker, M., et al.: Scaling genomics data processing with memory-driven computing to accelerate computational biology. In: Sadayappan, P., Chamberlain, B.L., Juckeland, G., Ltaief, H. (eds.) High Performance Computing. Lecture Notes in Computer Science, vol. 12151, pp. 328–344. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50743-5_17

    Chapter  Google Scholar 

  15. OpenFAM. https://github.com/OpenFAM. Accessed 06 Sep 2020

  16. Yoo, A., Jette, M., Grondona, M.: SLURM: simple linux utility for resource management. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 44–60. Springer, Heidelberg (2003). https://doi.org/10.1007/10968987_3

    Chapter  Google Scholar 

  17. Libfabric Programmer’s Manual. https://ofiwg.github.io/libfabric/. Accessed 05 Sep 2020

  18. “PMI v2 API - Mpich. https://wiki.mpich.org/mpich/index.php/PMI_v2_API. Accessed 21 June 2021

  19. “gRPC: A high performance, open source universal RPC framework. gRPC. https://grpc.io/. Accessed 05 Sep 2020

  20. HewlettPackard/gull. Hewlett Packard Enterprise (2021). https://github.com/HewlettPackard/gull. Accessed 21 June 2021

  21. HewlettPackard/meadowlark. Hewlett Packard Enterprise (2021). https://github.com/HewlettPackard/meadowlark. Accessed 21 June 2021

  22. OpenFAM API Examples. OpenFAM (2021). https://github.com/OpenFAM/OpenFAM/tree/master/examples/api. Accessed 03 Sep 2021

  23. OpenFAM Example Applications. OpenFAM (2021). https://github.com/OpenFAM/OpenFAM/tree/master/test/apps. Accessed 03 Sep 2021

  24. “ibv_rc_pingpong(1) - Linux manual page. https://man7.org/linux/man-pages/man1/ibv_rc_pingpong.1.html. Accessed 21 June 2021

  25. Large Atomic Transfer Library for OpenFAM. OpenFAM (2021). https://github.com/OpenFAM/OpenFAM_ATL. Accessed 06 July 2021

  26. OpenSHMEM Memory Spaces. GitHub: OpenSHMEM (2021). https://github.com/openshmem-org/specification/wiki/Memory-Spaces. Accessed 21 June 2021

  27. Ma, T., Zhang, M., Chen, K., Song, Z., Wu, Y., Qian, X.: AsymNVM: an efficient framework for implementing persistent data structures on asymmetric NVM architecture. In: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, New York, NY, USA, March 2020, pp. 757–773 (2020). https://doi.org/10.1145/3373376.3378511.

  28. Shan, Y., Tsai, S.-Y., Zhang, Y.: Distributed shared persistent memory. In: Proceedings of the 2017 Symposium on Cloud Computing, Santa Clara, California, September 2017, pp. 323–337 (2017). https://doi.org/10.1145/3127479.3128610

  29. Calciu, I., et al.: Rethinking software runtimes for disaggregated memory. In: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, New York, NY, USA, pp. 79–92. Association for Computing Machinery (2021). https://doi.org/10.1145/3445814.3446713. Accessed 14 May 2021

  30. Kommareddy, V.R., Hughes, C, Hammond, S.D., Awad, A.: DeACT: Architecture-Aware Virtual Memory Support for Fabric Attached Memory Systems, August 2020. http://arxiv.org/abs/2008.00171. Accessed 09 Nov 2020

  31. Aguilera, M.K., et al.: Remote regions: a simple abstraction for remote memory, pp. 775–787 (2018). https://www.usenix.org/conference/atc18/presentation/aguilera. Accessed 02 May 2020

  32. Yu, S., Xiao, N., Deng, M., Xing, Y., Liu, F., Chen, W.: Megalloc: fast distributed memory allocator for NVM-based cluster. In: 2017 International Conference on Networking, Architecture, and Storage (NAS), August 2017, pp. 1–9 (2017). https://doi.org/10.1109/NAS.2017.8026865

  33. Giles, E., Doshi, K., Varman, P.: Bridging the programming gap between persistent and volatile memory using WrAP. In: Proceedings of the ACM International Conference on Computing Frontiers, New York, NY, USA, May 2013, pp. 1–10. https://doi.org/10.1145/2482767.2482806.

  34. Tsai, S.-Y., Zhang, Y.: Building Atomic, Crash-Consistent Data Stores with Disaggregated Persistent Memory, January 2019. https://arxiv.org/abs/1901.01628v1. Accessed 12 May 2020

  35. Ruan, Z., Schwarzkopf, M., Aguilera, M.K., Belay, A.: AIFM: high-performance, application-integrated far memory, pp. 315–332 (2020). https://www.usenix.org/conference/osdi20/presentation/ruan. Accessed 22 June 2021

  36. Zamanian, E., Yu, X., Stonebraker, M., Kraska, T.: Rethinking database high availability with RDMA networks. Proc. VLDB Endow. 12(11), 1637–1650 (2019). https://doi.org/10.14778/3342263.3342639

    Article  Google Scholar 

  37. Tsai, S.-Y., Shan, Y., Zhang, Y.: Disaggregating Persistent Memory and Controlling Them Remotely: An Exploration of Passive Disaggregated Key-Value Stores, pp. 33–48 (2020). https://www.usenix.org/conference/atc20/presentation/tsai. Accessed 30 Oct 2020

Download references

Author information

Authors and Affiliations

  1. Hewlett Packard Enterprise, San Jose, CA, 95002, USA

    Sharad Singhal & Dave Emberson

  2. Hewlett Packard Enterprise, Bangalore, 560 048, Karnataka, India

    Mashood Abdulla, Faizan Barmawer, Ramya Ahobala, Gautham Bhat, Rishi kesh K. Rajak & P. N. Soumya

  3. Hewlett Packard Enterprise, Highland, NY, 12528, USA

    Clarete R. Crasta

Authors
  1. Sharad Singhal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Clarete R. Crasta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mashood Abdulla
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Faizan Barmawer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dave Emberson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ramya Ahobala
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gautham Bhat
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Rishi kesh K. Rajak
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. P. N. Soumya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sharad Singhal .

Editor information

Editors and Affiliations

  1. Los Alamos National Laboratory, Los Almos, NM, USA

    Stephen Poole

  2. NVIDIA Corporation, Santa Clara, CA, USA

    Oscar Hernandez

  3. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Matthew Baker

  4. Stony Brook University, Stony Brook, NY, USA

    Tony Curtis

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Singhal, S. et al. (2022). OpenFAM: A Library for Programming Disaggregated Memory. In: Poole, S., Hernandez, O., Baker, M., Curtis, T. (eds) OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Exascale and Smart Networks. OpenSHMEM 2021. Lecture Notes in Computer Science, vol 13159. Springer, Cham. https://doi.org/10.1007/978-3-031-04888-3_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-04888-3_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-04887-6

  • Online ISBN: 978-3-031-04888-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation