Marcos K. Aguilera
Nadav Amit
Gerd Zellweger
Skip Abstract Section
Abstract
Hardware disaggregation has emerged as one of the most fundamental shifts in how we build computer systems over the past decades. While disaggregation has been successful for several types of resources (storage, power, and others), memory disaggregation has yet to happen. We make the case that the time for memory disaggregation has arrived. We look at past successful disaggregation stories and learn that their success depended on two requirements: addressing a burning issue and being technically feasible. We examine memory disaggregation through this lens and find that both requirements are finally met. Once available, memory disaggregation will require software support to be used effectively. We discuss some of the challenges of designing an operating system that can utilize disaggregated memory for itself and its applications.
- Emmanuel Amaro, Christopher Branner-Augmon, Zhihong Luo, Amy Ousterhout, Marcos K Aguilera, Aurojit Panda, Sylvia Ratnasamy, and Scott Shenker. Can far memory improve job throughput? In European Conference on Computer Systems, pages 1--16, April 2020.Google Scholar
Digital Library
- Cristiana Amza, Alan L. Cox, Shandya Dwarkadas, Pete Keleher, Honghui Lu, Ramakrishnan Rajamony, Weimin Yu, and Willy Zwaenepoel. Tread- Marks: Shared memory computing on networks of workstations. IEEE Computer, 29(2):18--28, February 1996.Google ScholarDigital Library
- Krste Asanovi´c. FireBox: A hardware building block for 2020 Warehouse-Scale computers. In USENIX Conference on File and Storage Technologies, February 2014. Keynote talk.Google Scholar
- J. K. Bennett, J. B. Carter, and W. Zwaenepoel. Munin: Distributed shared memory based on typespecific memory coherence. In ACM Symposium on Principles and Practice of Parallel Programming, pages 168--176, March 1990.Google ScholarDigital Library
- Maciej Bielski, Ilias Syrigos, Kostas Katrinis, Dimitris Syrivelis, Andrea Reale, Dimitris Theodoropoulos, Nikolaos Alachiotis, Dionisios N. Pnevmatikatos, Evert H. Pap, Georgios Zervas, Vaibhawa Mishra, Arsalan Saljoghei, Alvise Rigo, Jose Fernando Zazo, Sergio Lopez-Buedo, Martí Torrents, Ferad Zyulkyarov, Michael Enrico, and Oscar Gonzalez de Dios. dReDBox: Materializing a full-stack rack-scale system prototype of a next-generation disaggregated datacenter. In Design, Automation & Test in Europe Conference & Exhibition, pages 1093--1098, March 2018.Google Scholar
- VMware Bitfusion. https://core.vmware.com/ bitfusion.Google Scholar
- Qingchao Cai,Wentian Guo, Hao Zhang, Divyakant Agrawal, Gang Chen, Beng Chin Ooi, Kian-Lee Tan, Yong Meng Teo, and Sheng Wang. Efficient distributed memory management with RDMA and caching. Proceedings of the VLDB Endowment, 11(11):1604--1617, July 2018.Google ScholarDigital Library
- Irina Calciu, M. Talha Imran, Ivan Puddu, Sanidhya Kashyap, Hasan Al Maruf, Onur Mutlu, and Aasheesh Kolli. Rethinking software runtimes for disaggregated memory. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, April 2021.Google ScholarDigital Library
- Compute eXpress Link. https://www. computeexpresslink.org.Google Scholar
- Paolo Faraboschi, Kimberly Keeton, Tim Marsland, and Dejan Milojicic. Beyond processor-centric operating systems. In Workshop on Hot Topics in Operating Systems, May 2015.Google Scholar
- E. Felten and J. Zahorjan. Issues in the implementation of a remote memory paging system. Technical Report CSE TR 91-03-09, University of Washington, March 1991.Google Scholar
- Gen-Z consortium. https://en.wikipedia. org/wiki/Gen-Z_(consortium).Google Scholar
- Donghyun Gouk, Sangwon Lee, Miryeong Kwon, and Myoungsoo Jung. Direct access, highperformance memory disaggregation with DirectCXL. In USENIX Annual Technical Conference, pages 287--294, June 2022.Google Scholar
- Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G Shin. Efficient memory disaggregation with Infiniswap. In Symposium on Networked Systems Design and Implementation, pages 649--667, March 2017.Google Scholar
- Hannes Hapke and Catherine Nelson. Building Machine Learning Pipelines. O'Reilly Media, Inc, July 2020.Google Scholar
- Intel rack scale architecture. https: //www-conf.slac.stanford.edu/ xldb2016/talks/published/Tues_6_ Mohan-Kumar-Rack-Scale-XLDB-Updated. pdf.Google Scholar
- Svilen Kanev, Juan Pablo Darago, Kim Hazelwood, Parthasarathy Ranganathan, Tipp Moseley, Gu-Yeon Wei, and David Brooks. Profiling a warehouse-scale computer. In International Symposium on Computer Architecture, pages 158--169, June 2015.Google ScholarDigital Library
- Kimberly Keeton. Memory driven computing. In USENIX Conference on File and Storage Technologies, February 2017. Keynote presentation.Google Scholar
- Seung-seob Lee, Yanpeng Yu, Yupeng Tang, Anurag Khandelwal, Lin Zhong, and Abhishek Bhattacharjee. MIND: In-network memory management for disaggregated data centers. In ACM Symposium on Operating Systems Principles, pages 488--504, October 2021.Google Scholar
- Huaicheng Li, Daniel S. Berger, Stanko Novakovic, Lisa Hsu, Dan Ernst, Pantea Zardoshti, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, and Ricardo Bianchini. Pond: CXL-based memory pooling systems for cloud platforms. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, March 2023.Google ScholarDigital Library
- Kevin Lim, Jichuan Chang, Trevor Mudge, Parthasarathy Ranganathan, Steven K Reinhardt, and Thomas F Wenisch. Disaggregated memory for expansion and sharing in blade servers. ACM SIGARCH Computer Architecture News, 37(3):267--278, June 2009.Google ScholarDigital Library
- Mark Mansi and Michael M. Swift. /0sim: Preparing system software for a world with terabytescale memories. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, page 267--282, March 2020.Google Scholar
- Hasan Al Maruf, HaoWang, Abhishek Dhanotia, JohannesWeiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowdhury, Shobhit Kanaujia, and Prakash Chauhan. TPP: Transparent page placement for CXL-enabled tiered-memory. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, page 742--755, March 2023.Google ScholarDigital Library
- Jacob Nelson, Brandon Holt, Brandon Myers, Preston Briggs, Luis Ceze, Simon Kahan, and Mark Oskin. Latency-tolerant software distributed shared memory. In USENIX Annual Technical Conference, pages 291--305, July 2015.Google ScholarDigital Library
- Open compute project. https://www. opencompute.org.Google Scholar
- Open19. https://www.open19.org.Google Scholar
- OpenVMS. https://en.wikipedia.org/wiki/ OpenVMS.Google Scholar
- Amanda Raybuck, Tim Stamler, Wei Zhang, Mattan Erez, and Simon Peter. HeMem: Scalable tiered memory management for big data applications and real NVM. In ACM Symposium on Operating Systems Principles, pages 392--407, October 2021.Google Scholar
- RDMA over Converged Ethernet. https: //en.wikipedia.org/wiki/RDMA_over_ Converged_Ethernet.Google Scholar
- Zhenyuan Ruan, Malte Schwarzkopf, Marcos K Aguilera, and Adam Belay. AIFM: Highperformance, application-integrated far memory. In Symposium on Operating Systems Design and Implementation, pages 315--332, November 2020.Google Scholar
- Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang. LegoOS: A disseminated, distributed os for hardware resource disaggregation. In Symposium on Operating Systems Design and Implementation, pages 69--87, October 2018.Google Scholar
- Silicon photonics. https://en.wikipedia.org/ wiki/Silicon_photonics.Google Scholar
- Scalable memory development kit. https:// github.com/OpenMPDK/SMDK.Google Scholar
- Paul Teich. HPE powers up The Machine architecture, January 2017. https: //www.nextplatform.com/2017/01/09/ hpe-powers-machine-architecture.Google Scholar
- Stephanie Wang, Eric Liang, Edward Oakes, Benjamin Hindman, Frank Sifei Luan, Audrey Cheng, and Ion Stoica. Ownership: A distributed futures system for fine-grained tasks. In Symposium on Networked Systems Design and Implementation, pages 671--686, April 2021.Google Scholar
- Zi Yan, Daniel Lustig, David Nellans, and Abhishek Bhattacharjee. Nimble page management for tiered memory systems. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 331--345, April 2019.Google ScholarDigital Library
- Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, and Ion Stoica. Spark: Cluster computing with working sets. In Workshop on Hot Topics in Cloud Computing, June 2010.Google Scholar
- Yang Zhou, Hassan MG Wassel, Sihang Liu, Jiaqi Gao, James Mickens, Minlan Yu, Chris Kennelly, Paul Turner, David E Culler, Henry M Levy, et al. Carbink: Fault-tolerant far memory. In Symposium on Operating Systems Design and Implementation, pages 55--71, July 2022. 46Google Scholar
Recommendations
Memory Disaggregation: Advances and Open Challenges
SIGOPSCompute and memory are tightly coupled within each server in traditional datacenters. Large-scale datacenter operators have identified this coupling as a root cause behind fleetwide resource underutilization and increasing Total Cost of Ownership (TCO). ...
Rcmp: Reconstructing RDMA-Based Memory Disaggregation via CXL
Memory disaggregation is a promising architecture for modern datacenters that separates compute and memory resources into independent pools connected by ultra-fast networks, which can improve memory utilization, reduce cost, and enable elastic scaling of ...
Poster: Designing a Memory Disaggregation System for Cloud
ACM SIGCOMM '23: Proceedings of the ACM SIGCOMM 2023 ConferenceMemory disaggregation is a new datacenter paradigm separating compute and memory nodes. While memory disaggregation improves memory utilization and scalability, it poses challenges for cloud applications, particularly in terms of high tail latency. ...
Comments