Skip to main content
SpringerLink
Log in

HMvisor: dynamic hybrid memory management for virtual machines

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Emerging non-volatile memory (NVM) technologies promise high density, low cost and dynamic random access memory (DRAM)-like performance, at the expense of limited write endurance and high write energy consumption. It is more practical to use NVM combining with the traditional DRAM. However, the hybrid memory management such as page migration becomes more challenging in a virtualization environment because virtual machines (VMs) are unaware of the memory heterogeneity. In this paper, we propose HMvisor, a hypervisor and VM coordinated hybrid memory management mechanism to better utilize DRAM and NVM resources. HMvisor exposes the memory heterogeneity to VMs by mapping virtual NUMA nodes to different physical NUMA nodes. We propose a lightweight and efficient page migration mechanism by decoupling page hotness tracking from page migration. HMvisor performs those operations in the hypervisor and VMs separately, without disrupting the execution of VMs. We also propose a memory resource trading policy to adjust the capacity of DRAM and NVM for each VM, with the monetary cost unchanged. We implement our prototype system based on QEMU/KVM and evaluate it with several benchmarks. Experimental results show that HMvisor can reduce 50% of write traffic to NVM with less than 5% performance overhead. Moreover, the hybrid memory adjustment scheme in HMvisor can significantly improve application performance by up to 30 ×.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. McKee S. Reflections on the memory wall. In: Proceedings of the 1st Conference on Computing Frontiers, Ischia, 2004. 162–167

  2. Meena J, Sze S, Chand U, et al. Overview of emerging nonvolatile memory technologies. Nanoscale Res Lett, 2014, 9: 526

    Article  Google Scholar 

  3. Pistol C, Chongchitmate W, Dwyer C, et al. Architectural implications of nanoscale integrated sensing and computing. SIGARCH Comput Archit News, 2009, 37: 13

    Article  Google Scholar 

  4. Li J, Lam C. Phase change memory. Sci China Inf Sci, 2011, 54: 1061–1072

    Article  Google Scholar 

  5. Hady F T, Foong A, Veal B, et al. Platform storage performance with 3D XPoint technology. Proc IEEE, 2017, 105: 1822–1833

    Article  Google Scholar 

  6. Dulloor S R, Roy A, Zhao Z, et al. Data tiering in heterogeneous memory systems. In: Proceedings of the 11th European Conference on Computer Systems, London, 2016. 1–16

  7. Qureshi M K, Srinivasan V, Rivers J A. Scalable high performance main memory system using phase-change memory technology. SIGARCH Comput Archit News, 2009, 37: 24–33

    Article  Google Scholar 

  8. Black B, Annavaram M, Brekelbaum N, et al. Die stacking (3D) microarchitecture. In: Proceedings of the 39th Annual IEEE/ACM International Symposium on Microarchitecture, Orlando, 2006. 469–479

  9. Radulovic M, Zivanovic D, Ruiz D, et al. Another trip to the wall: how much will stacked DRAM benefit HPC? In: Proceedings of 2015 International Symposium on Memory Systems, Washington, 2015. 31–36

  10. Kannan S, Gavrilovska A, Schwan K. pVM: persistent virtual memory for efficient capacity scaling and object storage. In: Proceedings of the 11th European Conference on Computer Systems, London, 2016. 1–16

  11. Lin F X, Liu X. Memif: towards programming heterogeneous memory asynchronously. SIGARCH Comput Archit News, 2016, 44: 369–383

    Article  Google Scholar 

  12. Coburn J, Caulfield A M, Akel A, et al. NV-Heaps: making persistent objects fast and safe with next-generation. SIGPLAN Not, 2012, 47: 105–118

    Article  Google Scholar 

  13. Kannan S, Gavrilovska A, Gupta V, et al. HeteroOS: OS design for heterogeneous memory management in datacenters. SIGOPS Oper Syst Rev, 2018, 51: 13–26

    Article  Google Scholar 

  14. Meswani M R, Blagodurov S, Roberts D, et al. Heterogeneous memory architectures: a HW/SW approach for mixing die-stacked and off-package memories. In: Proceedings of 2015 IEEE 21st International Symposium on High Performance Computer Architecture, San Francisco, 2015. 126–136

  15. Chou C, Jaleel A, Qureshi M. BATMAN: techniques for maximizing system bandwidth of memory systems with stacked-DRAM. In: Proceedings of International Symposium on Memory Systems, Alexandria, 2017. 268–280

  16. Meza J, Chang J, Yoon H B, et al. Enabling efficient and scalable hybrid memories using fine-granularity DRAM cache management. IEEE Comput Arch Lett, 2012, 11: 61–64

    Article  Google Scholar 

  17. Ramos L E, Gorbatov E, Bianchini R. Page placement in hybrid memory systems. In: Proceedings of International Conference on Supercomputing, Tucson, 2011. 85–95

  18. Dong X, Xie Y, Muralimanohar N, et al. Simple but effective heterogeneous main memory with on-chip memory controller support. In: Proceedings of 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, New Orleans, 2010. 1–11

  19. Agarwal N, Wenisch T F. Thermostat: application-transparent page management for two-tiered main memory. In: Proceedings of the 22nd International Conference on Architectural Support for Programming Languages and Operating Systems, Xi’an, 2017. 631–644

  20. Gandhi J, Basu A, Hill M D, et al. BadgerTrap. SIGARCH Comput Archit News, 2014, 42: 20–23

    Article  Google Scholar 

  21. Gupta V, Lee M, Schwan K. BadgerTrap: a tool to instrument x86-64 TLB misses. SIGPLAN Not, 2015, 50: 79–92

    Article  Google Scholar 

  22. Hirofuchi T, Takano R. RAMinate: hypervisor-based virtualization for hybrid main memory systems. In: Proceedings of the 7th ACM Symposium on Cloud Computing, Santa Clara, 2016. 112–125

  23. Kivity A, Kamay Y, Laor D, et al. KVM: the Linux virtual machine monitor. In: Proceedings of Linux Symposium, Ottawa, 2007. 225–230

  24. Dhiman G, Ayoub R, Rosing T. PDRAM: a hybrid PRAM and DRAM main memory system. In: Proceedings of the 46th Annual Design Automation Conference, San Francisco, 2009. 664–669

  25. Zhang W, Li T. Exploring phase change memory and 3D die-stacking for power/thermal friendly, fast and durable memory architectures. In: Proceedings of 2009 18th International Conference on Parallel Architectures and Compilation Techniques, Raleigh, 2009. 101–112

  26. Lee S, Bahn H, Noh S H. CLOCK-DWF: a write-history-aware page replacement algorithm for hybrid PCM and DRAM memory architectures. IEEE Trans Comput, 2014, 63: 2187–2200

    Article  MathSciNet  Google Scholar 

  27. Dybdahl H, Stenstrom P, Natvig L. An LRU-based replacement algorithm augmented with frequency of access in shared chip-multiprocessor caches. SIGARCH Comput, 2007, 35: 45–52

    Article  Google Scholar 

  28. Liu H, He B. Reciprocal resource fairness: towards cooperative multiple resource fair sharing in IaaS clouds. In: Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis, New Orleans, 2014. 970–981

  29. Liu H, He B. F2C: enabling fair and fine-grained resource sharing in multi-tenant IaaS clouds. IEEE Trans Parallel Distrib Syst, 2016, 37: 2589–2602

    Article  Google Scholar 

  30. Chen H G, Wang X L, Wang Z L, et al. DMM: a dynamic memory mapping model for virtual machines. Sci China Inf Sci, 2010, 53: 1097–1108

    Article  Google Scholar 

  31. Russell R. Virtio: towards a de-facto standard for virtual I/O devices. ACM SIGOPS Operat Syst Rev, 2008, 42: 95–103

    Article  Google Scholar 

  32. Liu H, Jin H, Liao X, et al. Hotplug or ballooning: a comparative study on dynamic memory management techniques for virtual machines. IEEE Trans Parallel Distrib Syst, 2015, 26: 1350–1363

    Article  Google Scholar 

  33. David R. Linux fake NUMA patch. 2007. https://www.kernel.org/doc/Documentation/x86/x86_64/fake-numa-for-cpusets

  34. Luk C K, Cohn R, Muth R, et al. Pin: building customized program analysis tools with dynamic instrumentation. In: Proceedings of 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, Chicago, 2005. 190–200

  35. Zhou Y, Chen Z, Li K. Second-level buffer cache management. IEEE Trans Parallel Distrib Syst, 2004, 15: 505–519

    Article  Google Scholar 

  36. Sanchez D, Kozyrakis C. ZSim: fast and accurate microarchitectural simulation of thousand-core systems. SIGARCH Comput Archit News, 2013, 41: 475–486

    Article  Google Scholar 

  37. Dulloor S R, Kumar S, Keshavamurthy A, et al. System software for persistent memory. In: Proceedings of the 9th European Conference on Computer Systems, Amsterdam, 2014. 1–15

  38. Volos H, Magalhaes G, Cherkasova L, et al. Quartz: a lightweight performance emulator for persistent memory software. In: Proceedings of the 16th Annual Middleware Conference, Vancouver, 2015. 37–49

  39. Duan Z, Liu H, Liao X, et al. HME: a lightweight emulator for hybrid memory. In: Proceedings of 2018 Design, Automation Test in Europe Conference Exhibition, Dresden, 2018. 1375–1380

  40. Ham T J, Chelepalli B K, Xue N, et al. Disintegrated control for energy-efficient and heterogeneous memory systems. In: Proceedings of 2013 IEEE 19th International Symposium on High Performance Computer Architecture, Shenzhen, 2013. 424–435

  41. Long L, Liu D, Liang L, et al. Morphable resistive memory optimization for mobile virtualization. IEEE Trans Comput-Aided Des Integr Circuits Syst, 2016, 35: 891–904

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported jointly by National Natural Science Foundation of China (Grants Nos. 61672251, 61732010, 61825202).

Author information

Authors and Affiliations

  1. National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, 430074, China

    Dang Yang, Haikun Liu, Hai Jin & Yu Zhang

Authors
  1. Dang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haikun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hai Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haikun Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, D., Liu, H., Jin, H. et al. HMvisor: dynamic hybrid memory management for virtual machines. Sci. China Inf. Sci. 64, 192104 (2021). https://doi.org/10.1007/s11432-019-2729-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-019-2729-5

Keywords

Search

Navigation