Vlad Nitu
Binoy Ravindran
ABSTRACT
The ability to quickly set up and tear down a virtual machine is critical for today's cloud elasticity, as well as in numerous other scenarios: guest migration/consolidation, event-driven invocation of micro-services, dynamically adaptive unikernel-based applications, micro-reboots for security or stability, etc.
In this paper, we focus on the process of setting up/freeing the hypervisor and host control layer data structures at boot/destruction time, showing that it does not scale in current virtualization solutions. In addition to the direct overhead of long VM set-up/destruction times, we demonstrate by experimentation the indirect costs on real world auto scaling systems. Focusing on the popular Xen hypervisor, we identify three critical issues hindering the scalability of the boot and destruction processes: serialized boot, unscalable interactions with the Xenstore at guest creation time, and remote NUMA memory scrubbing at destruction time. For each of these issues we present the design and implementation of a solution in the Xen infrastructure: parallel boot with fine-grained locking, caching of Xenstore data, and local NUMA scrubbing. We evaluate these solutions using micro-benchmarks, macro-benchmarks, and real world datacenter traces. Results show that our work improves the current Xen implementation by a significant factor, for example macro-benchmarks indicate a speedup of more than 4X in high-load scenarios.
- Amazon EC2 Spot Instances. https://aws.amazon.com/ec2/spot/, accessed 2016-11-30.Google Scholar
- AWS Auto Scaling. https://aws.amazon.com/autoscaling/, accessed 2016-11-30.Google Scholar
- Why does azure deployment take so long? http://stackoverflow.com/questions/5080445/why-does-azure-deployment-take-so-long, accessed 2016-11-30.Google Scholar
- CRIU Wiki. https://criu.org/Main_Page, accessed 2016-11-30.Google Scholar
- Amazon Web Services. https://aws.amazon.com/, accessed 2016-11-30.Google Scholar
- Unikernel.org website. http://unikernel.org/, accessed 2016-11-30.Google Scholar
- Xen on NUMA Machines. https://wiki.xen.org/wiki/Xen_on_NUMA_Machines, accessed 2016-11-30.Google Scholar
- Xen 4.3 NUMA Aware Scheduling. https://wiki.xen.org/wiki/Xen_4.3_NUMA_Aware_Scheduling, accessed 2016-11-30, 2016.Google Scholar
- M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia. A view of cloud computing. Commun. ACM, 53(4):50--58, Apr. 2010. ISSN 0001-0782. doi: 10. 1145/1721654.1721672. http://doi.acm.org/10.1145/1721654.1721672.Google ScholarDigital Library
- D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S. Schreiber, et al. The nas parallel benchmarks. International Journal of High Performance Computing Applications, 5(3):63--73, 1991. Google ScholarDigital Library
- N. Chohan, C. Castillo, M. Spreitzer, M. Steinder, A. N. Tantawi, and C. Krintz. See spot run: Using spot instances for mapreduce workflows. HotCloud, 10:7--7, 2010.Google ScholarDigital Library
- P. Colp, M. Nanavati, J. Zhu, W. Aiello, G. Coker, T. Deegan, P. Loscocco, and A. Warfield. Breaking up is hard to do: security and functionality in a commodity hypervisor. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, pages 189--202. ACM, 2011. Google ScholarDigital Library
- B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, I. Pratt, A. Warfield, P. Barham, and R. Neugebauer. Xen and the art of virtualization. In Proceedings of the ACM Symposium on Operating Systems Principles, October 2003.Google Scholar
- B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, I. Pratt, A. Warfield, P. Barham, and R. Neugebauer. Xen and the art of virtualization. In Proceedings of the ACM Symposium on Operating Systems Principles, October 2003.Google Scholar
- T. Gazagnaire and V. Hanquez. Oxenstored: an efficient hierarchical and transactional database using functional programming with reference cell comparisons. In ACM Sigplan Notices, volume 44, pages 203--214. ACM, 2009. Google ScholarDigital Library
- Google. Build What's Next Better software. Faster. https://cloud.google.com/, accessed 2016-11-30.Google Scholar
- Z. Hill, J. Li, M. Mao, A. Ruiz-Alvarez, and M. Humphrey. Early observations on the performance of windows azure. Scientific Programming, 19(2-3):121--132, 2011. Google ScholarDigital Library
- T. Hoff. Are long vm instance spin-up times in the cloud costing you money? http://highscalability.com/blog/2011/3/17/are-long-vm-instance-spin-up-times-in-the-cloud-costing-you.html, accessed 2016-11-30.Google Scholar
- K. Z. Ibrahim, S. Hofmeyr, and C. Iancu. Characterizing the performance of parallel applications on multi-socket virtual machines. In Cluster, Cloud and Grid Computing (CCGrid), 2011 11th IEEE/ACM International Symposium on, pages 1--12. IEEE, 2011. Google ScholarDigital Library
- A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. kvm: the linux virtual machine monitor. In Proceedings of the Linux Symposium, volume 1, pages 225--230, Ottawa, Ontario, Canada, June 2007.Google Scholar
- T. Knauth and C. Fetzer. Dreamserver: Truly on-demand cloud services. In Proceedings of International Conference on Systems and Storage, pages 1--11. ACM, 2014. Google ScholarDigital Library
- T. Knauth, P. Kiruvale, M. Hiltunen, and C. Fetzer. Sloth: Sdn-enabled activity-based virtual machine deployment. In Proceedings of the third workshop on Hot topics in software defined networking, pages 205--206. ACM, 2014. Google ScholarDigital Library
- I. Krsul, A. Ganguly, J. Zhang, J. A. Fortes, and R. J. Figueiredo. Vmplants: Providing and managing virtual machine execution environments for grid computing. In Supercomputing, 2004. Proceedings of the ACM/IEEE SC2004 Conference, pages 7--7. IEEE, 2004.Google ScholarDigital Library
- L. Kurth. Xen Wiki - StubDom page. https://wiki.xenproject.org/wiki/StubDom, accessed 2016-11-30.Google Scholar
- H. A. Lagar-Cavilla, J. A. Whitney, A. M. Scannell, P. Patchin, S. M. Rumble, E. De Lara, M. Brudno, and M. Satyanarayanan. Snowflock: rapid virtual machine cloning for cloud computing. In Proceedings of the 4th ACM European conference on Computer systems, pages 1--12. ACM, 2009. Google ScholarDigital Library
- J. Levon and P. Elie. Oprofile: A system profiler for linux, 2004.Google Scholar
- B. Liu. xen: free_domheap_pages: delay page scrub to idle loop. Xen development mailing list, https://lists.xenproject.org/archives/html/xen-devel/2014-05/msg02436.html, accessed 2016-11-30.Google Scholar
- A. Madhavapeddy, T. Leonard, M. Skjegstad, T. Gazagnaire, D. Sheets, D. Scott, R. Mortier, A. Chaudhry, B. Singh, J. Ludlam, et al. Jitsu: Just-in-time summoning of unikernels. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pages 559--573, 2015.Google Scholar
- M. Mao and M. Humphrey. A performance study on the vm startup time in the cloud. In Cloud Computing (CLOUD), 2012 IEEE 5th International Conference on, pages 423--430. IEEE, 2012. Google ScholarDigital Library
- D. Merkel. Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239):2, 2014.Google ScholarDigital Library
- S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D. Epema. A performance analysis of ec2 cloud computing services for scientific computing. In International Conference on Cloud Computing, pages 115--131. Springer, 2009.Google Scholar
- S. Popuri. A tour of the mini-os kernel. https://www.cs.uic.edu/~spopuri/minios.html, accessed 2016-11-30.Google Scholar
- D. E. Porter, S. Boyd-Wickizer, J. Howell, R. Olinsky, and G. C. Hunt. Rethinking the library os from the top down. In ACM SIGPLAN Notices, volume 46, pages 291--304. ACM, 2011. Google ScholarDigital Library
- I. Pratt, D. Magenheimer, H. Blanchard, J. Xenidis, J. Nakajima, and A. Liguori. The ongoing evolution of xen. In Proceedings of the Ottawa Linux Symposium, 2006.Google Scholar
- C. Reiss, J. Wilkes, and J. L. Hellerstein. Google cluster-usage traces: format + schema. Technical report, Google Inc., Mountain View, CA, USA, Nov. 2011. Revised 2012.03.20. Posted at https://github.com/google/cluster-data, accessed 2016-11/30.Google Scholar
- M. Russinovich. Inside windows azure: the cloud operating system. https://channel9.msdn.com/events/Build/BUILD2011/SAC-853T, accessed 2016-11-30, 2011.Google Scholar
- K. Rzeszutek Wilk. Xen profiling: Oprofile and perf. https://wiki.xenproject.org/wiki/Xen_Profiling:_oprofile_and_perf, accessed 2016-11-30.Google Scholar
- The Netflix Tech Blog. Auto scaling in the amazon cloud. http://techblog.netflix.com/2012/01/autoscaling-in-amazon-cloud.html, accessed 2016-11-30.Google Scholar
- V. M. Weaver. Linux perf_event features and overhead. In The 2nd International Workshop on Performance Analysis of Workload Optimized Systems, FastPath, page 80, 2013.Google Scholar
- J. Weinman. Time is Money: The Value of "on-demand". http://www.joeweinman.com/Resources/Joe_Weinman_Time_Is_Money.pdf, accessed 2016-11-30, 2011.Google Scholar
- A. Whitaker, M. Shaw, and S. Gribble. Denali: Lightweight virtual machines for distributed and networked applications. In Proceedings of the 2002 USENIX Annual Technical Conference, 2002. Google ScholarDigital Library
- J. Wilkes. More Google cluster data. Google research blog, Nov. 2011. Posted at http://googleresearch.blogspot.com/2011/11/more-google-cluster-data.html, accessed 2016-11-30.Google Scholar
- S. Yi, D. Kondo, and A. Andrzejak. Reducing costs of spot instances via checkpointing in the amazon elastic compute cloud. In 2010 IEEE 3rd International Conference on Cloud Computing, pages 236--243. IEEE, 2010. Google ScholarDigital Library
- I. Zhang, A. Garthwaite, Y. Baskakov, and K. C. Barr. Fast restore of checkpointed memory using working set estimation. In ACM SIGPLAN Notices, volume 46, pages 87--98. ACM, 2011. Google ScholarDigital Library
- I. Zhang, T. Denniston, Y. Baskakov, and A. Garthwaite. Optimizing vm checkpointing for restore performance in vmware esxi. In Presented as part of the 2013 USENIX Annual Technical Conference (USENIX ATC 13), pages 1--12, 2013.Google Scholar
- L. Zhang, J. Litton, F. Cangialosi, T. Benson, D. Levin, and A. Mislove. Picocenter: Supporting long-lived, mostly-idle applications in cloud environments. In Proceedings of the Eleventh European Conference on Computer Systems, page 37. ACM, 2016. Google ScholarDigital Library
- J. Zhu, Z. Jiang, and Z. Xiao. Twinkle: A fast resource provisioning mechanism for internet services. In Proceedings of IEEE INFOCOM, pages 802--810. IEEE, 2011. Google ScholarCross Ref
Recommendations
Swift Birth and Quick Death: Enabling Fast Parallel Guest Boot and Destruction in the Xen Hypervisor
VEE '17The ability to quickly set up and tear down a virtual machine is critical for today's cloud elasticity, as well as in numerous other scenarios: guest migration/consolidation, event-driven invocation of micro-services, dynamically adaptive unikernel-...
Transparently bridging semantic gap in CPU management for virtualized environments
Consolidated environments are progressively accommodating diverse and unpredictable workloads in conjunction with virtual desktop infrastructure and cloud computing. Unpredictable workloads, however, aggravate the semantic gap between the virtual ...
Enabling Instantaneous Relocation of Virtual Machines with a Lightweight VMM Extension
CCGRID '10: Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid ComputingWe are developing an efficient resource management system with aggressive virtual machine (VM) relocation among physical nodes in a data center. Existing live migration technology, however, requires a long time to change the execution host of a VM, it ...
Comments