Yi Ren
Ling Liu
Skip Abstract Section
Abstract
Virtual machines (VMs) and virtualization are one of the core computing technologies today. Inter-VM communication is not only prevalent but also one of the leading costs for data-intensive systems and applications in most data centers and cloud computing environments. One way to improve inter-VM communication efficiency is to support coresident VM communication using shared-memory-based methods and resort to the traditional TCP/IP for communications between VMs that are located on different physical machines. In recent years, several independent kernel development efforts have been dedicated to improving communication efficiency between coresident VMs using shared-memory channels, and the development efforts differ from one another in terms of where and how the shared-memory channel is established. In this article, we provide a comprehensive overview of the design choices and techniques for performance optimization of coresident inter-VM communication. We examine the key issues for improving inter-VM communication using shared-memory-based mechanisms, such as implementation choices in the software stack, seamless agility for dynamic addition or removal of coresident VMs, and multilevel transparency, as well as advanced requirements in reliability, security, and stability. An in-depth comparison of state-of-the-art research efforts, implementation techniques, evaluation methods, and performance is conducted. We conjecture that this comprehensive survey will not only provide the foundation for developing the next generation of inter-VM communication optimization mechanisms but also offers opportunities to both cloud infrastructure providers and cloud service providers and consumers for improving communication efficiency between coresident VMs in virtualized computing platforms.
- T. Anderson, K. Birman, R. Broberg, M. Caesar, D. Comer, C. Cotton, M. J. Freedman, A. Haeberlen, Z. G. Ives, A. Krishnamurthy, W. Lehr, B. T. Loo, D. Mazieres, A. Nicolosi, J. M. Smith, I. Stoica, R. V. Renesse, M. Walfish, H. Weatherspoon, and C. S. Yoo. 2013. The NEBULA future internet architecture. Lecture Notes in Computer Science 7858, 16--26.Google Scholar
Cross Ref
- M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia. 2010. Above the clouds: A Berkeley view of cloud computing. Communications of the ACM. 53, 4 (April 2010), 50--58. Google ScholarDigital Library
- P. N. Ayuso. 2006. Netfilter's connection tracking system. USENIX 31, 3.Google Scholar
- P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. 2003. Xen and the art of virtualization. In Proceedings of the 9th ACM Symposium on Operating Systems Principles (SOSP’03). ACM, New York, NY, 164--177. Google ScholarDigital Library
- A. Burtsev, K. Srinivasan, P. Radhakrishnan, L. N. Bairavasundaram, K. Voruganti, and G. R. Goodson. 2009. Fido: Fast inter virtual-machine communication for enterprise appliances. In Proceedings of the 2009 Conference on USENIX Annual Technical Conference. Google ScholarDigital Library
- F. Diakhaté, M. Perache, R. Namyst, and H. Jourdren. 2008. Efficient shared memory message passing for inter-VM communications. In Proceedings of 3rd Workshop on Virtualization in High-Performance Cluster and Grid Computing (VHPC’08). Springer-Verlag, Berlin. 53--62. Google ScholarDigital Library
- H. Eiraku, Y. Shinjo, C. Pu, Y. Koh, and K. Kato. 2009. Fast networking with socket-outsourcing in hosted virtual machine environments. In Proceedings of ACM Symposium on Applied Computing (SAC’09). ACM, New York, NY. 310--317. Google ScholarDigital Library
- C. Gebhardt and A. Tomlinson. 2010. Challenges for inter virtual machine communication. Technical Report, RHUL-MA-2010-12. Retrieved from http://www.ma.rhul.ac.uk/static/techrep/2010/RHUL-MA-2010-12.pdf.Google Scholar
- A. W. Gordon. 2011a. Enhancing cloud environments with inter-virtual machine shared memory. M.S. thesis, Department of Computing Science, University of Alberta.Google Scholar
- A. W. Gordon and P. Lu. 2011b. Low-latency caching for cloud-based web applications. In Proceedings of 6th International Workshop on Networking Meets Databases (NetDB’11).Google Scholar
- S. Govindan, A. R. Nath, A. Das, B. Urgaonkar, and A. Sivasubramaniam. 2007. Xen and Co.: Communication-aware CPU scheduling for consolidated Xen-based hosting platforms. In Proceedings of the 3rd International Conference on Virtual Execution Environments (VEE’07). ACM, 126--136. Google ScholarDigital Library
- U. Gurav and R. Shaikh. 2010. Virtualization: A key feature of cloud computing. In Proceedings of the International Conference and Workshop on Emerging Trends in Technology (ICWET’10). 227--229. Google ScholarDigital Library
- W. Huang. 2008. High performance network I/O in virtual machines over modern interconnects. Ph.D. thesis, Department of Computer Science and Engineering, Ohio State University. Google ScholarDigital Library
- W. Huang, M. Koop, Q. Gao, and D. K. Panda. 2007. Virtual machine aware communication libraries for high performance computing. In Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC’07). ACM, New York, NY. Article No. 9. Google ScholarDigital Library
- J. Hwang, K. Ramakrishnan, and T. Wood. 2014. NetVM: High performance and flexible networking using virtualization on commodity platforms. In Proceedings of 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI’14). USENIX Association. 445--458. Google ScholarDigital Library
- S. Imai, T. Chestna, and C. A. Varela. 2013. Accurate resource prediction for hybrid IaaS clouds using workload-tailored elastic compute units. In Proceedings of 6th IEEE/ACM International Conference on Utility and Cloud Computing (UCC’13). IEEE, 171--178. Google ScholarDigital Library
- Infiniband Trade Association. 2015. Homepage. Retrieved from http://www.infinibandta.org.Google Scholar
- Intel Corporation. 2013. Intel data plane development kit: Getting started guide. Retrieved from http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/intel-dpdk-programmers-guide.html.Google Scholar
- J. Jose, M. Li, X. Lu, K. Kandalla, M. Arnold, and D. K. Panda. 2013. SR-IOV support for virtualization on InfiniBand clusters: Early experience. In Proceedings of 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid’13). IEEE, 385--392.Google Scholar
- X. Ke. 2011. Interprocess communication mechanisms with inter-virtual machine shared memory. M.S. thesis, Department of Computing Science, University of Alberta.Google Scholar
- H. Kim, H. Lim, J. Jeong, H. Jo, and J. Lee. 2009. Task-aware virtual machine scheduling for I/O performance. In Proceedings of the 5th International Conference on Virtual Execution Environments (VEE’09). ACM, 101--110. Google ScholarDigital Library
- K. Kim. 2013. XWAY project. Retrieved from http://xway.sourceforge.net/.Google Scholar
- K. Kim, C. Kim, S. Jung, H. Shin, and J. Kim. 2008. Inter-domain socket communications supporting high performance and full binary compatibility on Xen. In Proceedings of the 4th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE’08). ACM, New York, NY. 11--20. Google ScholarDigital Library
- A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. 2007. KVM: The Linux virtual machine monitor. In Proceedings of the Linux Symposium, volume 1. 225--230.Google Scholar
- Y. Koh. 2010. Kernel service outsourcing: An approach to improve performance and reliability of virtualized systems. PhD thesis, School of Computer Science, College of Computing, Georgia Institute of Technology. Google ScholarDigital Library
- M. Lee, A. S. Krishnakumar, P. Krishnan, N. Singh, and S. Yajnik. 2010. Supporting soft real-time tasks in the Xen hypervisor. In Proceedings of the 6th International Conference on Virtual Execution Environments (VEE’10). Pittsburgh, PA, March 17--19, 2010. 97--108. Google ScholarDigital Library
- J. Levon. 2014. OProfile manual. Retrieved from http://oprofile.sourceforge.net/doc/index.html.Google Scholar
- D. Li, H. Jin, Y. Shao, X. Liao, Z. Han, and K. Chen. 2010. A high-performance inter-domain data transferring system for virtual machines. Journal of Software 5, 2 (February 2010), 206--213.Google ScholarCross Ref
- J. Liu, W. Huang, B. Abali, and D. K. Panda. 2006. High performance VMM bypass I/O in virtual machines. In Proceedings of the Annual Conference on USENIX’06 Annual Technical Conference (ATEC’06). 29--42. Google ScholarDigital Library
- A. C. Macdonell. 2011. Shared-memory optimizations for virtual machines. PhD thesis, Department of Computing Science, University of Alberta. Google ScholarDigital Library
- A. C. Macdonell. 2014. Nahanni: the KVM/Qemu inter-VM shared memory PCI device. Retrieved from http://gitorious.org/nahanni/pages/Home.Google Scholar
- Y. Mei, L. Liu, X. Pu, S. Sivathanu, and X. Dong. 2013. Performance analysis of network I/O workloads in virtualized data centers. IEEE Transactions on Services Computing 6, 1, 48--63. Google ScholarDigital Library
- A. Menon, A. L. Cox, and W. Zwaenepoel. 2006. Optimizing network virtualization in Xen. In Proceedings of the 2006 Conference on USENIX Annual Technical Conference. USENIX Association, Berkeley, CA. 15--28. Google ScholarDigital Library
- MSDN. 2014. Overview of single root I/O virtualization (SR-IOV). Retrieved from http://msdn.microsoft.com/ en-us/library/windows/hardware/hh440148%28v=vs.85%29.aspx.Google Scholar
- Netperf. 2015. http://www.netperf.org/netperf/.Google Scholar
- D. Ongaro, A. L. Cox, and S. Rixner. 2008. Scheduling I/O in virtual machine monitors. In Proceedings of the 4th International Conference on Virtual Execution Environments (VEE’08). Seattle, WA, March 5--7, 2008. ACM, 2008, ISBN 978-1-59593-796-4. 1--10. Google ScholarDigital Library
- M. Pearce, S. Zeadally, and R. Hunt. 2013. Virtualization: Issues, security threats, and solutions. ACM Computing Surveys 45, 2 (February 2013), 17. Google ScholarDigital Library
- X. Pu, L. Liu, Y. Mei, S. Sivathanu, Y. Koh, C. Pu, and Y. Cao. 2012. Who is your neighbor: Net I/O performance interference in virtualized clouds. IEEE Transactions on Services Computing 6, 3, 314--329. Google ScholarDigital Library
- P. Radhakrishnan and K. Srinivasan. 2008. MMNet: An efficient inter-VM communication mechanism. Xen Summit, Boston, 2008.Google Scholar
- Y. Ren, L. Liu, X. Liu, J. Kong, H. Dai, Q. Wu, and Y. Li. 2012. A fast and transparent communication protocol for co-resident virtual machines. In Proceedings of 8th IEEE International Conference on Collaborative Computing (CollaborateCom’12). 70--79.Google Scholar
- R. V. Renesse. 2012. Fact-based inter-process communication primitives for programming distributed systems. In Proceedings of Workshop on Languages for Distributed Algorithms (LADA’12). http://www.cs. cornell.edu/home/rvr/newpapers/lada2012.pdf.Google Scholar
- R. Russell. 2008. Virtio: Towards a de-facto standard for virtual I/O devices. ACM SIGOPS Operating Systems Review 42, 5 (July 2008), 95--103. Google ScholarDigital Library
- J. R. Santos, Y. Turner, G. Janakiraman, and I. Pratt. 2008. Bridging the gap between software and hardware techniques for I/O virtualization. In Proceedings of USENIX 2008 Annual Technical Conference on Annual Technical Conference (ATC’08). 29--42. Google ScholarDigital Library
- R. Sanger, Ed. 2013. Notes on libtrace Intel data plane development kit (DPDK) support - experimental. https://github.com/wanduow/libtrace/wiki/DPDK-Notes—Experimental.Google Scholar
- Z. Su, L. Liu, M. Li, X. Fan, and Y. Zhou. 2015. Reliable and resilient trust management in distributed service provision networks. http://www.cc.gatech.edu/∼lingliu/papers/2015/TWEB-ServiceTrust.pdf.Google Scholar
- VMware Inc. 2007. VMCI Overview. Retrieved from http://pubs.vmware.com/vmci-sdk/VMCI_intro.html.Google Scholar
- J. Wang. 2009. Survey of state-of-the-art in inter-VM communication mechanisms. Research Report (September 2009). Retrieved from http://www.cs.binghamton.edu/∼jianwang/papers/proficiency.pdf.Google Scholar
- J. Wang, K. Wright, and K. Gopalan. 2008a. XenLoop source code. Retrieved from http://osnet.cs.binghamton. edu/projects/xenloop-2.0.tgz.Google Scholar
- J. Wang, K. Wright, and K. Gopalan. 2008b. XenLoop: A transparent high performance inter-VM network loopback. In Proceedings of the 17th ACM International Symposium on High Performance Distributed Computing (HPDC’08). ACM, New York, NY. 109--118. Google ScholarDigital Library
- Q. Wang and C. A. Varela. 2011. Impact of cloud computing virtualization strategies on workloads’ performance. 2011. In Proceedings of4th IEEE/ACM International Conference on Utility and Cloud Computing (UCC’11). 130--137. Google ScholarDigital Library
- M. B. Yehuda, J. Mason, J. Xenidis, O. Krieger, L. V. Doorn, J. Nakajima, A. Mallick, and E. Wahlig. 2006. Utilizing IOMMUs for virtualization in Linux and Xen. In Proceedings of the 2006 Ottawa Linux Symposium (OLS’06). 71--86.Google Scholar
- A. J. Younge, R. Henschel, J. T. Brown, G. Laszewski, J. Qiu, and G. C. Fox. 2011. Analysis of virtualization technologies for high performance computing environments. In Proceedings of IEEE 4th International Conference on Cloud Computing (CLOUD’11). 9--16. Google ScholarDigital Library
- Q. Zhang, L. Liu, Y. Ren, K. Lee, Y. Tang, X. Zhao, and Y. Zhou. 2013a. Residency aware inter-VM communication in virtualized cloud: Performance measurement and analysis. In Proceedings of IEEE 6th International Conference on Cloud Computing (CLOUD’13). IEEE. 204--211. Google ScholarDigital Library
- Q. Zhang and L. Liu. 2015. Shared memory optimization in virtualized clouds. In Proceedings of IEEE 2015 International Conference on Cloud Computing (CLOUD’15). Google ScholarDigital Library
- J. Zhang, X. Lu, J. Jose, R. Shi, and D. K. Panda. 2014a. Can inter-VM shmem benefit MPI applications on SR-IOV based virtualized Infiniband clusters? In Proceedings of Euro-Par 2014 Parallel Processing, 20th International Conference. Springer, 342--353.Google Scholar
- J. Zhang, X. Lu, J. Jose, M. Li, R. Shi, and D. K. Panda. 2014b. High performance MPI library over SR-IOV enabled InfiniBand clusters. In Proceedings of 21st Annual IEEE International Conference on High Performance Computing (HiPC’14).Google Scholar
- X. Zhang and S. Mcintosh. 2013b. XVMSocket. Retrieved from http://sourceforge.net/projects/xvmsocket/.Google Scholar
- X. Zhang, S. Mcintosh, P. Rohatgi, and J. L. Griffin. 2007. XenSocket: A high-throughput inter domain transport for virtual machines. In Proceedings of the ACM/IFIP/USENIX 2007 International Conference on Middleware (Middleware’07). Springer-Verlag, New York, NY. 184--203. Google ScholarDigital Library
Index Terms
- Shared-Memory Optimizations for Inter-Virtual-Machine Communication
Recommendations
Residency-Aware Virtual Machine Communication Optimization: Design Choices and Techniques
CLOUD '13: Proceedings of the 2013 IEEE Sixth International Conference on Cloud ComputingNetwork I/O workloads are dominating in many data centers and cloud computing environments today. One way to improve inter Virtual Machine (VM) communication efficiency is to support co-resident VM communication by using shared memory based approaches ...
A high performance inter-domain communication approach for virtual machines
Highlights Presented a novel inter-domain communication approach (IVCOM) for virtual machine. Using bypassing protocol stacks, shunning page flipping methods. IVCOM applies a direct communication channel between domain 0 and U. IVOCM can improve the ...
Comments