Skip to main content
Springer Nature Link
Log in

Improving performance by network-aware virtual machine clustering and consolidation

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Modern data center consists of thousands of servers, racks and switches. Complicated structure means it requires well-designed algorithms to utilize resources of data centers efficiently. Current virtual machine scheduling algorithms mainly focus on the initial allocation of virtual machines based on the CPU, memory and network bandwidth requirements. However, when tasks finished or lease expired, related virtual machines would be deleted from the system which would generate resource fragments. Such fragments lead to unbalanced resource utilization and decline of communication performance. This paper investigates the network influence on typical applications in data centers and proposed a self-adaptive network-aware virtual machine clustering and consolidation algorithm to maintain an optimal system-wide status. Our consolidation algorithm periodically checks whether consolidation is necessary and then clusters and consolidates virtual machines to lower communication cost with an online heuristic. We used two benchmarks in a real environment to examine network influence on different tasks. To evaluate the advantages of the proposed algorithm, we also built a cloud computing testbed. Real workload trace-driven simulations and testbed-based experiments showed that, our algorithm greatly shortened the average finish time of map-reduce tasks and reduced time delay of web applications. Simulation results showed that our algorithm considerably reduced the amount of high-delay jobs, lowered the average traffic passed through aggregate switches and improved the communication ability among virtual machines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Purdue MapReduce Benchmarks Suite http://web.ics.purdue.edu/fahmad/benchmarks.htm.

  2. Rice University Bidding System, http://rubis.ow2.org/.

  3. http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Data_Center/DC_Infra2_5/DCI_SRND_2_5a_book.html.

  4. http://www.cs.huji.ac.il/labs/parallel/workload/logs.html.

  5. https://computing.llnl.gov/?set=resources&page=index.

References

  1. Alicherry M, Lakshman TV (2012) Network aware resource allocation in distributed clouds. In: Proceedings of International Conference on Computer Communications, IEEE

  2. Beloglazov A, Buyya R (2015) OpenStack Neat: a framework for dynamic and energy-efficient consolidation of virtual machines in OpenStack clouds. Concurr Comput Pract Exp 27(5):1310–1333

    Article  Google Scholar 

  3. Biran O et al. (2012) A stable network-aware vm placement for cloud systems. In: Proceedings of the IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, IEEE Computer Society

  4. Breitgand D, Kutiel G, Raz D (2010) Cost-aware live migration of services in the cloud. In: Proceedings of the 3rd Annual Haifa Experimental Systems Conference, ACM

  5. Dutta S, Verma A (2011) Service deactivation aware placement and defragmentation in enterprise clouds. In: Proceedings of the 7th International Conference on Network and Services Management, International Federation for Information Processing

  6. Ferdaus MH, Murshed M, Calheiros RN, et al. (2015) Network-aware virtual machine placement and migration in cloud data centers. In: Emerging research in cloud distributed computing systems, Chap 2, pp 42–91. doi:10.4018/978-1-4666-8213-9.ch002

  7. Iqbal W, Dailey MN, Carrera D (2010) Sla-driven dynamic resource management for multi-tier web applications in a cloud. In: 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid), IEEE

  8. Jiang JW et al (2012) Joint VM placement and routing for data center traffic engineering. In: Proceedings of International Conference on Computer Communications, IEEE

  9. Kliazovich D, Bouvry P, Khan SU (2013) DENS: data center energy-efficient network-aware scheduling. J Cluster Comput 16(1):65–75

    Article  Google Scholar 

  10. Meng X, Pappas V, Zhang L (2010) Improving the scalability of data center networks with traffic-aware virtual machine placement. In: Proceedings of International Conference on Computer Communications, IEEE

  11. Murty KG, Kabadi SN (1987) Some NP-complete problems in quadratic and nonlinear programming. Math Program 39(2):117–129

    Article  MathSciNet  Google Scholar 

  12. Nguyen Van H, Dang Tran F, Menaud JM (2009) Autonomic virtual resource management for service hosting platforms. In: Proceedings of the ICSE Workshop on Software Engineering Challenges of Cloud Computing, IEEE

  13. Shrivastava V et al. (2011) Application-aware virtual machine migration in data centers. In: Proceedings of International Conference on Computer Communications, IEEE

  14. Stage A, Setzer T (2009) Network-aware migration control and scheduling of differentiated virtual machine workloads. In: Proceedings of the ICSE Workshop on Software Engineering Challenges of Cloud Computing. IEEE Computer Society

  15. Steiner M et al (2012) Network-aware service placement in a distributed cloud environment. J ACM SIGCOMM Comput Commun Rev 42(4):73–74

    Article  Google Scholar 

  16. Stoer M, Wagner F (1997) A simple min-cut algorithm. J ACM (JACM) 44(4):585–591

    Article  MathSciNet  Google Scholar 

  17. Wang M, Meng X, Zhang L (2011) Consolidating virtual machines with dynamic bandwidth demand in data centers. In: Proceedings of International Conference on Computer Communications, IEEE

  18. Wilson C et al (2011) Better never than late: meeting deadlines in data center networks. J ACM SIGCOMM Comput Commun Rev 41(4):50–61

    Article  Google Scholar 

  19. Xia M, Shirazipour M, Zhang Y, Green H, Takacs A (2015) Network function placement for NFV chaining in packet/optical data centers. J Lightwave Technol 33(8):1565–1570

    Article  Google Scholar 

  20. Xu J, Fortes JAB (2010) Multi-objective virtual machine placement in virtualized data center environments. In: IEEE/ACM Int’l Conference on Green Computing and Communications, & Int’l Conference on Cyber, Physical and Social Computing, IEEE

  21. Zhu J et al (2012) Towards bandwidth guarantee in multi-tenancy cloud computing networks. In: Proceedings of 20th IEEE International Conference on Network Protocols, IEEE

Download references

Acknowledgements

The authors want to thank Weicheng Huai, Zhigang Jiang, Kaiyuan Wen and Shen Zhang for their novel suggestions and kind assistance. This work is partially supported by the National Natural Science Foundation of China under Grant No. 61472181, 61321491; Jiangsu Natural Science Foundation under Grant No. BK20151392; JSPS KAKENHI Grant Number JP16K00117, JP15K15976, KDDI Foundation. And this work is also partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization.

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, 163 Xianlin Avenue, Nanjing, 210023, China

    Gangyi Luo, Zhuzhong Qian & Sanglu Lu

  2. Muroran Institute of Technology, 27-1 Mizumoto-cho, Muroran, Hokkaido, 050-8585, Japan

    Mianxiong Dong & Kaoru Ota

Authors
  1. Gangyi Luo
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Zhuzhong Qian
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Mianxiong Dong
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Kaoru Ota
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Sanglu Lu
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Zhuzhong Qian.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, G., Qian, Z., Dong, M. et al. Improving performance by network-aware virtual machine clustering and consolidation. J Supercomput 74, 5846–5864 (2018). https://doi.org/10.1007/s11227-017-2104-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-017-2104-9

Keywords