Skip to main content
SpringerLink
Log in

Virtual machine placement based on multi-objective reinforcement learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Multi-objective virtual machine (VM) placement is a powerful tool, which can achieve different goals in data centers. It is an NP-hard problem, and various works have been proposed to solve it. However, almost all of them ignore the selection of weights. The selection of weights is difficult, but it is essential for multi-objective optimization. The inappropriate weights will cause the obtained solution set deviating from the Pareto optimal set. Fortunately, we find that this problem can be easily solved by using the Chebyshev scalarization function in multi-objective reinforcement learning (RL). In this paper, we propose a VM placement algorithm based on multi-objective RL (VMPMORL). VMPMORL is designed based on the Chebyshev scalarization function. We aim to find a Pareto approximate set to minimize energy consumption and resource wastage simultaneously. Compared with other multi-objective RL algorithms in the field of VM placement, VMPMORL not only uses the concept of the Pareto set but also solves the weight selection problem. Finally, VMPMORL is compared with some state-of-the-art algorithms in recent years. The results show that VMPMORL can achieve better performance than the approaches above.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Make IT Green (2010) Cloud computing and its contribution to climate change. Greenpeace International, p 83

  2. Wang J, Gong B, Liu H, Li S (2019) Intelligent scheduling with deep fusion of hardware-software energy-saving principles for greening stochastic nonlinear heterogeneous super-systems. Appl Intell, 1–14

  3. El Motaki S, Yahyaouy A, Gualous H, Sabor J (2019) Comparative study between exact and metaheuristic approaches for virtual machine placement process as knapsack problem. J Supercomput, 1–21

  4. Shaw R, Howley E, Barrett E (2019) An energy efficient anti-correlated virtual machine placement algorithm using resource usage predictions. Simul Model Pract Theory 93:322–342

    Article  Google Scholar 

  5. Xu J, Fortes JAB (2010) Multi-objective virtual machine placement in virtualized data center environments. In: 2010 IEEE/ACM int’l conference on & int’l conference on cyber, physical and social computing (CPSCom) green computing and communications (GreenCom). IEEE, pp 179–188

  6. Gao Y, Guan H, Qi Z, Hou Y, Liu L (2013) A multi-objective ant colony system algorithm for virtual machine placement in cloud computing. J Comput Syst Sci 79(8):1230–1242

    Article  MathSciNet  Google Scholar 

  7. Zheng Q, Li R, Li X, Shah N, Zhang J, Tian F, Chao K-M, Li J (2016) Virtual machine consolidated placement based on multi-objective biogeography-based optimization. Futur Gener Comput Syst 54:95–122

    Article  Google Scholar 

  8. Jamali S, Malektaji S, Analoui M (2017) An imperialist competitive algorithm for virtual machine placement in cloud computing. J Exper Theor Artif Intell 29(3):575–596

    Article  Google Scholar 

  9. Li Z, Li Y, Yuan T, Chen S, Jiang S (2019) Chemical reaction optimization for virtual machine placement in cloud computing. Appl Intell 49(1):220–232

    Article  Google Scholar 

  10. Filho MCS, Monteiro CC, Inácio PRM, Freire MM (2018) Approaches for optimizing virtual machine placement and migration in cloud environments: a survey. J Parallel Distrib Comput 111:222–250

    Article  Google Scholar 

  11. Das I, Dennis JE (1997) A closer look at drawbacks of minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems. Struct Optim 14(1):63–69

    Article  Google Scholar 

  12. Lee K, Lee S, Lee J (2019) Interactive character animation by learning multi-objective control. ACM Trans Graph (TOG) 37(6):180

    Google Scholar 

  13. Van Moffaert K, Drugan MM, Nowé A (2013) Scalarized multi-objective reinforcement learning Novel design techniques. In: 2013 IEEE Symposium on adaptive dynamic programming and reinforcement learning (ADPRL). IEEE, pp 191–199

  14. Pan A, Xu W, Wang L, Ren H (2019) Additional planning with multiple objectives for reinforcement learning. Knowl-Based Syst, 105392

  15. Farahnakian F, Liljeberg P, Plosila J (2014) Energy-efficient virtual machines consolidation in cloud data centers using reinforcement learning. In: 2014 22nd Euromicro international conference on parallel, distributed and network-based processing (PDP). IEEE, pp 500–507

  16. Shaw R, Howley E, Barrett E (2017) An advanced reinforcement learning approach for energy-aware virtual machine consolidation in cloud data centers. In: 2017 12th International conference for internet technology and secured transactions (ICITST). IEEE, pp 61–66

  17. Mannion P, Devlin S, Duggan J, Howley E (2018) Reward shaping for knowledge-based multi-objective multi-agent reinforcement learning. Knowl Eng Rev 33:e23

    Article  Google Scholar 

  18. Zhou J, Yao X (2017) Multi-objective hybrid artificial bee colony algorithm enhanced with lévy flight and self-adaption for cloud manufacturing service composition. Appl Intell 47(3):721–742

    Article  Google Scholar 

  19. Mossalam H, Assael YM, Roijers DM, Whiteson S (2016) Multi-objective deep reinforcement learning. arXiv:1610.02707

  20. Hsu C-H, Chang S-H, Liang J-H, Chou H-P, Liu C-H, Chang S-C, Pan J-Y, Chen Y-T, Wei W, Monas D-CJ (2018) Multi-objective neural architecture search using reinforcement learning. arXiv:abs/1806.10332

  21. Noothigattu R, Bouneffouf D, Mattei N, Chandra R, Madan P, Varshney K, Campbell M, Singh M, Rossi F (2018) Interpretable multi-objective reinforcement learning through policy orchestration

  22. García J, Iglesias R, Rodríguez MA, Regueiro CV, et al. (2019) Directed exploration in black-box optimization for multi-objective reinforcement learning. Int J Inform Technol Decis Mak (IJITDM) 18(03):1045–1082

    Article  Google Scholar 

  23. Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT Press

  24. Nguyen TT (2018) A multi-objective deep reinforcement learning framework

  25. Watkins CJCH (1989) Learning from delayed rewards

  26. Tsitsiklis JN (1994) Asynchronous stochastic approximation and q-learning. Mach Learn 16(3):185–202

    MATH  Google Scholar 

  27. Wiering MA, De Jong ED (2007) Computing optimal stationary policies for multi-objective Markov decision processes. In: 2007 IEEE International symposium on approximate dynamic programming and reinforcement learning. IEEE, pp 158–165

  28. Voß T, Beume N, Rudolph G, Igel C (2008) Scalarization versus indicator-basedselection in multi-objective cma evolution strategies. In: 2008 IEEE Congress on evolutionary computation (IEEE World Congress on Computational Intelligence). IEEE, pp 3036–3043

  29. Liu C, Shen C, Li S, Wang S (2014) A new evolutionary multi-objective algorithm to virtual machine placement in virtualized data center. In: 2014 IEEE 5th international conference on software engineering and service science. IEEE, pp 272–275

  30. Dashti SE, Rahmani AM (2016) Dynamic vms placement for energy efficiency by pso in cloud computing. J Exper Theor Artif Intell 28(1-2):97–112

    Article  Google Scholar 

  31. Alharbi F, Tian Y-C, Tang M, Zhang W-Z, Peng C, Fei M (2019) An ant colony system for energy-efficient dynamic virtual machine placement in data centers. Expert Syst Appl 120:228–238

    Article  Google Scholar 

  32. Ajiro Y, Tanaka A (2007) Improving packing algorithms for server consolidation. In: Int. CMG conference, vol 253

  33. Liu X-F, Zhan Z-H, Deng JD, Li Y, Gu T, Zhang J (2016) An energy efficient ant colony system for virtual machine placement in cloud computing. IEEE Transactions on Evolutionary Computation

  34. Fan X, Weber W-D, Barroso LA (2007) Power provisioning for a warehouse-sized computer. In: ACM SIGARCH computer architecture news, vol 35. ACM, pp 13–23

  35. Greenberg A, Hamilton J, Maltz DA, Patel P (2008) The cost of a cloud: research problems in data center networks. ACM SIGCOMM Comput Commun Rev 39(1):68–73

    Article  Google Scholar 

  36. Gao C, Wang H, Zhai L, Gao Y, Yi S (2016) An energy-aware ant colony algorithm for network-aware virtual machine placement in cloud computing. In: 2016 IEEE 22nd International conference on parallel and distributed systems (ICPADS). IEEE, pp 669–676

  37. Van Veldhuizen DA (1999) Multiobjective evolutionary algorithms: classifications, analyzes, and new innovations. Air Force Inst. Technol., Dayton, OH, Tech. Rep AFIT/DS/ENG/99-01

  38. Schott JR (1995) Fault tolerant design using single and multicriteria genetic algorithm optimization. Technical report, AIR FORCE INST OF TECH WRIGHT-PATTERSON AFB OH

    Google Scholar 

  39. Zitzler E, Thiele L, Laumanns M, Fonseca CM, Da Fonseca Grunert V (2002) Performance assessment of multiobjective optimizers: An analysis and review. TIK-Report, 139

Download references

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China under Grant NSFC 61672323, the Fundamental Research Funds of Shandong University under Grant 2017JC043, the Key Research and Development Program of Shandong Province under Grant 2017GGX10122 and Grant 2017GGX10142, and the Natural Science Foundation of Shandong Province Grant ZR2019MF072.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong University, Jinan, 250101, China

    Yao Qin & Shanwen Yi

  2. School of Software, Shandong University, Jinan, 250101, China

    Hua Wang

  3. School of Information Science and Engineering, Linyi University, Linyi, 276005, China

    Xiaole Li

  4. School of Information Science and Engineering, Shandong Normal University, Jinan, 250014, China

    Linbo Zhai

Authors
  1. Yao Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hua Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shanwen Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaole Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Linbo Zhai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hua Wang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, Y., Wang, H., Yi, S. et al. Virtual machine placement based on multi-objective reinforcement learning. Appl Intell 50, 2370–2383 (2020). https://doi.org/10.1007/s10489-020-01633-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01633-3

Keywords

Search

Navigation