Abstract
Reinforcement learning has been recently a very active field of research. Thanks to combining it with Deep Learning, many newly designed algorithms improve the state of the art. In this paper we present the results of our attempt to use the recent advancements in Reinforcement Learning to automate the management of heterogeneous resources in an environment which hosts a compute-intensive evolutionary process. We describe the architecture of our system and present evaluation results. The experiments include autonomous management of a sample workload and a comparison of its performance to the traditional automatic management approach. We also provide the details of training of the management policy using the Proximal Policy Optimization algorithm. Finally, we discuss the feasibility to extend the presented approach to other scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Amazon Web Services Elastic Compute Cloud. https://aws.amazon.com/ec2/. Accessed 30 Dec 2019
Graphite Project. https://graphiteapp.org/. Accessed 28 Nov 2019
PyTorch DNN Evolution. https://gitlab.com/pkoperek/pytorch-dnn-evolution. Accessed 01 Dec 2019
Ashraf, A., et al.: CRAMP: cost-efficient resource allocation for multiple web applications with proactive scaling. In: 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings, pp. 581–586, December 2012
Brockman, G., et al.: OpenAI Gym (2016). http://arxiv.org/abs/1606.01540
Chen, L., et al.: AuTO: scaling deep reinforcement learning for datacenter-scale automatic traffic optimization. In: SIGCOMM 2018, New York, USA, pp. 191–205 (2018)
Ferretti, S., et al.: Qos-aware clouds. In: 2010 IEEE 3rd International Conference on Cloud Computing, pp. 321–328, July 2010
Filho, M.C.S., et al.: Cloudsim plus: a cloud computing simulation framework pursuing software engineering principles for improved modularity, extensibility and correctness. In: IFIP/IEEE Symposium on Integrated Network and Service Management, pp. 400–406, May 2017
Funika, W., et al.: Towards autonomic semantic-based management of distributed applications. Comput. Sci. 11, 51–64 (2010)
Funika, W., Koperek, P.: Evaluating the use of policy gradient optimization approach for automatic cloud resource provisioning. In: Wyrzykowski, R., Deelman, E., Dongarra, J., Karczewski, K. (eds.) PPAM 2019. LNCS, vol. 12043, pp. 467–478. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43229-4_40
Gu, S., et al.: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: Proceedings 2017 IEEE International Conference on Robotics and Automation (ICRA), Piscataway, NJ, USA. IEEE, May 2017
Kaelbling, L.P., et al.: Reinforcement learning: a survey. CoRR cs.AI/9605103 (1996). http://arxiv.org/abs/cs.AI/9605103
Kitowski, J., et al.: Computer simulation of heuristic reinforcement learning system for nuclear plant load changes control. Comput. Phys. Commun. 18, 339–352 (1979)
Koperek, P., Funika, W.: Dynamic business metrics-driven resource provisioning in cloud environments. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds.) PPAM 2011. LNCS, vol. 7204, pp. 171–180. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-31500-8_18
LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010). http://yann.lecun.com/exdb/mnist/
Minarolli, D., Freisleben, B.: Distributed resource allocation to virtual machines via artificial neural networks. In: 2014 Proceedings of the 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2014, pp. 490–499. IEEE Computer Society, Washington, DC (2014)
Mnih, V., et al.: Playing Atari with deep reinforcement learning (2013). http://arxiv.org/abs/1312.5602
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: Balcan, M.F., Weinberger, K.Q. (eds.) Proceedings of the 33rd International Conference on Machine Learning, vol. 48, pp. 1928–1937. PMLR, 20–22 June 2016
Schulman, J., et al.: Proximal policy optimization algorithms. CoRR abs/1707.06347 (2017). http://arxiv.org/abs/1707.06347
Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550, 354–359 (2017)
Sutton, R.S.: Temporal credit assignment in reinforcement learning. Ph.D. thesis (1984)
Wang, Z., et al.: Automated cloud provisioning on AWS using deep reinforcement learning. CoRR abs/1709.04305 (2017). http://arxiv.org/abs/1709.04305
Xiong, P., et al.: SmartSLA: cost-sensitive management of virtualized resources for CPU-bound database services. IEEE Trans. Parallel Distrib. Syst. 26, 1441–1451 (2014)
Acknowledgements
The research presented in this paper was supported by the funds assigned to AGH University of Science and Technology by the Polish Ministry of Science and Higher Education. The experiments have been carried out on the PL-Grid infrastructure resources of ACC Cyfronet AGH and on the Amazon Web Services Elastic Compute Cloud.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Funika, W., Koperek, P., Kitowski, J. (2021). Management of Heterogeneous Cloud Resources with Use of the PPO. In: Balis, B., et al. Euro-Par 2020: Parallel Processing Workshops. Euro-Par 2020. Lecture Notes in Computer Science(), vol 12480. Springer, Cham. https://doi.org/10.1007/978-3-030-71593-9_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-71593-9_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-71592-2
Online ISBN: 978-3-030-71593-9
eBook Packages: Computer ScienceComputer Science (R0)