Skip to main content
SpringerLink
Log in

Intelligent microservices autoscaling module using reinforcement learning

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Cloud applications developed using a microservices approach require a scaling policy that adapts dynamically to the changing resource demand of the microservice while satisfying the application’s quality of service (QoS) agreement. Current cloud technologies, including Kubernetes Horizontal Pod Autoscaler (HPA), autoscale microservices based on predefined autoscaling metrics, which require a very good knowledge of the application’s resource usage. There is no built-in QoS autoscaling mechanism. In this work, we introduce an intelligent autoscaling module for the horizontal autoscaling of microservices in the cloud. We develop, train, validate, and deploy reinforcement learning (RL) agents on a real-time application using response time as a QoS. The agents, deployed on Google Cloud, can learn the microservices autoscaling resource usage metrics from the cloud environment and customize the Kubernetes HPA while minimizing response time. Our results show a decrease in response time compared to the default Kubernetes HPA. The agents provide an extendible autoscaling plug-in module that can adapt to new QoS metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

Enquiries about data availability should be directed to the authors.

References

  1. Khaleq, A. A., Ra, I.: Agnostic approach for microservices autoscaling in cloud applications. In :Proc. CSCI. pp. 1411–1415. Las Vegas, NV, USA (2019). https://doi.org/10.1109/CSCI49370.2019.00264.

  2. Gan, Y., Delimitrou, C.: The architectural implications of microservices in the cloud, arXiv preprint arXiv:1805.10351Y (2018)

  3. Ghahramani, Mohammad Hossein, Zhou, MengChu, Hon, Chi Tin: Toward cloud computing QoS architecture: analysis of cloud systems and cloud services. IEEE/CAA J. Autom. Sinica 4(1), 6–18 (2017)

    Article  MathSciNet  Google Scholar 

  4. Singh, Sukhpal, Chana, Inderveer: QoS-aware autonomic resource management in cloud computing: a systematic review. ACM Comput. Surv. (CSUR) 48(3), 1–46 (2015)

    Article  Google Scholar 

  5. Horovitz, S., Arian, Y.: Efficient cloud auto-scaling with SLA objective using Q-Learning. In: 2018 IEEE 6th International Conference on Future Internet of Things and Cloud (FiCloud). pp. 85–92. IEEE (2018 August)

  6. Yang, Z., Nguyen, P., Jin, H., Nahrstedt, K.: MIRAS: Model-based reinforcement learning for microservice resource allocation over scientific workflows. In: 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). pp. 122–132. Dallas, TX, USA (2019)

  7. Barrett, Enda, Howley, Enda, Duggan, Jim: Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurr. Comput.: Pract. Exp. 25(12), 1656–1674 (2013)

    Article  Google Scholar 

  8. Zheng, T., Zheng, X., Zhang, Y., Deng, Y., Dong, E., Zhang, R., Liu, X.: SmartVM: a SLA-aware microservice deployment framework. World Wide Web 22(1), 275–293 (2019)

    Article  Google Scholar 

  9. Rossi, F.: Auto-scaling Policies to Adapt the application deployment in kubernetes, pp. 30–38. ZEUS, Olympia (2020)

    Google Scholar 

  10. Rossi, F., Nardelli, M., Cardellini, V.: Horizontal and vertical scaling of container-based applications using reinforcement learning. In: Proc. IEEE CLOUD. pp. 329–338. (2019)

  11. Jamshidi, P., Sharifloo, A. M., Pahl, C., Metzger, A., Estrada, G.: Self-learning cloud controllers: fuzzy q-learning for knowledge evolution. In: International Conference on Cloud and Autonomic Computing. pp. 208–211. Boston, MA, USA (2015) https://doi.org/10.1109/ICCAC.2015.35

  12. Rzadca, K., Findeisen, P., Swiderski, J., Zych, P., Broniek, P., Kusmierek, J., Nowak, P., et al.: Autopilot: workload autoscaling at Google. In: Proc. of the Fifteenth European Conference on Computer Systems. pp. 1–16. (2020)

  13. Khaleq, A.A., Ra, I.: Intelligent autoscaling of microservices in the cloud for real-time applications. IEEE Access (2021). https://doi.org/10.1109/ACCESS.2021.3061890

    Article  Google Scholar 

  14. Train a deep Q-Network with TF-Agents. The TF-Agents Authors. https://www.tensorflow.org/agents/tutorials/1_dqn_tutorial. Last Accessed 17 Mar 2021

  15. Abdel Khaleq, A., Ra, I.: 20018, November. Twitter analytics for disaster relevance and disaster phase discovery. In: Proceeding of the Future Technologies conference. pp. 401–417. Springer, Cham

  16. Khaleq, A.A., Ra, I.: Cloud-based disaster management as a service A microservice approach for hurricane Twitter data analysis. In: IEEE Global Humanitarian Technology Conference (GHTC). San Jose, CA 2018. 1–8 (2018)

  17. Multi-layer perceptron classifier, Scikit Learn. https://scikit-learn.org/stable/modules/neural_networks_supervised.html. Last Accessed 17 Mar 2021

  18. Imran, M., Elbassuoni S., Castillo, C., Diaz, F., Meier, P.: Practical extraction of disaster-relevant information from social media. In: 22nd International Conference on World Wide Web. pp. 1021–1024. ACM, Rio de Janeiro, Brazil (2013)

  19. Vertical pod autoscaling, Google cloud. https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler. Last Accessed 17 Mar 2021

  20. Best practices for running cost-optimized Kubernetes applications on GKE, Cloud Architecture Center. https://cloud.google.com/solutions/best-practices-for-running-cost-effective-kubernetes-applications-on-gke. Last Accessed 21 Mar 2021

  21. Cloud computing QoS for real-time data applications, European commission. https://cordis.europa.eu/article/id/124056-cloud-computing-qos-for-realtime-data-applications. Last Accessed 15 Apr 2021

  22. Kho Lin, S. et al.: Auto-scaling a defence application across the cloud using Docker and Kubernetes. In: 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion). Zurich, pp. 327–334 (2018)

  23. Toka, L., Dobreff, G., Fodor, B., Sonkoly, B.: Adaptive AI-based auto-scaling for Kubernetes, p. 599. IEEE, Piscataway (2020)

    Google Scholar 

  24. Lee, Do-Young., et al.: Deep Q-network-based auto scaling for service in a multi-access edge computing environment. Int. J. Netw. Manag. 31(6), e2176 (2021)

    Article  Google Scholar 

  25. Kwan, Anthony, et al.: Hyscale: Hybrid and network scaling of dockerized microservices in cloud data centres. In: 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, (2019)

  26. Khaleq, Abeer Abdel: Design and Evaluation of QoS Aware Intelligent Autoscaling Services for Microservices in Cloud Computing Environments. University of Colorado at Denver, PhD diss. (2021)

  27. Sachidananda, Vighnesh, Sivaraman, Anirudh: Learned Autoscaling for Cloud Microservices with Multi-Armed Bandits. arXiv preprint arXiv:2112.14845 (2021)

  28. Wu, Qiang, et al.: Dynamically adjusting scale of a kubernetes cluster under qos guarantee. In: 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS). IEEE, (2019)

  29. Kingma, Diederik P., Ba, Jimmy: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  30. Khaleq, A. A., Ra, I.: Development of QoS-aware agents with reinforcement learning for autoscaling of microservices on the cloud. In: IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C). pp. 13–19. DC, USA (2021) https://doi.org/10.1109/ACSOS-C52956.2021.00025

Download references

Funding

The authors have not disclosed any funding.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Colorado, Denver, CO, USA

    Abeer Abdel Khaleq & Ilkyeun Ra

Authors
  1. Abeer Abdel Khaleq
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ilkyeun Ra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abeer Abdel Khaleq.

Ethics declarations

Competing interests

The authors have not disclosed any competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abdel Khaleq, A., Ra, I. Intelligent microservices autoscaling module using reinforcement learning. Cluster Comput 26, 2789–2800 (2023). https://doi.org/10.1007/s10586-023-03999-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-023-03999-8

Keywords

Search

Navigation