Abstract
Cloud computing resource management is a critical component of the modern cloud computing platforms, aimed to manage computing resources for a given application by minimizing the cost of the infrastructure while maintaining a Quality-of-Service (QoS) conditions. This task is usually solved using rule-based policies. Due to their limitations more complex solutions, such as Deep Reinforcement Learning (DRL) agents are being researched. Unfortunately, deploying such agents in a production environment can be seen as risky because of the lack of transparency of DRL decision-making policies. There is no way to know why a certain decision is made. To foster the trust in DRL generated policies it is important to provide means of explaining why certain decisions were made given a specific input. In this paper we present a tool applying the Integrated Gradients (IG) method to Deep Neural Networks used by DRL algorithms. This allowed to obtain feature attributions that show the magnitude and direction of each feature’s influence on the agent’s decision. We verify the viability of the proposed solution by applying it to a number of sample use cases with different DRL agents.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barredo Arrieta, A., et al.: Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58, 82–115 (2020). https://doi.org/10.1016/j.inffus.2019.12.012
Cheng, M., et al.: DRL-cloud: deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers. In: 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 129–134 (2018)
Cobbe, K., et al.: Quantifying Generalization in Reinforcement Learning (2018). https://doi.org/10.48550/ARXIV.1812.02341
Cuayáhuitl, H.: SimpleDS: A Simple Deep Reinforcement Learning Dialogue System (2016). https://doi.org/10.48550/ARXIV.1601.04574
Dutta, S., et al.: SmartScale: automatic application scaling in enterprise clouds. In: 2012 IEEE Fifth International Conference on Cloud Computing, pp. 221–228 (2012)
Funika, W., Koperek, P.: Evaluating the use of policy gradient optimization approach for automatic cloud resource provisioning. In: Wyrzykowski, R., Deelman, E., Dongarra, J., Karczewski, K. (eds.) PPAM 2019. LNCS, vol. 12043, pp. 467–478. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43229-4_40
Funika, W., Koperek, P., Kitowski, J.: Automatic management of cloud applications with use of proximal policy optimization. In: Krzhizhanovskaya, V.V., et al. (eds.) ICCS 2020. LNCS, vol. 12137, pp. 73–87. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50371-0_6
Gregurić, M., et al.: Application of deep reinforcement learning in traffic signal control: an overview and impact of open traffic data. Appl. Sci. 10(11) (2020)
Greydanus, S., et al.: Visualizing and Understanding Atari Agents (2017). https://doi.org/10.48550/ARXIV.1711.00138
Guo, W., et al.: EDGE: Explaining Deep Reinforcement Learning Policies. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 12222–12236. Curran Associates, Inc. (2021)
Heuillet, A., et al.: Explainability in Deep Reinforcement Learning. CoRR abs/2008.06693 (2020)
Hilton, J., et al.: Understanding RL Vision. Distill (2020). https://doi.org/10.23915/distill.00029
Małota, A., et al.: Trainloop-driver (2023). https://github.com/andrzejmalota/trainloop-driver/tree/master/examples
Milani, S., et al.: A Survey of Explainable Reinforcement Learning (2022). https://doi.org/10.48550/ARXIV.2202.08434
Mnih, V., et al.: Playing Atari with deep reinforcement learning. In: NIPS Deep Learning Workshop (2013). http://arxiv.org/abs/1312.5602
Mott, A., et al.: Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019). https://doi.org/10.48550/ARXIV.1906.02500
Olah, C., et al.: The Building Blocks of Interpretability. Distill (2018). https://doi.org/10.23915/distill.00010
OpenAI, et al.: Solving Rubik’s Cube with a Robot Hand (2019). https://doi.org/10.48550/ARXIV.1910.07113
Ribeiro, M.T., et al.: “Why Should I Trust You?”: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD 2016, New York, NY, USA, pp. 1135–1144. Association for Computing Machinery (2016)
Schulman, J., et al.: Proximal Policy Optimization Algorithms (2017). https://doi.org/10.48550/ARXIV.1707.06347
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vision 128(2), 336–359 (2019). https://doi.org/10.1007/s11263-019-01228-7
Campos da Silva Filho, M., et al.: CloudSim plus: a cloud computing simulation framework pursuing software engineering principles for improved modularity, extensibility and correctness. In: 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), pp. 400–406 (2017)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. The MIT Press (2018)
Tighe, M., Bauer, M.: Integrating cloud application autoscaling with dynamic VM allocation. In: 2014 IEEE Network Operations and Management Symposium (NOMS), pp. 1–9 (2014)
Wang, Z., et al.: Automated Cloud Provisioning on AWS using Deep Reinforcement Learning (2017). https://doi.org/10.48550/ARXIV.1709.04305
Zhang, Y., et al.: Intelligent cloud resource management with deep reinforcement learning. IEEE Cloud Comput. 4(6), 60–69 (2017). https://doi.org/10.1109/MCC.2018.1081063
Acknowledgements
The research presented in this paper was supported by the funds assigned to AGH University of Krakow by the Polish Ministry of Education and Science.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Małota, A., Koperek, P., Funika, W. (2023). Towards Understanding of Deep Reinforcement Learning Agents Used in Cloud Resource Management. In: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2023. ICCS 2023. Lecture Notes in Computer Science, vol 14074. Springer, Cham. https://doi.org/10.1007/978-3-031-36021-3_55
Download citation
DOI: https://doi.org/10.1007/978-3-031-36021-3_55
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36020-6
Online ISBN: 978-3-031-36021-3
eBook Packages: Computer ScienceComputer Science (R0)