Unveiling the Key Features Influencing Game Agents with Different Levels of Robustness
Authors: 
Alexandre M. M. Santos, Halisson R. Rodrigues, Romulo F. Férrer Filho, Yuri L. B. Nogueira, Creto A. Vidal, Joaquim B. Cavalcante-Neto, Artur O. R. Franco, Paulo B. S. SerafimAuthors Info & Claims
Alexandre M. M. Santos, Halisson R. Rodrigues, Romulo F. Férrer Filho, Yuri L. B. Nogueira, Creto A. Vidal, Joaquim B. Cavalcante-Neto, Artur O. R. Franco, Paulo B. S. SerafimAuthors Info & ClaimsPages 86 - 95
Abstract
Training agents using Deep Reinforcement Learning methods is rapidly progressing in several fields and techniques like domain randomization have been demonstrated to improve the generalization ability of these agents. However, due to the black-box nature of the models, it is not easy to understand why an action was selected from a given input. Although prior research on Explainable Artificial Intelligence presents efforts to bridge this gap, is unclear what particular input features that contribute to a model’s generalizability. This work examines the main aspects that affect the behavior of game agents with varying robustness levels. By comparing specialized and generalized agents, we investigate what are the main differences and similarities present in these models when they select an action. To achieve this goal, we trained two agents with different robustness levels and applied Explainable Artificial Intelligence methods to highlight the key features on the input screen. We employed a mixed methods analysis, which provided important quantitative results on the agents’ performance as well as qualitative insights about their behavior. We are able to show that the visualization of generalized agents tends to be more interpretable since they concentrate on the game objects, whereas specialized agents are more spread along the whole input screen. This result constitutes an important step to understanding the behavior of game agents trained using Deep Reinforcement Learning with different training procedures.
References
[1]
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. Advances in neural information processing systems 31 (2018).
[2]
Laura Almón-Manzano, Rafael Pastor-Vargas, and José M. C. Troncoso. 2022. Deep Reinforcement Learning in Agents’ Training: Unity ML-Agents. In Bio-inspired Systems and Applications: from Robotics to Ambient Intelligence: 9th International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2022. Springer, 391–400.
[3]
Alejandro B. Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion 58 (2020), 82–115.
[4]
Vittoria Bruni, Maria Lucia Cardinali, and Domenico Vitulano. 2022. A short review on minimum description length: an application to dimension reduction in PCA. Entropy 24, 2 (2022), 269.
[5]
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. 2020. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning. PMLR, 2048–2056.
[6]
Tianhong Dai, Kai Arulkumaran, Tamara Gerbert, Samyakh Tukra, Feryal Behbahani, and Anil Anthony Bharath. 2022. Analysing deep reinforcement learning agents trained with domain randomisation. Neurocomputing 493 (2022), 143–165.
[7]
Linus Gisslén, Andy Eakins, Camilo Gordillo, Joakim Bergdahl, and Konrad Tollmar. 2021. Adversarial reinforcement learning for procedural content generation. In 2021 IEEE Conference on Games (CoG). IEEE, 1–8.
[8]
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. 2018. Visualizing and understanding atari agents. In International conference on machine learning. PMLR, 1792–1801.
[9]
Jinlei Gu, Jiacun Wang, Xiwang Guo, Guanjun Liu, Shujin Qin, and Zhiliang Bi. 2023. A Metaverse-Based Teaching Building Evacuation Training System With Deep Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics: Systems (2023).
[10]
Piyush Gupta, Nikaash Puri, Sukriti Verma, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. 2020. Explain your move: Understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations (ICLR).
[11]
Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. 2021. Explainability in deep reinforcement learning. Knowledge-Based Systems 214 (2021).
[12]
Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. 2020. Understanding RL Vision. Distill (2020). https://doi.org/10.23915/distill.00029 https://distill.pub/2020/understanding-rl-vision.
[13]
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. 2022. CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. Journal of Machine Learning Research 23, 274 (2022), 1–18.
[14]
Tobias Huber, Benedikt Limmer, and Elisabeth André. 2021. Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents. arXiv preprint 2101.07312 (2021).
[15]
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. 2018. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 144–150.
[16]
Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016. Examples are not enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems 29 (2016).
[17]
Xiao Li, Hanchen Xu, Jinming Zhang, and Hua-hua Chang. 2023. Deep reinforcement learning for adaptive learning systems. Journal of Educational and Behavioral Statistics 48, 2 (2023), 220–243.
[18]
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017).
[19]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence 267 (2019), 1–38.
[20]
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The Building Blocks of Interpretability. Distill (2018). https://doi.org/10.23915/distill.00010 https://distill.pub/2018/building-blocks.
[21]
Vitali Petsiuk, Abir Das, and Kate Saenko. 2018. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint 1806.07421 (2018).
[22]
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. 2017. Robust adversarial reinforcement learning. In International Conference on Machine Learning. PMLR, 2817–2826.
[23]
Geeta Rani, Upasana Pandey, Aniket Anil Wagde, and Vijaypal Singh Dhaka. 2023. A deep reinforcement learning technique for bug detection in video games. International Journal of Information Technology 15, 1 (2023), 355–367.
[24]
Marco T. Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135–1144.
[25]
Furkat Safarov, Alpamis Kutlimuratov, Akmalbek Bobomirzaevich Abdusalomov, Rashid Nasimov, and Young-Im Cho. 2023. Deep Learning Recommendations of E-Education Based on Clustering and Sequence. Electronics 12, 4 (2023), 809.
[26]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint 1707.06347 (2017).
[27]
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision. 618–626.
[28]
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In International Conference on Learning Representations.
[29]
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning. 3319–3328.
[30]
Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 5026–5033.
[31]
Changnan Xiao, Yongxin Zhang, Xuefeng Huang, Qinhan Huang, Jie Chen, and Peng Sun. 2023. Mastering Strategy Card Game (Hearthstone) with Improved Techniques. arXiv preprint 2303.05197 (2023).
[32]
Wenzhuo Yang, Hung Le, Silvio Savarese, and Steven Hoi. 2022. OmniXAI: A Library for Explainable AI. arXiv preprint 2206.01612 (2022).
[33]
Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 818–833.
Index Terms
- Unveiling the Key Features Influencing Game Agents with Different Levels of Robustness
Recommendations
Towards Understanding of Deep Reinforcement Learning Agents Used in Cloud Resource Management
Computational Science – ICCS 2023AbstractCloud computing resource management is a critical component of the modern cloud computing platforms, aimed to manage computing resources for a given application by minimizing the cost of the infrastructure while maintaining a Quality-of-Service (...
Deep Reinforcement Learning in Agents’ Training: Unity ML-Agents
Bio-inspired Systems and Applications: from Robotics to Ambient IntelligenceAbstractVideo games are an area where Artificial Intelligence has multiple application scenarios, allowing to add improvements that can be applied to provide greater realism in the game experience, accelerate its development (even automate it) and save ...
Learning to Assist Agents by Observing Them
Artificial Neural Networks and Machine Learning – ICANN 2021AbstractThe ability of an AI agent to assist other agents, such as humans, is an important and challenging goal, which requires the assisting agent to reason about the behavior and infer the goals of the assisted agent. Training such an ability by using ...
Comments
Information & Contributors
Information
Published In
November 2023
176 pages
ISBN:9798400716270
DOI:10.1145/3631085
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 January 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SBGames 2023
SBGames 2023: 22nd Brazilian Symposium on Games and Digital Entertainment
November 6 - 9, 2023
Rio Grande (RS), Brazil
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 26Total Downloads
- Downloads (Last 12 months)21
- Downloads (Last 6 weeks)2
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format