Alexandre M. M. Santos
ABSTRACT
Training agents using Deep Reinforcement Learning methods is rapidly progressing in several fields and techniques like domain randomization have been demonstrated to improve the generalization ability of these agents. However, due to the black-box nature of the models, it is not easy to understand why an action was selected from a given input. Although prior research on Explainable Artificial Intelligence presents efforts to bridge this gap, is unclear what particular input features that contribute to a model’s generalizability. This work examines the main aspects that affect the behavior of game agents with varying robustness levels. By comparing specialized and generalized agents, we investigate what are the main differences and similarities present in these models when they select an action. To achieve this goal, we trained two agents with different robustness levels and applied Explainable Artificial Intelligence methods to highlight the key features on the input screen. We employed a mixed methods analysis, which provided important quantitative results on the agents’ performance as well as qualitative insights about their behavior. We are able to show that the visualization of generalized agents tends to be more interpretable since they concentrate on the game objects, whereas specialized agents are more spread along the whole input screen. This result constitutes an important step to understanding the behavior of game agents trained using Deep Reinforcement Learning with different training procedures.
- Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. Advances in neural information processing systems 31 (2018).Google Scholar
- Laura Almón-Manzano, Rafael Pastor-Vargas, and José M. C. Troncoso. 2022. Deep Reinforcement Learning in Agents’ Training: Unity ML-Agents. In Bio-inspired Systems and Applications: from Robotics to Ambient Intelligence: 9th International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2022. Springer, 391–400.Google Scholar
- Alejandro B. Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion 58 (2020), 82–115.Google ScholarDigital Library
- Vittoria Bruni, Maria Lucia Cardinali, and Domenico Vitulano. 2022. A short review on minimum description length: an application to dimension reduction in PCA. Entropy 24, 2 (2022), 269.Google ScholarCross Ref
- Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. 2020. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning. PMLR, 2048–2056.Google Scholar
- Tianhong Dai, Kai Arulkumaran, Tamara Gerbert, Samyakh Tukra, Feryal Behbahani, and Anil Anthony Bharath. 2022. Analysing deep reinforcement learning agents trained with domain randomisation. Neurocomputing 493 (2022), 143–165.Google ScholarDigital Library
- Linus Gisslén, Andy Eakins, Camilo Gordillo, Joakim Bergdahl, and Konrad Tollmar. 2021. Adversarial reinforcement learning for procedural content generation. In 2021 IEEE Conference on Games (CoG). IEEE, 1–8.Google ScholarDigital Library
- Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. 2018. Visualizing and understanding atari agents. In International conference on machine learning. PMLR, 1792–1801.Google Scholar
- Jinlei Gu, Jiacun Wang, Xiwang Guo, Guanjun Liu, Shujin Qin, and Zhiliang Bi. 2023. A Metaverse-Based Teaching Building Evacuation Training System With Deep Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics: Systems (2023).Google ScholarCross Ref
- Piyush Gupta, Nikaash Puri, Sukriti Verma, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. 2020. Explain your move: Understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations (ICLR).Google Scholar
- Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. 2021. Explainability in deep reinforcement learning. Knowledge-Based Systems 214 (2021).Google Scholar
- Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. 2020. Understanding RL Vision. Distill (2020). https://doi.org/10.23915/distill.00029 https://distill.pub/2020/understanding-rl-vision.Google ScholarCross Ref
- Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. 2022. CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. Journal of Machine Learning Research 23, 274 (2022), 1–18.Google Scholar
- Tobias Huber, Benedikt Limmer, and Elisabeth André. 2021. Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents. arXiv preprint 2101.07312 (2021).Google Scholar
- Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. 2018. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 144–150.Google ScholarDigital Library
- Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016. Examples are not enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems 29 (2016).Google Scholar
- Xiao Li, Hanchen Xu, Jinming Zhang, and Hua-hua Chang. 2023. Deep reinforcement learning for adaptive learning systems. Journal of Educational and Behavioral Statistics 48, 2 (2023), 220–243.Google ScholarCross Ref
- Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017).Google Scholar
- Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence 267 (2019), 1–38.Google Scholar
- Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The Building Blocks of Interpretability. Distill (2018). https://doi.org/10.23915/distill.00010 https://distill.pub/2018/building-blocks.Google ScholarCross Ref
- Vitali Petsiuk, Abir Das, and Kate Saenko. 2018. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint 1806.07421 (2018).Google Scholar
- Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. 2017. Robust adversarial reinforcement learning. In International Conference on Machine Learning. PMLR, 2817–2826.Google Scholar
- Geeta Rani, Upasana Pandey, Aniket Anil Wagde, and Vijaypal Singh Dhaka. 2023. A deep reinforcement learning technique for bug detection in video games. International Journal of Information Technology 15, 1 (2023), 355–367.Google ScholarCross Ref
- Marco T. Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135–1144.Google ScholarDigital Library
- Furkat Safarov, Alpamis Kutlimuratov, Akmalbek Bobomirzaevich Abdusalomov, Rashid Nasimov, and Young-Im Cho. 2023. Deep Learning Recommendations of E-Education Based on Clustering and Sequence. Electronics 12, 4 (2023), 809.Google ScholarCross Ref
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint 1707.06347 (2017).Google Scholar
- Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision. 618–626.Google ScholarCross Ref
- Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In International Conference on Learning Representations.Google Scholar
- Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning. 3319–3328.Google Scholar
- Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 5026–5033.Google ScholarCross Ref
- Changnan Xiao, Yongxin Zhang, Xuefeng Huang, Qinhan Huang, Jie Chen, and Peng Sun. 2023. Mastering Strategy Card Game (Hearthstone) with Improved Techniques. arXiv preprint 2303.05197 (2023).Google Scholar
- Wenzhuo Yang, Hung Le, Silvio Savarese, and Steven Hoi. 2022. OmniXAI: A Library for Explainable AI. arXiv preprint 2206.01612 (2022).Google Scholar
- Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 818–833.Google ScholarCross Ref
Index Terms
- Unveiling the Key Features Influencing Game Agents with Different Levels of Robustness
Recommendations
Towards Understanding of Deep Reinforcement Learning Agents Used in Cloud Resource Management
Computational Science – ICCS 2023AbstractCloud computing resource management is a critical component of the modern cloud computing platforms, aimed to manage computing resources for a given application by minimizing the cost of the infrastructure while maintaining a Quality-of-Service (...
Deep Reinforcement Learning in Agents’ Training: Unity ML-Agents
Bio-inspired Systems and Applications: from Robotics to Ambient IntelligenceAbstractVideo games are an area where Artificial Intelligence has multiple application scenarios, allowing to add improvements that can be applied to provide greater realism in the game experience, accelerate its development (even automate it) and save ...
Learning to Assist Agents by Observing Them
Artificial Neural Networks and Machine Learning – ICANN 2021AbstractThe ability of an AI agent to assist other agents, such as humans, is an important and challenging goal, which requires the assisting agent to reason about the behavior and infer the goals of the assisted agent. Training such an ability by using ...
Comments