Abstract
Swarm robotic systems (SRS) are a type of multi-robot systems that consist of many homogeneous autonomous robots inspired by social insects. In our pervious study, we succeeded in developing end-to-end control policies for SRS using Deep Q-Network (DQN) algorithm. However, since DQN is totally a black box, it is difficult to understand what were learnt through the learning process. Therefore, in this paper, a novel method of visualizing the decision making process in the DQN is proposed by combining Deconvolutional Network (Deconvnet) and Gradient-weighted Class Activation Mapping (Grad-CAM). Then we show what are being preserved as the deep features and which part of input image is concerned to make an action decision. The proposed method is demonstrated by conducting the computer simulations of a round trip task, in which the swarm robots need to visit two different locations alternatively as many times as possible. The computer simulations might also be explained that the proposed method visualizes the policies learned by DQN.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sahin, E.: Swarm robotics: from sources of inspiration to domains of application. In: International Workshop on Swarm Robotics. LNCS, vol. 3342, pp. 10–20 (2004)
Francesca, G., Brambilla, M., Trianni, V., Dorigo, M., Birattari, M.: Analysing an evolved robotic behaviour using a biological model of collegial decision making. In: International Conference on Simulation of Adaptive Behavior, pp. 381–390 (2012)
Brambilla, M., Ferrante, E., Birattari, M., Dorigo, M.: Swarmrobotics: a review from the swarm engineering perspective. Swarm Intell. 7(1), 1–41 (2013)
Wei, Y., Nie, X., Hiraga, M., Ohkura, K., Car, Z.: Developing end-to-end control policies for robotic swarms using deep Q-learning. J. Adv. Comput. Intell. Intell. Inf. 23, 920–927 (2019)
Gunning, D.: Explainable artificial intelligence (XAI). Defense Advanced Research Projects Agency (DARPA) (2017)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision (ECCV), pp. 818–833 (2014)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929 (2016)
Lin, M., Chen, Q., Yan, S.: Network in network. In: International Conference on Learning Representations (ICLR) (2014)
Selvaraju, R.R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., Batra, D.: Grad-CAM: why did you say that? Visual explanations from deep networks via gradient-based localization. arXiv preprint arXiv:1610.02391 (2016)
Fong, R.C., Vedaldi, A.: Interpretable explanations of black boxes by meaningful perturbation. arXiv preprint arXiv:1704.03296 (2017)
Zahavy, T., Zrihem, N.B., Mannor, S.: Graying the black box: understanding DQNs. In: International Conference on Machine Learning (ICML), pp. 1899–1908 (2016)
Van der Maaten, L., Hinton, G.E.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Nie, X., Hiraga, M., Ohkura, K. (2020). Visualizing Deep Q-Learning to Understanding Behavior of Swarm Robotic System. In: Sato, H., Iwanaga, S., Ishii, A. (eds) Proceedings of the 23rd Asia Pacific Symposium on Intelligent and Evolutionary Systems. IES 2019. Proceedings in Adaptation, Learning and Optimization, vol 12. Springer, Cham. https://doi.org/10.1007/978-3-030-37442-6_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-37442-6_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37441-9
Online ISBN: 978-3-030-37442-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)