Abstract
Deep Reinforcement Learning (DRL) combines deep neural networks with reinforcement learning. These methods, unlike their predecessors, learn end-to-end by extracting high-dimensional representations from raw sensory data to directly predict the actions. DRL methods were shown to master most of the ATARI games, beating humans in a good number of them, using the same algorithm, network architecture and hyper-parameters. However, why DRL works on some games better than others has not been fully investigated. In this paper, we propose that the complexity of each game is defined by a number of factors (the size of the search space, existence/absence of enemies, existence/absence of intermediate reward, and so on) and we posit that how fast and well a game is learned by DRL depends on these factors. Towards this aim, we use simplified Maze and Pacman environments and we conduct experiments to see the effect of such factors on the convergence of DRL. Our results provide a first step in a better understanding of how DRL works and as such will be informative in the future in determining scenarios where DRL can be applied effectively e.g., outside of games.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We expand and use Nathan Sprague’s replication: https://github.com/spragunr/deep_q_rl.
- 2.
We use the Pacman environment prepared for the course UC Berkeley CS188 Introduction to AI, available at http://ai.berkeley.edu/reinforcement.html.
References
Anderson, D., Stephenson, M., Togelius, J., Salge, C., Levine, J., Renz, J.: Deceptive games. In: Sim, K., Kaufmann, P. (eds.) EvoApplications 2018. LNCS, vol. 10784, pp. 376–391. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77538-8_26
Bellemare, M.G., Dabney, W., Munos, R.: A distributional perspective on reinforcement learning. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 449–458. JMLR. org. (2017)
Berndt, D.J., Clifford, J.: Using dynamic time warping to find patterns in time series. In: Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining, AAAIWS 1994, pp. 359–370. AAAI Press (1994)
Elias, G.S., Garfield, R., Gutschera, K.R.: Characteristics of Games. The MIT Press, Cambridge (2012)
Hasselt, H.V., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 2016, pp. 2094–2100. AAAI Press (2016)
Hessel, M., Modayil, J., van Hasselt, H., et al.: Rainbow: combining improvements in deep reinforcement learning. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Levine, S., Finn, C., Darrell, T., et al.: End-to-end training of deep visuomotor policies. J. Mach. Learn. Res. 17(39), 1–40 (2016)
Lillicrap, T.P., Hunt, J.J., Pritzel, A., et al.: Continuous control with deep reinforcement learning. ArXiv e-prints, September 2015
Mnih, V., Badia, A.P., Mirza, M., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, 19–24 June 2016, pp. 1928–1937 (2016)
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Nair, A., Srinivasan, P., Blackwell, S., et al.: Massively parallel methods for deep reinforcement learning. ArXiv e-prints, July 2015
Peng, X.B., Berseth, G., van de Panne, M.: Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Trans. Graph. 35(4), 81:1–81:12 (2016). https://doi.org/10.1145/2897824.2925881
Schaul, T., Quan, J., Antonoglou, I., et al.: Prioritized experience replay. ArXiv e-prints, November 2015
Silver, D., Huang, A., Maddison, C.J., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Silver, D., Hubert, T., Schrittwieser, J., et al.: Mastering Chess and Shogi by self-play with a general reinforcement learning algorithm. ArXiv e-prints, December 2017
Silver, D., Schrittwieser, J., Simonyan, K., et al.: Mastering the game of go without human knowledge. Nature 550, 354–359 (2017)
Wang, Z., Schaul, T., Hessel, M., et al.: Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning, ICML 2016, vol. 48, pp. 1995–2003. JMLR.org (2016)
Yannakakis, G.N., Togelius, J.: Artificial Intelligence and Games. Springer, Heidelberg (2018). http://gameaibook.org
Acknowledgements
We would like to express our gratitude to Yapı Kredi Teknoloji A.Ş. for supporting us in participating to IJCAI 2018 events.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Emekligil, E., Alpaydın, E. (2019). What’s in a Game? The Effect of Game Complexity on Deep Reinforcement Learning. In: Cazenave, T., Saffidine, A., Sturtevant, N. (eds) Computer Games. CGW 2018. Communications in Computer and Information Science, vol 1017. Springer, Cham. https://doi.org/10.1007/978-3-030-24337-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-24337-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-24336-4
Online ISBN: 978-3-030-24337-1
eBook Packages: Computer ScienceComputer Science (R0)