Abstract
For the last decade, Deep Reinforcement Learning (DRL) has undergone very rapid development. However, less has been done to integrate linear methods into it. Our research aims at a simple and practical Wide and Deep Reinforcement Learning framework to extend DRL algorithms by combining linear (wide) and non-linear (deep) methods. This framework can help to integrate expert knowledge or to fuse sensor information while at the same time improving the performance of existing DRL algorithms. To test this framework we have developed an extension of the popular Deep Q-Networks Algorithm, which we call Wide Deep Q-Networks. We analyze its performance compared to Deep Q-Networks and Linear Agents, as well as human agents by applying our new algorithm to Berkeley’s Pac-Man environment. Our algorithm considerably outperforms Deep Q-Networks both in terms of learning speed and ultimate performance, showing its potential for boosting existing algorithms. Furthermore, it is robust to the failure of one of its components.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
- 2.
References
Bohez, S., Verbelen, T., De Coninck, E., Vankeirsbilck, B., Simoens, P., Dhoedt, B.: Sensor fusion for robot control through deep reinforcement learning. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2365–2370. IEEE, September 2017
Cheng, H.T., et al.: Wide & deep learning for recommender systems. In: Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS 2016, pp. 7–10. ACM, New York (2016)
DeNero, J., Klein, D.: Teaching introductory artificial intelligence with pac-man. In: Proceedings of the Symposium on Educational Advances in Artificial Intelligence, pp. 1885–1889 (2010)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
van Hasselt, H.P., Guez, A., Hessel, M., Mnih, V., Silver, D.: Learning values across many orders of magnitude. In: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December 2016, pp. 4287–4295 (2016)
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., Meger, D.: Deep reinforcement learning that matters. In: Proceedings of the Thirtieth-Second AAAI Conference on Artificial Intelligence, AAAI 2018. AAAI Press (2018)
Kalashnikov, D., et al.: QT-Opt: scalable deep reinforcement learning for vision-based robotic. CoRR abs/1806.10293 (2018)
Kim, H.J., Jordan, M.I., Sastry, S., Ng, A.Y.: Autonomous helicopter flight via reinforcement learning. In: Thrun, S., Saul, L.K., Schölkopf, B. (eds.) Advances in Neural Information Processing Systems 16, pp. 799–806. MIT Press (2004)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lin, L.J.: Self-improving reactive agents based on reinforcement learning, Plann. Teach. Machine Learning 8(3), 293–321 (1992). https://doi.org/10.1007/BF00992699
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Montoya., J.M., Borgelt., C.: Wide and deep reinforcement learning for grid-based action games. In: Proceedings of the 11th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, pp. 50–59. INSTICC, SciTePress (2019). https://doi.org/10.5220/0007313200500059
van der Ouderaa, T.: Deep Reinforcement Learning in Pac-Man (2016). Bachelor thesis, University of Amsterdam
Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach. Pearson Education, 3 edn. (2003)
Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning (71 2018), working Second Edition
Watkins, C.J.C.H.: Learning from Delayed Rewards. Ph.D. thesis, King’s College, Cambridge, UK (1989). http://www.cs.rhul.ac.uk/~chrisw/new_thesis.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Montoya, J.M., Doell, C., Borgelt, C. (2019). Wide and Deep Reinforcement Learning Extended for Grid-Based Action Games. In: van den Herik, J., Rocha, A., Steels, L. (eds) Agents and Artificial Intelligence. ICAART 2019. Lecture Notes in Computer Science(), vol 11978. Springer, Cham. https://doi.org/10.1007/978-3-030-37494-5_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-37494-5_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37493-8
Online ISBN: 978-3-030-37494-5
eBook Packages: Computer ScienceComputer Science (R0)