Skip to main content
SpringerLink
Log in

A Survey of Reinforcement Learning Toolkits for Gaming: Applications, Challenges and Trends

  • Conference paper
  • First Online:
Proceedings of the Future Technologies Conference (FTC) 2022, Volume 1 (FTC 2022 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 559))

Included in the following conference series:

Abstract

The gaming industry has become one of the most exciting and creative industries. The annual revenue has crossed $200 billion in recent years and has created a lot of jobs globally. Many games are using Artificial Intelligence (AI) and techniques like Machine Learning (ML), Reinforcement Learning (RL) gained popularity among researchers and game development community to enable smart games involving AI-based agents at a faster rate. Although, many toolkits are available for use, a framework to evaluate, compare and advise on these toolkits is still missing. In this paper, we present a comprehensive overview of ML/RL toolkits for games with an emphasis on their applications, challenges, and trends. We propose a qualitative evaluation methodology, discuss the obtained analysis results, and conclude with future work and perspectives.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. AlphaZero: shedding new light on chess, shogi, and go. https://deepmind.com/blog/article/alphazero-shedding-new-light-grand-games-chess-shogi-and-go

  2. Facebook, Carnegie Mellon build first AI that beats pros in 6-player poker. https://ai.facebook.com/blog/pluribus-first-ai-to-beat-pros-in-6-player-poker/

  3. MIT 6.S191: Introduction to deep learning. https://introtodeeplearning.com/

  4. OpenAI

    Google Scholar 

  5. OpenAI five defeats dota 2 world champions. https://openai.com/blog/openai-five-defeats-dota-2-world-champions/

  6. Unity machine learning agents

    Google Scholar 

  7. Arulkumaran, K., Cully, A., Togelius, J. : Alphastar: an evolutionary computation perspective. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 314–315 (2019)

    Google Scholar 

  8. Baby, N., Goswami, B.: Implementing artificial intelligence agent within connect 4 using unity3D and machine learning concepts. Int. J. Recent Technol. Eng. 7(6S3), 193–200 (2019)

    Google Scholar 

  9. Barth-Maron G., et al.: Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617, 2018

  10. Bellemare, M. G., Dabney, W., Munos, R.: A distributional perspective on reinforcement learning. In: International Conference on Machine Learning, pp. 449–458. PMLR (2017)

    Google Scholar 

  11. Bertens, P., Guitart, A., Chen, P. P., Periáñez, Á.: A machine-learning item recommendation system for video games. In: 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1–4. IEEE (2018)

    Google Scholar 

  12. Booth J., Booth, J.: Marathon environments: multi-agent continuous control benchmarks in a modern video game engine. arXiv preprint arXiv:1902.09097 (2019)

  13. Bornemark, O.: Success factors for e-sport games. In: Umeå’s 16th Student Conference in Computing Science, pp. 1–12 (2013)

    Google Scholar 

  14. Borovikov, I., Harder, J., Sadovsky, M., Beirami, A.: Towards interactive training of non-player characters in video games. arXiv preprint arXiv:1906.00535 (2019)

  15. Borowy, M., et al.: Pioneering eSport: the experience economy and the marketing of early 1980s arcade gaming contests. Int. J. Commun. 7, 21 (2013)

    Google Scholar 

  16. Brockman, G., et al.:Openai gym. arXiv preprint arXiv:1606.01540 (2016)

  17. Cao, Z., Lin, C. -T.: Reinforcement learning from hierarchical critics. IEEE Trans. Neural Netw. Learn. Syst. (2021)

    Google Scholar 

  18. Castro, P. S., Moitra, S., Gelada, C., Kumar, S., Bellemare, M. G.: A Research framework for deep reinforcement learning, dopamine (2018)

    Google Scholar 

  19. Dabney, W., Ostrovski, G., Silver, D., Munos, R.: Implicit quantile networks for distributional reinforcement learning. In: International conference on machine learning, pages 1096–1105. PMLR (2018)

    Google Scholar 

  20. Dhariwal, P., et al.: OpenAI Baselines, Szymon Sidor (2022)

    Google Scholar 

  21. Frank, A. B.: Gaming AI without AI. J. Defense Mod. Simul., p. 15485129221074352 (2022)

    Google Scholar 

  22. Moreno, S. E. G., Montalvo, J. A. C., Palma-Ruiz, J. M.: La industria cultural y la industria de los videojuegos. JUEGOS Y SOCIEDAD: DESDE LA INTERACCIÓN A LA INMERSIÓN PARA EL CAMBIO SOCIAL, pp. 19–26 (2019)

    Google Scholar 

  23. Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning, pp. 1861–1870. PMLR (2018)

    Google Scholar 

  24. Hessel, M., et al.: Rainbow: combining improvements in deep reinforcement learning. In: Thirty-second AAAI conference on artificial intelligence (2018)

    Google Scholar 

  25. Ho, J., Ermon, S.: Generative adversarial imitation learning. Adv. Neural Info. Proc. Syst. 29 (2016)

    Google Scholar 

  26. Juliani, A., et al.: Unity: a general platform for intelligent agents. arXiv preprint arXiv:1809.02627 (2018)

  27. Lanham, M.: Learn Unity ML-Agents-Fundamentals of Unity Machine Learning: Incorporate New Powerful ML Algorithms Such as Deep Reinforcement Learning for Games. Packt Publishing Ltd., Birmingham (2018)

    Google Scholar 

  28. Li, R.: Good Luck Have Fun: The Rise of eSports. Simon and Schuster, New York (2017)

    Google Scholar 

  29. Lillicrap, T. P.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

  30. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275 (2017)

  31. Lyle, D., et al.: Chess and strategy in the age of artificial intelligence. In: Lai, D. (eds) US-China Strategic Relations and Competitive Sports, pages 87–126. Palgrave Macmillan, Cham (2022). https://doi.org/10.1007/978-3-030-92200-9_5

  32. Mekni, M.: An artificial intelligence based virtual assistant using conversational agents. J. Softw. Eng. Appl. 14(9), 455–473 (2021)

    Article  Google Scholar 

  33. Mekni, M., Jayan, A.: Automated modular invertebrate research environment using software embedded systems. In: Proceedings of the 2nd International Conference on Software Engineering and Information Management, pp. 85–90 (2019)

    Google Scholar 

  34. Mitchell, T. M., et al.: Machine learning (1997)

    Google Scholar 

  35. Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937. PMLR (2016)

    Google Scholar 

  36. Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

  37. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  38. Newzoo: Global games market report (2021)

    Google Scholar 

  39. Nichol, A., Pfau, V., Hesse, C., Klimov, O., Schulman J.: Gotta learn fast: a new benchmark for generalization in RL. arXiv preprint arXiv:1804.03720 (2018)

  40. Nowé, A., Vrancx, P., De Hauwere, Y. M.: Game theory and multi-agent reinforcement learning. In: Wiering, M., van Otterlo, M. (eds) Reinforcement Learning, pp. 441–470. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-27645-3_14

  41. O’Donoghue, B., Munos, R., Kavukcuoglu, K., Mnih, V.: Combining policy gradient and Q-learning. arXiv preprint arXiv:1611.01626 (2016)

  42. Palma-Ruiz, J. M., Torres-Toukoumidis, A., González-Moreno, S. E., Valles-Baca, H. G.: An overview of the gaming industry across nations: using analytics with power bi to forecast and identify key influencers, p. e08959. Heliyon (2022)

    Google Scholar 

  43. Ray, A., Achiam, J., Amodei, D.: Benchmarking safe exploration in deep reinforcement learning, p. 7. arXiv preprint arXiv:1910.01708 (2019)

  44. Saiz-Alvarez, J.M., Palma-Ruiz, J.M., Valles-Baca, H.G., Fierro-Ramírez, L.A.: Knowledge management in the esports industry: sustainability, continuity, and achievement of competitive results. Sustainability 13(19), 10890 (2021)

    Article  Google Scholar 

  45. Samara, F., Ondieki, S., Hossain, A. M., Mekni, M.: Online social network interactions (OSNI): a novel online reputation management solution. In: 2021 International Conference on Engineering and Emerging Technologies (ICEET), pp. 1–6. IEEE (2021)

    Google Scholar 

  46. Scholz, T. M., Scholz, T. M., Barlow: eSports is Business. Springer (2019)

    Google Scholar 

  47. Schrittwieser, J., et al.: Mastering atari, go, chess and shogi by planning with a learned model. Nature 588(7839), 604–609 (2020)

    Article  Google Scholar 

  48. Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.:. Trust region policy optimization. In: International Conference on Machine Learning, pp. 1889–1897. PMLR (2015)

    Google Scholar 

  49. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

  50. Shabbir, J., Anwer, T.: Artificial intelligence and its role in near future (2018)

    Google Scholar 

  51. Shao, K., Tang, Z., Zhu, Y., Li, N., Zhao, D.: A survey of deep reinforcement learning in video games. arXiv preprint arXiv:1912.10944 (2019)

  52. Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)

    Article  Google Scholar 

  53. Silver, D., et al.: A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362(6419), 1140–1144 (2018)

    Article  MathSciNet  Google Scholar 

  54. Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)

    Article  Google Scholar 

  55. Silver, T., Chitnis, R.:. PDDLGym: Gym environments from PDDL problems. arXiv preprint arXiv:2002.06432 (2020)

  56. Sweetser, P., Wiles, J.: Current AI in games: a review. Australian J. Intell. Info. Proc. Syst. 8(1), 24–42 (2002)

    Google Scholar 

  57. Tazouti, Y., Boulaknadel, S., Fakhri, Y.: Design and implementation of ImALeG serious game: behavior of non-playable characters (NPC). In: Saeed, F., Al-Hadhrami, T., Mohammed, E., Al-Sarem, M. (eds.) Advances on Smart and Soft Computing. AISC, vol. 1399, pp. 69–77. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-5559-3_7

    Chapter  Google Scholar 

  58. Terry, J., et al. Pettingzoo: Gym for multi-agent reinforcement learning. Adv. Neural Inf. Proc. Syst. 34 (2021)

    Google Scholar 

  59. Tucker, A., Gleave, A., Russell, S.: Inverse reinforcement learning for video games. arXiv preprint arXiv:1810.10593 (2018)

  60. Wang, Z., et al.: Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224 (2016)

  61. Wu, Y., Mansimov, E., Grosse, R. B., Liao, S., Ba, J.: Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation. Adv. Neural Inf. Proc. Syst. 30 (2017)

    Google Scholar 

  62. Yannakakis, G. N.: Game AI revisited. In: Proceedings of the 9th Conference on Computing Frontiers, pp. 285–292 (2012)

    Google Scholar 

  63. Yannakakis, G.N., Togelius, J.: A panorama of artificial and computational intelligence in games. IEEE Trans. Comput. Intell. AI in Games 7(4), 317–335 (2014)

    Article  Google Scholar 

  64. Yohanes, D.N., Rochmawati, N.: Implementasi algoritma collision detection dan a*(a star) pada non player character game world of new normal. J. Inf. Comput. Sci. (JINACS) 3(03), 322–333 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of New Haven, West Haven, CT, 06516, USA

    Charitha Sree Jayaramireddy, Sree Veera Venkata Sai Saran Naraharisetti, Mohamad Nassar & Mehdi Mekni

Authors
  1. Charitha Sree Jayaramireddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sree Veera Venkata Sai Saran Naraharisetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamad Nassar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mehdi Mekni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehdi Mekni .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jayaramireddy, C.S., Naraharisetti, S.V.V.S.S., Nassar, M., Mekni, M. (2023). A Survey of Reinforcement Learning Toolkits for Gaming: Applications, Challenges and Trends. In: Arai, K. (eds) Proceedings of the Future Technologies Conference (FTC) 2022, Volume 1. FTC 2022 2022. Lecture Notes in Networks and Systems, vol 559. Springer, Cham. https://doi.org/10.1007/978-3-031-18461-1_11

Download citation

Publish with us

Policies and ethics

Search

Navigation