Skip to main content
SpringerLink
Log in

Deep Reinforced Navigation of Agents in 2D Platform Video Games

  • Conference paper
  • First Online:
Advances in Computer Graphics (CGI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14497))

Included in the following conference series:

  • 176 Accesses

Abstract

The use of Artificial Intelligence in Computer Graphics can be applied to video games to a great extent, from human-computer interaction to character animation. The development of increasingly complex environments and, consequently, ever increasing state-spaces, brought the necessity of new AI approaches. This is why Deep Reinforcement Learning is becoming widespread also in this domain, by enabling training of agents capable of out-performing humans. This work aims to develop a methodology to train intelligent agents, allowing them to perform the task of interacting and navigating through complex 2D environments, achieving different goals. Two platform video games have been examined: one is a level-based platformer, which provides a “static” environment, while the other is an endless-type video game, in which elements change randomly every game, making the environment more “dynamic”. Different experiments have been performed, with different configuration settings; in both cases, trained agents showed good performance results, proving the effectiveness of the proposed method. In particular, in both scenarios the stable cumulative reward achieved corresponds to the highest value of all the trainings performed, and the policy and value loss obtained show really low values.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://gymnasium.farama.org/.

  2. 2.

    Components are implemented in Unity ML-Agents through pre-defined classes.

  3. 3.

    https://github.com/Zj-Lan/Unity_Platform-game.

  4. 4.

    https://github.com/BayatGames/RedRunner.

References

  1. Introducing NVIDIA DLSS 3 (2022). https://www.nvidia.com/en-us/geforce/news/dlss3-ai-powered-neural-graphics-innovations/

  2. Aouaidjia, K., Sheng, B., Li, P., Kim, J., Feng, D.D.: Efficient body motion quantification and similarity evaluation using 3-D joints skeleton coordinates. IEEE Trans. Syst. Man Cybern. Syst. 51(5), 2774–2788 (2021). https://doi.org/10.1109/TSMC.2019.2916896

    Article  Google Scholar 

  3. Arjona-Medina, J.A., Gillhofer, M., Widrich, M., Unterthiner, T., Brandstetter, J., Hochreiter, S.: Rudder: return decomposition for delayed rewards (2019)

    Google Scholar 

  4. Beattie, C., et al.: Deepmind lab (2016)

    Google Scholar 

  5. Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013). https://doi.org/10.1613/jair.3912

    Article  Google Scholar 

  6. Berner, C., et al.: DOTA 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 (2019)

  7. Brockman, G., et al.: OpenAI gym (2016)

    Google Scholar 

  8. ElDahshan, K.A., Farouk, H., Mofreh, E.: Deep reinforcement learning based video games: a review. In: 2022 2nd International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), pp. 302–309 (2022). https://doi.org/10.1109/MIUCC55081.2022.9781752

  9. Espié, E., Guionneau, C., Wymann, B., Dimitrakakis, C., Coulom, R., Sumner, A.: TORCS, the open racing car simulator (2005)

    Google Scholar 

  10. Ha, D., Schmidhuber, J.: Recurrent world models facilitate policy evolution. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianch, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc. (2018)

    Google Scholar 

  11. Hessel, M., et al.: Rainbow: combining improvements in deep reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1 (2018). https://ojs.aaai.org/index.php/AAAI/article/view/11796

  12. Johnson, M., Hofmann, K., Hutton, T., Bignell, D., Hofmann, K.: The Malmo platform for artificial intelligence experimentation. In: 25th International Joint Conference on Artificial Intelligence (IJCAI 2016). AAAI - Association for the Advancement of Artificial Intelligence (2016)

    Google Scholar 

  13. Juliani, A., et al.: Unity: a general platform for intelligent agents (2020)

    Google Scholar 

  14. Kempka, M., Wydmuch, M., Runc, G., Toczek, J., Jaśkowski, W.: ViZDoom: a doom-based AI research platform for visual reinforcement learning. In: 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1–8 (2016). https://doi.org/10.1109/CIG.2016.7860433

  15. Kurach, K., et al.: Google research football: a novel reinforcement learning environment (2020)

    Google Scholar 

  16. Liu, Y., Long, W., Shu, Z., Yi, S., Xin, S.: Voxel-based 3D shape segmentation using deep volumetric convolutional neural networks. In: Magnenat-Thalmann, N., et al. (eds.) CGI 2022. LNCS, vol. 13443, pp. 489–500. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-23473-6_38

    Chapter  Google Scholar 

  17. Mirzaei, M.S., Meshgi, K., Frigo, E., Nishida, T.: Animgan: a spatiotemporally-conditioned generative adversarial network for character animation. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 2286–2290 (2020). https://doi.org/10.1109/ICIP40778.2020.9190871

  18. Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML 2016, pp. 1928–1937. JMLR.org (2016)

    Google Scholar 

  19. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015). https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  20. Nardelli, N., Synnaeve, G., Lin, Z., Kohli, P., Torr, P.H., Usunier, N.: Value propagation networks (2019)

    Google Scholar 

  21. Pohlen, T., et al.: Observe and look further: achieving consistent performance on atari (2018)

    Google Scholar 

  22. Schrittwieser, J., et al.: Mastering atari, go, chess and shogi by planning with a learned model. Nature 588(7839), 604–609 (2020). https://doi.org/10.1038/s41586-020-03051-4

    Article  Google Scholar 

  23. Schulman, J., Chen, X., Abbeel, P.: Equivalence between policy gradients and soft Q-learning (2018)

    Google Scholar 

  24. Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: F. Bach, D. Blei (eds.) Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Lille, France, vol. 37, pp. 1889–1897. PMLR (2015). https://proceedings.mlr.press/v37/schulman15.html

  25. Schulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P.: High-dimensional continuous control using generalized advantage estimation (2018)

    Google Scholar 

  26. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017)

    Google Scholar 

  27. Shao, K., Tang, Z., Zhu, Y., Li, N., Zhao, D.: A survey of deep reinforcement learning in video games, p. 2 (2019)

    Google Scholar 

  28. Suta, A., Hlavacs, H.: Comparing traditional rendering techniques to deep learning based super-resolution in fire and smoke animations. In: Magnenat-Thalmann, N., Zhang, J., Kim, J., Papagiannakis, G., Sheng, B., Thalmann, D., Gavrilova, M. (eds.) CGI 2022. LNCS, vol. 13443, pp. 199–210. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-23473-6_16

    Chapter  Google Scholar 

  29. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)

    Google Scholar 

  30. Synnaeve, G., et al.: Torchcraft: a library for machine learning research on real-time strategy games (2016)

    Google Scholar 

  31. Vinyals, O., et al.: StarCraft II: a new challenge for reinforcement learning (2017)

    Google Scholar 

  32. Wang, J., Xiang, N., Kukreja, N., Yu, L., Liang, H.N.: LVDIF: a framework for real-time interaction with large volume data. Vis. Comput. 39(8), 3373–3386 (2023). https://doi.org/10.1007/s00371-023-02976-x

    Article  Google Scholar 

  33. Wang, S., Jiang, H., Wang, Z.: Resilient navigation among dynamic agents with hierarchical reinforcement learning. In: Magnenat-Thalmann, N., et al. (eds.) CGI 2021. LNCS, vol. 13002, pp. 504–516. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-89029-2_39

    Chapter  Google Scholar 

  34. Wen, Y., et al.: Structure-aware motion deblurring using multi-adversarial optimized CycleGAN. IEEE Trans. Image Process. 30, 6142–6155 (2021). https://doi.org/10.1109/TIP.2021.3092814

    Article  Google Scholar 

  35. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8(3–4), 229–256 (1992). https://doi.org/10.1007/BF00992696

    Article  Google Scholar 

  36. Yadav, K.S., Kirupakaran, A.M., Laskar, R.H.: End-to-end bare-hand localization system for human–computer interaction: a comprehensive analysis and viable solution. Vis. Comput. (2023). https://doi.org/10.1007/s00371-023-02837-7

Download references

Author information

Authors and Affiliations

  1. Università Politecnica delle Marche, Ancona, Italy

    Emanuele Balloni, Marco Mameli, Adriano Mancini & Primo Zingaretti

Authors
  1. Emanuele Balloni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Mameli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adriano Mancini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Primo Zingaretti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Emanuele Balloni .

Editor information

Editors and Affiliations

  1. Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  2. Shanghai Jiao Tong University, Shanghai, China

    Lei Bi

  3. University of Sydney, Sydney, NSW, Australia

    Jinman Kim

  4. MIRALab-CUI, University of Geneva, Carouge, Geneve, Switzerland

    Nadia Magnenat-Thalmann

  5. Swiss Federal Institute of Technology, Lausanne, Switzerland

    Daniel Thalmann

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Balloni, E., Mameli, M., Mancini, A., Zingaretti, P. (2024). Deep Reinforced Navigation of Agents in 2D Platform Video Games. In: Sheng, B., Bi, L., Kim, J., Magnenat-Thalmann, N., Thalmann, D. (eds) Advances in Computer Graphics. CGI 2023. Lecture Notes in Computer Science, vol 14497. Springer, Cham. https://doi.org/10.1007/978-3-031-50075-6_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50075-6_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50074-9

  • Online ISBN: 978-3-031-50075-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation