Skip to main content

Advertisement

Deep reinforcement learning in real-time strategy games: a systematic literature review

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Reinforcement learning is a field of Machine Learning in which agents learn from interacting with the environment. These agents can deal with more complex problems when their decision-making process is combined with deep learning. While deep reinforcement learning can be used in many real-world applications, games often provide a good source of simulation environments for testing such algorithms. Among all game categories, real-time strategy games usually pose a difficult challenge since they have large state and action spaces, partial observation maps, sparse reward, and Multi-Agent problems, where the events occur continuously simultaneously. Thus, this paper provides a systematic literature review of deep reinforcement learning related to real-time strategy games. The main goals of this review are presented as follows: (a) identify the games used in recent works; (b) summarize the architectures and techniques used; (c) identify the simulation environments adopted and (d) understand whether the works focus on micromanagement or macromanagement tasks when dealing with real-time strategy games. The results show that some architectures have achieved better performance overall when handling both micro and macromanagement tasks, and that techniques for reducing the training time and the state space may improve the agents learning. This paper may help to guide future research on developing strategies to build agents for complex scenarios such as those faced in real-time strategy games.

Graphical abstract

Visual summary of the Systematic Literature Review methodology and results. It presents the objective of the review, the research questions, the protocol parameters and criteria, and the results

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

Not applicable

Code availability

Not applicable

Notes

  1. www.deepmind.com

  2. www.blizzard.com

  3. https://github.com/deepmind/PySC2

  4. https://github.com/oxwhirl/smac

References

  1. Sutton R, Barto A (2018) Reinforcement Learning: An Introduction (2nd Edition) (MIT Press)

  2. Li Y (2018) Deep reinforcement learning. arXiv. arxiv:1810.06339

  3. Shao K, Tang Z, Zhu Y, Li N, Zhao D (2019) A survey of deep reinforcement learning in video games. arxiv:1912.10944

  4. Szita I (2012) Reinforcement Learning in Games, 539–577 (Springer Berlin Heidelberg, Berlin, Heidelberg). https://doi.org/10.1007/978-3-642-27645-3_17

  5. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444. https://www.nature.com/articles/nature14539

  6. Mnih V et al (2013) Playing atari with deep reinforcement learning. NIPS Deep Learning Workshop 2013

  7. Mnih V, Kavukcuoglu K, Silver D et al (2015) Human-level control through deep reinforcement learning. Nature 518:529–533. www.nature.com/articles/nature14236

  8. van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence 30. https://ojs.aaai.org/index.php/AAAI/article/view/10295

  9. Silver D et al (2016) Mastering the game of go with deep neural networks and tree search. Proceedings of the AAAI Conference on Artificial Intelligence 529:484–489. https://www.nature.com/articles/nature16961

  10. Silver D et al (2017) Mastering the game of go without human knowledge. Nature. https://www.nature.com/articles/nature24270

  11. Vinyals O et al (2017) Starcraft ii: A new challenge for reinforcement learning. arxiv:1708.04782

  12. Vinyals O et al (2019) Grandmaster level in starcraft ii using multi-agent reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence 575:350–354. https://www.nature.com/articles/s41586-019-1724-z

  13. Ye D et al (2020) Towards playing full moba games with deep reinforcement learning. In: Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H (eds) Advances in Neural Information Processing Systems, vol 33, 621–632 (Curran Associates, Inc.). https://proceedings.neurips.cc/paper_files/paper/2020/file/06d5ae105ea1bea4d800bc96491876e9-Paper.pdf

  14. Zha, D. et al. Meila, M. & Zhang, T. (eds) Douzero: Mastering doudizhu with self-play deep reinforcement learning. (eds Meila, M. & Zhang, T.) Proceedings of the 38th International Conference on Machine Learning, Vol. 139 of Proceedings of Machine Learning Research, 12333–12344 (PMLR, 2021). https://proceedings.mlr.press/v139/zha21a.html

  15. Perolat J et al (2022) Mastering the game of stratego with model-free multiagent reinforcement learning. Science 378:990–996. https://www.science.org/doi/abs/10.1126/science.add4679

  16. Wurman PR et al (2022) Outracing champion gran turismo drivers with deep reinforcement learning. Nature 602:223–228. https://www.nature.com/articles/s41586-021-04357-7

  17. Sethy H, Patel A, Padmanabhan V (2015) Real time strategy games: A reinforcement learning approach. Procedia Computer Science 54:257–264. https://www.sciencedirect.com/science/article/pii/S187705091501354X

  18. Robertson G, Watson I (2014) A review of real-time strategy game ai. AI Magazine 35:75–104. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2478

  19. Ontañón S et al (2015) RTS AI Problems and Techniques, 1–12 (Springer International Publishing, Cham). https://link.springer.com/referenceworkentry/10.1007/978-3-319-08234-9_17-1

  20. Ontañón S et al (2013) A survey of real-time strategy game ai research and competition in starcraft. IEEE Transactions on Computational Intelligence and AI in Games 5:293–311

    Article  MATH  Google Scholar 

  21. Churchill DG (2016) Heuristic Search Techniques for Real-Time Strategy Games. Ph.D. thesis, University of Alberta

  22. Ashraf NM et al (2021) A State-of-the-Art Review of Deep Reinforcement Learning Techniques for Real-Time Strategy Games, 285–307 (Springer International Publishing, Cham). https://link.springer.com/chapter/10.1007/978-3-030-72080-3_17

  23. Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report

  24. Huang S, Ontañón S, Bamford C, Grela L (2021) Gym-µrts: Toward affordable full game real-time strategy games research with deep reinforcement learning, 1–8 (IEEE Press). https://ieeexplore.ieee.org/document/9619076

  25. Andersen P-A, Goodwin M, Granmo O-C (2018) Deep rts: A game environment for deep reinforcement learning in real-time strategy games, 1–8

  26. Araújo MAS, Alves LPC, Madeira CAG, Nóbrega MM (2020) Urnai: A multi-game toolkit for experimenting deep reinforcement learning algorithms, 178–187

  27. Ramadhan F, Suyanto S (2020) Royale heroes: A unique rts game using deep reinforcement learning-based autonomous movement, 494–498

  28. Han L et al (2019) Chaudhuri K, Salakhutdinov R (eds) Grid-wise control for multi-agent reinforcement learning in video game AI. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th International Conference on Machine Learning, vol 97 of Proceedings of Machine Learning Research, 2576–2585 (PMLR). https://proceedings.mlr.press/v97/han19a.html

  29. Kanervisto A, Scheller C, Hautamäki V (2020) Action space shaping in deep reinforcement learning 2004:00980

    Google Scholar 

  30. Ng AY, Harada D, Russell SJ (1999) Policy invariance under reward transformations: Theory and application to reward shaping, ICML ’99, 278–287. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA

    Google Scholar 

  31. Huang S, Ontañón S (2022) A closer look at invalid action masking in policy gradient algorithms 35. https://journals.flvc.org/FLAIRS/article/view/130584

  32. Hao D, Sweetser P, Aitchison M (2020) Designing curriculum for deep reinforcement learning in starcraft ii. In: Gallagher M, Moustafa N, Lakshika E (eds) AI 2020: Advances in Artificial Intelligence, 243–255 (Springer International Publishing, Cham)

  33. Waytowich N, Barton SL, Lawhern V, Stump E, Warnell G (2019) Grounding natural language commands to StarCraft II game states for narration-guided reinforcement learning. In: Pham T (ed) Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, vol. 11006, 110060S. International Society for Optics and Photonics (SPIE). https://doi.org/10.1117/12.2519138

  34. Zhang F, Yang Q, An D (2022) A leader-following paradigm based deep reinforcement learning method for multi-agent cooperation games. Neural Networks 156:1–12. https://www.sciencedirect.com/science/article/pii/S089360802200346X

  35. yang Zhao L et al (2022) Targeted multi-agent communication algorithm based on state control. Defence Technology. https://www.sciencedirect.com/science/article/pii/S2214914722001490

  36. Li Y, Fang Y, Akhtar Z (2020) Accelerating deep reinforcement learning model for game strategy. Neurocomputing 408:157–168. https://www.sciencedirect.com/science/article/pii/S0925231220303337

  37. Zhang J, Chen J, Huang Y, Wan W, Li T (2018) Applying online expert supervision in deep actor-critic reinforcement learning. In: Lai J-H et al (eds) Pattern Recognition and Computer Vision, 469–478 (Springer International Publishing, Cham)

  38. Wang H et al (2020) Large scale deep reinforcement learning in war-games, 1693–1699

  39. Li C, Wei X, Zhao Y, Geng X (2020) An effective maximum entropy exploration approach for deceptive game in reinforcement learning. Neurocomputing 403:98–108. https://www.sciencedirect.com/science/article/pii/S0925231220306536

  40. Hu C (2020) A confrontation decision-making method with deep reinforcement learning and knowledge transfer for multi-agent system. Symmetry 12. https://www.mdpi.com/2073-8994/12/4/631

  41. Kelly R, Churchill D (2020) Transfer learning between rts combat scenarios using component-action deep reinforcement learning. https://ceur-ws.org/Vol-2862/

  42. Lee D et al (2018) Modular architecture for starcraft ii with deep reinforcement learning, AIIDE’18 (AAAI Press)

  43. Chen L, LIU T, Liu Y-t (2020) Research on the starcraft ii decision method based on hierarchical reinforcement learning 582–586

  44. Xu, S. et al. Macro action selection with deep reinforcement learning in starcraft. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment15, 94–99 (2019). https://ojs.aaai.org/index.php/AIIDE/article/view/5230

  45. Liu T, Zheng Z, Li H, Bian K, Song L (2019) Playing card-based rts games with deep reinforcement learning, pp 4540–4546 (Int Joint Conf Artif Intell Org). https://www.ijcai.org/proceedings/2019/631

  46. Hu H, Wang Q (2020) Implementation on benchmark of sc2le environment with advantage actor – critic method, pp 362–366

  47. Hao D, Sweetser P, Aitchison M (2022) Curriculum generation and sequencing for deep reinforcement learning in starcraft ii, ACSW ’22:1–11 (Association for Computing Machinery, New York, NY, USA). https://dl.acm.org/doi/10.1145/3511616.3513093

  48. Harris A, Liu S (2021) Maidrl: Semi-centralized multi-agent reinforcement learning using agent influence, pp 01–08

  49. Nipu AS, Liu S, Harris A (2022) Maidcrl: Semi-centralized multi-agent influence dense-cnn reinforcement learning, pp 512–515

  50. Sun Y, Yuan B, Zhang Y et al (2021) Research on action strategies and simulations of drl and mcts-based intelligent round game. Int J Control Autom Syst 19:2984–2998. https://link.springer.com/article/10.1007/s12555-020-0277-0

  51. Sun Y et al (2023) Intelligent decision-making and human language communication based on deep reinforcement learning in a wargame environment. IEEE Transactions on human-machine systems 53:201–214

    Article  MATH  Google Scholar 

  52. Andersen P-A, Goodwin M, Granmo O-C (2021) Increasing sample efficiency in deep reinforcement learning using generative environment modelling. Exp Syst 38. https://onlinelibrary.wiley.com/doi/abs/10.1111/exsy.12537

  53. Fu Y, Liang X, Ma Y, Huang K, Li Y (2021) Coordinating multi-agent deep reinforcement learning in wargame, ACAI ’20 (Association for Computing Machinery, New York, NY, USA). https://dl.acm.org/doi/10.1145/3446132.3446137

  54. Boron J, Darken C (2020) Developing combat behavior through reinforcement learning in wargames and simulations, pp 728–731

  55. Huang W, Yin Q, Zhang J, Huang K (2021) Learning macromanagement in starcraft by deep reinforcement learning. Sensors 21. https://www.mdpi.com/1424-8220/21/10/3332

  56. Samvelyan M et al (2019) The starcraft multi-agent challenge 1902:04043

    Google Scholar 

  57. Rashid T et al (2018) Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. Proc Mach Learn Res. https://proceedings.mlr.press/v80/rashid18a.html

  58. Yun WJ, Yi S, Kim J (2021) Multi-agent deep reinforcement learning using attentive graph neural architectures for real-time strategy games pp 2967–2972

  59. Barriga NA, Stanescu M, Besoain F, Buro M (2019) Improving rts game ai by supervised policy learning, tactical search, and deep reinforcement learning. IEEE Comput Intell Mag 14:8–18

    Article  MATH  Google Scholar 

  60. Zhou Y et al (2020) Towards a distributed framework for multi-agent reinforcement learning research pp 1–9

  61. Shen X, Yin C, Hou X (2019) Self-attention for deep reinforcement learning, ICMAI ’19, 71–75 (Association for Computing Machinery, New York, NY, USA). https://dl.acm.org/doi/10.1145/3325730.3325743

Download references

Funding

Not applicable

Author information

Authors and Affiliations

Authors

Contributions

\(\bullet \) Gabriel Caldas Barros e Sá: Literature search; data analysis; writing - original draft; writing - review and editing.\(\bullet \) Charles Andrye Galvão Madeira: Conceptualization; supervision; review.

Corresponding author

Correspondence to Gabriel Caldas Barros e Sá.

Ethics declarations

Competing Interests

The authors declare no conflict of interest.

Ethical approval

This is a review paper and we do not use any data.

Consent to participate

Not applicable

Consent for publication

Not applicable

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A  Studies categorization

Table 5 Reference of the studies and their category according to the clusters defined on this SLR

Appendix B  Scenarios and architectures

Table 6 Architectures implemented and/or analyzed by the selected studies and the scenarios in which they were applied at

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barros e Sá, G.C., Madeira, C.A.G. Deep reinforcement learning in real-time strategy games: a systematic literature review. Appl Intell 55, 243 (2025). https://doi.org/10.1007/s10489-024-06220-4

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-06220-4

Keywords