Abstract
Reinforcement learning is a field of Machine Learning in which agents learn from interacting with the environment. These agents can deal with more complex problems when their decision-making process is combined with deep learning. While deep reinforcement learning can be used in many real-world applications, games often provide a good source of simulation environments for testing such algorithms. Among all game categories, real-time strategy games usually pose a difficult challenge since they have large state and action spaces, partial observation maps, sparse reward, and Multi-Agent problems, where the events occur continuously simultaneously. Thus, this paper provides a systematic literature review of deep reinforcement learning related to real-time strategy games. The main goals of this review are presented as follows: (a) identify the games used in recent works; (b) summarize the architectures and techniques used; (c) identify the simulation environments adopted and (d) understand whether the works focus on micromanagement or macromanagement tasks when dealing with real-time strategy games. The results show that some architectures have achieved better performance overall when handling both micro and macromanagement tasks, and that techniques for reducing the training time and the state space may improve the agents learning. This paper may help to guide future research on developing strategies to build agents for complex scenarios such as those faced in real-time strategy games.
Graphical abstract
Visual summary of the Systematic Literature Review methodology and results. It presents the objective of the review, the research questions, the protocol parameters and criteria, and the results
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Availability of data and materials
Not applicable
Code availability
Not applicable
References
Sutton R, Barto A (2018) Reinforcement Learning: An Introduction (2nd Edition) (MIT Press)
Li Y (2018) Deep reinforcement learning. arXiv. arxiv:1810.06339
Shao K, Tang Z, Zhu Y, Li N, Zhao D (2019) A survey of deep reinforcement learning in video games. arxiv:1912.10944
Szita I (2012) Reinforcement Learning in Games, 539–577 (Springer Berlin Heidelberg, Berlin, Heidelberg). https://doi.org/10.1007/978-3-642-27645-3_17
Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444. https://www.nature.com/articles/nature14539
Mnih V et al (2013) Playing atari with deep reinforcement learning. NIPS Deep Learning Workshop 2013
Mnih V, Kavukcuoglu K, Silver D et al (2015) Human-level control through deep reinforcement learning. Nature 518:529–533. www.nature.com/articles/nature14236
van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence 30. https://ojs.aaai.org/index.php/AAAI/article/view/10295
Silver D et al (2016) Mastering the game of go with deep neural networks and tree search. Proceedings of the AAAI Conference on Artificial Intelligence 529:484–489. https://www.nature.com/articles/nature16961
Silver D et al (2017) Mastering the game of go without human knowledge. Nature. https://www.nature.com/articles/nature24270
Vinyals O et al (2017) Starcraft ii: A new challenge for reinforcement learning. arxiv:1708.04782
Vinyals O et al (2019) Grandmaster level in starcraft ii using multi-agent reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence 575:350–354. https://www.nature.com/articles/s41586-019-1724-z
Ye D et al (2020) Towards playing full moba games with deep reinforcement learning. In: Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H (eds) Advances in Neural Information Processing Systems, vol 33, 621–632 (Curran Associates, Inc.). https://proceedings.neurips.cc/paper_files/paper/2020/file/06d5ae105ea1bea4d800bc96491876e9-Paper.pdf
Zha, D. et al. Meila, M. & Zhang, T. (eds) Douzero: Mastering doudizhu with self-play deep reinforcement learning. (eds Meila, M. & Zhang, T.) Proceedings of the 38th International Conference on Machine Learning, Vol. 139 of Proceedings of Machine Learning Research, 12333–12344 (PMLR, 2021). https://proceedings.mlr.press/v139/zha21a.html
Perolat J et al (2022) Mastering the game of stratego with model-free multiagent reinforcement learning. Science 378:990–996. https://www.science.org/doi/abs/10.1126/science.add4679
Wurman PR et al (2022) Outracing champion gran turismo drivers with deep reinforcement learning. Nature 602:223–228. https://www.nature.com/articles/s41586-021-04357-7
Sethy H, Patel A, Padmanabhan V (2015) Real time strategy games: A reinforcement learning approach. Procedia Computer Science 54:257–264. https://www.sciencedirect.com/science/article/pii/S187705091501354X
Robertson G, Watson I (2014) A review of real-time strategy game ai. AI Magazine 35:75–104. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2478
Ontañón S et al (2015) RTS AI Problems and Techniques, 1–12 (Springer International Publishing, Cham). https://link.springer.com/referenceworkentry/10.1007/978-3-319-08234-9_17-1
Ontañón S et al (2013) A survey of real-time strategy game ai research and competition in starcraft. IEEE Transactions on Computational Intelligence and AI in Games 5:293–311
Churchill DG (2016) Heuristic Search Techniques for Real-Time Strategy Games. Ph.D. thesis, University of Alberta
Ashraf NM et al (2021) A State-of-the-Art Review of Deep Reinforcement Learning Techniques for Real-Time Strategy Games, 285–307 (Springer International Publishing, Cham). https://link.springer.com/chapter/10.1007/978-3-030-72080-3_17
Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report
Huang S, Ontañón S, Bamford C, Grela L (2021) Gym-µrts: Toward affordable full game real-time strategy games research with deep reinforcement learning, 1–8 (IEEE Press). https://ieeexplore.ieee.org/document/9619076
Andersen P-A, Goodwin M, Granmo O-C (2018) Deep rts: A game environment for deep reinforcement learning in real-time strategy games, 1–8
Araújo MAS, Alves LPC, Madeira CAG, Nóbrega MM (2020) Urnai: A multi-game toolkit for experimenting deep reinforcement learning algorithms, 178–187
Ramadhan F, Suyanto S (2020) Royale heroes: A unique rts game using deep reinforcement learning-based autonomous movement, 494–498
Han L et al (2019) Chaudhuri K, Salakhutdinov R (eds) Grid-wise control for multi-agent reinforcement learning in video game AI. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th International Conference on Machine Learning, vol 97 of Proceedings of Machine Learning Research, 2576–2585 (PMLR). https://proceedings.mlr.press/v97/han19a.html
Kanervisto A, Scheller C, Hautamäki V (2020) Action space shaping in deep reinforcement learning 2004:00980
Ng AY, Harada D, Russell SJ (1999) Policy invariance under reward transformations: Theory and application to reward shaping, ICML ’99, 278–287. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA
Huang S, Ontañón S (2022) A closer look at invalid action masking in policy gradient algorithms 35. https://journals.flvc.org/FLAIRS/article/view/130584
Hao D, Sweetser P, Aitchison M (2020) Designing curriculum for deep reinforcement learning in starcraft ii. In: Gallagher M, Moustafa N, Lakshika E (eds) AI 2020: Advances in Artificial Intelligence, 243–255 (Springer International Publishing, Cham)
Waytowich N, Barton SL, Lawhern V, Stump E, Warnell G (2019) Grounding natural language commands to StarCraft II game states for narration-guided reinforcement learning. In: Pham T (ed) Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, vol. 11006, 110060S. International Society for Optics and Photonics (SPIE). https://doi.org/10.1117/12.2519138
Zhang F, Yang Q, An D (2022) A leader-following paradigm based deep reinforcement learning method for multi-agent cooperation games. Neural Networks 156:1–12. https://www.sciencedirect.com/science/article/pii/S089360802200346X
yang Zhao L et al (2022) Targeted multi-agent communication algorithm based on state control. Defence Technology. https://www.sciencedirect.com/science/article/pii/S2214914722001490
Li Y, Fang Y, Akhtar Z (2020) Accelerating deep reinforcement learning model for game strategy. Neurocomputing 408:157–168. https://www.sciencedirect.com/science/article/pii/S0925231220303337
Zhang J, Chen J, Huang Y, Wan W, Li T (2018) Applying online expert supervision in deep actor-critic reinforcement learning. In: Lai J-H et al (eds) Pattern Recognition and Computer Vision, 469–478 (Springer International Publishing, Cham)
Wang H et al (2020) Large scale deep reinforcement learning in war-games, 1693–1699
Li C, Wei X, Zhao Y, Geng X (2020) An effective maximum entropy exploration approach for deceptive game in reinforcement learning. Neurocomputing 403:98–108. https://www.sciencedirect.com/science/article/pii/S0925231220306536
Hu C (2020) A confrontation decision-making method with deep reinforcement learning and knowledge transfer for multi-agent system. Symmetry 12. https://www.mdpi.com/2073-8994/12/4/631
Kelly R, Churchill D (2020) Transfer learning between rts combat scenarios using component-action deep reinforcement learning. https://ceur-ws.org/Vol-2862/
Lee D et al (2018) Modular architecture for starcraft ii with deep reinforcement learning, AIIDE’18 (AAAI Press)
Chen L, LIU T, Liu Y-t (2020) Research on the starcraft ii decision method based on hierarchical reinforcement learning 582–586
Xu, S. et al. Macro action selection with deep reinforcement learning in starcraft. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment15, 94–99 (2019). https://ojs.aaai.org/index.php/AIIDE/article/view/5230
Liu T, Zheng Z, Li H, Bian K, Song L (2019) Playing card-based rts games with deep reinforcement learning, pp 4540–4546 (Int Joint Conf Artif Intell Org). https://www.ijcai.org/proceedings/2019/631
Hu H, Wang Q (2020) Implementation on benchmark of sc2le environment with advantage actor – critic method, pp 362–366
Hao D, Sweetser P, Aitchison M (2022) Curriculum generation and sequencing for deep reinforcement learning in starcraft ii, ACSW ’22:1–11 (Association for Computing Machinery, New York, NY, USA). https://dl.acm.org/doi/10.1145/3511616.3513093
Harris A, Liu S (2021) Maidrl: Semi-centralized multi-agent reinforcement learning using agent influence, pp 01–08
Nipu AS, Liu S, Harris A (2022) Maidcrl: Semi-centralized multi-agent influence dense-cnn reinforcement learning, pp 512–515
Sun Y, Yuan B, Zhang Y et al (2021) Research on action strategies and simulations of drl and mcts-based intelligent round game. Int J Control Autom Syst 19:2984–2998. https://link.springer.com/article/10.1007/s12555-020-0277-0
Sun Y et al (2023) Intelligent decision-making and human language communication based on deep reinforcement learning in a wargame environment. IEEE Transactions on human-machine systems 53:201–214
Andersen P-A, Goodwin M, Granmo O-C (2021) Increasing sample efficiency in deep reinforcement learning using generative environment modelling. Exp Syst 38. https://onlinelibrary.wiley.com/doi/abs/10.1111/exsy.12537
Fu Y, Liang X, Ma Y, Huang K, Li Y (2021) Coordinating multi-agent deep reinforcement learning in wargame, ACAI ’20 (Association for Computing Machinery, New York, NY, USA). https://dl.acm.org/doi/10.1145/3446132.3446137
Boron J, Darken C (2020) Developing combat behavior through reinforcement learning in wargames and simulations, pp 728–731
Huang W, Yin Q, Zhang J, Huang K (2021) Learning macromanagement in starcraft by deep reinforcement learning. Sensors 21. https://www.mdpi.com/1424-8220/21/10/3332
Samvelyan M et al (2019) The starcraft multi-agent challenge 1902:04043
Rashid T et al (2018) Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. Proc Mach Learn Res. https://proceedings.mlr.press/v80/rashid18a.html
Yun WJ, Yi S, Kim J (2021) Multi-agent deep reinforcement learning using attentive graph neural architectures for real-time strategy games pp 2967–2972
Barriga NA, Stanescu M, Besoain F, Buro M (2019) Improving rts game ai by supervised policy learning, tactical search, and deep reinforcement learning. IEEE Comput Intell Mag 14:8–18
Zhou Y et al (2020) Towards a distributed framework for multi-agent reinforcement learning research pp 1–9
Shen X, Yin C, Hou X (2019) Self-attention for deep reinforcement learning, ICMAI ’19, 71–75 (Association for Computing Machinery, New York, NY, USA). https://dl.acm.org/doi/10.1145/3325730.3325743
Funding
Not applicable
Author information
Authors and Affiliations
Contributions
\(\bullet \) Gabriel Caldas Barros e Sá: Literature search; data analysis; writing - original draft; writing - review and editing.\(\bullet \) Charles Andrye Galvão Madeira: Conceptualization; supervision; review.
Corresponding author
Ethics declarations
Competing Interests
The authors declare no conflict of interest.
Ethical approval
This is a review paper and we do not use any data.
Consent to participate
Not applicable
Consent for publication
Not applicable
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A Studies categorization
Appendix B Scenarios and architectures
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Barros e Sá, G.C., Madeira, C.A.G. Deep reinforcement learning in real-time strategy games: a systematic literature review. Appl Intell 55, 243 (2025). https://doi.org/10.1007/s10489-024-06220-4
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-06220-4