skip to main content
10.1145/3507623.3507629acmotherconferencesArticle/Chapter ViewAbstractPublication PagesciisConference Proceedingsconference-collections
research-article

Developing an Adaptive AI Agent using Supervised and Reinforcement Learning with Monte Carlo Tree Search in FightingICE

Published: 11 April 2022 Publication History

Abstract

Reinforcement Learning (RL) and Monte Carlo Tree Search (MCTS) are efficient algorithms for video game artificial intelligence (AI) agents, while Supervised Learning (SL) would make a video game AI agent visual-based. Combining SL and RL with MCTS has been tested in Computer Go, but this has yet to be thoroughly explored for fighting games. FightingICE, a 2D fighting game, serves as an ideal testing environment because of its complex action and observation spaces. In this paper, we use a Convolutional Neural Network (CNN) and Deep Q-Learning with MCTS (DQCN with MCTS) to create three models for FightingICE and compare their performance with an MCTS agent when playing against the same set of human testers. Our best performing model achieved a 58.57% game win-rate in 70 testing games after 7 training games. Although the model did not beat the MCTS agent's performance, it demonstrates the potential of combining SL, RL, and MCTS to develop an AI agent for fighting games.

References

[1]
Wang, J.Y. and Tseng, Y.R. 2013. Dynamic difficulty adjustment by fuzzy rules using in a neural network controlled game. Proceedings - International Conference on Natural Computation. (2013), 277–281.
[2]
Yoon, S., and Kim, K.J. 2017. Deep Q networks for visual fighting game AI. 2017 IEEE Conference on Computational Intelligence and Games, CIG 2017. (2017), 306–308.
[3]
Miyashita, S., Lian, X., Zeng, X., Matsubara, T. and Uehara, K. 2017. Developing game AI agent behaving like human by mixing reinforcement learning and supervised learning. Proceedings - 18th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2017. (2017), 489–494.
[4]
Elnaggar, A.A. and Gadallah, M. 2014. A Comparative Study of Game Tree Searching Methods. 5, 5 (2014), 68–77.
[5]
Demediuk, S., Tamassia, M., Raffe, W.L., Zambetta, F. and Li, X. 2017. Monte Carlo Tree Search Based Algorithms for Dynamic Difficulty Adjustment. (2017), 53–59.
[6]
Silver, D., Huang, A., Maddison, C. 2016. Mastering the game of Go with deep neural networks and tree search. Nature. 529, 7587 (2016), 484–489.
[7]
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F. and Sifre, L. 2017. Article Mastering the game of Go without human knowledge. Nature Publishing Group. 550, 7676 (2017), 354–359.
[8]
Quang, V. N. 2016. Building a Strong Fighting Game Player. JAIST Repository.
[9]
Asayama, K., Moriyama, K., Ken-ichi F. & Numao M. 2015. Prediction as Faster Perception in a Real-time Fighting Video Game. IEEE Conference on Computational Intelligence and Game. (2015).
[10]
Li, Y., Chang, H., Lin, Y., Wu, P., and Wang, Y.F. 2018. DEEP REINFORCEMENT LEARNING FOR PLAYING 2.5D FIGHTING GAMES. Graduate Institute of Communication Engineering, National Taiwan University Department of Computer Science & Information Engineering, National Taiwan University. (2018), 3778–3782.
[11]
Intelligent Computer Lab, Ristumeikan University. Welcome to the Fighting Game AI Competition. Fighting Game AI Competition. Retrieved from http://www.ice.ci.ritsumei.ac.jp/∼ftgaic/index.htm.
[12]
Ishihara, M., Miyazaki, T., Chu, C.Y., Harada, T. and Thawonmas, R. 2016. Applying and Improving Monte-Carlo Tree Search in a Fighting Game AI. Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology - ACE2016. (2016), 1–6.
[13]
Demediuk, S., Tamassia, M., Raffe, W.L., Zambetta, F. and Li, X. 2017. Monte Carlo Tree Search Based Algorithms for Dynamic Difficulty Adjustment. (2017), 53–59.
[14]
Ito, S., Ishihara M., Tamassia, M., Harada, T., Thawomnas, R., and Zambetta, F. 2017. Procedural Play Generation According to Play Arcs Using Monte-Carlo Tree Search. Proc. of the 18th International Conference on Intelligent Games Simulation (GAME-ON’2017), Carlow, Ireland, pp. 67-71.
[15]
OpenAI. Background: Why Gym? (2016). Retreved from https://gym.openai.com/docs/.
[16]
Parr, B., Dilipkumar, D. and Liu, Y. 2017. Nintendo Super Smash Bros. Melee: An "Untouchable" Agent. arXiv:1712.03280. Retrieved from https://arxiv.org/abs/1712.03280.
[17]
Martínez-Arellano, G., Cant, R., and Woods, D. 2017. Creating AI Characters for Fighting Games using Genetic Programming. IEEE Transactions on Computational Intelligence and AI in Games, Vol. 9, No. 4 (Dec. 2017), 423-434.
[18]
Zuin, G.L. and Macedo, Y.P.A. 2016. Attempting to discover infinite combos in fighting games using hidden markov models. Brazilian Symposium on Games and Digital Entertainment, SBGAMES. (2016), 80–88.

Cited By

View all
  • (2024)Adapting to the human: A systematic review of a decade of human factors research on adaptive autonomyApplied Ergonomics10.1016/j.apergo.2024.104336120(104336)Online publication date: Oct-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
CIIS '21: Proceedings of the 2021 4th International Conference on Computational Intelligence and Intelligent Systems
November 2021
95 pages
ISBN:9781450385930
DOI:10.1145/3507623
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 April 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Artificial Intelligence
  2. Computer Vision
  3. Machine Learning
  4. Video Game Agents

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

CIIS 2021

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)43
  • Downloads (Last 6 weeks)1
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Adapting to the human: A systematic review of a decade of human factors research on adaptive autonomyApplied Ergonomics10.1016/j.apergo.2024.104336120(104336)Online publication date: Oct-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media