skip to main content
10.1145/3564719.3568687acmconferencesArticle/Chapter ViewAbstractPublication PagesgpceConference Proceedingsconference-collections
research-article

A Model-Driven Generative Self Play-Based Toolchain for Developing Games and Players

Published: 01 December 2022 Publication History

Abstract

Turn-based games such as chess are very popular, but tool-chains tailored for their development process are still rare. In this paper we present a model-driven and generative toolchain aiming to cover the whole development process of rule-based games. In particular, we present a game description language enabling the developer to model the game in a logics-based syntax. An executable game interpreter is generated from the game model and can then act as an environment for reinforcement learning-based self-play training of players. Before the training, the deep neural network can be modeled manually by a deep learning developer or generated using a heuristics estimating the complexity of mapping the state space to the action space. Finally, we present a case study modeling three games and evaluate the language features as well as the player training capabilities of the toolchain.

References

[1]
Accel Brain Co., Ltd. 2017. Reinforcement Learning Library: pyqlearning. https://github.com/accel-brain/accel-brain-code/tree/master/Reinforcement-Learning Accessed: 2022-11-08
[2]
Abdallah Atouani, Jörg Christian Kirchhof, Evgeny Kusmenko, and Bernhard Rumpe. 2021. Artifact and Reference Models for Generative Machine Learning Frameworks and Build Systems. In Proceedings of the 20th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE 21), Eli Tilevich and Coen De Roover (Eds.). ACM SIGPLAN, 55–68. http://www.se-rwth.de/publications/Artifact-and-Reference-Models-for-Generative-Machine-Learning-Frameworks-and-Build-Systems.pdf
[3]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540.
[4]
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. 2018. Dopamine: A Research Framework for Deep Reinforcement Learning. arxiv:1812.06110
[5]
Claudio Ciancio, Giuseppina Ambrogio, Francesco Gagliardi, and Roberto Musmanno. 2016. Heuristic techniques to optimize neural network architecture in manufacturing applications. Neural Computing and Applications, 27, 7 (2016), 2001–2015.
[6]
Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. 2017. AdaNet: Adaptive structural learning of artificial neural networks. In International conference on machine learning. 874–883.
[7]
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, and Mahmoud Adada. 2018. Textworld: A learning environment for text-based games. In Workshop on Computer Games. 41–75.
[8]
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. 2017. OpenAI Baselines. https://github.com/openai/baselines
[9]
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning. 1–16.
[10]
Nicola Gatto, Evgeny Kusmenko, and Bernhard Rumpe. 2019. Modeling Deep Reinforcement Learning Based Architectures for Cyber-Physical Systems. In Proceedings of MODELS 2019. Workshop MDE Intelligence. 196–202. http://www.se-rwth.de/publications/Modeling-Deep-Reinforcement-Learning-based-Architectures-for-Cyber-Physical-Systems.pdf
[11]
Michael Genesereth, Nathaniel Love, and Barney Pell. 2005. General game playing: Overview of the AAAI competition. AI magazine, 26, 2 (2005), 62–62.
[12]
Adrian Goldwaser and Michael Thielscher. 2020. Deep reinforcement learning for general game playing. In Proceedings of the AAAI conference on artificial intelligence. 34, 1701–1708.
[13]
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Efi Kokiopoulou, Luciano Sbaiz, Jamie Smith, Gábor Bartók, Jesse Berent, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. 2018. TF-Agents: A library for Reinforcement Learning in TensorFlow. https://github.com/tensorflow/agents [Online; accessed 25-June-2019]
[14]
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. 2018. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence.
[15]
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. 2018. Stable Baselines. https://github.com/hill-a/stable-baselines
[16]
Katrin Hölldobler, Oliver Kautz, and Bernhard Rumpe. 2021. MontiCore Language Workbench and Library Handbook: Edition 2021. Shaker Verlag. isbn:978-3-8440-8010-0 http://www.monticore.de/handbook.pdf
[17]
Eddie Huang. 2019. GymGo. https://github.com/aigagror/GymGo Accessed: 2022-11-08
[18]
Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. 2016. ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning. In IEEE Conference on Computational Intelligence and Games. IEEE, Santorini, Greece. 341–348. arxiv:1605.02097 The best paper award
[19]
Jörg Christian Kirchhof, Evgeny Kusmenko, Jonas Ritz, Bernhard Rumpe, Armin Moin, Atta Badii, Stephan Günnemann, and Moharram Challenger. 2022. MDE for Machine Learning-Enabled Software Systems: A Case Study and Comparison of MontiAnna & ML-Quadrat. arXiv preprint arXiv:2209.07282.
[20]
Alexander Kuhnle, Michael Schaarschmidt, and Kai Fricke. 2017. Tensorforce: a TensorFlow library for applied reinforcement learning. Web page. https://github.com/tensorforce/tensorforce
[21]
Evgeny Kusmenko. 2021. Model-Driven Development Methodology and Domain-Specific Languages for the Design of Artificial Intelligence in Cyber-Physical Systems. Shaker Verlag. isbn:978-3-8440-8286-9
[22]
Evgeny Kusmenko, Sebastian Nickels, Svetlana Pavlitskaya, Bernhard Rumpe, and Thomas Timmermanns. 2019. Conference on Model Driven Engineering Languages and Systems (MODELS’19). IEEE, 283–293. http://www.se-rwth.de/publications/Modeling-and-Training-of-Neural-Processing-Systems.pdf
[23]
Evgeny Kusmenko, Svetlana Pavlitskaya, Bernhard Rumpe, and Sebastian Stüber. 2019. On the Engineering of AI-Powered Systems. In ASE19. Software Engineering Intelligence Workshop (SEI19), Lisa O’Conner (Ed.). IEEE, 126–133.
[24]
Evgeny Kusmenko, Jean-Marc Ronck, Bernhard Rumpe, and Michael von Wenckstern. 2018. EmbeddedMontiArc: Textual Modeling Alternative to Simulink. In Proceedings of MODELS 2018. Workshop EXE. http://www.se-rwth.de/publications/EmbeddedMontiArc-Textual-Modeling-Alternative-to-Simulink.pdf
[25]
Evgeny Kusmenko, Alexander Roth, Bernhard Rumpe, and Michael von Wenckstern. 2017. Modeling Architectures of Cyber-Physical Systems. In European Conference on Modelling Foundations and Applications (ECMFA’17) (LNCS 10376). Springer, 34–50. http://www.se-rwth.de/publications/Modeling-Architectures-of-Cyber-Physical-Systems.pdf
[26]
Evgeny Kusmenko, Bernhard Rumpe, Sascha Schneiders, and Michael von Wenckstern. 2018. Highly-Optimizing and Multi-Target Compiler for Embedded System Models: C++ Compiler Toolchain for the Component and Connector Language EmbeddedMontiArc. In Conference on Model Driven Engineering Languages and Systems (MODELS’18). ACM, 447 – 457. http://www.se-rwth.de/publications/Highly-Optimizing-and-Multi-Target-Compiler-for-Embedded-System-Models.pdf
[27]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.
[28]
DeepMind Technologies Limited. 2018. Tensorflow Reinforcement Learning: TRFL. https://github.com/deepmind/trfl Accessed: 2022-11-08
[29]
MathWorks. 2022. Reinforcement Learning Toolbox. https://www.mathworks.com/products/reinforcement-learning.html Accessed: 2022-11-08
[30]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
[31]
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. 2018. Gotta Learn Fast: A New Benchmark for Generalization in RL. arXiv preprint arXiv:1804.03720.
[32]
Matthias Plappert. 2016. keras-rl. https://github.com/keras-rl/keras-rl
[33]
Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y Ng. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software. 3, 5.
[34]
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, and Thore Graepel. 2020. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588, 7839 (2020), 604–609.
[35]
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, and Thore Graepel. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362, 6419 (2018), 1140–1144.
[36]
Norman Tasfi. 2016. PyGame Learning Environment. https://github.com/ntasfi/PyGame-Learning-Environment
[37]
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. 2020. dmcontrol: Software and Tasks for Continuous Control. arxiv:2006.12983.
[38]
Michael Thielscher. 2011. GDL-II. KI-Künstliche Intelligenz, 25, 1 (2011), 63–66.
[39]
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, and Julian Schrittwieser. 2017. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782.
[40]
Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. 2019. Rlcard: A toolkit for reinforcement learning in card games. arXiv preprint arXiv:1910.04376.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GPCE 2022: Proceedings of the 21st ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences
November 2022
186 pages
ISBN:9781450399203
DOI:10.1145/3564719
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. code generation
  2. games
  3. reinforcement learning

Qualifiers

  • Research-article

Conference

GPCE '22
Sponsor:

Acceptance Rates

Overall Acceptance Rate 56 of 180 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 79
    Total Downloads
  • Downloads (Last 12 months)17
  • Downloads (Last 6 weeks)3
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media