Skip to main content
SpringerLink
Log in

Intelligent Adjustment of Game Properties at Run Time Using Multi-armed Bandits

  • Research
  • Published:
The Computer Games Journal

Abstract

Dynamic modification of game properties based on the preferences of players can be an essential factor of successful game design. This paper proposes a technique based on the multi-armed bandit (MAB) approach for intelligent and dynamic theme selection in a video game. The epsilon-greedy algorithm is exploited in order to implement the MAB approach and apply players’ preferences in the game. A 3D-Roll ball game with four different themes has been developed for the purpose of evaluating the efficacy of the proposed technique. In this game, the color of the gaming environment and the speed of a player are defined as two game properties that determine game themes. The results of a user study performed on this system show that our technique has the potential of being used as a toolkit for determining the preferences of players at real-time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Agrawal, R. (1995). Sample mean-based index policies with O(log n) regret for the multi-armed bandit problem. Advances in Applied Probability,27, 1054–1078.

    Article  MathSciNet  Google Scholar 

  • Andersen, E., Liu, Y. E., Apter, E., Boucher-Genesse, F. & Popovic, Z. (2010). Gameplay analysis through state projection. In Proceedings of FDG (pp. 1–8). ACM Press.

  • Audibert, J. Y., Bubeck, S., & Munos, R. (2010). Best arm identification in multi-armed bandits. In Proceedings of the 23rd conference on learning theory.

  • Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning,47, 235–256.

    Article  Google Scholar 

  • Belluz, J., Gaudesi, M., & Tonda, A. (2015). Operator selection using improved dynamic multi-armed bandit. In GECCO’15, July 11–15, 2015, Madrid, Spain. Copyright ©ACM.

  • Bouneffouf, P. D., & Feraud, R. (2016). Multi-armed bandit problem with known trend. Neurocomputing Archive,205:C, 16–21.

    Article  Google Scholar 

  • Broden, B., Hammar, M., Nilsson, B. J., & Paraschakis, D. (2017). Bandit algorithms for e-Commerce recommender systems. In: RecSys 2017, Como, Italy.

  • Bubeck, S., & Cesa-Bianchi, N. (2012). Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning,5(1), 1–122.

    Article  Google Scholar 

  • Cesa-Bianchi, N., & Lugosi, G. (2012). Combinatorial bandits. Journal of Computer and System Sciences,78, 1404–1422.

    Article  MathSciNet  Google Scholar 

  • Chen, W., Wang, Y., Yuan, Y., & Wang, Q. (2016). Combinatorial multi-armed bandit and its extension to probabilistically triggered arms. Journal of Machine Learning Research,17, 1–33.

    MathSciNet  MATH  Google Scholar 

  • Desurvire, H., Caplan, M., & Toth, J. A. (2004). Using heuristics to evaluate the playability of games. In: Extended abstracts CHI 2004 (pp. 1509–1512). ACM Press.

  • Desurvire, H., & Wiberg, C. (2009). Game usability heuristics (play) for evaluating and designing better games: the next iteration. In Proceedings of OCSC 2009 (pp. 557–566).

  • Dixit, P. N., Youngblood, G. M. (2008). Understanding playtest data through visual data mining in interactive 3D environments. In: Proceedings of CGAMES.

  • Drachen, A., & Canossa, A. (2009a). Towards gameplay analysis via gameplay metrics. In: Proceedings of Mind Trek (pp. 202–209). ACM Press.

  • Drachen, A., & Canossa, A. (2009b). Analyzing spatial user behavior in computer games using geo-graphic information systems. In: Proceedings of MindTrek (pp. 182–189). ACM Press.

  • Dumas, J. S. (2002). User-based evaluations. In The human–computer interaction handbook (pp. 1093–1117). L. Erlbaum Associates Inc.

  • Fialho, A., Da Costa, L., Schoenauer, M., & Sebag, M. (2010). Analyzing bandit-based adaptive operator selection mechanisms. Annals of Mathematics and Artificial Intelligence,60(1–2), 25–64.

    Article  MathSciNet  Google Scholar 

  • Gajos, K., & Weld, D. S. (2005). Preference elicitation for interface optimization. In Proceedings of the 18th annual ACM symposium on user interface software and technology (pp. 173–182).

  • Garivier, A., Kaufmann, E., & Koolen, W. (2016). Maximum action identification: A new bandit framework for games. In JMLR: Workshop and Conference Proceedings (vol 49, pp. 1–23).

  • Geslin, E., Jegou, L., & Beaudoin, D. (2016). How color properties can be used to elicit emotions in video games. International Journal of Computer Games Technology. https://doi.org/10.1155/2016/5182768.

    Article  Google Scholar 

  • Gilleade, K. M., & Dix, A. (2004). Using frustration in the design of adaptive videogames (pp. 228–232). New York: ACM Press.

    Google Scholar 

  • Ijsselsteijn, W. A., de Kort, Y. A. W., & Poels, K. (2013). The game experience questionnaire. Technische Universiteit Eindhoven.

  • Kaufmann, E., & Garivier, A. (2017). Learning the distribution with largest mean: Two bandit frameworks. In ESAIM: Proceedings and surveys (pp. 1–10).

  • Kaufmann, E., & Kalyanakrishnan, S. (2013). Information complexity in bandit subset selection. In Proceedings of the 26th conference on learning theory (pp. 228–251).

  • Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Ya, X. (2012) Trustworthy online controlled experiments: Five puzzling outcomes explained. In Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 786–794).

  • Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery,18, 140–181.

    Article  MathSciNet  Google Scholar 

  • Kuleshov, V., & Precup, D. (2000). Algorithms for the multi-armed bandit problem. Journal of Machine Learning Research,1, 1–48.

    Google Scholar 

  • Kuniavsky, M. (2003). Observing the user experience: A practitioner’s guide to user research. Amsterdam: Elsevier.

    Google Scholar 

  • Tran-Thanh, L. Chapman, A., de Cote, E. M., Rogers, A., & Jennings, N. R. (2010). First policies for budget-limited multi-armed bandits. In Proceedings of the 24th AAAI conference on artificial intelligence, 11–15 July 2010, Georgia, USA (pp. 1211–1216).

  • Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World Wide Web (pp. 661–670).

  • Liu, J., Togelius, J., Perez-Liebana, D., & Lucas, S. M. (2017). Evolving game skill-depth using general video game AI agents. In: IEEE congress on evolutionary computation (CEC) San Sebastian, Spain.

  • Liu, Y., Mandel, T., Brunskill, E., & Popovic, Z. (2014). Trading off scientific knowledge and user learning with multi-armed bandits. In EDM (pp. 161–168).

  • Lomas, D., Forlizzi, J., Poonawala, N., Patel, N., Shodhan, S., Patel, K., Koedinger, K. R., & Brunskill, E. (2016). Interface design optimization as a multi-armed bandit problem. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 4142–4153).

  • Loren, R., & Benson, D. B. (Eds.). (1999). Introduction to string field theory (2nd ed.). New York: Springer.

    Google Scholar 

  • Lu, J., Li, L., Shen, D., Chen, G., Jia, B., Blasch, E., & Pham, K. (2017). Dynamic multi-arm bandit game based multi-agents spectrum sharing strategy design. Cornell University. Submitted 12 Nov 2017.

  • Nakamura, A., Helmbold, D. P., & Warmuth, M. K. (2016). Noise-free multi-armed bandit game (pp. 412–423). Switzerland: Springer.

    MATH  Google Scholar 

  • Ontanon, S. (2017). Combinatorial multi-armed bandits for real-time strategy games. Journal of Artificial Intelligence Research,58, 665–702.

    Article  MathSciNet  Google Scholar 

  • Ontanon, S., & Zhu, J. (2011). The SAM algorithm for analogy-based story generation. In Proceedings of the seventh AAAI conference on artificial intelligence and interactive digital entertainment, AIIDE 2011, 10–14 October 2011, Stanford, California, USA (pp. 67–72).

  • Pendharkar, P. C., & Cusatis, P. (2017). Trading financial indices with reinforcement learning agents. Expert Systems with Applications,103, 1–13.

    Article  Google Scholar 

  • Raharjo, K. (2002). Using confidence bounds for exploitation–exploration trade-offs. Journal of Machine Learning Research,3, 397–422.

    MathSciNet  Google Scholar 

  • Raharjo, K., & Lawrence, R. (2016). Using multi-armed bandits to optimize game play metrics and effective game design. International Journal of Computer and Information Engineering,10(10), 1758–1761.

    Google Scholar 

  • Ramirez, A. J., & Bulitko, V. (2014). Automated planning and player modeling for interactive story telling. In IEEE transactions on computer intelligence and AI in games (pp. 375–386).

  • Sekhavat, Y. A. (2017). Behavior trees for computer games. International Journal on Artificial Intelligence Tools,26, 1–28.

    Article  Google Scholar 

  • Sekhavat, Y. A., & Namani, M. S. (2018). Projection-based AR: Effective visual feedback in gait rehabilitation. IEEE Transactions on Human–Machine Systems,48(6), 626–636.

    Article  Google Scholar 

  • Sutton, R. S., & Barto, A. G. (1988). Introduction to reinforcement learning. Cambridge, MA: MIT Press.

    MATH  Google Scholar 

  • Sweetser, P., & Wyeth, P. (2005). Gameflow: A model for evaluating player enjoyment in games. Computer Entertainment,3, 1–24.

    Article  Google Scholar 

  • Thawonmas, R., Kurashige, M., & Chen, K. T. (2007). Detection of landmarks for clustering of online-game players. International Journal of Virtual Reality,6, 11–16.

    Google Scholar 

  • Trappl, R., & Petta, P. (1997). Creating personalities for synthetic actors: Towards autonomous personality agents (p. 119). Berlin: Springer.

    Book  Google Scholar 

  • Tychsen, A., & Canossa, A. (2008). Defining personas in games using metrics. In Proceedings of future play 2008 (pp. 400–433). ACM Press.

  • Vermeulen, I. E., Roth, C., Vorderer, P., & Klimmt, C. (2010). Measuring user responses to inter-active stories: Towards a standardized assessment tool. In Proceedings of the international conference on interactive digital storytelling (ICIDS) (pp. 38–43).

  • Zhao, Z., & Liu, A. L. (2017) Intelligent demand response for electricity consumers: A multi-armed bandit game approach. In Intelligent system application to power systems (ISAP).

Download references

Acknowledgements

This work has been carried out in the Cognitive Augmented Reality Lab (www.carlab.ir) at the Faculty of Multimedia, Tabriz Islamic Art University.

Author information

Authors and Affiliations

  1. Faculty of Multimedia, Tabriz Islamic Art University, Hakim Nezami Square, Azadi Boulevard, Tabriz, East Azerbaijan, 51647-36931, Iran

    Zahra Amiri & Yoones A. Sekhavat

Authors
  1. Zahra Amiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yoones A. Sekhavat
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yoones A. Sekhavat.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amiri, Z., Sekhavat, Y.A. Intelligent Adjustment of Game Properties at Run Time Using Multi-armed Bandits. Comput Game J 8, 143–156 (2019). https://doi.org/10.1007/s40869-019-00083-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40869-019-00083-3

Keywords

Search

Navigation