Diversifying dynamic difficulty adjustment agent by integrating player state models into Monte-Carlo tree search

https://doi.org/10.1016/j.eswa.2022.117677Get rights and content

Highlights

  • Adapting real-time video game difficulty with player’s affective states-based MCTS.

  • DDA agent predicts the players’ state using occurred logs and simulated logs by MCTS.

  • DDA agent adapts game difficulty based on predicted player’s state to enhance it.

  • DDA strategies can be diversified by focusing on different player states.

  • The possibility of balancing the game difficulty while satisfying diverse preference.

Abstract

Game developers have employed dynamic difficulty adjustment (DDA) in designing game artificial intelligence (AI) to improve players’ game experience by adjusting the skill of game agents. Traditional DDA agents depend on player proficiency only to balance game difficulty, and this does not always lead to improved enjoyment for the players. To improve game experience, there is a need to design game AIs that consider players’ affective states. Herein, we propose AI opponents that decide their next actions according to a player’s affective states, in which the Monte-Carlo tree search (MCTS) algorithm exploits the states estimated by machine learning models referencing in-game features. We targeted four affective states to build the model: challenge, competence, valence, and flow. The results of our user study demonstrate that the proposed approach enables the AI opponents to play automatically and adaptively with respect to the players’ states, resulting in an enhanced game experience.

Introduction

A balanced challenge is an important factor for the satisfaction of players’ gaming experience (Ang and Mitchell, 2017, Chen, 2007, Cowley et al., 2008, Csikszentmihalyi, 1991). Game developers use predefined curves that manipulate the level of difficulty as they undergo trial and error, but developing the curves is a time-consuming and challenging task due to differing players’ needs (Ang & Mitchell, 2017). Therefore, many studies (Ang and Mitchell, 2019, Constant and Levieux, 2019, Frommel et al., 2018, Hunicke, 2005, Xue et al., 2017, Zohaib and Nakanishi, 2018) have employed dynamic difficulty adjustment (DDA) methods to automatically control game components to improve each unique player experience (PX). DDA methods measure how well a player plays a game through a fitness function (herein, called score function) and manipulate game components, such as game parameters (game speed), game environment (next block of Tetris), or the game AI itself (beginner-level or high-level strategies of agents in fighting games), to adjust the difficulty. As AI technology advances, there has been a growing interest in direct game AI manipulation for DDA methods to achieve more effective game difficulty adjustment (Demediuk et al., 2017, Ishihara et al., 2018, Moon and Seo, 2020, Pfau et al., 2020, Zohaib and Nakanishi, 2018). Most of these studies concentrate on adapting game components for game difficulty by focusing on the player’s proficiency.

Many heuristic approaches have been employed to determine a player’s proficiency, such as measuring win rate (Duque et al., 2020, Sarkar and Cooper, 2019), game score (Hagelback and Johansson, 2009, Silva et al., 2017), and health points (HP) (Demediuk et al., 2017, Ishihara et al., 2018). For example, games with opponent agents, such as fighting games, adjust the difficulty by changing the skills of the game agents using tree search (Demediuk et al., 2017, Ishihara et al., 2018) or deep learning (Pfau et al., 2020), according to heuristic measures like HP-difference. Independent of the model used, if the model is based on a simple heuristic score function, the gameplay is relatively static and has predictable difficulty adaptations (Ang and Mitchell, 2017, Constant and Levieux, 2019, Hunicke, 2005, Stephenson and Renz, 2019). Players have certain difficulty expectations from games. For instance, some players may prefer challenging games, whereas some enjoy more relaxed, easily winnable games. Simple heuristic measurements are often too limited to satisfy the wide breadth of player expectations.

Some researchers have developed DDA models to estimate players’ affective states using sensors (Afergan et al., 2014, Chanel et al., 2011, Fernandez B. et al., 2017, Kivikangas et al., 2011, Kneller et al., 2012, Parnandi and Gutierrez-Osuna, 2015, Rani et al., 2005, Stein et al., 2018, Tognetti et al., 2010). Instead of measuring player proficiency, they focus on how their DDA model improves the actual PX. However, they employed simple adjustment mechanisms, which manually controlled game parameters, such as game speed. In this case, the affective model to estimate player state is outside the adjustment mechanism and does not change according to the individual player. Thus, the game difficulty does not accurately match the player’s estimated state, but rather operates based on a pre-designed function designed by the developer. Therefore, it is necessary to apply the player states directly to the game AI mechanism, and automatically generate strategies to satisfy players’ complex preference instead of manual design.

To easily and precisely control the game difficulty and satisfy players’ diverse preferences we designed a DDA-based game AI mechanism that directly considers various players’ affective states and adjusts the game difficulty according to the recognized state. The player state model based on machine learning (ML) methods predicted the affective state of a game player, and the Monte-Carlo tree search (MCTS) algorithm (Chaslot et al., 2008) exploited the ML model as its scoring function. We trained four ML models for four states, including challenge (CH), competence (CO), valence (VA), and flow (FL). Instead of relying on external sensors, our player state model estimates player’s states only using in-game features (or logs). To evaluate the performance of the proposed approach and the effect on PX, we conducted a user study with 20 human players, where our game agent played diverse action strategies according to each player’s state and improved their PX.

The contributions of this study are as follows:

  • This is the first attempt to incorporate the player’s affective states directly into DDA agents and automatically generate game AI appropriate for a specific player state.

  • Game AI with diverse skill strategies was generated to meet a wide breadth of players’ preferences and improve the PX.

Section snippets

Background and related work

Dynamic Difficulty Adjustment (DDA) consists of two steps (Zohaib & Nakanishi, 2018). First, measuring the players’ skill levels through a score function. Then, adjusting the game difficulty in various ways according to the measured players’ skill levels. Section 2.1 demonstrates the DDA methods which mainly concentrated on the difficulty adjustment methods (second step), and Section 2.2 introduces the DDA methods which mainly concentrated on the player modeling for measuring players’ level

Architecture of the DDA model for an AI opponent

Herein, we show the development of our model. Fig. 1 shows the overall architecture of the model, which incorporates an MCTS algorithm and a player state prediction model. The main difference between our model and the heuristic DDA method is that we directly used players’ state as a score function for MCTS instead of using heuristic functions, such as game score, win rate, and HP differences. The player state model uses a combination of the real-played and simulated game logs by MCTS to predict

Environmental settings

We employed a fighting game for our experiment, which is a representative genre in which players play the game against AI opponent. There are four player states we aimed to encourage; Challenge (CH), Competence (CO), Valence (VA), and Flow (FL) which were derived from the Game Experience Questionnaire (GEQ) survey (IJsselsteijn et al., 2013). MCTS time limit for the entire searching calculation was set as 165 ms to meet the real-time performance of the agent. In other words, the four steps of

Materials

A counter-balanced single factor within-subjects design was conducted to validate the performance and PX of our DDA model compared to the current heuristic DDA models. For our model, four game agents were designed each for CH (CH-agent), CO (CO-agent), VA (VA-agent), and FL (FL-agent), which is the principal dimension of PX. Each agent played the game aimed to elevate the players’ target state. We employed prior work by Demediuk et al. (2017) as a baseline (HP-agent), as often exploited in many

Analysis and result

We collected 200 samples of the questionnaire and log data (20 participants × 5 game conditions × 2 sessions). Scale reliability tests were performed for the five subcategories of GEQ. Cronbach’s alpha (Cronbach, 1951) showed strong reliability for all subcategories (Challenge: α = 0.88; Competence: α = 0.93; Positive effect: α = 0.96; Negative effect: α = 0.86; Flow: α = 0.94), indicating consistent answers. The analysis was conducted with two main research questions to analyze whether

Player preference for AI opponents

We found that individual player preferences for state-specific DDA agents were varied. Fig. 5 exhibits the diversity of the preferences. The overall preferences were 15% (three participants) for CH, 20% (four participants) for CO, 35% (seven participants) for VA, and 30% (six participants) for FL, indicating diverse preferences for the four agents. Players’ second and third preference rankings were evenly distributed. The preference ranking for each agent was influenced by individual

Conclusion

Herein, we propose state-specific (CH, CO, VA, and FL) DDA agents that aim to improve PX by observing the player state. This was accomplished by a simple concept that uses player states as a score function to determine the action of game agents to automatically adjust difficulty, unlike manual modification on the heuristics. The proposed approach adjusted the AI opponents according to their target states and improved gameplay compared to a heuristic-based baseline (HP) agent. Despite being

CRediT authorship contribution statement

JaeYoung Moon: Conceptualization, Methodolgy, Software, Investigation, Writing – original draft, Project administration. YouJin Choi: Conceptualization, Validation, Formal analysis, Investigation, Writing – original draft, Visualization. TaeHwa Park: Software, Resources. JunDoo Choi: Resources, Data curation. Jin-Hyuk Hong: Supervision. Kyung-Joong Kim: Supervision, Funding acquisition.

Declaration of Competing Interest

One or more of the authors of this paper have disclosed potential or pertinent conflicts of interest, which may include receipt of payment, either direct or indirect, institutional support, or association with an entity in the biomedical field which may be perceived to have potential conflict of interest with this work. For full disclosure statements refer to https://doi.org/10.1016/j.eswa.2022.117677. KyungJoong Kim reports financial support was provided by National Research Foundation of

Acknowledgments

This research was supported by the National Research Foundation of Korea (NRF) funded by the MSIT (2021R1A4A1030075).

References (53)

  • NackeL.E. et al.

    More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game

    Interactive Computing

    (2010)
  • SilvaM.P. et al.

    Dynamic difficulty adjustment on MOBA games

    Entertainment Computing

    (2017)
  • SteinA. et al.

    EEG-triggered dynamic difficulty adjustment for multiplayer games

    Entertainment Computing

    (2018)
  • AferganD. et al.

    Dynamic difficulty using brain metrics of workload

  • AngD. et al.

    Comparing effects of dynamic difficulty adjustment systems on video game experience

  • AngD. et al.

    Representation and frequency of player choice in player-oriented dynamic difficulty adjustment systems

  • AuerP.

    Using confidence bounds for exploitation-exploration trade-offs

    Journal of Machine Learning Research

    (2003)
  • BonferroniC.

    Teoria statistica delle classi e calcolo delle probabilita

    Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze

    (1936)
  • BreimanL.

    Random forests

    Machine Learning

    (2001)
  • ChanelG. et al.

    Emotion assessment from physiological signals for adaptation of game difficulty

    IEEE Transactions on Systems, Man, and Cybernetics. A: Systems and Humans

    (2011)
  • ChaslotG. et al.

    Monte-Carlo tree search: A new framework for game AI

  • ChenJ.

    Flow in games (and everything else)

    Communications of the ACM

    (2007)
  • ConstantT. et al.

    Dynamic difficulty adjustment impact on players’ confidence

  • CowleyB. et al.

    Toward an understanding of flow in video games

    Computers in Entertainment

    (2008)
  • CronbachL.J.

    Coefficient alpha and the internal structure of tests

    Psychometirka

    (1951)
  • CsikszentmihalyiM.

    Flow: The psychology of optimal experience

    (1991)
  • DemediukS. et al.

    Monte Carlo tree search based algorithms for dynamic difficulty adjustment

  • DenisovaA. et al.

    “Whatever the emotional experience, it’s up to them”: Insights from designers of emotionally impactful games

  • DenisovaA. et al.

    Adaptation in digital games: The effect of challenge adjustment on player performance and experience

  • DuqueM.G. et al.

    Finding game levels with the right difficulty in a few trials through intelligent trial-and-error

  • Fernandez B.H.D. et al.

    Adaptable game experience based on player’s performance and EEG

  • FriedmanM.

    The use of ranks to avoid the assumption of normality implicit in the analysis of variance

    Journal of the American Statistical Association

    (1937)
  • FrommelJ. et al.

    Emotion-based dynamic difficulty adjustment using parameterized difficulty and self-reports of emotion

  • HagelbackJ. et al.

    Measuring player experience on runtime dynamic difficulty scaling in an RTS game

  • HunickeR.

    The case for dynamic difficulty adjustment in games

  • HunickeR. et al.

    Ai for dynamic difficulty adjustment in games

  • Cited by (9)

    View all citing articles on Scopus
    1

    J. Moon and Y. Choi contributed equally to this work.

    View full text