Online behavior change detection in computer games
Introduction
Modeling a player’s behavior in a particular game scenario has been the focus of attention of many research work developed in the recent years (Smith, Lewis, Hullett, Smith, & Sullivan, 2011). The act of tracking game metrics and using them afterwards to construct representative models of players is generally known as Player Modeling (PM). In order to learn from the collected game metrics, PM usually relies on Machine Learning (ML) techniques. The types of models learned vary, because PM can be applied to different objectives.
Generally, there are three major objectives for the application of PM techniques. First, game metrics can be used to learn a player’s style or strategy. This is particularly useful for automatically building game characters that present a more human-like behavior (Galli et al., 2009, Weber and Mateas, 2009, Zanetti and Rhalibi, 2004). Second, it can be applied on metrics collected after a game has been launched in order to check whether the game design is appropriate and allows the existence of different player styles (Drachen, Canossa, & Yannakakis, 2009). In this case, the idea is to explore the types of players encountered and use this information to improve next games of the same series or genre. There is still another objective of PM, which is to be able to track or identify characteristics of a player while the game is in progress (Delalleau et al., 2011, Hunicke and Chapman, 2004, Martinez et al., 2010, Yannakakis and Hallam, 2009, Yannakakis and Maragoudakis, 2005). This is, perhaps, the most interesting application of PM, allowing for the adaptation of a game’s responses to different players’ needs along time.
Although they have different objectives, all the aforementioned approaches share the same basic idea: employ the statistics collected through the interaction of a player to improve the game’s quality and/or responses. The data collection process is generally separated from the model learning step, which means that models are induced in an offline manner, using data previously gathered from players. This allows the application of traditional ML techniques, which require a training set of examples and multiple iterations to induce an accurate model. Some exceptions to this methodology do exist, like the work by Hunicke and Chapman (2004), in which game metrics are online collected and used to induce models.
Following the same steps as Hunicke and Chapman (2004), this work proposes to track the player’s behavior during his/her interaction with the game environment. This means that model induction is done completely online. Our objective is to be able to identify moments of possible changes in the behavior of a player based on data collected due to player/environment interactions. In order to accomplish this objective, a change detection mechanism from the area of Data Stream Mining is applied (Vallim, Filho, de Mello, de Carvalho, & Gama, 2014). This method, called M-DBScan, is based on incremental online clustering and novelty detection. We apply the method on the Unreal Tournament 20041 (UT2004) game scenario with three proposed extensions. The first one attempts to improve the rate of truly detected changes, while the second defines a more formal way of detecting when a change happens. Finally, the third modification intends to reduce the delay for change detection. Our experimental results confirm these modifications increase the capability of the method to detect true behavior changes under a small delay. We also apply two other change detection techniques from the Data Stream Mining area and compare results with the extended version of M-DBScan proposed in this work.
This paper is organized as follows. Section 2 presents related work in the area of PM. Next, Section 3 brings the main concepts of M-DBScan, i.e., the change detection algorithm used in this work. Section 4 presents how M-DBScan tracks a player’s behavior, together with the extensions proposed to improve the algorithm’s performance. Experimental results obtained with the Unreal Tournament game are shown and discussed in Section 5. Finally, Section 6 concludes this work and draws some future research directions.
Section snippets
Player modeling
The term Player Modeling (PM) can be used in a variety of scenarios. In the work by Smith et al. (2011), a taxonomy is proposed in order to distinguish the major existing PM applications and techniques. This taxonomy is based on four independent facets that distinguish player models based on who they apply to, what they are used for, the details they model and, finally, how the models are induced. Generally, PM has been used for designing AI players with human-like behavior (Galli et al., 2009,
Micro-clustering DBScan
In order to build online models to represent the evolving behavior of players, the learning algorithm needs to constantly add new data to the induced model, updating its structure incrementally without the need to re-induce the whole model from scratch. In this scenario, traditional ML algorithms are not appropriate, since they need to be re-trained everytime new data becomes available. To deal with this scenario, algorithms from the area known as Data Stream Mining are more suitable.
Data
Detecting changing player behavior
In order to detect changes in the behavior of a player during gameplay, this work proposes the application of the M-DBScan algorithm. In this section, we first explain how to apply the original M-DBScan algorithm to game scenarios. Afterwards, we present the extensions proposed in this work.
Experiments
In order to validate the application of M-DBScan to the games domain, as well as the extensions presented in this work, simulations were created in the First Person Shooter (FPS) Unreal Tournament 2004 (UT2004) using the Pogamut IDE (Gemrot et al., 2009). Our hypothesis is that M-DBScan, and the extensions proposed in this work, will prove to be useful on the task of online learning of a player’s behavior and detecting possible behavior changes.
Conclusions
The online construction of player’s behavioral models is an interesting approach to PM. Being able of tracking the player in an online manner, and constantly updating the model that represents this player, makes it possible to understand how the player behavior is evolving and to capture subtle changes. However, only a few attempts have been made in this direction, with the majority of works aiming at using player models induced offline. In this work, we proposed the application of the M-DBScan
Acknowledgment
The authors thank the Brazilian Research Councils Fapesp (Fundação de Amparo à Pesquisa do Estado de São Paulo) and CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico).
References (27)
- et al.
A framework for clustering evolving data streams
- et al.
A self-organising network that grows when required
Neural Networks
(2002) - et al.
A self-organizing neural network to approach novelty detection
- Bifet, A., & Gavaldà, R. (2007). Learning from time-changing data with adaptive windowing. In SIAM international...
- Cao, F., Ester, M., Qian, W., & Zhou, A. (2006). Density-based clustering over and evolving data stream with noise. In...
- Charles, D., & Black, M. (2004). Dynamic player modeling: a framework for player-centric digital games. In Proceedings...
- Delalleau, O., Contal, E., Thibodeau-Laufer, E., Ferrari, R. C., Bengio, Y., & Zhang, F. (2011). Beyond skill rating:...
Statistical comparisons of classifiers over multiple data sets
The Journal of Machine Learning Research
(2006)- et al.
Player modeling using self-organization in tomb raider: underworld
- et al.
A density-based algorithm for discovering clusters in large spatial databases with noise
Learning a context-aware weapon selection policy for unreal tournament III
Knowledge discovery from data streams
Data stream processing
Cited by (11)
Varying density method for data stream clustering
2020, Applied Soft Computing JournalDetecting pattern changes in individual travel behavior: A Bayesian approach
2018, Transportation Research Part B: MethodologicalCitation Excerpt :Beyond travel behavior, a few studies have been done to detect pattern changes in other aspects of human behavior. For example, Vallim et al. (2013) also used an entropy-based measure to track changes in the behavior of computer game players. The use of entropy is a convenient choice for behavior change detection.
Multidimensional surrogate stability to detect data stream concept drift
2017, Expert Systems with ApplicationsUsing dynamical systems tools to detect concept drift in data streams
2016, Expert Systems with ApplicationsAutomatic visual detection of human behavior: A review from 2000 to 2014
2015, Expert Systems with ApplicationsCitation Excerpt :Different machine learning techniques have been used over the time to build models of human behavior. Vallim, Andrade Filho, De Mello, and De Carvalho (2013) proposed a method that uses incremental learning technique to track a player’s behavior during his interaction with the game. A change detection technique from the area of stream mining was applied based on an incremental clustering and novelty detection method.
Proposal of a new stability concept to detect changes in unsupervised data streams
2014, Expert Systems with ApplicationsCitation Excerpt :In the particular case of learning from unsupervised data, change detection is even more challenging because there is no label information available in the stream. Attempting to accurately detect when data characteristics change, several recent works apply clustering techniques in the stream of data and, afterwards, monitor cluster evolution by means of novelty detection methods (Albertini & Mello, 2010; Marsland et al., 2002; Vallim et al., 2013). The rationale assumed by these algorithms consider that changes observed in induced clustering models correspond to changes in data characteristics.