skip to main content
research-article

Usertesting Without the User: Opportunities and Challenges of an AI-Driven Approach in Games User Research

Published: 10 April 2018 Publication History

Abstract

The use of human participants in game evaluation can be costly, time-consuming, and present challenges for constructing representative player samples. These challenges may be overcome by using computer-controlled agents in place of human users for certain stages of the usertesting process. This article explores opportunities and challenges in the use of behavioural modelling to create independent “user” agents driven by artificial intelligence (AI). We highlight the utility of imitating cognitive processes such as spatial reasoning, memory, and goal-oriented decision-making as a means to increase the viability of independent agents as a tool in usertesting. Specifically, we investigate the possible design and use of proxy AI “users” that mimic human navigational behaviour to assist in the evaluation of level designs. Ultimately, we propose that a configurable population of AI players can provide a data-rich supplement to current approaches in games user research.

References

[1]
E. Adar, D. S. Weld, B. N. Bershad, and S. S. Gribble. 2007. Why we search: Visualizing and predicting user behavior. In Proceedings of the 16th International Conference on World Wide Web Pages (WWW’07), 161--167.
[2]
Z. A. Algfoor, M. S. Sunar, and H. Kolivand. 2015. A comprehensive study on pathfinding techniques for robotics and video games. Int. J. Comput. Games Technol. Article 736138 (2015).
[3]
R. Bartle. 1996. Hearts, clubs, diamonds, spades: Players who suit MUDs. Retrieved from http://mud.co.uk/richard/hcds.htm.
[4]
M. J. Chadwick, R. S. Anjum, D. Kumaran, D. L. Schacter, H. J. Spiers, and D. Hassabis. 2016. Semantic representations in the temporal pole predict false memories. Proc. Natl. Acad. Sci. U.S.A. 113(36): 10180--10185.
[5]
B. Drenikow and P. Mirza-Babaei. 2017. Vixen: Interactive visualization of gameplay experiences. In Proceedings of the 12th International Conference on the Foundations of Digital Games (FDG’17). Article 3.
[6]
I. Driscoll, D. A. Hamilton, R. A. Yeo, W. M. Brooks, and R. J. Sutherland. 2004. Virtual navigation in humans: The impact of age, sex, and hormones on place learning. Hormones Behav. 47, 3, 326--335.
[7]
C. Eliasmith, T. C. Stewart, X. Choo, T. Bekolay, T. DeWolf, Y. Tang, and D. Rasmussen. 2012. A large-scale model of the functioning brain. Science 338(6111): 1202--1205.
[8]
IGDA Games User Research SIG (n.d.) What is GUR? Retrieved from http://gamesuserresearchsig.org/what-is-gur/.
[9]
C. Giovannangeli and P. Gaussier. 2008. Autonomous vision-based navigation: Goal-oriented action planning by transient states prediction, cognitive map building, and sensory-motor learning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’08).
[10]
D. L. Hintzman. 1984. MINERVA 2: A simulation model of human memory. Behav. Res. Methods Instrum. Comput. 16, 2, 96--101.
[11]
B. A. Kuhl, N. M. Dudukovic, I. Kahn, and A. D. Wagner. 2007. Decreased demands on cognitive control reveal the neural processing benefits of forgetting. Nature Neurosci. 10, 908-914.
[12]
T. Machado, A. Nealen, and J. Togelius. 2017. SeekWhence: A retrospective analysis tool for general game design. In Proceedings of the 12th International Conference on the Foundations of Digital Games (FDG’17): Article 4.
[13]
T. Mahlmann, A. Drachen, J. Togelius, A. Canossa, and G. N. Yannakakis. 2010. Predicting player behavior in tomb raider: Underworld. In Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games. 178--185.
[14]
V. Mnih, K. Kavukcuoglu, D. Silver et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 529--533H.
[15]
J. McCarthy and P. Wright. 2004. Technology as experience. Interactions 11, 5, 42--43.
[16]
A. Mendes, J. Togelius, and A. Nealen. 2016. Hyper-heuristic general video game playing. 2016 In IEEE Conference on Computational Intelligence and Games (CIG’16).
[17]
G. McAllister, P. Mirza-Babaei, and J. Avent. 2013. Improving gameplay with game metrics and player metrics. In Game Analytics, M. Seif El-Nasr, A. Drachen, and A. Canossa (Eds.). Springer-Verlag London, London, 621--638
[18]
P. Mirza-Babaei. 2013. Biometric storyboards: A games user research approach for improving qualitative evaluations of player experience. Ph.D. Dissertation. University of Sussex, United Kingdom.
[19]
P. Mirza-Babaei, L. E. Nacke, J. Gregory, N. Collins, and G. Fitzpatrick. 2013. How does it play better? Exploring user testing and biometric storyboards in games user research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1499--1508.
[20]
P. Mirza-Babaei, V. Zammitto, J. Niesenhaus, M. Sangin, and L. E. Nacke. 2013. Games user research: practice, methods, and applications. In Proceedings of the 2013 CHI Conference on Human Factors in Computing Systems: Extended Abstracts (CHI’13). 3219--3222.
[21]
N. Moosajee and P. Mirza-Babaei. 2016. Games user research (GUR) for indie studios. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems: Extended Abstracts (CHI’16). 3159--3165.
[22]
L. E. Nacke, S. Engels, and P. Mirza-Babaei. 2015. Actionable inexpensive games user research. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems: Extended Abstracts. 2461--246.
[23]
J. Ortega, N. Shaker, J. Togelius, and G. N. Yannakakis. 2013. Imitating human playing styles in Super Mario Bros. Entertain. Comput. 4, 2, 93--104.
[24]
R. J. Pagulayan, K. Keeker, D. Wixon, R. L. Romero, and T. Fuller. 2003. User-centered design in games. In The Human-computer Interaction Handbook, J. A. Jacko and A. Sears (eds.). L. Erlbaum Associates Inc., Hillsdale, NJ, 883--906
[25]
J. Renaudin. 2016. The role of artificial intelligence in testing: An interview with Jason Arbon. Retrieved from https://www.stickyminds.com/interview/role-artificial-intelligence-testing-interview-jason-arbon.
[26]
M. Seif El-Nasr, A. Drachen, and A. Canossa. (eds.) 2013. Game Analytics. Springer-Verlag, London.
[27]
C. T. Tan, T. W. Leong, and S. Shen. 2014. Combining think-aloud and physiological data to understand video game experiences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14). 381--390.
[28]
C. T. Tan, P. Mirza-Babaei, V. Zammitto, A. Canossa, G. Conley, and G. Wallner. 2015. Tool design jam: Designing tools for games user research. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (CHIPLAY’15). 827--831.
[29]
D. Tisserand. (In Press) It's All About the Process. In Games User Research, A. Drachen, P. Mirza-Babaei, and L. E. Nacke (eds.). Oxford University Press, UK, 31--44.
[30]
G. F. Tondello, R. R. Wehbe, L. Diamond, M. Busch, A. Marczewski, and L. E. Nacke. 2016. The gamification user types hexad scale. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (CHIPLAY’16). 229--243.
[31]
I. J. M. van der Ham, A. M. E. Faber, M. Venselaar, M. J. van Kreveld, M. Loffler. 2015. Ecological validity of virtual environments to assess human navigation ability. Front. Psychol. 2015, 6.
[32]
A. Vieira. 2015. Predicting online user behaviour using deep learning algorithms. arXiv:1511.06247. Retrieved from https://arxiv.org/abs/1511.06247.
[33]
M. Vijesh, S. Iyengar, S. M. Vijay Mahantesh, A. Ramesh, C. Pandurangan, and V. Madhavan. 2012. A navigation algorithm inspired by human navigation. In Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. 1309--1314.
[34]
T. Wolbers and M. Hegarty. 2010. What determines our navigational abilities? Trends Cogn. Sci. 14, 3, 138--146.
[35]
G. N. Yannakakis, P. Spronck, D. Loiacono, and E. Andre. 2013. Player modeling. Dagstuhl Follow-Ups 6: 45--59.
[36]
S. Yildrim and S. B. Stene 2008. A survey on the need and use of AI in game agents. Proceedings of the 2008 Spring Simulation Multiconference (SpringSim’08). 124--131.
[37]
C. Yun, P. Trevino, W. Holtkamp, and Z. Deng. 2010. PADS: Enhancing gaming experience using profile-based adaptive difficulty system. In Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games. 31--36.
[38]
V. Zammitto, P. Mirza-Babaei, I. Livingston, M. Kobayashi, and L. E. Nacke. 2014. Player experience: mixed methods and reporting results. In Proceedings of the 2014 CHI Conference on Human Factors in Computing Systems: Extended Abstracts (CHI’14). 147--150.

Cited By

View all
  • (2024)Automated Game Design Testing Using Machine LearningEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_385(199-208)Online publication date: 5-Jan-2024
  • (2023)Beyond Equilibrium: Utilizing AI Agents in Video Game Economy BalancingCompanion Proceedings of the Annual Symposium on Computer-Human Interaction in Play10.1145/3573382.3616092(155-160)Online publication date: 6-Oct-2023
  • (2023)Offline and online user experience of gamified robotics for introducing computational thinking: Comparing engagement, game mechanics and coding motivationComputers & Education10.1016/j.compedu.2022.104664193(104664)Online publication date: Feb-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Computers in Entertainment
Computers in Entertainment   Volume 16, Issue 2
Special Issue: Deep Learning, Ubiquitous and Toy Computing
April 2018
152 pages
EISSN:1544-3574
DOI:10.1145/3181320
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 April 2018
Accepted: 01 January 2018
Received: 01 January 2018
Published in CIE Volume 16, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Artificial intelligence
  2. computer modeling
  3. games user research

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)103
  • Downloads (Last 6 weeks)6
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Automated Game Design Testing Using Machine LearningEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_385(199-208)Online publication date: 5-Jan-2024
  • (2023)Beyond Equilibrium: Utilizing AI Agents in Video Game Economy BalancingCompanion Proceedings of the Annual Symposium on Computer-Human Interaction in Play10.1145/3573382.3616092(155-160)Online publication date: 6-Oct-2023
  • (2023)Offline and online user experience of gamified robotics for introducing computational thinking: Comparing engagement, game mechanics and coding motivationComputers & Education10.1016/j.compedu.2022.104664193(104664)Online publication date: Feb-2023
  • (2023)Automated Game Design Testing Using Machine LearningEncyclopedia of Computer Graphics and Games10.1007/978-3-319-08234-9_385-1(1-10)Online publication date: 11-Apr-2023
  • (2023)Model-based Player Experience Testing with Emotion Pattern VerificationFundamental Approaches to Software Engineering10.1007/978-3-031-30826-0_9(151-172)Online publication date: 22-Apr-2023
  • (2021)PathOS+: A New Realm in Expert EvaluationExtended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play10.1145/3450337.3483495(122-127)Online publication date: 15-Oct-2021
  • (2021)An Appraisal Transition System for Event-Driven Emotions in Agent-Based Player Experience TestingEngineering Multi-Agent Systems10.1007/978-3-030-97457-2_9(156-174)Online publication date: 3-May-2021
  • (2020)Artificial Players in the Design Process: Developing an Automated Testing Tool for Game Level and World DesignProceedings of the Annual Symposium on Computer-Human Interaction in Play10.1145/3410404.3414249(267-280)Online publication date: 2-Nov-2020
  • (2020)Defining playability of gamesProceedings of the 23rd International Conference on Academic Mindtrek10.1145/3377290.3377309(55-64)Online publication date: 29-Jan-2020
  • (2020)Visualizing AI Playtesting Data of 2D Side-scrolling Games2020 IEEE Conference on Games (CoG)10.1109/CoG47356.2020.9231915(572-575)Online publication date: Aug-2020
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media