skip to main content
10.1145/3686215.3690151acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
short-paper
Open access

Comparing Subjective Measures of Workload in Video Game Play: Evaluating the Test-Retest Reliability of the VGDS and NASA-TLX

Published: 04 November 2024 Publication History

Abstract

The measurement of player engagement has been the concern of academics and industry professionals alike. Specifically, there is interest towards the best way to measure engagement through surveys or questionnaires. Recently, a new tool was published that attempts to measure workload experienced during video game play, however, it has thus far only been used for exploring the retrospective recollection of gameplay. In this paper, we present, for the first time, a proof of concept study of the test-retest reliability of the Video Game Demand Scale (VGDS) and NASA Task Load Index (NASA-TLX) during natural gameplay with Minecraft. Surveys were administered immediately after gameplay, and then once again approximately one month later. The participants completed a building task in Minecraft whilst being assisted by a companion. Results revealed some test-retest reliability of the VGDS and NASA-TLX, as well as strong correlations between the totals at the two time points. This work is an initial starting point for exploring the test-retest reliability of subjective measures, and as such further experiments are planned and underway to gain a broader understanding of VGDS and NASA-TLX in this context.

References

[1]
Orly Adler and Ainat Pansky. 2020. A “rosy view” of the past: Positive memory biases. In Cognitive Biases in Health and Psychiatric Disorders, Tatjana Aue and Hadas Okon-Singer (Eds.). Academic Press, 139–171. https://doi.org/10.1016/B978-0-12-816660-4.00007-6
[2]
Haldun Akoglu. 2018. User’s guide to correlation coefficients. Turkish journal of emergency medicine 18, 3 (2018), 91–93.
[3]
Victoria K Aldridge, Terence M Dovey, and Angie Wade. 2017. Assessing test-retest reliability of psychological measures. European Psychologist (2017).
[4]
Regina Bernhaupt. 2010. Evaluating user experience in games: Concepts and methods. Springer Science & Business Media.
[5]
Nicholas David Bowman, Jihhsuan Tammy Lin, and Chieh Wu. 2021. A chinese-language validation of the video game demand scale (vgds-c) measuring the cognitive, emotional, physical, and social demands of video games. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–10.
[6]
Nicholas D Bowman and Ron Tamborini. 2012. Task demand and mood repair: The intervention potential of computer games. New Media & Society 14, 8 (2012), 1339–1357.
[7]
Nicholas David Bowman, Joseph Wasserman, and Jaime Banks. 2018. Development of the video game demand scale. In Video Games. Routledge, 208–233.
[8]
Alena Denisova, Paul Cairns, Christian Guckelsberger, and David Zendle. 2020. Measuring perceived challenge in digital games: Development & validation of the challenge originating from recent gameplay interaction scale (CORGIS). International Journal of Human-Computer Studies 137 (2020), 102383.
[9]
Katharina Emmerich, Patrizia Ring, and Maic Masuch. 2018. I’m Glad You Are on My Side: How to Design Compelling Game Companions. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA) (CHI PLAY ’18). Association for Computing Machinery, 141–152. https://doi.org/10.1145/3242671.3242709
[10]
Alex Flint, Alena Denisova, and Nick Bowman. 2023. Comparing Measures of perceived challenge and demand in video games: Exploring the conceptual dimensions of CORGIS and VGDS. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–19.
[11]
Aqeel Haider, Casper Harteveld, Daniel Johnson, Max V Birk, Regan L Mandryk, Magy Seif El-Nasr, Lennart E Nacke, Kathrin Gerling, and Vero Vanden Abeele. 2022. miniPXI: Development and validation of an eleven-item measure of the player experience inventory. Proceedings of the ACM on Human-Computer Interaction 6, CHI PLAY (2022), 1–26.
[12]
Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
[13]
Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139–183.
[14]
Kevin Koban and Nicholas D Bowman. 2020. Further validation and cross-cultural replication of the video game demand scale. Journal of Media Psychology (2020).
[15]
Mehmet Kosa and Nicholas David Bowman. 2023. Replication and Extension of Video Game Demand Scale with a Turkish-Speaking Gamer Population. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–11.
[16]
Heejae Lee, Yoon Esther Lee, Nicholas David Bowman, Shengjie Yao, and Siyang Chen. 2023. Gaming on the Go? Translation and Validation of the Video Game Demand Scale to Korean. International Journal of Human–Computer Interaction (2023), 1–12.
[17]
Francisco J López-Fernández, Laura Mezquita, Mark D Griffiths, Generós Ortet, and Manuel I Ibáñez. 2020. The development and validation of the Videogaming Motives Questionnaire (VMQ). PLoS One 15, 10 (2020), e0240726.
[18]
Gerald Matthews, Joost De Winter, and Peter A Hancock. 2020. What do subjective workload scales really measure? Operational and representational solutions to divergence of workload measures. Theoretical issues in ergonomics science 21, 4 (2020), 369–396.
[19]
Mary Beth Oliver, Nicholas David Bowman, Julia K Woolley, Ryan Rogers, Brett I Sherrick, and Mun-Young Chung. 2016. Video games as meaningful entertainment experiences.Psychology of popular media culture 5, 4 (2016), 390.
[20]
Fred GWC Paas. 1992. Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach.Journal of educational psychology 84, 4 (1992), 429.
[21]
Emma J Pretty, Haytham M Fayek, and Fabio Zambetta. 2023. A Case for Personalized Non-Player Character Companion Design. International Journal of Human–Computer Interaction (2023), 1–20.
[22]
Richard M Ryan, C Scott Rigby, and Andrew Przybylski. 2006. The motivational pull of video games: A self-determination theory approach. Motivation and emotion 30 (2006), 344–360.
[23]
Kevin Mendez Tellez. 2021. Methodologies for evaluating the playability of mobile games: systematic literature review.https://oulurepo.oulu.fi/handle/10024/19184 Publisher: University of Oulu.
[24]
Alexander Thorpe, Keith Nesbitt, and Ami Eidels. 2019. Assessing game interface workload and usability: A cognitive science perspective. In Proceedings of the australasian computer science week multiconference. 1–8.
[25]
Henrik Warpefelt. 2016. The Non-Player Character : Exploring the believability of NPC presentation and behavior. http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-128079 Publisher: Department of Computer and Systems Sciences, Stockholm University.

Index Terms

  1. Comparing Subjective Measures of Workload in Video Game Play: Evaluating the Test-Retest Reliability of the VGDS and NASA-TLX

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ICMI Companion '24: Companion Proceedings of the 26th International Conference on Multimodal Interaction
      November 2024
      252 pages
      ISBN:9798400704635
      DOI:10.1145/3686215
      This work is licensed under a Creative Commons Attribution International 4.0 License.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 04 November 2024

      Check for updates

      Author Tags

      1. Quantitative Methods
      2. User Experience
      3. Video Games
      4. Workload

      Qualifiers

      • Short-paper
      • Research
      • Refereed limited

      Conference

      ICMI '24
      Sponsor:
      ICMI '24: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
      November 4 - 8, 2024
      San Jose, Costa Rica

      Acceptance Rates

      Overall Acceptance Rate 453 of 1,080 submissions, 42%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 148
        Total Downloads
      • Downloads (Last 12 months)148
      • Downloads (Last 6 weeks)46
      Reflects downloads up to 19 Feb 2025

      Other Metrics

      Citations

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Login options

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media