skip to main content
research-article

GAME: Game As a Measurement Environment: Scheme to Evaluate Interfaces and Game Contents Based on Test Theories

Published: 18 December 2020 Publication History

Abstract

Although many computer games have become diversified in recent years, much effort and ingenuity are needed to produce games that persons with visual impairments can enjoy. However, game developers face difficulties in adjusting the parameters of these games, including difficulty and interface features, because the control skills and disability conditions of gamers vary. Thus, it is essential to establish a scheme to evaluate game interfaces and stages with individuals trials with a short time. In this study, our purpose is to propose and establish the concept of GAME (Game As a Measurement Environment) that can evaluate game interfaces and stages. This concept aims to divide a game stage into substages and then randomly allocate them to the stage. Then, the generated stage records the play logs and play conditions and then analyzes them logs based on test theories, including classical test theory (CTT) and item response theory (IRT). Based on the analysis of the gathered play logs of side-scroller action games, our concept could clarify the relationship between the stage contents, interface effects, and players skills and disabilities.

Supplementary Material

miura (miura.zip)
Supplemental movie, appendix, image and software files for, GAME: Game As a Measurement Environment: Scheme to Evaluate Interfaces and Game Contents Based on Test Theories

References

[1]
Accessible game wiki for blind gamers. http://mm-galabo.com/AcGameWiki/. (in Japanese)
[2]
Audiogame.net. http://www.audiogames.net/.
[3]
BBC: Accessible Games Standard v1.0. http://www.bbc.co.uk/guidelines/futuremedia/accessibility/games.shtml.
[4]
Game accessibility guidelines. http://gameaccessibilityguidelines.com/.
[5]
Ryo-Gei-Kan. http://www.eonet.ne.jp/~akiraworld0719/gindex.html. (in Japanese)
[6]
Summary Wiki of accessible games for visually impaired gamers. http://mm-galabo.com/AcGameWiki/.
[7]
Web Content Accessibility Guidelines (WCAG) 2.1. https://www.w3.org/TR/WCAG21/.
[8]
Dominique Archambault, Roland Ossmann, Thomas Gaudy, and Klaus Miesenberger. 2007. Computer games and visually impaired people. Upgrade 8, 2 (2007), 43--53.
[9]
Frank B Baker and Seock-Ho Kim. 2004. Item response theory: Parameter estimation techniques. CRC Press.
[10]
Jen Beeston, Christopher Power, Paul Cairns, and Mark Barlet. 2018. Accessible Player Experiences (APX): The Players. In International Conference on Computers Helping People with Special Needs. Springer, 245--253.
[11]
Isaac I Bejar. 1985. Test speededness under number-right scoring: An analysis of the Test of English as a Foreign Language. ETS Research Report Series 1985, 1 (1985), i-57.
[12]
A Lord Birnbaum. 1968. Some latent trait models and their use in inferring an examinee's ability. Statistical theories of mental test scores (1968).
[13]
R Darrell Bock and Murray Aitkin. 1981. Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika 46, 4 (1981), 443--459.
[14]
R Darrell Bock and Marcus Lieberman. 1970. Fitting a response model forn dichotomously scored items. Psychometrika 35, 2 (1970), 179--197.
[15]
R Darrell Bock and Robert J Mislevy. 1982. Adaptive EAP estimation of ability in a microcomputer environment. Applied psychological measurement 6, 4 (1982), 431--444.
[16]
Jacob Cohen. 1992. A power primer. Psychological bulletin 112, 1 (1992), 155.
[17]
Linda Crocker and James Algina. 1986. Introduction to classical and modern test theory ERIC.
[18]
Márcia de Borba Campos and Juliana Damasio Oliveira. 2016. Usability, accessibility and gameplay heuristics to evaluate audiogames for users who are blind. In International Conference on Universal Access in Human-Computer Interaction. Springer, 38--48.
[19]
Anders Drachen, Magy Seif El-Nasr, and Alessandro Canossa. 2013. Game analytics-the basics. In Game analytics. Springer, 13--40.
[20]
Magy Seif El-Nasr, Anders Drachen, and Alessandro Canossa. 2016. Game analytics. Springer.
[21]
Lydia Ijeoma Eleje, Frederick Ekene Onah, and Chidiebere Christopher Abanobi. 2018. Comparative study of classical test theory and item response theory using diagnostic quantitative economics skill test item analysis results. European Journal of Educational and Social Sciences 3, 1 (2018), 57--75.
[22]
Leonardo Garcez, Marcello Thiry, and Anita Fernandes. 2020. Accessible Features to Support Visually Impaired People in Game Development:: A Systematic Literature Review of the last 15 years. In 2020 15th Iberian Conference on Information Systems and Technologies (CISTI). IEEE, 1--6.
[23]
Kathrin Gerling, Kieran Hicks, Michael Kalyn, Adam Evans, and Conor Linehan. 2016. Designing movement-based play with young people using powered wheelchairs. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4447--4458.
[24]
Kathrin M Gerling, Regan L Mandryk, and Michael R Kalyn. 2013. Wheelchair-based game design for older adults. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 27.
[25]
Bernadette Gray-Little, Valerie SL Williams, and Timothy D Hancock. 1997. An item response theory analysis of the Rosenberg Self-Esteem Scale. Personality and social psychology bulletin 23, 5 (1997), 443--451.
[26]
Ervin R Hafter, Raymond H Dye, John M Neutzel, and Howard Aronow. 1977. Difference thresholds for interaural intensity. The Journal of the Acoustical Society of America 61, 3 (1977), 829--834.
[27]
Ronald K Hambleton and Russell W Jones. 1993. Comparison of classical test theory and item response theory and their applications to test development. Educational measurement: issues and practice 12, 3 (1993), 38--47.
[28]
Kyung T Han. 2007. WinGen: Windows software that generates item response theory parameters and item responses. Applied Psychological Measurement 31, 5 (2007), 457--459.
[29]
Akio Honda, Hiroshi Shibata, Jiro Gyoba, Kouji Saitou, Yukio Iwaya, and Yôiti Suzuki. 2007. Transfer effects on sound localization performances from playing a virtual three-dimensional auditory game. Applied Acoustics 68, 8 (2007), 885--896.
[30]
In Hong Hwang. 2000. The Usability of Item-Total Correlation as the Index of Item Discrimination. Korean Journal of Medical Education 12, 1 (2000), 45--51.
[31]
Wijnand IJsselsteijn, Yvonne De Kort, Karolien Poels, Audrius Jurgelionis, and Francesco Bellotti. 2007. Characterising and measuring user experiences in digital games. In International conference on advances in computer entertainment technology, Vol. 2. 27.
[32]
Daniel Johnson, M John Gardner, and Ryan Perry. 2018. Validation of two game experience scales: the player experience of need satisfaction (PENS) and game experience questionnaire (GEQ). International Journal of Human-Computer Studies 118 (2018), 38--46.
[33]
Natalie Jorion, Brian D Gane, Katie James, Lianne Schroeder, Louis V DiBello, and James W Pellegrino. 2015. An analytic framework for evaluating the validity of concept inventory claims. Journal of Engineering Education 104, 4 (2015), 454--496.
[34]
Russell V. Lenth. 2016. Least-Squares Means: The R Package lsmeans. Journal of Statistical Software 69, 1 (2016), 1--33.
[35]
N Lessard, M Pare, F Lepore, and M Lassonde. 1998. Early-blind human subjects localize sound sources better than sighted subjects. Nature 395, 6699 (1998), 278--280. https://doi.org/10.1038/26228
[36]
Frederic Lord. 1952. A theory of test scores. Psychometric monographs (1952).
[37]
Frederic M Lord. 2012. Applications of item response theory to practical testing problems. Routledge.
[38]
David Magis, Gilles Raîche, et al. 2012. Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software 48, 8 (2012), 1--31.
[39]
Gary L Marco. 1977. Item characteristic curve solutions to three intractable testing problems 1. ETS Research Bulletin Series 1977, 1 (1977), i-41.
[40]
Masaki Matsuo, Takahiro Miura, Masatsugu Sakajiri, Junji Onishi, and Tsukasa Ono. 2016. Audible Mapper & ShadowRine: Development of Map Editor Using only Sound in Accessible Game for Blind Users, and Accessible Action RPG for Visually Impaired Gamers. In Computers Helping People with Special Needs, LNCS, Vol. 9759. Springer, 537--544.
[41]
Masaki Matsuo, Takahiro Miura, Masatsugu Sakajiri, Junji Onishi, and Tsukasa Ono. 2017. Inclusive Side-Scrolling Action Game Securing Accessibility for Visually Impaired People. In IFIP Conference on Human-Computer Interaction. Springer, 410--414.
[42]
Masaki Matsuo, Masatsugu Sakajiri, Junji Onishi, Tsukasa Ono, and Takahiro Miura. 2017. Experience Report of a Blind Gamer to Develop and Improve the Accessible Action RPG ShadowRine for Visually Impaired Gamers. Journal on Technology and Persons with Disabilities (2017).
[43]
Roderick P McDonald. 1981. The dimensionality of tests and items. British Journal of mathematical and statistical Psychology 34, 1 (1981), 100--117.
[44]
Roderick P McDonald. 2013. Test theory: A unified treatment. psychology press.
[45]
Klaus Miesenberger, Roland Ossmann, Dominique Archambault, Gig Searle, and Andreas Holzinger. 2008. More than just a game: accessibility in computer games. In Symposium of the Austrian HCI and Usability Engineering Group. Springer, 247--260.
[46]
George A Miller. 1947. Sensitivity to changes in the intensity of white noise and its relation to masking and loudness. The Journal of the Acoustical Society of America 19, 4 (1947), 609--619.
[47]
Takahiro Miura, Tohru Ifukube, and Shigeto Furukawa. 2011. Contribution of acoustical characteristics to auditory perception of silent object. In Systems, Man, and Cybernetics (SMC), 2011 IEEE International Conference on. IEEE, 1074--1079.
[48]
Takahiro Miura, Teruo Muraoka, and Tohru Ifukube. 2010. Comparison of obstacle sense ability between the blind and the sighted: A basic psychophysical study for designs of acoustic assistive devices. Acoustical Science and Technology 31, 2 (2010), 137--147.
[49]
Tony Morelli, John Foley, Luis Columna, Lauren Lieberman, and Eelke Folmer. 2010. VI-Tennis: a vibrotactile/audio exergame for players who are visually impaired. In Proceedings of the Fifth International Conference on the Foundations of Digital Games. ACM, 147--154.
[50]
Tony Morelli, John Foley, and Eelke Folmer. 2010. Vi-bowling: a tactile spatial exergame for individuals with visual impairments. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility. ACM, 179--186.
[51]
Tony Morelli and Eelke Folmer. 2014. Real-time sensory substitution to enable players who are blind to play video games using whole body gestures. Entertainment Computing 5, 1 (2014), 83--90.
[52]
Lennart E Nacke. 2015. Games user research and physiological game evaluation. In Game user experience evaluation. Springer, 63--86.
[53]
Daniel Natapov, Steven J Castellucci, and I Scott MacKenzie. 2009. ISO 9241-9 evaluation of video game controllers. In Proceedings of Graphics Interface 2009. Canadian Information Processing Society, 223--230.
[54]
Makoto Ohuchi, Yukio Iwaya, Yôiti Suzuki, and Tetsuya Munekata. 2006. A comparative study of sound localization acuity of congenital blind and sighted people. Acoustical Science and Technology 27, 5 (2006), 290--293.
[55]
Per Anders Östblad, Henrik Engström, Jenny Brusk, Per Backlund, and Ulf Wilhelmsson. 2014. Inclusive game design: audio interface in a graphical adventure game. In Proceedings of the 9th Audio Mostly: A Conference on Interaction With Sound. 1--8.
[56]
Insu Paek and Ki Cole. 2019. Using R for item response theory model applications. Routledge.
[57]
John R Porter and Julie A Kientz. 2013. An empirical study of issues and barriers to mainstream video game accessibility. In Proceedings of the 15th international ACM SIGACCESS conference on computers and accessibility. ACM, 3.
[58]
Caterina Primi, Kinga Morsanyi, Francesca Chiesi, Maria Anna Donati, and Jayne Hamilton. 2016. The development and testing of a new version of the cognitive reflection test applying item response theory (IRT). Journal of Behavioral Decision Making 29, 5 (2016), 453--469.
[59]
Georg Rasch. 1993. Probabilistic models for some intelligence and attainment tests. ERIC.
[60]
Kyle Rector, Cynthia L Bennett, and Julie A Kientz. 2013. Eyes-free yoga: an exergame using depth cameras for blind & low vision exercise. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 12.
[61]
Mary Roznowski. 1989. Examination of the measurement properties of the Job Descriptive Index with experimental items. Journal of Applied Psychology 74, 5 (1989), 805.
[62]
Richard M Ryan, C Scott Rigby, and Andrew Przybylski. 2006. The motivational pull of video games: A self-determination theory approach. Motivation and emotion 30, 4 (2006), 344--360.
[63]
Sujit K Sahu. 2002. Bayesian estimation and model choice in item response models. Journal of Statistical Computation and Simulation 72, 3 (2002), 217--232.
[64]
Martin Schmettow and Wolfgang Vietze. 2008. Introducing item response theory for measuring usability inspection processes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 893--902.
[65]
Didier Schwab, Amela Fejza, Loïc Vial, and Yann Robert. 2018. The GazePlay Project: Open and Free Eye-trackers Games and a Community for People with Multiple Disabilities. In 16th International Conference on Computers Helping People with Special Needs.
[66]
Yoshikazu Seki and Norio Nakamura. 2000. Obstacle Sense in Sound Field with Plural Circumstance Noise. Transactions of the Virtual Reality Society of Japan 5, 3 (2000), 989--995.
[67]
Yoshikazu Seki and Tetsuji Sato. 2011. A training system of orientation and mobility for blind people using acoustic virtual reality. IEEE Transactions on neural systems and rehabilitation engineering 19, 1 (2011), 95--104.
[68]
Rafael Tezza, Antonio Cezar Bornia, and Dalton Francisco De Andrade. 2011. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers 23, 2 (2011), 167--175.
[69]
Michael Urbanek and Florian Güldenpfennig. 2019. Celebrating 20 Years of Computer-based Audio Gaming. In Proceedings of the 14th International Audio Mostly Conference: A Journey in Sound. 90--97.
[70]
Wim J Van Der Linden, Bernard P Veldkamp, and Lynda M Reese. 2000. An integer programming approach to item bank design. Applied Psychological Measurement 24, 2 (2000), 139--150.
[71]
Matthias von Davier, Kentaro Yamamoto, Hyo Jeong Shin, Henry Chen, Lale Khorramdel, Jon Weeks, Scott Davis, Nan Kong, and Mat Kandathil. 2019. Evaluating item response theory linking and model fit for data from PISA 2000-2012. Assessment in Education: Principles, Policy & Practice 26, 4 (2019), 466--488.
[72]
N Watanabe. 1994. Application of item response theory to questionnaire translations: An aid to com parative studies of management in the United States and Japan. Association of Japanese Business Studies 1994 Best Paper Proceedings (1994), 687--707.
[73]
David J Weiss and G Gage Kingsbury. 1984. Application of computerized adaptive testing to educational problems. Journal of Educational Measurement 21, 4 (1984), 361--375.
[74]
Gyoergy Wersenyi. 2012. Virtual localization by blind persons. Journal of the Audio Engineering Society 60, 7/8 (2012), 568--579.
[75]
Thomas Westin, JaEun Jemma Ku, Jérôme Dupire, and Ian Hamilton. 2018. Game Accessibility Guidelines and WCAG 2.0-A Gap Analysis. In International Conference on Computers Helping People with Special Needs. Springer, 270--279.
[76]
Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems. 143--146.
[77]
Keith A Yeomans and Paul A Golder. 1982. The Guttman-Kaiser criterion as a predictor of the number of common factors. The Statistician (1982), 221--229.
[78]
William A Yost and Raymond H Dye Jr. 1988. Discrimination of interaural differences of level as a function of frequency. The Journal of the Acoustical Society of America 83, 5 (1988), 1846--1851.
[79]
Bei Yuan, Eelke Folmer, and Frederick C Harris. 2011. Game accessibility: a survey. Universal Access in the Information Society 10, 1 (2011), 81--100.
[80]
BRANNON Zahand. 2007. Making video games accessible: business justifications and design considerations. MSDN Library Technical Articles. Accessed 6 (2007).

Cited By

View all
  • (2023)Logs Based Verification Tool of Serious Game for Autistic ChildrenQuality of Information and Communications Technology10.1007/978-3-031-43703-8_9(121-129)Online publication date: 13-Sep-2023
  • (2022)Stepping Into the Next Decade of Ubiquitous and Pervasive Computing: UbiComp and ISWC 2021IEEE Pervasive Computing10.1109/MPRV.2022.316006321:2(87-99)Online publication date: 1-Apr-2022
  • (2021)Inclusive Action Game Presenting Real-time Multimodal Presentations for Sighted and Blind PersonsProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479912(62-70)Online publication date: 18-Oct-2021

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 4, Issue 4
December 2020
1356 pages
EISSN:2474-9567
DOI:10.1145/3444864
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 December 2020
Published in IMWUT Volume 4, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Usability evaluation
  2. classical test theory
  3. game controllers
  4. inclusive game
  5. item response theory

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)48
  • Downloads (Last 6 weeks)5
Reflects downloads up to 30 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Logs Based Verification Tool of Serious Game for Autistic ChildrenQuality of Information and Communications Technology10.1007/978-3-031-43703-8_9(121-129)Online publication date: 13-Sep-2023
  • (2022)Stepping Into the Next Decade of Ubiquitous and Pervasive Computing: UbiComp and ISWC 2021IEEE Pervasive Computing10.1109/MPRV.2022.316006321:2(87-99)Online publication date: 1-Apr-2022
  • (2021)Inclusive Action Game Presenting Real-time Multimodal Presentations for Sighted and Blind PersonsProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479912(62-70)Online publication date: 18-Oct-2021

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media