extended-abstract

Reflections on Rigor and Reproducibility: Moving Toward a Community Standard for the Description of Artifacts in Experimental Games Research

Authors:

Kathrin Gerling,

Max V. BirkAuthors Info & Claims

CHI PLAY '22: Extended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in Play

Pages 266 - 267

https://doi.org/10.1145/3505270.3558360

Published: 07 November 2022 Publication History

Get Access

CHI PLAY '22: Extended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in Play

Reflections on Rigor and Reproducibility: Moving Toward a Community Standard for the Description of Artifacts in Experimental Games Research

Pages 266 - 267

Abstract
References

Abstract

Games research routinely employs custom-designed artifacts as research tools, but there is no standard for their description. To improve rigor and reproducibility of experimental games research within the CHI PLAY community, this piece of work argues for the development of a community standard for the description of games that are used as research tools, allowing authors to describe their interventions in a structured way, and making it easier for all of us to understand and reproduce each other’s work.

References

[1]

Ernest Adams. 2013. Fundamentals of Game Design(3rd ed.). New Riders.

Digital Library

Google Scholar

[2]

Lena Fanya Aeschbach, Sebastian A.C. Perrig, Lorena Weder, Klaus Opwis, and Florian Brühlmann. 2021. Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY. Proc. ACM Hum.-Comput. Interact. 5, CHI PLAY, Article 233 (oct 2021), 21 pages. https://doi.org/10.1145/3474660

Digital Library

Google Scholar

[3]

American Psychological Association. 2022. Journal Article Reporting Standards. Retrieved July 19, 2022 from https://apastyle.apa.org/jars

Google Scholar

[4]

Tracy Fullerton. 2018. Game Design Workshop: A Playcentric Approach to Creating Innovative Games (4th ed.). CRC Press.

Google Scholar

[5]

Andrey Krekhov, Katharina Emmerich, Philipp Bergmann, Sebastian Cmentowski, and Jens Krüger. 2017. Self-Transforming Controllers for Virtual Reality First Person Shooters. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Amsterdam, The Netherlands) (CHI PLAY ’17). Association for Computing Machinery, New York, NY, USA, 517–529. https://doi.org/10.1145/3116595.3116615

Digital Library

Google Scholar

[6]

Stefan Liszio, Oliver Basu, and Maic Masuch. 2020. A Universe Inside the MRI Scanner: An In-Bore Virtual Reality Game for Children to Reduce Anxiety and Stress. Association for Computing Machinery, New York, NY, USA, 46–57. https://doi.org/10.1145/3410404.3414263

Digital Library

Google Scholar

[7]

Mitchell McEwan, Alethea Blackler, Peta Wyeth, and Daniel Johnson. 2020. Intuitive Interaction with Motion Controls in a Tennis Video Game. Association for Computing Machinery, New York, NY, USA, 321–333. https://doi.org/10.1145/3410404.3414242

Digital Library

Google Scholar

[8]

Nature. 2022. Reporting standards and availability of data, materials, code and protocols. Retrieved July 19, 2022 from https://www.nature.com/commschem/editorial-policies/reporting-standards

Google Scholar

[9]

Jesse Schell. 2008. The Art of Game Design: A Book of Lenses(1st ed.). Morgan Kaufmann.

Digital Library

Google Scholar

[10]

SIGCHI. 2015. Improving the replicability of HCI research with supplementary material. Retrieved July 19, 2022 from https://sigchi.tumblr.com/post/129729411430/improving-the-replicability-of-hci

Google Scholar

[11]

Jan B. Vornhagen, April Tyack, and Elisa D. Mekler. 2020. Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency. Association for Computing Machinery, New York, NY, USA, 4–18. https://doi.org/10.1145/3410404.3414229

Digital Library

Google Scholar

[12]

Chat Wacharamanotham, Lukas Eisenring, Steve Haroz, and Florian Echtler. 2020. Transparency of CHI Research Artifacts: Results of a Self-Reported Survey. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376448

Digital Library

Google Scholar

Cited By

View all

Hougaard BKnoche H(2024)Aiming, Pointing, Steering: A Core Task Analysis Framework for GameplayProceedings of the ACM on Human-Computer Interaction10.1145/36770578:CHI PLAY(1-48)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677057
Karaosmanoglu SCmentowski SNacke LSteinicke F(2024)Born to Run, Programmed to Play: Mapping the Extended Reality Exergames LandscapeProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642124(1-28)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642124
Pais PGonçalves DReis DGodinho JMorais JPiçarra MTrindade PAlexandrovsky DGerling KGuerreiro JRodrigues A(2024)A Living Framework for Understanding Cooperative GamesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641953(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641953
Show More Cited By

Index Terms

Reflections on Rigor and Reproducibility: Moving Toward a Community Standard for the Description of Artifacts in Experimental Games Research
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Virtual worlds software
        Interactive games

Recommendations

Towards reproducibility in recommender-systems research

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system ...
Reproducibility in learning
STOC 2022: Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing

We introduce the notion of a reproducible algorithm in the context of learning. A reproducible learning algorithm is resilient to variations in its samples — with high probability, it returns the exact same output when run on two samples from the same ...
Rigor, Relevance and Impact: The Tensions and Trade-Offs Between Research in the Lab and in the Wild
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

As an interdisciplinary field, CHI embraces diverse research practices ranging from controlled lab experiments to field studies in homes and communities. While quantitative research in the lab emphasizes the scientific rigor for testing hypotheses, ...

Comments

Information & Contributors

Information

Published In

CHI PLAY '22: Extended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in Play

November 2022

419 pages

ISBN:9781450392112

DOI:10.1145/3505270

Editors:
Kathrin Gerling
KU Leuven, Belgium
,
Jo Iacovides
University of York, UK
,
Rainer Malaka
University of Bremen, Germany
,
Beth Bonsignore
University of Maryland, College Park, USA
,
Julian Frommel
Utrecht University, The Netherlands

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2022

Check for updates

Author Tags

Qualifiers

Extended-abstract
Research
Refereed limited

Conference

CHI PLAY '22

Sponsor:

SIGCHI

CHI PLAY '22: The Annual Symposium on Computer-Human Interaction in Play

November 2 - 5, 2022

Bremen, Germany

Acceptance Rates

Overall Acceptance Rate 421 of 1,386 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
123
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)3

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Hougaard BKnoche H(2024)Aiming, Pointing, Steering: A Core Task Analysis Framework for GameplayProceedings of the ACM on Human-Computer Interaction10.1145/36770578:CHI PLAY(1-48)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677057
Karaosmanoglu SCmentowski SNacke LSteinicke F(2024)Born to Run, Programmed to Play: Mapping the Extended Reality Exergames LandscapeProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642124(1-28)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642124
Pais PGonçalves DReis DGodinho JMorais JPiçarra MTrindade PAlexandrovsky DGerling KGuerreiro JRodrigues A(2024)A Living Framework for Understanding Cooperative GamesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641953(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641953
Seaborn K(2023)Super Synthesis Pros., or Why CHI PLAY Needs Research SynthesisCompanion Proceedings of the Annual Symposium on Computer-Human Interaction in Play10.1145/3573382.3616051(235-237)Online publication date: 6-Oct-2023
https://dl.acm.org/doi/10.1145/3573382.3616051
Rogers KSeaborn K(2023)The Systematic Review-lution: A Manifesto to Promote Rigour and Inclusivity in Research SynthesisExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3582733(1-11)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3582733
Spors VLaato SBuruk OHamari J(2023)Longing to be the Mountain: A Scoping Review about Nature-Centric, Health-Minded TechnologiesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581479(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581479

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Index Terms

Recommendations

Towards reproducibility in recommender-systems research

Reproducibility in learning

Rigor, Relevance and Impact: The Tensions and Trade-Offs Between Research in the Lab and in the Wild

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations