skip to main content
10.1145/1920778.1920786acmotherconferencesArticle/Chapter ViewAbstractPublication PagesfutureplayConference Proceedingsconference-collections
research-article

Critic-proofing: how using critic reviews and game genres can refine heuristic evaluations

Published: 06 May 2010 Publication History

Abstract

Heuristic evaluation -- a technique where experts inspect software and determine where the application violates predetermined policies for good usability - is an effective technique for evaluating productivity software. The technique has recently been applied to video games, examining playability and usability for both single and multiplayer games. However, the severity ratings assigned to usability problems and used as a coarse categorization method for triage are still subjectively and somewhat arbitrarily assigned by evaluators, offering limited organizational value. In addition, they fail to account for the diversity found between games and game genres. In this paper we present a modified heuristic evaluation technique, which produces a prioritized list of heuristic violations based on the problem's frequency, impact, persistence, the heuristic it violates, and the game's genre. We evaluate our technique in a case study and show that the technique provides substantial value with little additional effort.

References

[1]
Desurvire, H., Caplan, M., & Toth, J. A., Using heuristics to evaluate the playability of games. Ext. Abstracts of CHI 2004, 1509--12.
[2]
Desurvire, H., & Wiberg, C. Game Usability Heuristics (PLAY) For Evaluating and Designing Better Games: The Next Iteration. Proc. of HCI Conference 2009.
[3]
Dumas, J. S., & Redish, J. C., A Practical Guide to Usability Testing, Ablex Publishing Corporation, Norwood, 1993.
[4]
Federoff, M. Heuristics and Usability Guidelines for the Creation and Evaluation of Fun in Video Games. Indiana University Master of Science Thesis, 2002.
[5]
Fullerton, T., Game Design Workshop, 2nd Edition: A Playcentric Approach to Creating Innovative Games, Morgan Kaufmann Publishers, Boston, 2008.
[6]
Koeffel, C., Hochleitner, W., Leitner, J., Haller, M., Geven, A., & Tscheligi, M., Using Heuristics to Evaluate the Overall User Experience of Video Games and Advanced Interaction Games, in Bernhaupt, R. ed. Evaluating User Experience in Games. Springer, 2009.
[7]
Korhonen, H., Koivisto, E. M. I., Playability Heuristics for Mobile Games. Proc. of MobileHCI 2006, 9--16.
[8]
Korhonen, H., Koivisto, E. M. I., Playability Heuristics for Mobile Multi-Player Games. Proc. of DIMEA 2007, 28--35.
[9]
Korhonen, H., Paavilainen, J., & Saarenpäää, H. Expert review method in game evaluations: comparison of two playability heuristic sets. Proc. of MindTrek 2009, 74--81.
[10]
Larsen JM (2008) Evaluating User Experience -- how game reviewers do it. In: Evaluating User Experiences in Games, Workshop at CHI 2008.
[11]
Nacke, L. From playability to a hierarchical game usability model. Proc. Of FuturePlay 2009, 11--12.
[12]
Nielsen, J. How to Conduct a Heuristic Evaluation (2007). Retrieved March 4, 2010, from UseIt: www.useit.com/papers/heuristic/heuristic_evaluation.html.
[13]
Nielsen, J. 1992. Reliability of severity estimates for usability problems found by heuristic evaluation. Proc. CHI 1992, 129--130.
[14]
Nielsen, J. Severity Ratings for Usability Problems (2007). Retrieved March 4, 2010, from UseIt: http://www.useit.com/papers/heuristic/severityrating.html.
[15]
Nielsen, J. Ten Usability Heuristics (2005). Retrieved March 4, 2010, from UseIt: www.useit.com/papers/heuristic/heuristic_list.html.
[16]
Nielsen, J., and Mack, R. L. Usability Inspection Methods, John Wiley & Sons, New York, 1994.
[17]
Nutt, C Back in Space: Bio Ware On Mass Effect 2. Retrieved March 4, 2010, from Gamasutra: www.gamasutra.com/view/feature/4251/back_in_space_bioware_on_mass_.php, 2010.
[18]
Pinelle, D., Wong, N., Stach, T., Heuristic Evaluation for Games: Usability Principles for Video Game Design. Proc. of CHI 2008, 1453--1462.
[19]
Pinelle, D., Wong, N., Stach, T., Gutwin, C. Usability Heuristics for Networked Multiplayer Games. Proc. of GROUP 2009, 169--178.
[20]
Pinelle, D., Wong, N., Stach, T., Using Genres to Customize Usability Evaluations of Video Games. Proc. of Future Play 2008, 129--136.

Cited By

View all
  • (2022)Echoing the Gameplay: Analyzing Gameplay Sessions across Genres by Reconstructing Them from Recorded DataInternational Journal of Human–Computer Interaction10.1080/10447318.2021.201623739:1(52-84)Online publication date: 3-Jan-2022
  • (2021)Heuristic Evaluation of a Mobile Game Developed to Help Battle the Pandemic2021 IEEE Conference on Games (CoG)10.1109/CoG52621.2021.9619103(1-8)Online publication date: 17-Aug-2021
  • (2020)A Postmortem on Playtesting: Exploring the Impact of Playtesting on the Critical Reception of Video GamesProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376831(1-12)Online publication date: 21-Apr-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
Futureplay '10: Proceedings of the International Academic Conference on the Future of Game Design and Technology
May 2010
282 pages
ISBN:9781450302357
DOI:10.1145/1920778
  • Conference Chairs:
  • Bill Kapralos,
  • Andrew Hogue,
  • Simon Xu
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • Algoma University College

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 May 2010

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. critic-proofing
  2. game genre
  3. heuristic evaluation
  4. severity rating
  5. usability evaluation
  6. video games

Qualifiers

  • Research-article

Conference

Futureplay '10
Sponsor:
Futureplay '10: Futureplay '10 @ GDC Canada
May 6 - 7, 2010
British Columbia, Vancouver, Canada

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)30
  • Downloads (Last 6 weeks)4
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Echoing the Gameplay: Analyzing Gameplay Sessions across Genres by Reconstructing Them from Recorded DataInternational Journal of Human–Computer Interaction10.1080/10447318.2021.201623739:1(52-84)Online publication date: 3-Jan-2022
  • (2021)Heuristic Evaluation of a Mobile Game Developed to Help Battle the Pandemic2021 IEEE Conference on Games (CoG)10.1109/CoG52621.2021.9619103(1-8)Online publication date: 17-Aug-2021
  • (2020)A Postmortem on Playtesting: Exploring the Impact of Playtesting on the Critical Reception of Video GamesProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376831(1-12)Online publication date: 21-Apr-2020
  • (2019)Heuristic usability evaluation on gamesMultimedia Tools and Applications10.1007/s11042-018-6593-178:4(4937-4964)Online publication date: 1-Feb-2019
  • (2018)Video Game Selection Procedures For Experimental ResearchProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173760(1-9)Online publication date: 21-Apr-2018
  • (2017)Probing user opinions in an indirect wayProceedings of the 21st International Academic Mindtrek Conference10.1145/3131085.3131121(1-7)Online publication date: 20-Sep-2017
  • (2017)Designing Mobile Applications with Empathizing User ExperienceProceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems10.1145/3027063.3053347(1108-1116)Online publication date: 6-May-2017
  • (2017)Academic methods for usability evaluation of serious gamesMultimedia Tools and Applications10.1007/s11042-016-3845-976:4(5755-5784)Online publication date: 1-Feb-2017
  • (2015)A Heuristic Framework for Evaluating User Experience in GamesGame User Experience Evaluation10.1007/978-3-319-15985-0_9(187-206)Online publication date: 5-Jun-2015
  • (2015)Games User Research and Physiological Game EvaluationGame User Experience Evaluation10.1007/978-3-319-15985-0_4(63-86)Online publication date: 5-Jun-2015
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media