Skip to main content

User Experience Evaluation Methods for Games in Serious Contexts

  • Chapter
  • First Online:
Software Engineering for Games in Serious Contexts

Abstract

User experience in digital games can be influenced by many factors such as flow [Csikszentmihalyi (Flow: the psychology of optimal experience. Harper Collins, 1990), Sweetser and Wyeth (Computers in Entertainment 3(3):1–24, 2005)], immersion [Brown and Cairns (ACM Conference on Human Factors in Computing Systems, CHI 2004, ACM Press, 2004), Ermi and Mayra (Proceedings of Chancing Views – Worlds in Play. Digital Games Research Association’s Second International Conference, 2005)], frustration or tension [Gilleade and Dix (Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology – ACE ’04, 2004)], psychological absorption [Funk et al. (Proceedings of the Second International Conference on Entertainment Computing Pittsburgh, Carnegie Mellon University, 2003)], and social game context [Bracken et al. (Online video games and gamers’ sensations of spatial, social, and copresence. FuturePlay 2005, 2005)]. Most of these factors should be present in a digital game in order to provide the optimal gaming experience [Kirginas (Contemporary Educational Technology 14(2):ep351, 2022), Kirginas et al. (International Journal of Child-Computer Interaction 28, 2021), Kirginas and Gouscos (The International Journal of Serious Games 4:53–69, 2017; International Journal of Serious Games 3:29–45, 2016)]. As there are many different game genres, sub-genres, and game types, user experience needs to be explored in more detail in research studies. This need is even greater when we talk about serious games. User experience is a multifactorial concept that is difficult to measure. This chapter aims to present a range of quantitative and qualitative/objective and subjective/short-term and long-term/formative and summative methods that can be used to evaluate users’ experience in serious games during and after the development process. It is also intended to provide insight into when the different user experience assessment methodologies should be employed in the development cycle.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Nacke, L.E.: Affective ludology: scientific measurement of user experience, interactive entertainment. Blekinge Institute of Technology (2009)

    Google Scholar 

  2. Mandryk, R.L., Atkins, M.S., Inkpen, K.M.A.: Continuous and objective evaluation of emotional experience with interactive play environments. In: Proceedings of CHI 2006, Montréal, Québec, Canada, April 2006, pp. 1027–1036. ACM (2006)

    Google Scholar 

  3. Drachen, A., Canossa, A.: Towards gameplay analysis via gameplay metrics. In: Proceedings of MindTrek, Tampere, Finland, October 1–2. ACM (2009)

    Google Scholar 

  4. Kim, J.H., Gunn, D.V., Schuh, E., Phillips, B., Pagulayan, R.J., Wixon, D.: Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In: Proceedings of CHI 2008, pp. 443–452. ACM, Florence, Italy (2008)

    Google Scholar 

  5. Nacke, L., Niesenhaus, J., Engl, S., Canossa, A., Kuikkaniemi, K., Immich, T.: Bringing digital games to user research and user experience. In: Proceedings of the Entertainment Interfaces Track 2010 at Interaktive Kulturen 2010 ceur workshop proceedings, 12–15 September (2010)

    Google Scholar 

  6. Pagulayan, R., Keeker, K., Wixon, D., Romero, R.L., Fuller, T.: User-centered design in games. In: The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. L, pp. 883–906. Erlbaum Associates, New York, NY (2003)

    Google Scholar 

  7. Pagulayan, R., Steury, K.R., Fulton, B., Romero, R.L.: Designing for fun: user-testing case studies. In: Funology: From Usability to Enjoyment, pp. 137–150. Kluwer Academic Publishers, Norwell, MA (2004)

    Google Scholar 

  8. Desurvire, H., Caplan, M., Toth, J.A.: Using heuristics to evaluate the playability of games. In: CHI ’04 Extended Abstracts, pp. 1509–1512. ACM, Vienna (2004)

    Google Scholar 

  9. Korhonen, H., Koivisto, E.M.I.: Playability heuristics for mobile games. In: Proceedings of Conference on HCI with mobile devices and services, pp. 9–16. ACM, Espoo, Finland (2006)

    Google Scholar 

  10. Csikszentmihalyi, M.: Flow: the psychology of optimal experience, vol. 39, 1st edn. Harper Collins, New York (1990)

    Google Scholar 

  11. Sweetser, P., Wyeth, P.: GameFlow: a model for evaluating player enjoyment in games. Comput. Entertain. 3(3), 1–24 (2005)

    Article  Google Scholar 

  12. Brown, E., Cairns, P.: A grounded investigation of immersion in games. In: ACM Conference on Human Factors in Computing Systems, CHI 2004, pp. 1297–1300. ACM Press (2004)

    Google Scholar 

  13. Ermi, L., Mayra, F.: Fundamental components of the gameplay experience: analysing immersion. In: de Castell, S., Jenson, J. (eds.) Proceedings of Chancing Views – Worlds in Play. Digital Games Research Association’s Second International Conference, Vancouver (2005)

    Google Scholar 

  14. Gilleade, Κ.Μ., Dix, Α.: Using frustration in the design of adaptive videogames. In: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology – ACE ’04, pp. 228–232. (2004)

    Google Scholar 

  15. Funk, J.B., Pasold, T., Baumgardner, J.: How children experience playing video games. In: Proceedings of the Second International Conference on Entertainment Computing Pittsburgh, Carnegie Mellon University, pp. 1–14 (2003)

    Google Scholar 

  16. Bracken, C., Lange, R.L., Denny, J.: Online video games and gamers’ sensations of spatial, social, and copresence. FuturePlay 2005. East Lansing (2005)

    Google Scholar 

  17. Nacke, L., Lindley, C., Stellmach, S.: Log who’s playing: psychophysiological game analysis made easy through event logging. In: Proceedings of Fun and Games, 2nd International Conference, Eindhoven, The Netherlands, October 20–21, pp. 150–157. Springer (2008)

    Chapter  Google Scholar 

  18. Almeida, S.: The player and video game interplay in the gameplay experience construct. PhD, Universidade de Aveiro (2013)

    Google Scholar 

  19. Kirginas, S., Gouscos, D.: Development and validation of a questionnaire to measure perceptions of freedom of choice in digital games. Int. J. Serious Games (IJSG). 3(2), 29–45 (2016)

    Google Scholar 

  20. Almeida, S., Veloso, A., Roque, L., Mealha, O., Moura, A.: The video game and player in a gameplay experience model proposal. In: Proceedings of Videojogos 2013 – 6th Annual Conference in the Science and Art of Video Games. University of Coimbra, Coimbra, Portugal (2013)

    Google Scholar 

  21. Kirginas, S., Psaltis, A., Gouscos, D., Mourlas, C.: Studying children’s experience during free-form and formally structured gameplay. Int. J. Child-Comput. Interact. 28 (2021)

    Google Scholar 

  22. Kirginas, S., Gouscos, D.: Exploring the impact of free-form and structured digital games on the player experience of kindergarten and primary school students. In: Russell, D., Laffey, J. (eds.) Handbook of Research on Gaming Trends in P-12 Education, pp. 394–420. Hershey, PA, Information Science Reference (2016)

    Chapter  Google Scholar 

  23. Roto, V.: User Experience from Product Creation Perspective. Towards a UX Manifesto, pp. 31–34 (2007)

    Google Scholar 

  24. Lallemand, C.: Towards consolidated methods for the design and evaluation of user experience. Doctoral dissertation, University of Luxembourg (2015)

    Google Scholar 

  25. Mirza-Babaei, P.: Getting ahead of the game: challenges and methods in games user research. User Exp. Magaz. 15(2) (2015) https://uxpamagazine.org/getting-ahead-of-the-game/

  26. Neill, J.: Qualitative & Quantitative Research. http://wilderdom.com/research/QualitativeVersusQuantitativeResearch.html (2009)

    Google Scholar 

  27. Bird, M., Hammersley, Μ., Gomm, R., Woods, P.: Educational Research in Action/ Fragkou E: Translate in Greek. Educational Research in Practice-Study Manual, Patras (1999)

    Google Scholar 

  28. Cacioppo, J., Tassinary, L., Berntson, G.: Handbook of Psychophysiology, 3rd edn. Cambridge University Press, New York (2007)

    Google Scholar 

  29. Vermeeren, A., Lai-Chong Law, E., Roto, V., Obrist, M., Hoonhaut, J., Väänänen-Vainio-Mattila, K.: User experience evaluation methods: current state and development needs. In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries: NordiCHI ’10 (2010)

    Google Scholar 

  30. Fenko, A., Schifferstein, H.N.J., Hekkert, P.: Shifts in sensory dominance between various stages of user-product interactions. Appl. Ergon. 41, 34–40 (2010)

    Article  Google Scholar 

  31. Joyce, A.: Formative vs. Summative Evaluations [blog]. https://www.nngroup.com/articles/formative-vs-summative-evaluations/ (2019, July 28)

  32. Pellicone, A., Weintrop, D., Ketelhut, D.J., Shokeen, E., Cukier, M., Plane, J.D., Rahimian, F.: Playing aloud: leveraging game commentary culture for playtesting. Int. J. Gaming Comput.-Mediat. Simulat. (IJGCMS). 14(1), 1–16 (2022)

    Article  Google Scholar 

  33. Guss, C.D.: What is going through your mind? Thinking Aloud as a method in cross-cultural psychology. Front. Psychol. 9, 1292 (2018)

    Article  Google Scholar 

  34. Johnson, W.R., Artino, A.R., Jr., Durning, S.J.: Using the think aloud protocol in health professions education: an interview method for exploring thought processes: AMEE Guide No. 151. Medical teacher, 1–12. Advance online publication (2022)

    Google Scholar 

  35. Lundgren-Laine, H., Salantera, S.: Think-aloud technique and protocol analysis in clinical decision-making research. Qual. Health Res. 20(4), 565–575 (2010)

    Article  Google Scholar 

  36. Zhang, X., Simeone, A.L.: Using the think aloud protocol in an immersive virtual reality evaluation of a virtual twin. In: Proceedings of the 2022 ACM Symposium on Spatial User Interaction (SUI ’22), pp. 1–8. Association for Computing Machinery, New York, NY (2022)

    Google Scholar 

  37. Avouris, N., Katsanos, C., Tselios, N., Moustakas, K.: Introduction to Human-Computer Interaction [Undergraduate textbook]. Chapter 11. Kallipos, Open Academic Editions (2015)

    Google Scholar 

  38. Jordan, P.W.: An introduction to usability. Taylor & Francis, London (1998)

    Google Scholar 

  39. Hashemi Farzaneh, H., Neuner, L.: Usability evaluation of software tools for engineering design. In: Proceedings of the 22nd International Conference on Engineering Design (ICED19), Delft, The Netherlands, 5–8 August (2019)

    Google Scholar 

  40. Nova, A., Sansalone, S., Robinson, R., Mirza-Babaei, P.: Charting the uncharted with GUR: how AI playtesting can supplement expert evaluation. In: Proceedings of the 17th International Conference on the Foundations of Digital Games (FDG ’22). Association for Computing Machinery, New York, NY, Article 28, pp. 1–12 (2022)

    Google Scholar 

  41. Klas, C.: Expert evaluation methods. In: Dobreva, M., O’Dwyer, A., Feliciati, P. (eds.) User Studies for Digital Library Development, pp. 75–84. Facet (2012)

    Chapter  Google Scholar 

  42. Rajanen, M., Rajanen, D.: Heuristic evaluation in game and gamification development. In: Proceedings of GamiFin 2018 Conference, Pori (2018)

    Google Scholar 

  43. Allendoerfer, K., Aluker, S., Panjwani, G., Proctor, J., Sturtz, D., Vukovic, M., Chen, C.: Adapting the cognitive walkthrough method to assess the usability of a knowledge domain visualization. In: IEEE Symposium on Information Visualization, 2005. INFOVIS 2005, pp. 195–202. IEEE (2005)

    Chapter  Google Scholar 

  44. Farrell, D., Moffat, D.C.: Adapting cognitive walkthrough to support game based learning design. Int. J. Game-Based Learn. (IJGBL). 4(3), 23–34 (2014)

    Article  Google Scholar 

  45. Salazar, K.: Evaluate Interface Learnability with Cognitive Walkthroughs. [online]. https://www.nngroup.com/articles/cognitive-walkthroughs/ (2022)

  46. Farzandipour, M., Nabovati, E., Sadeqi Jabali, M.: Comparison of usability evaluation methods for a health information system: heuristic evaluation versus cognitive walkthrough method. BMC Med. Inf. Decis. Making. 22(1), 1–1 (2022)

    Google Scholar 

  47. Yanez-Gomez, R., Cascado-Caballero, D., Sevillano, J.L.: Academic methods for usability evaluation of serious games: a systematic review. Multimed. Tools Appl. 76, 5755–5784 (2017)

    Article  Google Scholar 

  48. Koutsabasis, P., Gardeli, A., Partheniadis, K., Vogiatzidakis, P., Nikolakopoulou, V., Chatzigrigoriou, P., Vosinakis, S.: Field playtesting with experts’ constructive interaction: an evaluation method for mobile games for cultural heritage. In: 2021 IEEE Conference on Games (CoG), pp. 1–9 (2021)

    Google Scholar 

  49. Korhonen, H.: Comparison of playtesting and expert review methods in mobile game evaluation. In: Fun and Games ’10: Proceedings of the 3rd International Conference on Fun and Games, pp. 18–27 (2010)

    Google Scholar 

  50. Drachen, A., Mirza-Babaei, P., Nacke, L.E.: Games User Research. Oxford University Press (2018)

    Book  Google Scholar 

  51. Carneiro, N., Darin, T., Pinheiro, M., Viana, W.: Using interviews to evaluate location-based games: lessons and challenges. J. Interact. Syst. 11(1), 125–138 (2020)

    Article  Google Scholar 

  52. Isbister, K., Schaffer, N.: Game usability: advice from the experts for advancing the player experience. Morgan Kaufmann Publishers, Burlington, MA (2008)

    Book  Google Scholar 

  53. Krueger, R.A., Casey, M.A.: Focus groups A practical guide for applied research. Sage Publications, Thousand Oaks (2000)

    Google Scholar 

  54. Bruhlmann, F., Mekler, E.D.: Surveys in games user research. In: Drachen, A., Mirza-Babaei, P., Nacke, L. (eds.) Games User Research, pp. 141–162. Oxford University Press, Oxford (2018)

    Chapter  Google Scholar 

  55. Rahman, M.M., Tabash, M.I., Salamzadeh, A., Abduli, S., Rahaman, M.S.: Sampling techniques (probability) for quantitative social science researchers: a conceptual guidelines with examples. Seeu Rev. 17(1), 42–51 (2022)

    Article  Google Scholar 

  56. Story, D.A., Tait, A.R.: Survey research. Anesthesiology. 130(2), 192–202 (2019)

    Article  Google Scholar 

  57. Hazan, E.: Contextualizing data. In: El-Nasr, M.S., et al. (eds.) Game analytics, pp. 477–496. Springer, London (2013)

    Chapter  Google Scholar 

  58. Rigby, S., Ryan, R.: The Player Experience of Need Satisfaction (PENS) Model, pp. 1–22. Immersyve Inc (2007)

    Google Scholar 

  59. Ryan, R., Rigby, S., Przybylski, A.: The motivational pull of video games: a self-determination theory approach. Motiv. Emot. (2006)

    Google Scholar 

  60. Denisova, A., Cairns, P., Guckelsberger, C., Zendle, D.: Measuring perceived challenge in digital games: development & validation of the challenge originating from recent gameplay interaction scale (CORGIS). Int. J. Hum.-Comput. Stud., 137 (2020)

    Google Scholar 

  61. Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., Walton, A.: Measuring and defining the experience of immersion in games. Int. J. Hum.-Comput. Stud. 66(9), 641–661 (2008)

    Article  Google Scholar 

  62. Isbister, K., Höök, K., Sharp, M., Laaksolahti, J.: The sensual evaluation instrument: developing an affective evaluation tool. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1163–1172 (2006)

    Google Scholar 

  63. Laaksolahti, J., Isbister, K., Höök, K.: Using the sensual evaluation instrument. Digit. Creativ. 20(3), 165–175 (2009)

    Article  Google Scholar 

  64. Ijsselsteijn, W.A., Poels, K., de Kort, Y.A.W.: The game experience questionnaire: development of a self-report measure to assess player experiences of digital games, deliverable 3.3. FUGA technical report, TU, Eindhoven, The Netherlands (2008)

    Google Scholar 

  65. Nacke, L.E., Grimshaw, M.N., Lindley, C.A.: More than a feeling: measurement of sonic user experience and psychophysiology in a first-person shooter game. Interact. Comput. 22(5), 336–343 (2010)

    Article  Google Scholar 

  66. Drachen, A., Nacke, L.E., Yannakakis, G., Lee Pedersen, A.: Correlation between heart rate, electrodermal activity and player experience in First-Person Shooter games. In: Spencer, S.N. (ed.) Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games, pp. 54–2010. ACM, Los Angeles, CA (49)

    Google Scholar 

  67. Ortega, M.C., Bruno, E., Richardson, M.P.: Electrodermal activity response during seizures: a systematic review and meta-analysis. Epilepsy Behav. 134, 108864 (2022)

    Article  Google Scholar 

  68. Paloniemi, S., Penttonen, M., Eteläpelto, A., Hökkä, P., Vähäsantanen, K.: Integrating self-reports and electrodermal activity (EDA) measurement in studying emotions in professional learning. In: Methods for Researching Professional Learning and Development, pp. 87–109. Springer, Cham (2022)

    Chapter  Google Scholar 

  69. Richter, H., Krukewitt, L., Müller-Graf, F., Zitzmann, A., Merz, J., Böhm, S., Kühn, V.: High resolution EIT based heart rate detection using Synchrosqueezing (2022)

    Google Scholar 

  70. Thissen, B.A., Schlotz, W., Abel, C., Scharinger, M., Frieler, K., Merrill, J., Haider, T., Menninghaus, W.: At the heart of optimal reading experiences: cardiovascular activity and flow experiences in fiction reading. Read. Res. Q. 57(3), 831–845 (2022)

    Article  Google Scholar 

  71. Fuentes del Toro, S., Wei, Y., Olmeda, E., Ren, L., Guowu, W., Díaz, V.: Validation of a low-cost electromyography (EMG) system via a commercial and accurate EMG device: pilot study. Sensors. 19(23), 5214 (2019)

    Article  Google Scholar 

  72. Tortora, G., Derrickson, B.: Principles of Anatomy and Physiology, 12th edn. Biological Sciences Textbook (2009)

    Google Scholar 

  73. Ramos, A.L.A., Dadiz, B.G., Santos, A.B.G.: Classifying emotion based on facial expression analysis using Gabor filter: a basis for adaptive effective teaching strategy. In: Computational Science and Technology, pp. 469–479. Springer, Singapore (2020)

    Chapter  Google Scholar 

  74. Westover, M.B., Gururangan, K., Markert, M.S., Blond, B.N., Lai, S., Benard, S., et al.: Diagnostic value of electroencephalography with ten electrodes in critically ill patients. Neurocrit. Care. 33(2), 479–490 (2020)

    Article  Google Scholar 

  75. Masood, K., Alghamdi, M.A.: Modeling mental stress using a deep learning framework. IEEE Access. 7, 68446–68454 (2019)

    Article  Google Scholar 

  76. Read, J.C., MacFarlane, S.J.: Using the Fun Toolkit and other survey methods to gather opinions in child computer interaction. In: Interaction Design and Children, IDC2006. ACM Press, Tampere (2006)

    Google Scholar 

  77. Lang, P.J.: The cognitive psychophysiology of emotion: fear and anxiety. In: Tuma, A.H., Maser, J.D. (eds.) Anxiety and the Anxiety Disorders, pp. 131–170. Lawrence Erlbaum Associates (1985)

    Google Scholar 

  78. Kujala, S., Roto, V., Mattila, K., Karapanos, E., Sinnela, A.: UX curve: a method for evaluating long-term user experience. Interact. Comput. 23, 473–483 (2011)

    Article  Google Scholar 

  79. Vissers, J., De Bot, L., Zaman, B.: MemoLine: evaluating long-term UX with children. In: Proceedings of the 12th International Conference on Interaction Design and Children, New York, pp. 285–288 (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sotiris Kirginas .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Kirginas, S. (2023). User Experience Evaluation Methods for Games in Serious Contexts. In: Cooper, K.M.L., Bucchiarone, A. (eds) Software Engineering for Games in Serious Contexts. Springer, Cham. https://doi.org/10.1007/978-3-031-33338-5_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-33338-5_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-33337-8

  • Online ISBN: 978-3-031-33338-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics