Abstract
User experience in digital games can be influenced by many factors such as flow [Csikszentmihalyi (Flow: the psychology of optimal experience. Harper Collins, 1990), Sweetser and Wyeth (Computers in Entertainment 3(3):1–24, 2005)], immersion [Brown and Cairns (ACM Conference on Human Factors in Computing Systems, CHI 2004, ACM Press, 2004), Ermi and Mayra (Proceedings of Chancing Views – Worlds in Play. Digital Games Research Association’s Second International Conference, 2005)], frustration or tension [Gilleade and Dix (Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology – ACE ’04, 2004)], psychological absorption [Funk et al. (Proceedings of the Second International Conference on Entertainment Computing Pittsburgh, Carnegie Mellon University, 2003)], and social game context [Bracken et al. (Online video games and gamers’ sensations of spatial, social, and copresence. FuturePlay 2005, 2005)]. Most of these factors should be present in a digital game in order to provide the optimal gaming experience [Kirginas (Contemporary Educational Technology 14(2):ep351, 2022), Kirginas et al. (International Journal of Child-Computer Interaction 28, 2021), Kirginas and Gouscos (The International Journal of Serious Games 4:53–69, 2017; International Journal of Serious Games 3:29–45, 2016)]. As there are many different game genres, sub-genres, and game types, user experience needs to be explored in more detail in research studies. This need is even greater when we talk about serious games. User experience is a multifactorial concept that is difficult to measure. This chapter aims to present a range of quantitative and qualitative/objective and subjective/short-term and long-term/formative and summative methods that can be used to evaluate users’ experience in serious games during and after the development process. It is also intended to provide insight into when the different user experience assessment methodologies should be employed in the development cycle.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Nacke, L.E.: Affective ludology: scientific measurement of user experience, interactive entertainment. Blekinge Institute of Technology (2009)
Mandryk, R.L., Atkins, M.S., Inkpen, K.M.A.: Continuous and objective evaluation of emotional experience with interactive play environments. In: Proceedings of CHI 2006, Montréal, Québec, Canada, April 2006, pp. 1027–1036. ACM (2006)
Drachen, A., Canossa, A.: Towards gameplay analysis via gameplay metrics. In: Proceedings of MindTrek, Tampere, Finland, October 1–2. ACM (2009)
Kim, J.H., Gunn, D.V., Schuh, E., Phillips, B., Pagulayan, R.J., Wixon, D.: Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In: Proceedings of CHI 2008, pp. 443–452. ACM, Florence, Italy (2008)
Nacke, L., Niesenhaus, J., Engl, S., Canossa, A., Kuikkaniemi, K., Immich, T.: Bringing digital games to user research and user experience. In: Proceedings of the Entertainment Interfaces Track 2010 at Interaktive Kulturen 2010 ceur workshop proceedings, 12–15 September (2010)
Pagulayan, R., Keeker, K., Wixon, D., Romero, R.L., Fuller, T.: User-centered design in games. In: The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. L, pp. 883–906. Erlbaum Associates, New York, NY (2003)
Pagulayan, R., Steury, K.R., Fulton, B., Romero, R.L.: Designing for fun: user-testing case studies. In: Funology: From Usability to Enjoyment, pp. 137–150. Kluwer Academic Publishers, Norwell, MA (2004)
Desurvire, H., Caplan, M., Toth, J.A.: Using heuristics to evaluate the playability of games. In: CHI ’04 Extended Abstracts, pp. 1509–1512. ACM, Vienna (2004)
Korhonen, H., Koivisto, E.M.I.: Playability heuristics for mobile games. In: Proceedings of Conference on HCI with mobile devices and services, pp. 9–16. ACM, Espoo, Finland (2006)
Csikszentmihalyi, M.: Flow: the psychology of optimal experience, vol. 39, 1st edn. Harper Collins, New York (1990)
Sweetser, P., Wyeth, P.: GameFlow: a model for evaluating player enjoyment in games. Comput. Entertain. 3(3), 1–24 (2005)
Brown, E., Cairns, P.: A grounded investigation of immersion in games. In: ACM Conference on Human Factors in Computing Systems, CHI 2004, pp. 1297–1300. ACM Press (2004)
Ermi, L., Mayra, F.: Fundamental components of the gameplay experience: analysing immersion. In: de Castell, S., Jenson, J. (eds.) Proceedings of Chancing Views – Worlds in Play. Digital Games Research Association’s Second International Conference, Vancouver (2005)
Gilleade, Κ.Μ., Dix, Α.: Using frustration in the design of adaptive videogames. In: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology – ACE ’04, pp. 228–232. (2004)
Funk, J.B., Pasold, T., Baumgardner, J.: How children experience playing video games. In: Proceedings of the Second International Conference on Entertainment Computing Pittsburgh, Carnegie Mellon University, pp. 1–14 (2003)
Bracken, C., Lange, R.L., Denny, J.: Online video games and gamers’ sensations of spatial, social, and copresence. FuturePlay 2005. East Lansing (2005)
Nacke, L., Lindley, C., Stellmach, S.: Log who’s playing: psychophysiological game analysis made easy through event logging. In: Proceedings of Fun and Games, 2nd International Conference, Eindhoven, The Netherlands, October 20–21, pp. 150–157. Springer (2008)
Almeida, S.: The player and video game interplay in the gameplay experience construct. PhD, Universidade de Aveiro (2013)
Kirginas, S., Gouscos, D.: Development and validation of a questionnaire to measure perceptions of freedom of choice in digital games. Int. J. Serious Games (IJSG). 3(2), 29–45 (2016)
Almeida, S., Veloso, A., Roque, L., Mealha, O., Moura, A.: The video game and player in a gameplay experience model proposal. In: Proceedings of Videojogos 2013 – 6th Annual Conference in the Science and Art of Video Games. University of Coimbra, Coimbra, Portugal (2013)
Kirginas, S., Psaltis, A., Gouscos, D., Mourlas, C.: Studying children’s experience during free-form and formally structured gameplay. Int. J. Child-Comput. Interact. 28 (2021)
Kirginas, S., Gouscos, D.: Exploring the impact of free-form and structured digital games on the player experience of kindergarten and primary school students. In: Russell, D., Laffey, J. (eds.) Handbook of Research on Gaming Trends in P-12 Education, pp. 394–420. Hershey, PA, Information Science Reference (2016)
Roto, V.: User Experience from Product Creation Perspective. Towards a UX Manifesto, pp. 31–34 (2007)
Lallemand, C.: Towards consolidated methods for the design and evaluation of user experience. Doctoral dissertation, University of Luxembourg (2015)
Mirza-Babaei, P.: Getting ahead of the game: challenges and methods in games user research. User Exp. Magaz. 15(2) (2015) https://uxpamagazine.org/getting-ahead-of-the-game/
Neill, J.: Qualitative & Quantitative Research. http://wilderdom.com/research/QualitativeVersusQuantitativeResearch.html (2009)
Bird, M., Hammersley, Μ., Gomm, R., Woods, P.: Educational Research in Action/ Fragkou E: Translate in Greek. Educational Research in Practice-Study Manual, Patras (1999)
Cacioppo, J., Tassinary, L., Berntson, G.: Handbook of Psychophysiology, 3rd edn. Cambridge University Press, New York (2007)
Vermeeren, A., Lai-Chong Law, E., Roto, V., Obrist, M., Hoonhaut, J., Väänänen-Vainio-Mattila, K.: User experience evaluation methods: current state and development needs. In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries: NordiCHI ’10 (2010)
Fenko, A., Schifferstein, H.N.J., Hekkert, P.: Shifts in sensory dominance between various stages of user-product interactions. Appl. Ergon. 41, 34–40 (2010)
Joyce, A.: Formative vs. Summative Evaluations [blog]. https://www.nngroup.com/articles/formative-vs-summative-evaluations/ (2019, July 28)
Pellicone, A., Weintrop, D., Ketelhut, D.J., Shokeen, E., Cukier, M., Plane, J.D., Rahimian, F.: Playing aloud: leveraging game commentary culture for playtesting. Int. J. Gaming Comput.-Mediat. Simulat. (IJGCMS). 14(1), 1–16 (2022)
Guss, C.D.: What is going through your mind? Thinking Aloud as a method in cross-cultural psychology. Front. Psychol. 9, 1292 (2018)
Johnson, W.R., Artino, A.R., Jr., Durning, S.J.: Using the think aloud protocol in health professions education: an interview method for exploring thought processes: AMEE Guide No. 151. Medical teacher, 1–12. Advance online publication (2022)
Lundgren-Laine, H., Salantera, S.: Think-aloud technique and protocol analysis in clinical decision-making research. Qual. Health Res. 20(4), 565–575 (2010)
Zhang, X., Simeone, A.L.: Using the think aloud protocol in an immersive virtual reality evaluation of a virtual twin. In: Proceedings of the 2022 ACM Symposium on Spatial User Interaction (SUI ’22), pp. 1–8. Association for Computing Machinery, New York, NY (2022)
Avouris, N., Katsanos, C., Tselios, N., Moustakas, K.: Introduction to Human-Computer Interaction [Undergraduate textbook]. Chapter 11. Kallipos, Open Academic Editions (2015)
Jordan, P.W.: An introduction to usability. Taylor & Francis, London (1998)
Hashemi Farzaneh, H., Neuner, L.: Usability evaluation of software tools for engineering design. In: Proceedings of the 22nd International Conference on Engineering Design (ICED19), Delft, The Netherlands, 5–8 August (2019)
Nova, A., Sansalone, S., Robinson, R., Mirza-Babaei, P.: Charting the uncharted with GUR: how AI playtesting can supplement expert evaluation. In: Proceedings of the 17th International Conference on the Foundations of Digital Games (FDG ’22). Association for Computing Machinery, New York, NY, Article 28, pp. 1–12 (2022)
Klas, C.: Expert evaluation methods. In: Dobreva, M., O’Dwyer, A., Feliciati, P. (eds.) User Studies for Digital Library Development, pp. 75–84. Facet (2012)
Rajanen, M., Rajanen, D.: Heuristic evaluation in game and gamification development. In: Proceedings of GamiFin 2018 Conference, Pori (2018)
Allendoerfer, K., Aluker, S., Panjwani, G., Proctor, J., Sturtz, D., Vukovic, M., Chen, C.: Adapting the cognitive walkthrough method to assess the usability of a knowledge domain visualization. In: IEEE Symposium on Information Visualization, 2005. INFOVIS 2005, pp. 195–202. IEEE (2005)
Farrell, D., Moffat, D.C.: Adapting cognitive walkthrough to support game based learning design. Int. J. Game-Based Learn. (IJGBL). 4(3), 23–34 (2014)
Salazar, K.: Evaluate Interface Learnability with Cognitive Walkthroughs. [online]. https://www.nngroup.com/articles/cognitive-walkthroughs/ (2022)
Farzandipour, M., Nabovati, E., Sadeqi Jabali, M.: Comparison of usability evaluation methods for a health information system: heuristic evaluation versus cognitive walkthrough method. BMC Med. Inf. Decis. Making. 22(1), 1–1 (2022)
Yanez-Gomez, R., Cascado-Caballero, D., Sevillano, J.L.: Academic methods for usability evaluation of serious games: a systematic review. Multimed. Tools Appl. 76, 5755–5784 (2017)
Koutsabasis, P., Gardeli, A., Partheniadis, K., Vogiatzidakis, P., Nikolakopoulou, V., Chatzigrigoriou, P., Vosinakis, S.: Field playtesting with experts’ constructive interaction: an evaluation method for mobile games for cultural heritage. In: 2021 IEEE Conference on Games (CoG), pp. 1–9 (2021)
Korhonen, H.: Comparison of playtesting and expert review methods in mobile game evaluation. In: Fun and Games ’10: Proceedings of the 3rd International Conference on Fun and Games, pp. 18–27 (2010)
Drachen, A., Mirza-Babaei, P., Nacke, L.E.: Games User Research. Oxford University Press (2018)
Carneiro, N., Darin, T., Pinheiro, M., Viana, W.: Using interviews to evaluate location-based games: lessons and challenges. J. Interact. Syst. 11(1), 125–138 (2020)
Isbister, K., Schaffer, N.: Game usability: advice from the experts for advancing the player experience. Morgan Kaufmann Publishers, Burlington, MA (2008)
Krueger, R.A., Casey, M.A.: Focus groups A practical guide for applied research. Sage Publications, Thousand Oaks (2000)
Bruhlmann, F., Mekler, E.D.: Surveys in games user research. In: Drachen, A., Mirza-Babaei, P., Nacke, L. (eds.) Games User Research, pp. 141–162. Oxford University Press, Oxford (2018)
Rahman, M.M., Tabash, M.I., Salamzadeh, A., Abduli, S., Rahaman, M.S.: Sampling techniques (probability) for quantitative social science researchers: a conceptual guidelines with examples. Seeu Rev. 17(1), 42–51 (2022)
Story, D.A., Tait, A.R.: Survey research. Anesthesiology. 130(2), 192–202 (2019)
Hazan, E.: Contextualizing data. In: El-Nasr, M.S., et al. (eds.) Game analytics, pp. 477–496. Springer, London (2013)
Rigby, S., Ryan, R.: The Player Experience of Need Satisfaction (PENS) Model, pp. 1–22. Immersyve Inc (2007)
Ryan, R., Rigby, S., Przybylski, A.: The motivational pull of video games: a self-determination theory approach. Motiv. Emot. (2006)
Denisova, A., Cairns, P., Guckelsberger, C., Zendle, D.: Measuring perceived challenge in digital games: development & validation of the challenge originating from recent gameplay interaction scale (CORGIS). Int. J. Hum.-Comput. Stud., 137 (2020)
Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., Walton, A.: Measuring and defining the experience of immersion in games. Int. J. Hum.-Comput. Stud. 66(9), 641–661 (2008)
Isbister, K., Höök, K., Sharp, M., Laaksolahti, J.: The sensual evaluation instrument: developing an affective evaluation tool. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1163–1172 (2006)
Laaksolahti, J., Isbister, K., Höök, K.: Using the sensual evaluation instrument. Digit. Creativ. 20(3), 165–175 (2009)
Ijsselsteijn, W.A., Poels, K., de Kort, Y.A.W.: The game experience questionnaire: development of a self-report measure to assess player experiences of digital games, deliverable 3.3. FUGA technical report, TU, Eindhoven, The Netherlands (2008)
Nacke, L.E., Grimshaw, M.N., Lindley, C.A.: More than a feeling: measurement of sonic user experience and psychophysiology in a first-person shooter game. Interact. Comput. 22(5), 336–343 (2010)
Drachen, A., Nacke, L.E., Yannakakis, G., Lee Pedersen, A.: Correlation between heart rate, electrodermal activity and player experience in First-Person Shooter games. In: Spencer, S.N. (ed.) Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games, pp. 54–2010. ACM, Los Angeles, CA (49)
Ortega, M.C., Bruno, E., Richardson, M.P.: Electrodermal activity response during seizures: a systematic review and meta-analysis. Epilepsy Behav. 134, 108864 (2022)
Paloniemi, S., Penttonen, M., Eteläpelto, A., Hökkä, P., Vähäsantanen, K.: Integrating self-reports and electrodermal activity (EDA) measurement in studying emotions in professional learning. In: Methods for Researching Professional Learning and Development, pp. 87–109. Springer, Cham (2022)
Richter, H., Krukewitt, L., Müller-Graf, F., Zitzmann, A., Merz, J., Böhm, S., Kühn, V.: High resolution EIT based heart rate detection using Synchrosqueezing (2022)
Thissen, B.A., Schlotz, W., Abel, C., Scharinger, M., Frieler, K., Merrill, J., Haider, T., Menninghaus, W.: At the heart of optimal reading experiences: cardiovascular activity and flow experiences in fiction reading. Read. Res. Q. 57(3), 831–845 (2022)
Fuentes del Toro, S., Wei, Y., Olmeda, E., Ren, L., Guowu, W., Díaz, V.: Validation of a low-cost electromyography (EMG) system via a commercial and accurate EMG device: pilot study. Sensors. 19(23), 5214 (2019)
Tortora, G., Derrickson, B.: Principles of Anatomy and Physiology, 12th edn. Biological Sciences Textbook (2009)
Ramos, A.L.A., Dadiz, B.G., Santos, A.B.G.: Classifying emotion based on facial expression analysis using Gabor filter: a basis for adaptive effective teaching strategy. In: Computational Science and Technology, pp. 469–479. Springer, Singapore (2020)
Westover, M.B., Gururangan, K., Markert, M.S., Blond, B.N., Lai, S., Benard, S., et al.: Diagnostic value of electroencephalography with ten electrodes in critically ill patients. Neurocrit. Care. 33(2), 479–490 (2020)
Masood, K., Alghamdi, M.A.: Modeling mental stress using a deep learning framework. IEEE Access. 7, 68446–68454 (2019)
Read, J.C., MacFarlane, S.J.: Using the Fun Toolkit and other survey methods to gather opinions in child computer interaction. In: Interaction Design and Children, IDC2006. ACM Press, Tampere (2006)
Lang, P.J.: The cognitive psychophysiology of emotion: fear and anxiety. In: Tuma, A.H., Maser, J.D. (eds.) Anxiety and the Anxiety Disorders, pp. 131–170. Lawrence Erlbaum Associates (1985)
Kujala, S., Roto, V., Mattila, K., Karapanos, E., Sinnela, A.: UX curve: a method for evaluating long-term user experience. Interact. Comput. 23, 473–483 (2011)
Vissers, J., De Bot, L., Zaman, B.: MemoLine: evaluating long-term UX with children. In: Proceedings of the 12th International Conference on Interaction Design and Children, New York, pp. 285–288 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Kirginas, S. (2023). User Experience Evaluation Methods for Games in Serious Contexts. In: Cooper, K.M.L., Bucchiarone, A. (eds) Software Engineering for Games in Serious Contexts. Springer, Cham. https://doi.org/10.1007/978-3-031-33338-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-33338-5_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-33337-8
Online ISBN: 978-3-031-33338-5
eBook Packages: Computer ScienceComputer Science (R0)