Skip to main content
SpringerLink
Log in

Operationalization and Measurement of Evaluation Constructs

  • Chapter
  • First Online:
Entertainment Computing and Serious Games

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9970))

Abstract

This chapter deals with the operationalization and measurement of evaluation constructs, an important and challenging part of the serious games evaluation process. Hereby, advices will be given on what has to be measured and how to quantify an abstract concept. Thus, the chapter makes two main contributions. First, general data gathering methods are described and discussed in terms of advantages and disadvantages. Second, main psychological concepts and evaluation constructs relevant in the context of serious games, as well as their theoretical foundations, are introduced. In order to support the reader on planning future serious game evaluations, a list and description of concrete techniques and questionnaires addressing concepts like motivation, player experience, learning outcomes, health, well-being, and attitudes are compiled.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.act.org/content/act/en.html.

  2. 2.

    http://cae.org/participating-institutions/cla-references.

  3. 3.

    https://www.ets.org/proficiencyprofile/about.

References

  1. Aronson, E., Wilson, T.D., Akert, R.M.: Social Psychology, 7th edn. Pearson, Boston (2010). Global edn

    Google Scholar 

  2. Bandura, A.: Guide for constructing self-efficacy scales. In: Pajares, F., Urdan, T.C. (eds.) Self-Efficacy Beliefs of Adolescents, pp. 307–337. Information Age Publishing, Charlotte (2006)

    Google Scholar 

  3. Bartle, R.: Hearts, clubs, diamonds, spades: players who suit muds. J. Virtual Environ. 1(1) (1996). http://mud.co.uk/richard/hcds.htm

  4. Bartle, R.A.: Designing Virtual Worlds. New Riders Pub., Indianapolis (2004)

    Google Scholar 

  5. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum. Comput. Interact. 2013, 1–11 (2013)

    Google Scholar 

  6. Bernhaupt, R. (ed.): Evaluating User Experience in Games: Concepts and Methods. Human-Computer Interaction Series. Springer, London, New York (2010)

    Google Scholar 

  7. Bernhaupt, R.: User experience evaluation in entertainment. In: Bernhaupt, R. (ed.) Evaluating User Experience in Games. Human-ComputerInteraction Series, pp. 3–7. Springer, London, New York (2010)

    Chapter  Google Scholar 

  8. Biocca, F., Harms, C., Burgoon, J.: Towards a more robust theory and measure of social presence: Review and suggested criteria. Presence Teleoper. Virtual Environ. 12(5), 456–480 (2003)

    Article  Google Scholar 

  9. Bircher, J., Kuruvilla, S.: Defining health by addressing individual, social, and environmental determinants: new opportunities for health care and public health. J. Public Health Policy 35(3), 363–386 (2014). http://www.palgrave-journals.com/jphp/journal/v35/n3/pdf/jphp.201419a.pdf

    Article  Google Scholar 

  10. Campbell, D.T., Stanley, J.C., Gage, N.L.: Experimental and Quasi-Experimental Designs for Research. Ravenio books, New York (2015)

    Google Scholar 

  11. Chin, J., Dukes, R., Gamson, W.: Assessment in simulation and gaming a review of the last 40 years. Simul. Gaming 40(4), 553–568 (2009)

    Article  Google Scholar 

  12. Connolly, T., Stansfield, M., Hainey, T.: Towards the development of a games-based learning evaluation framework. In: Games-Based Learning Advancements for Multisensory Human Computer Interfaces: Techniques and Effective Practices (2009)

    Google Scholar 

  13. Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper Perennial, New York (1990)

    Google Scholar 

  14. De Kort, Y.A.W., Ijsselsteijn, W.A.: People, places, and play: player experience in a socio-spatial context. Comput. Entertain. 6(2), 18 (2008)

    Google Scholar 

  15. De Kort, Y.A.W., Ijsselsteijn, W.A., Poels, K.: Digital games as social presence technology: development of the social presence in gaming questionnaire (SPGQ). In: Proceedings of the 10th Annual International Workshop on Presence, pp. 195–203 (2007)

    Google Scholar 

  16. Dignan, A.: Game frame: Using Games as a Strategy for Success. Free Press, New York (2011). 1st free press hardcover edn

    Google Scholar 

  17. Dillon, R.: On The Way to Fun: An Emotion-Based Approach to Successful Game Design. A K Peters, Natick (2010)

    Book  Google Scholar 

  18. Dodge, R., Daly, A., Huyton, J., Sanders, L.: The challenge of defining wellbeing. Int. J. Wellbeing 2(3), 222–235 (2012)

    Article  Google Scholar 

  19. Dugard, P., Todman, J.: Analysis of pre-test-post-test control group designs in educational research. Educ. Psychol. 15(2), 181–198 (1995)

    Article  Google Scholar 

  20. Ermi, L., Mäyrä, F.: Fundamental components of the gameplay experience: Analysing immersion. In: DiGRA 2005, Proceedings of the 2005 DiGRA International Conference: Changing Views: Worlds in Play (2005). http://www.digra.org/wp-content/uploads/digital-library/06276.41516.pdf

  21. Fernandez, A.: Fun experience with digital games: a model proposition. In: Leino, O., Wirman, H., Fernandez, A. (eds.) Extending Experiences, pp. 181–190. Lapland University Press, Rovaniemi (2008)

    Google Scholar 

  22. Ferrara, J.: Playful Design: Creating Game Experiences in Everyday Interfaces. Rosenfeld Media, Brooklyn (2012)

    Google Scholar 

  23. Garfield, S., Clifford, S., Eliasson, L., Barber, N., Willson, A.: Suitability of measures of self-reported medication adherence for routine clinical use: a systematic review. BMC Med. Res. Methodol. 11, 149 (2011)

    Article  Google Scholar 

  24. Garris, R., Ahlers, R., Driskell, J.E.: Games, motivation, and learning: a research and practice model. Simul. Gaming 33(4), 441–467 (2002)

    Article  Google Scholar 

  25. Gee, J.P.: What video games have to teach us about learning and literacy. Comput. Entertain. 1(1), 20 (2003)

    Article  Google Scholar 

  26. Gekker, A.: Health games. In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 13–30. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33687-4_2

    Chapter  Google Scholar 

  27. Gill, P., Stewart, K., Treasure, E., Chadwick, B.: Methods of data collection in qualitative research: interviews and focus groups. Br. Dent. J. 204(6), 291–295 (2008)

    Article  Google Scholar 

  28. Girard, C., Ecalle, J., Magnan, A.: Serious games as new educational tools: how effective are they? a meta-analysis of recent studies. J. Comput. Assist. Learn. 29(3), 207–219 (2013)

    Article  Google Scholar 

  29. Goldstein, G., Hersen, M.: Handbook of Psychological Assessment. Elsevier, Hoboken (2000)

    Google Scholar 

  30. Goodwin, C.J., Goodwin, K.A.: Research in Psychology: Methods and Design, 7th edn. Wiley, Hoboken (2012)

    Google Scholar 

  31. Greenwald, A.G., McGhee, D.E., Schwartz, J.L.: Measuring individual differences in implicit cognition: the implicit association test. J. Person. Soc. Psychol. 74(6), 1464–1480 (1998)

    Article  Google Scholar 

  32. Guilford, J.P.: Psychometric Methods, 2nd edn. McGraw-Hill, New York (1954)

    Google Scholar 

  33. Hudson, M., Cairns, P.: Measuring social presence in team-based digital games. In: Riva, G., Waterworth, J., Murray, D. (eds.) Interacting with Presence: HCI and the Sense of Presence in Computer-Mediated Environments. DE GRUYTER OPEN, Warsaw, Poland (2014)

    Google Scholar 

  34. Igroup Project Consortium: Igroup presence questionnaire (ipq)

    Google Scholar 

  35. IJsselsteijn, W., de Kort, Y., Poels, K., Jurgelionis, A., Bellotti, F: Characterising and measuring user experiences in digital games. In: ACE 2007 International Conference on Advances in Computer Entertainment Technology, Workshop ‘Methods for Evaluating Games - How to measure Usability and User Experience in Games’ (2007)

    Google Scholar 

  36. Ijsselsteijn, W.A., De Kort, Y.A.W., Poels, K.: The game experience questionnaire: Development of a self-report measure to assess the psychological impact of digital games. Manuscript in preparation

    Google Scholar 

  37. Kahneman, D., Krueger, A.B.: Developments in the measurement of subjective well-being. J. Econ. Perspect. 20(1), 3–24 (2006)

    Article  Google Scholar 

  38. Kato, P.M., Cole, S.W., Bradlyn, A.S., Pollock, B.H.: A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics 122(2), e305–e317 (2008). 18676516

    Article  Google Scholar 

  39. Kirkpatrick, D.L.: Evaluating Training Programs. Tata McGraw-Hill Education, San Francisco (1975)

    Google Scholar 

  40. Kivikangas, J.M., Chanel, G., Cowley, B., Ekman, I., Salminen, M., Järvelä, S., Ravaja, N.: A review of the use of psychophysiological methods in game research. J. Gaming Virtual World 3(3), 181–199 (2011)

    Article  Google Scholar 

  41. Kline, P.: Psychometrics and Psychology. Academy Press, Washington, D.C. (1979)

    Google Scholar 

  42. Koster, R.: A Theory of Fun for Game Design, 2nd edn. O’Reilly Media Inc., Sebastopol (2014)

    Google Scholar 

  43. Krosnick, J.A., Presser, S.: Question and questionnaire design. In: Marsden, P.V. (ed.) Handbook of Survey Research. Emerald Group Publishing, Bingley (2010)

    Google Scholar 

  44. Landers, R.N., Bauer, K.N.: Quantitative methods and analyses for the study of players and their behaviour. In: Lankoski, P., Björk, S. (eds.) Game Research Methods, pp. 151–174. ETC Press, Pittsburgh (2015)

    Google Scholar 

  45. Law, E.L.C., Roto, V., Hassenzahl, M., Vermeeren, A.P., Kort, J.: Understanding, scoping and defining user experience. In: Olsen, D.R., Arthur, R.B., Hinckley, K., Morris, M.R., Hudson, S., Greenberg, S. (eds.) The SIGCHI Conference, p. 719 (2009)

    Google Scholar 

  46. Lazzaro, N.: Why we play games: Four keys to more emotion without story (2004)

    Google Scholar 

  47. Loh, C.S., Anantachai, A., Byun, J., Lenox, J.: Assessing what players learned in serious games: in situ data collection, information trails, and quantitative analysis. In: 10th International Conference on Computer Games: AI, Animation, Mobile, Educational & Serious Games (CGAMES 2007), pp. 25–28 (2007)

    Google Scholar 

  48. Loh, C.S., Sheng, Y., Ifenthaler, D. (eds.): Serious Games Analytics: Methodologies for Performance Measurement, Assessment, and Improvement. Springer, Heidelberg (2015)

    Google Scholar 

  49. Maki, P.L.: Assessing for Learning: Building a Sustainable Commitment Across the Institution. Stylus Publishing, LLC, Menlo Park (2012)

    Google Scholar 

  50. Mandryk, R.L., Inkpen, K.M.: Physiological indicators for the evaluation of co-located collaborative play. In: Herbsleb, J., Olson, G. (eds.) The 2004 ACM Conference, p. 102 (2004)

    Google Scholar 

  51. McAuley, E., Duncan, T., Tammen, V.V.: Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: a confirmatory factor analysis. Res. Q. Exerc. Sport 60(1), 48–58 (1989)

    Article  Google Scholar 

  52. Nacke, L.E., Bateman, C., Mandryk, R.L.: BrainHex: preliminary results from a neurobiological gamer typology survey. In: Anacleto, J.C., Fels, S., Graham, N., Kapralos, B., Saif El-Nasr, M., Stanley, K. (eds.) ICEC 2011. LNCS, vol. 6972, pp. 288–293. Springer, Heidelberg (2011). doi:10.1007/978-3-642-24500-8_31

    Chapter  Google Scholar 

  53. Nacke, L.E.: Affective ludology: scientific measurement of user experience in interactive entertainment, Blekinge Institute of Technology doctoral dissertation series, vol. 2009: 04. School of Computing, Blekinge Institute of Technology, Karlskrona (2010)

    Google Scholar 

  54. O’Neil, H.F., Wainess, R., Baker, E.L.: Classification of learning outcomes: evidence from the computer games literature. Cirriculum J. 16(4), 455–474 (2005)

    Article  Google Scholar 

  55. Plass, J.L., Homer, B.D., Kinzer, C.K., Chang, Y.K., Frye, J., Kaczetow, W., Isbister, K., Perlin, K.: Metrics in simulations and games for learning. In: El-Nasr, M.S., Drachen, A., Canossa, A. (eds.) Game Analytics, pp. 697–729. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  56. Poels, K., de Kort, Y., Ijsselsteijn, W.: It is always a lot of fun! In: Kapralos, B., Katchabaw, M., Rajnovich, J. (eds.) The 2007 Conference, p. 83 (2007)

    Google Scholar 

  57. Rammstedt, B., Kemper, C.J., Klein, M.C., Beierlein, C., Kovaleva, A.: A short scale for assessing the big five dimensions of personality

    Google Scholar 

  58. Ravaja, N., Saari, T., Turpeinen, M., Laarni, J., Salminen, M., Kivikangas, M.: Spatial presence and emotions during video game playing: does it matter with whom you play? Presence Teleoper. Virtual Environ. 15(4), 381–392 (2006)

    Article  Google Scholar 

  59. Rheinberg, F.: Motivation, Kohlhammer-Urban-Taschenbücher, vol. 555. Kohlhammer, Stuttgart, 7, aktualisierte aufl. edn. (2008)

    Google Scholar 

  60. Ryan, D.: Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp. Educ. Psychol. 25(1), 54–67 (2000)

    Article  Google Scholar 

  61. Ryan, R.M., Rigby, C.S., Przybylski, A.: The motivational pull of video games: a self-determination theory approach. Motiv. Emot. 30(4), 344–360 (2006)

    Article  Google Scholar 

  62. Sadler, D.: Formative assessment and the design of instructional systems. Instr. Sci. 18(2), 119–144 (1989). http://dx.doi.org/10.1007/BF00117714

    Article  Google Scholar 

  63. Sano, A., Picard, R.W.: Stress recognition using wearable sensors and mobile phones. In: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 671–676 (2013)

    Google Scholar 

  64. Shute, V.J., Ventura, M., Bauer, M., Zapata-Rivera, D.: Melding the power of serious games and embedded assessment to monitor and foster learning. Serious Games Mech. Effects 2, 295–321 (2009)

    Google Scholar 

  65. Slater, M.: A note on presence terminology. Presence Connect 3(3), 1–5 (2003)

    MathSciNet  Google Scholar 

  66. Spielberger, C.D.: State-trait anxiety inventory. In: Weiner, I.B., Craighead, W.E. (eds.) The Corsini Encyclopedia of Psychology. Wiley, Hoboken (2010)

    Google Scholar 

  67. Strube, M.J., Newman, L.C.: Psychometrics. In: Cacioppo, J., Tassinary, L.G., Berntson, G.G. (eds.) The Handbook of Psychophysiology, pp. 789–811. Cambridge University Press, Cambridge (2007)

    Google Scholar 

  68. Sweetser, P., Wyeth, P.: Gameflow: a model for evaluating player enjoyment in games. Comput. Entertain. 3(3), 3 (2005)

    Article  Google Scholar 

  69. Takatalo, J., Häkkinen, J., Kaistinen, J., Nyman, G.: Presence, involvement, and flow in digital games. In: Bernhaupt, R. (ed.) Evaluating User Experience in Games, pp. 23–46. Human-Computer Interaction Series, Springer, London, New York (2010)

    Chapter  Google Scholar 

  70. Takatalo, J., Häkkinen, J., Komulainen, J., Särkelä, H., Nyman, G.: Involvement and presence in digital gaming. In: Mørch, A., Morgan, K., Bratteteig, T., Ghosh, G., Svanaes, D. (eds.) The 4th Nordic Conference, pp. 393–396 (2006)

    Google Scholar 

  71. UQO Cyberpsychology Lab: Immersive tendencies questionnaire (itq) (2004)

    Google Scholar 

  72. Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: a presence questionnaire. Presence Teleoper. Virtual Environ. 7(3), 225–240 (1998)

    Article  Google Scholar 

  73. Yee, N.: Motivations for play in online games. Cyberpsychol. Behav. 9(6), 772–775 (2006). The impact of the Internet, multimedia and virtual reality on behavior and society

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Entertainment Computing Group, University of Duisburg-Essen, Duisburg, Germany

    Katharina Emmerich

  2. Moscow State University, Moscow, Russia

    Natalya Bogacheva

  3. Graph Theory and Complex Network Analysis Group, TU Kaiserslautern, Kaiserslautern, Germany

    Mareike Bockholt

  4. Multimedia Communications Lab, TU Darmstadt, Darmstadt, Germany

    Viktor Wendel

Authors
  1. Katharina Emmerich
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Natalya Bogacheva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mareike Bockholt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Viktor Wendel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katharina Emmerich .

Editor information

Editors and Affiliations

  1. RheinMain University of Applied Sciences , Wiesbaden, Germany

    Ralf Dörner

  2. Technische Universität Darmstadt , Darmstadt, Germany

    Stefan Göbel

  3. TU Graz , Graz, Austria

    Michael Kickmeier-Rust

  4. University of Duisburg-Essen , Duisburg, Germany

    Maic Masuch

  5. TU Kaiserslautern , Kaiserslautern, Germany

    Katharina Zweig

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this chapter

Cite this chapter

Emmerich, K., Bogacheva, N., Bockholt, M., Wendel, V. (2016). Operationalization and Measurement of Evaluation Constructs. In: Dörner, R., Göbel, S., Kickmeier-Rust, M., Masuch, M., Zweig, K. (eds) Entertainment Computing and Serious Games. Lecture Notes in Computer Science(), vol 9970. Springer, Cham. https://doi.org/10.1007/978-3-319-46152-6_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-46152-6_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-46151-9

  • Online ISBN: 978-3-319-46152-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation