Skip to main content

A Quantitative Approach for Developing Serious Games for Aptitude and Trait Assessment

  • Conference paper
  • First Online:
Games and Learning Alliance (GALA 2019)

Abstract

We describe a development process for serious games to create psychometrically rigorous measures of individual aptitudes (abilities, skills) and traits (habits, tendencies, behaviors). We begin with a discussion of serious games and how they can instantiate appropriate cognitive states for relevant aptitudes and traits to manifest. This can have numerous advantages over traditional assessment modalities. We then describe the iterative approach to aptitude and trait measurement that emphasizes (1) careful definition and specification of the traits and aptitudes to be measured, (2) rigorous assessment of reliability and validity, and (3) revision of gameplay elements and metrics to improve measurement properties.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Kozlowski, S.W.J., DeShon, R.P.: A psychological fidelity approach to simulation-based training: theory, research and principles. In: Schiflett, S.G., Elliott, L.R., Salas, E., Coovert, M.D. (eds.) Scaled Worlds: Development, Validation and Applications. Routledge, London (2017). https://doi.org/10.4324/9781315243771

  2. Society for Industrial and Organizational Psychology, American Psychological Association: Principles for the validation and use of personnel selection procedures (Fifth edition) (2018). https://www.apa.org/ed/accreditation/about/policies/personnel-selection-procedures.pdf

  3. Ford, J.K., Meyer, T.: Advances in training technology: meeting the workplace challenges of talent development, deep specialization, and collaborative learning. In: Coovert, M.D., Thompson, L.F. (eds.) The Psychology of Workplace Technology. Routledge, New York (2013). https://doi.org/10.4324/9780203735565

  4. Long, D.T., Mulch, C.M.: Interactive wargaming cyberwar: 2025 (2017). https://apps.dtic.mil/docs/citations/AD1053350

  5. Wiemeyer, J., Hardy, S.: Serious games and motor learning: concepts, evidence, technology. In: Bredl, B., Bösche, W. (eds.) Serious Games and Virtual Worlds in Education, Professional Development, and Healthcare, pp. 197–220. IGI Global, Hershey (2013). https://doi.org/10.4018/978-1-4666-3673-6.ch013

  6. Wiemeyer, J., Kliem, A.: Serious games in prevention and rehabilitation—a new panacea for elderly people? Eur. Rev. Aging Phys. Act. 9, 41–50 (2012). https://doi.org/10.1007/s11556-011-0093-x

  7. The O*NET® Content Model. https://www.onetcenter.org/content.html

  8. Ludoscience: A collaborative classification of serious games. http://serious.gameclassification.com/

  9. Dörner, R., Göbel, S., Effelsberg, W., Wiemeyer, J. (eds.): Serious Games: Foundations, Concepts and Practice. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40612-1

  10. Coovert, M.D., Winner, J., Bennett, W.: Construct development and validation in game-based research. Simul. Gaming. 48, 236–248 (2017). https://doi.org/10.1177/1046878116682661

  11. Lievens, F., Patterson, F.: The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. J. Appl. Psychol. 96, 927–940 (2011). https://doi.org/10.1037/a0023496

  12. Schell, J.: The Art of Game Design: A Book of Lenses CRC Press, Boca Raton (2008). https://doi.org/10.1201/9780080919171

  13. Anderson, N., Salgado, J.F., Hülsheger, U.R.: Applicant reactions in selection: comprehensive meta-analysis into reaction generalization versus situational specificity. Int. J. Sel. Assess. 18, 291–304 (2010). https://doi.org/10.1111/j.1468-2389.2010.00512.x

  14. Anderson, N.: Applicant and recruiter reactions to new technology in selection: a critical review and agenda for future research. Int. J. Sel. Assess. 11, 121–136 (2003). https://doi.org/10.1111/1468-2389.00235

  15. Gilliland, S.W.: Fairness from the applicant’s perspective: reactions to employee selection procedures. Int. J. Sel. Assess. 3, 11–18 (1995). https://doi.org/10.1111/j.1468-2389.1995.tb00002.x

  16. Ones, D.S., Viswesvaran, C., Reiss, A.D.: Role of social desirability in personality testing for personnel selection: the red herring. J. Appl. Psychol. 81, 660–679 (1996). https://doi.org/10.1037/0021-9010.81.6.660

  17. Eklöf, H.: Skill and will: test taking motivation and assessment quality. Assess. Educ. Princ. Policy Pract. 17, 345–356 (2010). https://doi.org/10.1080/0969594X.2010.516569

  18. McFarland, L.A., Yun, G.J., Harold, C.M., Viera, L., Moore, L.G.: An examination of impression management use and effectiveness across assessment center exercises: the role of competency demands. Pers. Psychol. 58, 949–980 (2005). https://doi.org/10.1111/j.1744-6570.2005.00374.x

  19. Wilson, M.A. (ed.): The Handbook of Work Analysis: Methods, Systems, Applications and Science of Work Measurement in Organizations. Routledge, New York (2013). https://doi.org/10.4324/9780203136324

  20. Coovert, M.D., Winner, J., Bennett, Jr., W., Howard, D.J.: Serious games are a serious tool for team research. Int. J. Serious Games. 4 (2017). https://doi.org/10.17083/ijsg.v4i1.141

  21. Campbell, J.P., Wiernik, B.M.: The modeling and assessment of work performance. Annu. Rev. Organ. Psychol. Organ. Behav. 2 47–74 (2015). https://doi.org/10.1146/annurev-orgpsych-032414-111427

  22. Wiemeyer, J., Kickmeier-Rust, M., Steiner, Christina M.: Performance assessment in serious games. In: Dörner, R., Göbel, S., Effelsberg, W., Wiemeyer, J. (eds.) Serious Games, pp. 273–302. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40612-1_10

    Chapter  Google Scholar 

  23. Nandakumar, R.: Assessing essential unidimensionality of real data. Appl. Psychol. Meas. 17, 29–38 (1993). https://doi.org/10.1177/014662169301700108

  24. Gignac, G.E., Watkins, M.W.: Bifactor modeling and the estimation of model-based reliability in the WAIS-IV. Multivar. Behav. Res. 48, 639–662 (2013). https://doi.org/10.1080/00273171.2013.804398

  25. Coovert, M.D., McNelis, K.: Determining the number of common factors in factor analysis: a review and program. Educ. Psychol. Meas. 48, 687–692 (1988). https://doi.org/10.1177/0013164488483012

  26. Comrey, A.L.: A First Course in Factor Analysis, 2 edn. Psychology Press, New York (2013). https://doi.org/10.4324/9781315827506

  27. Reise, S.P.: The rediscovery of bifactor measurement models. Multivar. Behav. Res. 47, 667–696 (2012). https://doi.org/10.1080/00273171.2012.715555

  28. Wiernik, B.M., Wilmot, M.P., Kostal, J.W.: How data analysis can dominate interpretations of dominant general factors. Ind. Organ. Psychol. 8, 438–445 (2015). https://doi.org/10.1017/iop.2015.60

  29. Giordano, C.A., Waller, N.G.: Recovering bifactor models: a comparison of seven methods. Psychol. Methods (2019). https://doi.org/10.1037/met0000227

    Article  Google Scholar 

  30. McArdle, J.J.: Latent variable modeling of differences and changes with longitudinal data. Annu. Rev. Psychol. 60, 577–605 (2009). https://doi.org/10.1146/annurev.psych.60.110707.163612

  31. Coovert, M., Miller, E., Bennett, Jr., W.: Assessing trust and effectiveness in virtual teams: latent growth curve and latent change score models. Soc. Sci. 6, 87 (2017). https://doi.org/10.3390/socsci6030087

  32. Bollen, K.A., Curran, P.J.: Latent Curve Models: A Structural Equation Perspective. Wiley, Hoboken (2005). https://doi.org/10.1002/0471746096

  33. Kuhn, M., Johnson, K.: Applied Predictive Modeling. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-6849-3

    Book  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Brenton M. Wiernik .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wiernik, B.M., Coovert, M.D. (2019). A Quantitative Approach for Developing Serious Games for Aptitude and Trait Assessment. In: Liapis, A., Yannakakis, G., Gentile, M., Ninaus, M. (eds) Games and Learning Alliance. GALA 2019. Lecture Notes in Computer Science(), vol 11899. Springer, Cham. https://doi.org/10.1007/978-3-030-34350-7_54

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-34350-7_54

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-34349-1

  • Online ISBN: 978-3-030-34350-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics