Abstract
We describe a development process for serious games to create psychometrically rigorous measures of individual aptitudes (abilities, skills) and traits (habits, tendencies, behaviors). We begin with a discussion of serious games and how they can instantiate appropriate cognitive states for relevant aptitudes and traits to manifest. This can have numerous advantages over traditional assessment modalities. We then describe the iterative approach to aptitude and trait measurement that emphasizes (1) careful definition and specification of the traits and aptitudes to be measured, (2) rigorous assessment of reliability and validity, and (3) revision of gameplay elements and metrics to improve measurement properties.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Kozlowski, S.W.J., DeShon, R.P.: A psychological fidelity approach to simulation-based training: theory, research and principles. In: Schiflett, S.G., Elliott, L.R., Salas, E., Coovert, M.D. (eds.) Scaled Worlds: Development, Validation and Applications. Routledge, London (2017). https://doi.org/10.4324/9781315243771
Society for Industrial and Organizational Psychology, American Psychological Association: Principles for the validation and use of personnel selection procedures (Fifth edition) (2018). https://www.apa.org/ed/accreditation/about/policies/personnel-selection-procedures.pdf
Ford, J.K., Meyer, T.: Advances in training technology: meeting the workplace challenges of talent development, deep specialization, and collaborative learning. In: Coovert, M.D., Thompson, L.F. (eds.) The Psychology of Workplace Technology. Routledge, New York (2013). https://doi.org/10.4324/9780203735565
Long, D.T., Mulch, C.M.: Interactive wargaming cyberwar: 2025 (2017). https://apps.dtic.mil/docs/citations/AD1053350
Wiemeyer, J., Hardy, S.: Serious games and motor learning: concepts, evidence, technology. In: Bredl, B., Bösche, W. (eds.) Serious Games and Virtual Worlds in Education, Professional Development, and Healthcare, pp. 197–220. IGI Global, Hershey (2013). https://doi.org/10.4018/978-1-4666-3673-6.ch013
Wiemeyer, J., Kliem, A.: Serious games in prevention and rehabilitation—a new panacea for elderly people? Eur. Rev. Aging Phys. Act. 9, 41–50 (2012). https://doi.org/10.1007/s11556-011-0093-x
The O*NET® Content Model. https://www.onetcenter.org/content.html
Ludoscience: A collaborative classification of serious games. http://serious.gameclassification.com/
Dörner, R., Göbel, S., Effelsberg, W., Wiemeyer, J. (eds.): Serious Games: Foundations, Concepts and Practice. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40612-1
Coovert, M.D., Winner, J., Bennett, W.: Construct development and validation in game-based research. Simul. Gaming. 48, 236–248 (2017). https://doi.org/10.1177/1046878116682661
Lievens, F., Patterson, F.: The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. J. Appl. Psychol. 96, 927–940 (2011). https://doi.org/10.1037/a0023496
Schell, J.: The Art of Game Design: A Book of Lenses CRC Press, Boca Raton (2008). https://doi.org/10.1201/9780080919171
Anderson, N., Salgado, J.F., Hülsheger, U.R.: Applicant reactions in selection: comprehensive meta-analysis into reaction generalization versus situational specificity. Int. J. Sel. Assess. 18, 291–304 (2010). https://doi.org/10.1111/j.1468-2389.2010.00512.x
Anderson, N.: Applicant and recruiter reactions to new technology in selection: a critical review and agenda for future research. Int. J. Sel. Assess. 11, 121–136 (2003). https://doi.org/10.1111/1468-2389.00235
Gilliland, S.W.: Fairness from the applicant’s perspective: reactions to employee selection procedures. Int. J. Sel. Assess. 3, 11–18 (1995). https://doi.org/10.1111/j.1468-2389.1995.tb00002.x
Ones, D.S., Viswesvaran, C., Reiss, A.D.: Role of social desirability in personality testing for personnel selection: the red herring. J. Appl. Psychol. 81, 660–679 (1996). https://doi.org/10.1037/0021-9010.81.6.660
Eklöf, H.: Skill and will: test taking motivation and assessment quality. Assess. Educ. Princ. Policy Pract. 17, 345–356 (2010). https://doi.org/10.1080/0969594X.2010.516569
McFarland, L.A., Yun, G.J., Harold, C.M., Viera, L., Moore, L.G.: An examination of impression management use and effectiveness across assessment center exercises: the role of competency demands. Pers. Psychol. 58, 949–980 (2005). https://doi.org/10.1111/j.1744-6570.2005.00374.x
Wilson, M.A. (ed.): The Handbook of Work Analysis: Methods, Systems, Applications and Science of Work Measurement in Organizations. Routledge, New York (2013). https://doi.org/10.4324/9780203136324
Coovert, M.D., Winner, J., Bennett, Jr., W., Howard, D.J.: Serious games are a serious tool for team research. Int. J. Serious Games. 4 (2017). https://doi.org/10.17083/ijsg.v4i1.141
Campbell, J.P., Wiernik, B.M.: The modeling and assessment of work performance. Annu. Rev. Organ. Psychol. Organ. Behav. 2 47–74 (2015). https://doi.org/10.1146/annurev-orgpsych-032414-111427
Wiemeyer, J., Kickmeier-Rust, M., Steiner, Christina M.: Performance assessment in serious games. In: Dörner, R., Göbel, S., Effelsberg, W., Wiemeyer, J. (eds.) Serious Games, pp. 273–302. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40612-1_10
Nandakumar, R.: Assessing essential unidimensionality of real data. Appl. Psychol. Meas. 17, 29–38 (1993). https://doi.org/10.1177/014662169301700108
Gignac, G.E., Watkins, M.W.: Bifactor modeling and the estimation of model-based reliability in the WAIS-IV. Multivar. Behav. Res. 48, 639–662 (2013). https://doi.org/10.1080/00273171.2013.804398
Coovert, M.D., McNelis, K.: Determining the number of common factors in factor analysis: a review and program. Educ. Psychol. Meas. 48, 687–692 (1988). https://doi.org/10.1177/0013164488483012
Comrey, A.L.: A First Course in Factor Analysis, 2 edn. Psychology Press, New York (2013). https://doi.org/10.4324/9781315827506
Reise, S.P.: The rediscovery of bifactor measurement models. Multivar. Behav. Res. 47, 667–696 (2012). https://doi.org/10.1080/00273171.2012.715555
Wiernik, B.M., Wilmot, M.P., Kostal, J.W.: How data analysis can dominate interpretations of dominant general factors. Ind. Organ. Psychol. 8, 438–445 (2015). https://doi.org/10.1017/iop.2015.60
Giordano, C.A., Waller, N.G.: Recovering bifactor models: a comparison of seven methods. Psychol. Methods (2019). https://doi.org/10.1037/met0000227
McArdle, J.J.: Latent variable modeling of differences and changes with longitudinal data. Annu. Rev. Psychol. 60, 577–605 (2009). https://doi.org/10.1146/annurev.psych.60.110707.163612
Coovert, M., Miller, E., Bennett, Jr., W.: Assessing trust and effectiveness in virtual teams: latent growth curve and latent change score models. Soc. Sci. 6, 87 (2017). https://doi.org/10.3390/socsci6030087
Bollen, K.A., Curran, P.J.: Latent Curve Models: A Structural Equation Perspective. Wiley, Hoboken (2005). https://doi.org/10.1002/0471746096
Kuhn, M., Johnson, K.: Applied Predictive Modeling. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-6849-3
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Wiernik, B.M., Coovert, M.D. (2019). A Quantitative Approach for Developing Serious Games for Aptitude and Trait Assessment. In: Liapis, A., Yannakakis, G., Gentile, M., Ninaus, M. (eds) Games and Learning Alliance. GALA 2019. Lecture Notes in Computer Science(), vol 11899. Springer, Cham. https://doi.org/10.1007/978-3-030-34350-7_54
Download citation
DOI: https://doi.org/10.1007/978-3-030-34350-7_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34349-1
Online ISBN: 978-3-030-34350-7
eBook Packages: Computer ScienceComputer Science (R0)