Abstract
This paper is about performance assessment in serious games. We conceive serious gaming as a process of player-lead decision taking. Starting from combinatorics and item-response theory we provide an analytical model that makes explicit to what extent observed player performances (decisions) are blurred by chance processes (guessing behaviors). We found large effects both theoretically and practically. In two existing serious games random guess scores were found to explain up to 41 % of total scores. Monte Carlo simulation of random game play confirmed the substantial impact of randomness on performance. For valid performance assessments, be it in-game or post-game, the effects of randomness should be included to produce re-calibrated scores that can reasonably be interpreted as the players’ achievements.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Abt, C. (1970). Serious games. New York: Viking.
Aldrich, C. (2005). Learning by doing: the essential guide to simulations, computer games, and pedagogy E-learning and other educational experiences. San Francisco: Wiley.
Becker, K., & Parker, J. R. (2011). The guide to computer simulations and games. Indianapolis: Wiley.
Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of Serious Games: An Overview. Advances in Human-Computer Interaction, 1–11. http://www.hindawi.com/journals/ahci/2013/136864/#B18.
Bente, G., & Breuer, J. (2009). Making the implicit explicit: Embedded measurement in serious games. In U. Ritterfield, M. J. Cody, & P. Vorderer (Eds.), Serious Games: Mechanisms and Effects (pp. 322–343). New York: Routledge.
Boston, C. (2002). The concept of formative assessment. Practical Assessment, Research & Evaluation, 8 (9). http://pareonline.net/getvn.asp?v=8&n=9.
Brennan, R. L. (2006). Educational measurement (4th ed.). Lanham: Rowman & Littlefield Publishers.
Chin, J., Dukes, R., & Gamson, W. (2009). Assessment in simulation and gaming: a review of the last 40 years. Simulation & Gaming, 40(4), 553–568.
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of the empirical evidence on computer games and serious games. Computers and Education, 59, 661–686.
Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41, 1040–1048.
Ebel, R. L., & Frisbie, D. A. (1991). Essentials of educational measurement (Vol. 5). New Jersey: Prentice Hall.
Farr, J. L., Hofmann, D. A., & Ringenbach, K. L. (1993). Goal orientation and action control theory: Implications for industrial and organizational psychology. In C. L. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology (pp. 193–232). New York: John Wiley.
Guttormsen Schär, S., Schluep, S., Schierz, C., & Krueger, H. (2000). Interaction for Computer-Aided Learning. Interactive Multimedia Electronic Journal of Computer-enhanced learning 2(1). http://imej.wfu.edu/articles/2000/1/03/.
Kolb, D. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs: Prentice Hall.
Nuffic (2006). Fact sheet cijfers ontcijferd. Nuffic: The Hague. http://www.nuffic.nl/bestanden/documenten/over-de-nuffic/publicaties/factsheet-cijfers-ontcijferd.pdf/view.
Redeker, C., Punie, Y., & Ferrari, A. (2012). eAssessment for 21st century learning and skills. In A. Ravenscroft, S. Lindsteadt, C. D. Kloos, & D. Hernandez-Leo (Eds.), 21st Century Learning for 21st Century Skills. Proceedings of the 7th European Conference on technology-enhanced learning EC-TEL, Saarbrücken, 2012 (pp. 292–305). Heidelberg: Springer.
Reese, H.W. (2011). The Learning-by-Doing Principle. Behavioral Development Bulletin, 11, 1–19. http://www.baojournal.com/BDB%20WEBSITE/archive/BDB-2011-11-01-001-019.pdf.
Schank, R. C. (1995). Engines for education. New York: Lawrence Erlbaum.
Schank, R. C., Berman, T. R., & Macpherson, K. A. (1999). Learning by doing. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory (Vol. II, pp. 161–181). Mahwah: Lawrence Erlbaum Associates.
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.
Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning: flow and grow. In U. Ritterfeld, M. Cody, & M. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295–321). New York: Routledge.
Vargas, J. S. (1986). Instructional Design Flaws in Computer-Assisted Instruction. The Phi Delta Kappan, 67(10), 738–744. Retrieved from http://www.jstor.org/stable/20403230.
Westera, W., Hommes, M. A., Houtmans, M., & Kurvers, H. J. (2003). Computer-supported training of psycho-diagnostic skills. Interactive Learning Environments, 11(3), 215–231.
Westera, W., Nadolski, R., Hummel, H., & Wopereis, I. (2008). Serious games for higher education: a framework for reducing design complexity. Journal of Computer-Assisted Learning, 24(5), 420–432.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Westera, W. Performance assessment in serious games: Compensating for the effects of randomness. Educ Inf Technol 21, 681–697 (2016). https://doi.org/10.1007/s10639-014-9347-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-014-9347-3