Skip to main content

Evaluation Methods for the Effective Assessment of Simulation Games

A Literature Review

  • Conference paper
  • First Online:
The Challenges of the Digital Transformation in Education (ICL 2018)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 916))

Included in the following conference series:

Abstract

Simulation games play an important role in the area of technology-based education. They allow the simulation of real-world problems in a risk-free environment and thereby intend to increase the learning experience of students. However, assessing the effectiveness of a simulation game is necessary to optimize elements of the game and increase the learning effect for students. For this, different evaluation methods exist, which do not always cover all phases when running a simulation game. In this study, we conduct a literature review to analyze evaluation methods for the pre-game, in-game, and post-game assessment of simulation games. In accordance with our inclusion criteria, we selected 31 peer-reviewed articles, and categorized them according to a didactic framework that describes four phases of running simulation games: Preparation, Introduction, Interaction and Conclusion phase. Based on the results, we provide a concrete evaluation strategy that can be used to assess simulation games throughout all phases. This study contributes to theory by providing an overview of evaluation methods for the assessment of simulation games within the different game phases. It contributes to practice by providing a concrete evaluation strategy that can be adapted and used to assess simulation games.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Hodgetts, R.: Management gaming for didactic purposes: a new look. Simul. Games 1(1), 55–66 (1970)

    Article  Google Scholar 

  2. Faria, A.J., Hutchinson, D., Wellington, W.J., Gold, S.: Developments in business gaming: a review of the past 40 years. Simul. Gaming 40(4), 464–487 (2009)

    Article  Google Scholar 

  3. Lillard, A.S.: Playful learning and Montessori education. Am. J. Play 5(2), 157 (2013)

    Google Scholar 

  4. Thatcher, D.C.: Promoting learning through games and simulations. Simul. Gaming 21(3), 262–273 (1990)

    Article  MathSciNet  Google Scholar 

  5. Marquardt, M.J., Kearsley, G.: Technology-Based Learning: Maximizing Human Performance and Corporate Success. CRC Press (1998)

    Google Scholar 

  6. Baume, M.: Computerunterstützte Planspiele für das Informationsmanagement: Realitätsnahe und praxisorientierte Ausbildung in der universitären Lehre am Beispiel der CIO-Simulation. BoD–Books on Demand (2009)

    Google Scholar 

  7. Aldrich, C.: Clark Aldrich’s Six Criteria of an Educational Simulation (2004)

    Google Scholar 

  8. Hoberg, P., Krcmar, H., Oswald, G., Welz, B.: Skills for Digital Transformation (2016)

    Google Scholar 

  9. Prifti, L., Knigge, M., Kienegger, H., Krcmar, H.: A competency model for Industrie 4.0 employees. In: 13th International Conference on Wirtschaftsinformatik (2017)

    Google Scholar 

  10. Riedel, J.C., Hauge, J.B.: State of the art of serious games for business and industry. In: 17th International Conference on Concurrent Enterprising (ICE), pp. 1–8 (2011)

    Google Scholar 

  11. Löffler, A., Prifti, L., Levkovskyi, B., Utesch, M., Krcmar, H.: Simulation games for the digital transformation of business processes. In: IEEE Global Engineering Education Conference (EDUCON), Santa Cruz de Tenerife, Canary Islands, Spain (2018)

    Google Scholar 

  12. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum. Comput. Interact. 2013, 1 (2013)

    Google Scholar 

  13. Utesch, M.C.: A successful approach to study skills: Go4C´ s projects strengthen teamwork. Int. J. Eng. Pedagog. (iJEP) 6(1), 35–43 (2016)

    Article  Google Scholar 

  14. OECD, CERI: Assessment for learning formative assessment. In: OECD/CERI International Conference on Learning in the 21st Century: Research, Innovation, Policy (2008)

    Google Scholar 

  15. Ifenthaler, D., Eseryel, D., Ge, X.: Assessment in Game-Based Learning, pp. 1–8. Springer (2012)

    Google Scholar 

  16. Shute, V.J.: Stealth assessment in computer-based games to support learning. Comput. Games Instr. 55(2), 503–524 (2011)

    Google Scholar 

  17. Webster, J., Watson, R.T.: Analyzing the past to prepare for the future: writing a literature review. MIS Q. xiii–xxiii (2002)

    Google Scholar 

  18. Vom Brocke, J., Simons, A., Niehaves, B., Riemer, K., Plattfaut, R., Cleven, A.: Reconstructing the giant: on the importance of rigour in documenting the literature search process. In: European Conference on Information Systems (ECIS), vol. 9, pp. 2206–2217 (2009)

    Google Scholar 

  19. Eseryel, D., Ifenthaler, D., Ge, X.: Alternative assessment strategies for complex problem solving in game-based learning environments. In: Multiple Perspectives on Problem Solving and Learning in the Digital Age, pp. 159–178 (2011)

    Google Scholar 

  20. Costantino, F., Di Gravio, G., Shaban, A., Tronci, M.: A simulation based game approach for teaching operations management topics. In: IEEE Winter Simulation Conference (WSC), pp. 1–12 (2012)

    Google Scholar 

  21. Smyrnaiou, Z., Petropoyloy, E., Menon, S., Zini, V.: From game to guidance: the innovative evaluation approach of the P4G simulation business game. In: Mathematics and Computers in Sciences and in Industry (MCSI), pp. 148–153 (2017)

    Google Scholar 

  22. Callaghan, M., Savin-Baden, M., McShane, N., Eguíluz, A.G.: Mapping learning and game mechanics for serious games analysis in engineering education. IEEE Trans. Emerg. Top. Comput. 5(1), 77–83 (2017)

    Article  Google Scholar 

  23. DiCerbo, K.E.: Building the evidentiary argument in game-based assessment. J. Appl. Test. Technol. 18(S1), 7–18 (2017)

    Google Scholar 

  24. Cowley, B., Fantato, M., Jennett, C., Ruskov, M., Ravaja, N.: Learning when serious: psychophysiological evaluation of a technology-enhanced learning game. Educ. Technol. Soc. 17(1), 3–16 (2014)

    Google Scholar 

  25. Tan, J.L., Goh, D.H.-L., Ang, R.P., Huan, V.S.: Participatory evaluation of an educational game for social skills acquisition. Comput. Educ. 64, 70–80 (2013)

    Article  Google Scholar 

  26. Duin, H., Pourabdollahian, B., Thoben, K.-D., Taisch, M.: On the effectiveness of teaching sustainable global manufacturing with serious gaming. In: Engineering, Technology and Innovation (ICE) & IEEE International Technology Management Conference, pp. 1–8 (2013)

    Google Scholar 

  27. Wilson, D.W., et al.: Serious games: an evaluation framework and case study. In: 49th Hawaii International Conference on System Sciences (HICSS), pp. 638–647 (2016)

    Google Scholar 

  28. Utesch, M., Heininger, R., Krcmar, H.: Strengthening study skills by using ERPsim as a new tool within the pupils’ academy of serious gaming. In: IEEE Global Engineering Education Conference (EDUCON), pp. 592–601 (2016)

    Google Scholar 

  29. Utesch, M., Heininger, R., Krcmar. H.: The pupils’ academy of serious gaming: strengthening study skills with ERPsim. In: 13th International Conference on Remote Engineering and Virtual Instrumentation (REV), pp. 93–102 (2016)

    Google Scholar 

  30. Krassmann, A.L., Paschoal, L.N., Falcade, A., Medina, R.D.: Evaluation of game-based learning approaches through digital serious games in computer science higher education: a systematic mapping. In: 14th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pp. 43–51 (2015)

    Google Scholar 

  31. Boyle, L., Hancock, F., Seeney, M., Allen, L.: The implementation of team based assessment in serious games. In: Games and Virtual Worlds for Serious Applications, pp. 28–35 (2009)

    Google Scholar 

  32. Bhardwaj, J.: Evaluation of the lasting impacts on employability of co-operative serious game-playing by first year computing students: an exploratory analysis. In: Frontiers in Education Conference (FIE), pp. 1–9 (2014)

    Google Scholar 

  33. Zolotaryova, I., Plokha, O.: Serious games: evaluation of the learning outcomes. In: 13th International Conference on Modern Problems of Radio Engineering. Telecommunications and Computer Science (TCSET), pp. 858–862 (2016)

    Google Scholar 

  34. Abdellatif, A.J., McCollum, B., McMullan, P.: Serious games: quality characteristics evaluation framework and case study. In: Integrated STEM Education Conference (ISEC), pp. 112–119 (2018)

    Google Scholar 

  35. Wideman, H.H., Owston, R.D., Brown, C., Kushniruk, A., Ho, F., Pitts, K.C.: Unpacking the potential of educational gaming: a new tool for gaming research. Simul. Gaming 38(1), 10–30 (2007)

    Article  Google Scholar 

  36. Meerbaum-Salant, O., Armoni, M., Ben-Ari, M.: Learning computer science concepts with scratch. Comput. Sci. Educ. 23(3), 239–264 (2013)

    Article  Google Scholar 

  37. Al-Smadi, M., Wesiak, G., Guetl, C.: Assessment in serious games: an enhanced approach for integrated assessment forms and feedback to support guided learning. In: 15th International Conference on in Interactive Collaborative Learning (ICL), pp. 1–6 (2012)

    Google Scholar 

  38. Zeng, L.Y.: An evaluation system of game-based learning based on data mining. In: International Conference on Computer Science and Network Technology (ICCSNT), pp. 1732–1736 (2012)

    Google Scholar 

  39. Hainey, T., Connolly, T.: Evaluating games-based learning (2010)

    Google Scholar 

  40. Merkuryev, Y., Bikovska, J.: Business simulation game development for education and training in supply chain management. In: Asia Modelling Symposium (AMS), pp. 179–184 (2012)

    Google Scholar 

  41. de Carvalho, C.V.: Is game-based learning suitable for engineering education? In: IEEE Global Engineering Education Conference (EDUCON), pp. 1–8 (2012)

    Google Scholar 

  42. Boughzala, I., Bououd, I., Michel, H.: Characterization and evaluation of serious games: a perspective of their use in higher education. In: 46th Hawaii International Conference on System Sciences (HICSS), pp. 844–852 (2013)

    Google Scholar 

  43. Michel, H.: Characterizing serious games implementation’s strategies: is higher education the new playground of serious games? In: 49th Hawaii International Conference on System Sciences (HICSS), pp. 818–826 (2016)

    Google Scholar 

  44. Chatterjee, S., Mohanty, A., Bhattacharya, B.: Computer game-based learning and pedagogical contexts: initial findings from a field study. In: IEEE International Conference on Technology for Education (T4E), pp. 109–115 (2011)

    Google Scholar 

  45. Mettler, T., Pinto, R.: Serious games as a means for scientific knowledge transfer: a case from engineering management education. IEEE Trans. Eng. Manag. 62(2), 256–265 (2015)

    Article  Google Scholar 

  46. Cleophas, C.: Designing serious games for revenue management training and strategy development. In: Proceedings of the Winter Simulation Conference, p. 140 (2012)

    Google Scholar 

  47. Escudeiro, P., Escudeiro, N.: Evaluation of serious games in mobile platforms with QEF: QEF (quantitative evaluation framework). In: IEEE 7th International Conference on Wireless, Mobile and Ubiquitous Technology in Education (WMUTE), pp. 268–271 (2012)

    Google Scholar 

  48. Tantan, O.C., Lang, D., Boughzala, I.: Learning business process management through serious games: feedbacks on the usage of INNOV8. In: 18th IEEE Conference on Business Informatics (CBI), vol. 1, pp. 248–254 (2016)

    Google Scholar 

  49. Yang, M.C., Xu, Z.T., Hsu, L.H.: On developing the learning game for graph theory: a new design model considering the learners’ reflexiveness. In: 5th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), pp. 418–422 (2016)

    Google Scholar 

  50. Cutumisu, M., Blair, K.P., Chin, D.B., Schwartz, D.L.: Posterlet: a game-based assessment of children’s choices to seek feedback and to revise. J. Learn. Anal. 2(1), 49–71 (2015)

    Article  Google Scholar 

  51. IJsselsteijn, W., De Kort, Y., Poels, K.: The Game Experience Questionnaire (2008)

    Google Scholar 

  52. The Glossary of Education Reform: Interim Assessment. https://www.edglossary.org/interim-assessment/ (2018)

  53. The University of Sheffield: Questionnaires. https://www.sheffield.ac.uk/lets/strategy/resources/evaluate/general/methods-collection/questionnaire (2018)

  54. Davis, G.A.: Using a retrospective pre-post questionnaire to determine program impact (2002)

    Google Scholar 

  55. Waite, L., Lyons, C.: The 4 questions of a retrospective and why they work. https://www.infoq.com/articles/4-questions-retrospective (2013)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alexander Löffler .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bas, N., Löffler, A., Heininger, R., Utesch, M., Krcmar, H. (2020). Evaluation Methods for the Effective Assessment of Simulation Games. In: Auer, M., Tsiatsos, T. (eds) The Challenges of the Digital Transformation in Education. ICL 2018. Advances in Intelligent Systems and Computing, vol 916. Springer, Cham. https://doi.org/10.1007/978-3-030-11932-4_59

Download citation

Publish with us

Policies and ethics