Skip to main content
Springer Nature Link
Log in

Utilizing Game Analytics to Inform and Validate Digital Game-based Assessment with Evidence-centered Game Design: A Case Study

  • ARTICLE
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

The purpose of this case study is to demonstrate how to utilize machine learning approaches to analyze student process data for validating and informing digital game-based assessments (DGBAs) with an evidence-centered game design (ECgD). The first analysis was conducted to examine whether students’ mastery of the overall skill required by the game can be well predicted by task-related behavioral features and if the selected key features map onto the evidence model of the ECgD. Specifically, we extracted 27 behavioral features as the indicators of students’ gameplay activities from the evidence trace files and modelled them using a machine learning algorithm—support vector machine with recursive feature elimination—to identify the key features for prediction. The key features were in turn used to predict students’ mastery of the overall skill. Results showed that students’ retry attempts on two assessment tasks were found to be most influential for prediction with a moderate to high training and testing accuracy. The second analysis was conducted to examine whether the number of learning opportunities is sufficient for evaluating students’ mastery of the overall skill as well as determine the optimal number of learning opportunities for evaluation. The approach of long short-term memory networks was used to model students’ time-series behavioral features across multiple learning opportunities for predicting their acquisition of the overall skill. Results suggested that five learning opportunities were a good balance between evaluation accuracy and practical feasibility, and they were sufficient for evaluating students’ mastery of the overall skill given the DGGA tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Alberta Education. (1996). Science (elementary). Alberta Education. Retrieved from http://www.education.alberta.ca/media/654825/elemsci.pdf.

  • Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., & Fernández-Manjón, B. (2019). Applications of data science to game learning analytics data: A systematic literature review. Computers & Education, 141, 103612.

    Google Scholar 

  • Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166.

    Google Scholar 

  • Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., & Runmble, M. (2014, December 8). Partnership for 21st century skills. Retrieved from http://www.p21.org/.

  • Bodily, R., Ikahihifo, T. K., Mackley, B., & Graham, C. R. (2018). The design, development, and implementation of student-facing learning analytics dashboards. Journal of Computing in Higher Education, 30(3), 572–598.

    Google Scholar 

  • Burisch, M. (1997). Test length and validity revisited. European Journal of Personality, 11(4), 303–315.

    Google Scholar 

  • Cano, A. R., Fernández-Manjón, B., & García‐Tejedor, ÁJ. (2018). Using game learning analytics for validating the design of a learning game for adults with intellectual disabilities. British Journal of Educational Technology, 49(4), 659–672.

    Google Scholar 

  • Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357.

    MATH  Google Scholar 

  • Chen, K., Zhou, Y., & Dai, F. (2015). A LSTM-based method for stock returns prediction: A case study of China stock market. In 2015 IEEE international conference on big data (big data) (pp. 2823–2824). IEEE.

  • Chollet, F. (2015). keras. GitHub repository. https://github.com/fchollet/keras.

  • Chu, M. W., & Chiang, A. (2018). Raging skies: Development of a digital game-based science assessment using evidence-centered game design. Alberta Science Education Journal, 45, 37–47. https://doi.org/10.11575/PRISM/32941.

    Article  Google Scholar 

  • Coelho, O. B., & Silveira, I. (2017). Deep Learning applied to Learning Analytics and Educational Data Mining: A Systematic Literature Review. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE) (Vol. 28, No. 1, p. 143).

  • Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661–686.

    Google Scholar 

  • Cui, Y., Chen, F., Shiri, A., & Fan, Y. (2019a). Predictive analytic models of student success in higher education: A review of methodology. Information and Learning Sciences, 120(3/4), 208–227. https://doi.org/10.1108/ILS-10-2018-0104.

    Article  Google Scholar 

  • Cui, Y., Chu, M.-W., & Chen, F. (2019b). Analyzing student process data in game-based assessments with Bayesian knowledge tracing and dynamic Bayesian network. JEDM | Journal of Educational Data Mining, 11(1), 80–100.

    Google Scholar 

  • de Klerk, S., Veldkamp, B. P., & Eggen, T. J. (2015). Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a Bayesian network example. Computers & Education, 85, 23–34.

    Google Scholar 

  • DiCerbo, K. E. (2014). Game-Based Assessment of Persistence. Educational Technology & Society, 17(1), 17–28.

    Google Scholar 

  • Divjak, B., & Tomić, D. (2011). The impact of game-based learning on the achievement of learning goals and motivation for learning mathematics-literature review. Journal of Information and Organizational Sciences, 35(1), 15–30.

    Google Scholar 

  • El-Nasr, M. S., Drachen, A., & Canossa, A. (Eds.). (2016). Game analytics – maximizing the value of player data. London: Springer.

    Google Scholar 

  • Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Computers & Education, 67, 156–167.

    Google Scholar 

  • Fu, J., Zapata-Rivera, D., & Mavronikolas, E. (2014). Statistical Methods for Assessments in Simulations and Serious Games (ETS Research Report Series No. RR-14-12). Princeton: Educational Testing Service.

  • Gal, Y., & Ghahramani, Z. (2016). A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems (pp. 1019–1027).

  • Gaydos, M. J. (2016). Developing a geography game for Singapore clasrooms. In C.-K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), Transforming learning, empowering learners: The International Conference of the Learning Sciences (ICLS) (Vol. 2, pp. 729–736). Singapore: International Society of the Learning Sciences.

  • Griffin, P., Care, E., & McGaw, B. (2012). The changing role of education and schools. In P. Griffin, B. McGaw & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 1–15). Dordrecht: Springer.

    Google Scholar 

  • Harpstead, E., Zimmermann, T., Nagapan, N., Guajardo, J. J., Cooper, R., Solberg, T., & Greenawalt, D. (2015). What drives people: Creating engagement profiles of players from game log data. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (pp. 369–379). New York: ACM.

  • Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.

    Google Scholar 

  • Hsiao, H. S., & Chen, J. C. (2016). Using a gesture interactive game-based learning approach to improve preschool children’s learning performance and motor skills. Computers & Education, 95, 151–162.

    Google Scholar 

  • Hsu, W. N., Zhang, Y., Lee, A., & Glass, J. (2016). Exploiting depth and highway connections in convolutional recurrent deep neural networks for speech recognition. Cell, 50(1), 395–399.

    Google Scholar 

  • Hwang, G. J., & Wu, P. H. (2012). Advancements and trends in digital game-based learning research: a review of publications in selected journals from 2001 to 2010. British Journal of Educational Technology, 43(1), E6–E10.

    Google Scholar 

  • Jackson, G. T., & McNamara, D. S. (2013). Motivation and performance in a game-based intelligent tutoring system. Journal of Educational Psychology, 105(4), 1036–1049.

    Google Scholar 

  • Kang, J., Liu, S., & Liu, M. (2017). Tracking students’ activities in serious games. In F. Q. Lai & J. Lehman (Eds.), Learning and knowledge analytics in open education. Cham: Springer.

    Google Scholar 

  • Kazanidis, I., Palaigeorgiou, G., Chintiadis, P., & Tsinakos, A. (2018). Á Pilot Evaluation of a Virtual Reality Educational Game for History Learning. In European Conference on e-Learning (pp. 245–253). Academic Conferences International Limited.

  • Kerr, D., & Chung, G. K. (2012). Identifying key features of student performance in educational video games and simulations through cluster analysis. JEDM | Journal of Educational Data Mining, 4(1), 144–182.

    Google Scholar 

  • Kerr, D., Chung, G. K., & Iseli, M. R. (2011). The feasibility of using cluster analysis to examine log data from educational video games (CRESST Report No. 790). Los Angeles: The National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Center for Studies in Education, UCLA.

  • Keshtkar, F., Burkett, C., Li, H., & Graesser, A. C. (2014). Using data mining techniques to detect the personality of players in an educational game. In A. Peña-Ayala (Ed.), Educational data mining (pp. 125–150). Cham: Springer.

    Google Scholar 

  • Kiili, K., & Ketamo, H. (2017). Evaluating cognitive and affective outcomes of a digital game-based math test. IEEE Transactions on Learning Technologies, 11(2), 255–263.

    Google Scholar 

  • Kim, Y. J., Almond, R. G., & Shute, V. J. (2016). Applying evidence-centered design for the development of game-based assessments in physics playground. International Journal of Testing, 16(2), 142–163.

    Google Scholar 

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

  • Kuhn, M. (2008). Caret package. Journal of Statistical Software, 28(5).

  • Lamb, R. L. (2013). The application of cognitive diagnostic approaches via neural network analysis of serious educational games. Doctoral dissertation. George Mason University.

  • Lamb, R. L., Annetta, L., Vallett, D. B., & Sadler, T. D. (2014). Cognitive diagnostic like approaches using neural-network analysis of serious educational videogames. Computers & Education, 70, 92–104.

    Google Scholar 

  • Liu, M., Kang, J., Liu, S., Zou, W., & Hodson, J. (2017). Learning analytics as an assessment tool in serious games: A review of literature. In Serious games and edutainment applications (pp. 537–563). Cham: Springer.

    Google Scholar 

  • Liu, M., Lee, J., Kang, J., & Liu, S. (2016). What we can learn from the data: A multiple-case study examining behavior patterns by students with different characteristics in using a serious game. Technology, Knowledge and Learning, 21(1), 33–57.

    Google Scholar 

  • Loh, C. S., & Sheng, Y. (2015). Measuring expert performance for serious games analytics: From data to insights. In C. S. Loh, Y. Sheng & D. Ifenthaler (Eds.), Serious games analytics. Cham: Springer.

    Google Scholar 

  • Loh, C. S., Sheng, Y., & Ifenthaler, D. (2015). Serious games analytics: Theoretical framework. In C. S. Loh, Y. Sheng & D. Ifenthaler (Eds.), Serious games analytics. Cham: Springer.

    Google Scholar 

  • Long, L., Siemens, L., Conole, G., & Gasevic, D. (2011). Proceedings of the 1st international conference on learning analytics and knowledge (Banff, AB, Canada). New York: ACM Press.

  • Lotfi, E., Amine, B., & Mohammed, B. (2014). Players performances analysis based on educational data mining case of study: Interactive waste sorting serious game. International Journal of Computer Applications, 108(11), 13–18.

    Google Scholar 

  • Mathrani, A., Christian, S., & Ponder-Sutton, A. (2016). PlayIT: game based learning approach for teaching programming concepts. Journal of Educational Technology & Society, 19(2), 5–17.

    Google Scholar 

  • Millman, J. (1989). If at first you don’t succeed setting passing scores when more than one attempt is permitted. Educational Researcher, 18(6), 5–9.

    Google Scholar 

  • Min, W., Frankosky, M., Mott, B. W., Rowe, J., Smith, P. A. M., Wiebe, E., … Lester, J. (2019). DeepStealth: Game-based learning stealth assessment with deep neural networks. IEEE Transactions on Learning Technologies. https://doi.org/10.1109/TLT.2019.2922356.

    Article  Google Scholar 

  • Minović, M., Milovanović, M., Šošević, U., & González, M.ÁC. (2015). Visualisation of student learning model in serious games. Computers in Human Behavior, 47, 98–107.

    Google Scholar 

  • Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1(1), 3–62.

    Google Scholar 

  • Nahid, A. A., Mehrabi, M. A., & Kong, Y. (2018). Histopathological breast cancer image classification by deep neural network techniques guided by local clustering. BioMed research international. https://doi.org/10.1155/2018/2362108.

    Article  Google Scholar 

  • Organization for Economic Co-operation and Development. (2005). The definition and selection of key competencies: Executive summary. Paris: Author.

    Google Scholar 

  • Petri, G., & von Wangenheim, C. G. (2017). How games for computing education are evaluated? A systematic literature review. Computers & Education, 107, 68–90.

    Google Scholar 

  • Plass, J., Homer, B. D., Kinzer, C. K., Chang, Y. K., Frye, J., Kaczetow, W., et al. (2013). Metrics in simulations and games for learning. In M. Seif El-Nasr, A. Drachen & A. Canossa (Eds.), Game analytics: Maximizing the value of player data (pp. 694–730). London: Springer.

  • Qian, M., & Clark, K. R. (2016). Game-based Learning and 21st century skills: A review of recent research. Computers in Human Behavior, 63, 50–58.

    Google Scholar 

  • Qu, Z., Haghani, P., Weinstein, E., & Moreno, P. (2017). Syllable-based acoustic modeling with CTC-SMBR-LSTM. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (pp. 173–177). Piscataway: IEEE.

  • R Core Team. (2018). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. https://www.R-project.org/.

  • Rowe, E., Asbell-Clarke, J., & Baker, R. S. (2015). Serious games analytics to measure implicit science learning. In C. S. Loh, Y. Sheng & D. Ifenthaler (Eds.), Serious games analytics. Cham: Springer.

    Google Scholar 

  • Shute, V. J. (2007). Tensions, trends, tools, and technologies: Time for an educational sea change. In C. A. Dwyer (Ed.), The future of assessment: Shaping teaching and learning (pp. 139–187). New York: Lawrence Erlbaum/Taylor & Francis.

    Google Scholar 

  • Shute, V. J., & Becker, B. J. (2010). Prelude: issues and assessment for the 21st century. In V. J. Shute & B. J. Becker (Eds.), Innovative assessment for the 21st century: Supporting educational needs (pp. 1e11). New York: Springer-Verlag.

    Google Scholar 

  • Shute, V., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning? Flow and Grow. Serious Games: Mechanisms and Effects, 1(1), 1–33.

    Google Scholar 

  • Snow, E. L., Allen, L. K., & McNamara, D. S. (2015). The dynamical analysis of log data within educational games. In C. S. Loh, Y. Sheng & D. Ifenthaler (Eds.), Serious games analytics. Cham: Springer.

    Google Scholar 

  • Tlili, A., Essalmi, F., & Jemni, M. (2015). An educational game for teaching computer architecture: Evaluation using learning analytics. In 2015 5th International Conference on Information & Communication Technology and Accessibility (ICTA) (pp. 1–6). IEEE.

  • Westera, W., Nadolski, R., & Hummel, H. (2013). Learning analytics in serious gaming: uncovering the hidden treasury of game log files. Paper presented at the International Conference on Games and Learning Alliance (pp. 41–52). Cham: Springer.

  • Westera, W., Nadolski, R., & Hummel, H. (2014). Serious gaming analytics: What students̈ log files tell us about gaming and learning. International Journal of Serious Games, 1(2), 35–50.

    Google Scholar 

  • Yu, Y., Si, X., Hu, C., & Zhang, J. (2019). A review of recurrent neural networks: LSTM cells and network architectures. Neural Computation, 31(7), 1235–1270.

    MathSciNet  MATH  Google Scholar 

  • Yukselturk, E., Altıok, S., & Başer, Z. (2018). Using game-based learning with kinect technology in foreign language education course. Journal of Educational Technology & Society, 21(3), 159–173.

    Google Scholar 

  • Zhou, Y., Huang, C., Hu, Q., Zhu, J., & Tang, Y. (2018). Personalized learning full-path recommendation model based on LSTM neural networks. Information Sciences, 444, 135–152.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Educational Psychology, University of Alberta, Edmonton, AB, T6G 2G5, Canada

    Fu Chen & Ying Cui

  2. Werklund School of Education, University of Calgary, Calgary, AB, Canada

    Man-Wai Chu

Authors
  1. Fu Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ying Cui
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Man-Wai Chu
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Fu Chen.

Ethics declarations

Conflict of Interest

The authors declared no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, F., Cui, Y. & Chu, MW. Utilizing Game Analytics to Inform and Validate Digital Game-based Assessment with Evidence-centered Game Design: A Case Study. Int J Artif Intell Educ 30, 481–503 (2020). https://doi.org/10.1007/s40593-020-00202-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40593-020-00202-6

Keywords