Skip to main content

Advertisement

Log in

Predictive Student Modeling in Game-Based Learning Environments with Word Embedding Representations of Reflection

  • Article
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

Game-based learning environments are designed to provide effective and engaging learning experiences for students. Predictive student models use trace data extracted from students’ in-game learning behaviors to unobtrusively generate early assessments of student knowledge and skills, equipping game-based learning environments with the capacity to anticipate student outcomes and proactively deliver adaptive scaffolding or notify instructors. Reflection is a key component of self-regulated learning, and it is critical in effective learning. However, there is currently limited work exploring the utility of reflection for inducing accurate predictive student models. This article presents a predictive student modeling framework that leverages natural language responses to in-game reflection prompts to predict student learning outcomes in a game-based learning environment for middle school microbiology, CRYSTAL ISLAND. With data from a pair of classroom studies involving 118 middle school students, we investigate the accuracy of early prediction models that utilize features extracted from student trace data combined with word embedding-based representations (i.e., GloVe, ELMo) of student reflection responses. We evaluate the accuracy of the predictive models over time using data from incremental segments of each student’s interaction with the game-based learning environment, and we compare against models that omit student reflection features. Results reveal that models encoding students’ natural language reflections with ELMo word embeddings yield significantly improved accuracy compared to other representations, with the greatest accuracy demonstrated by an ensemble of predictive models. We discuss the implications of these results for the design of game-based learning environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Acharya, A., & Sinha, D. (2014). Early prediction of students’ performance using machine learning techniques. International Journal of Computer Applications, 107(1), 37–43.

  • Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., & Goldberg, Y. (2017). Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of the 5th International Conference on Learning Representations (pp. 1–13).

  • Akram, B., Min, W., Wiebe, E., Mott, B., Boyer, K. E., & Lester, J. (2018). Improving stealth assessment in game-based learning with LSTM-based analytics. In Proceedings of the 11th International Educational Data Mining, (pp. 208-218).

  • Bannert, M. (2006). Effects of reflection prompts when learning with hypermedia. Journal of Educational Computing Research, 35(4), 359–375.

    Google Scholar 

  • Bannert, M., & Reimann, P. (2012). Supporting self-regulated hypermedia learning through prompts. Instructional Science, 40(1), 193–211.

    Google Scholar 

  • Baneres, D., Rodríguez-Gonzalez, M., & Serra, M. (2019). An early feedback prediction system for learners at-risk within a first-year higher education course. IEEE Transactions on Learning Technologies, 12(2), 249–263.

    Google Scholar 

  • Baker, R., Corbett, A., & Aleven, V. (2008). More accurate student modeling through contextual estimation of slip and guess probabilities in Bayesian knowledge tracing. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 406–415).

  • Bell, A., Kelton, J., McDonagh, N., Mladenovic, R., & Morrison, K. (2011). A critical evaluation of the usefulness of a coding scheme to categorise levels of reflective thinking. Assessment & Evaluation in Higher Education, 36(7), 797–815.

    Google Scholar 

  • Bisman, J. (2011). Engaged pedagogy: A study of the use of reflective journals in accounting education. Assessment & Evaluation in Higher Education, 36(3), 315–330.

    Google Scholar 

  • Biswas, G., Baker, R. S., & Paquette, L. (2018). Data mining methods for assessing self-regulated learning. In D. H. Schunk & J. A. Greene (Eds.), Educational Psychology handbook series. Handbook of Self-regulation of learning and performance, 388–403. Routledge/Taylor & Francis Group.

  • Blaney, J., Filer, K., & Lyon, J. (2014). Assessing high impact practices using NVivo: An automated approach to analyzing student reflections for program improvement. Research & Practice in Assessment, 9, 97–100.

    Google Scholar 

  • Blaylock, N. and Allen, J. 2003. Corpus-based, statistical goal recognition. In Proceedings of the 18th International Joint Conference on Artificial Intelligence (pp. 1303–1308).

  • Brezovszky, B., McMullen, J., Veermans, K., Hannula-Sormunen, M. M., Rodríguez-Aflecht, G., Pongsakdi, N., Laakkonen, E., & Lehtinen, E. (2019). Effects of a mathematics game-based learning environment on primary school students' adaptive number knowledge. Computers & Education, 128, 63–74.

    Google Scholar 

  • Boenink, A., Oderwald, A., De Jonge, P., Van Tilburg, W., & Smal, J. (2004). Assessing student reflection in medical practice. The development of an observer-rated instrument: Reliability, validity and initial experiences. Medical Education, 38(4), 368–377.

    Google Scholar 

  • Botelho, A., Varatharaj, A., Patikorn, T., Doherty, D., Adjei, S. A., & Beck, J. E. (2019). Developing early detectors of student attrition and wheel spinning using deep learning. IEEE Transactions on Learning Technologies, 12(2), 158–170.

    Google Scholar 

  • Boud, D., Keogh, R., & Walker, D. (2013). Reflection: Turning experience into learning. New York: Routledge.

    Google Scholar 

  • Burrows, V., McNeill, B., Hubele, N., & Bellamy, L. (2001). Statistical evidence for enhanced learning of content through reflective journal writing. Journal of Engineering Education, 90(4), 661–667.

    Google Scholar 

  • Chen, C., & Law, V. (2016). Scaffolding individual and collaborative game-based learning in learning performance and intrinsic motivation. Computers in Human Behavior, 55, 1201–1212.

    Google Scholar 

  • Cheng, G. (2017). Towards an automatic classification system for supporting the development of critical reflective skills in L2 learning. Australasian Journal of Educational Technology, 33(4), 1–21.

    Google Scholar 

  • Cisero, C. (2006). Does reflective journal writing improve course performance? College Teaching, 54(2), 231–236.

    Google Scholar 

  • Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.

  • Costa, E., Fonseca, B., Santana, M., de Araújo, F., & Rego, J. (2017). Evaluating the effectiveness of educational data mining techniques for early prediction of students' academic failure in introductory programming courses. Computers in Human Behavior, 73, 247–256.

    Google Scholar 

  • Desmarais, M. C., & Baker, R. S. (2012). A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction, 22, 9–38.

    Google Scholar 

  • Emerson, A., Smith, A., Smith, C., Rodríguez, F., Min, W., Wiebe, E., Mott, B., Boyer, K., & Lester, J. (2019). Predicting early and often: Predictive student modeling for block-based programming environments. In Proceedings of the 12th International Conference on Educational Data Mining (pp. 39–48).

  • Falakmasir, M. H., Gonzalez-Brenes, J. P., Gordon, G. J., & DiCerbo, K. E. (2016). A data-driven approach for inferring student proficiency from game activity logs. In Proceedings of the Third ACM Conference on Learning@ Scale (pp. 341–349).

  • de Freitas, S. (2018). Are games effective learning tools? A review of educational games. Educational Technology & Society, 21(2), 74–84.

    Google Scholar 

  • Gitinabard, N., Xu, Y., Heckman, S., Barnes, T., & Lynch, C. F. (2019). How widely can prediction models be generalized? Performance prediction in blended courses. IEEE Transactions on Learning Technologies, 12(2), 184–197.

    Google Scholar 

  • Geden, M., Emerson, A., Rowe, J., Azevedo, R., & Lester, J. (2020). Predictive student modeling in educational games with multi-task learning. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (pp. 654–661).

  • Graesser, A. C. (2016). Conversations with AutoTutor help students learn. International Journal of Artificial Intelligence in Education, 26(1), 124–132.

    Google Scholar 

  • Greene, J., Moos, D., & Azevedo, R. (2011). Self-regulation of learning with computer-based learning environments. In New Directions for Teaching and Learning, 107–115.

  • Greene, J., & Azevedo, R. (2007). A theoretical review of Winne and Hadwin’s model of self-regulated learning: New perspectives and directions. Review of Educational Research, 77(3), 334–372.

    Google Scholar 

  • Hume, A. (2009). Promoting higher levels of reflective writing in student journals. Higher Education Research & Development, 28(3), 247–260.

    Google Scholar 

  • Jiménez, F., Paoletti, A., Sánchez, G., & Sciavicco, G. (2019). Predicting the risk of academic dropout with temporal multi-objective optimization. IEEE Transactions on Learning Technologies, 12(2), 225–236.

    Google Scholar 

  • Jung, Y. Wise, A., & (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69.

  • Karumbaiah, S., Baker, R., & Shute, V. (2018). Predicting quitting in students playing a learning game. In Proceedings of the Twelfth International Conference on Educational Data Mining, 167–176.

  • Kember, D. (1999). Determining the level of reflective thinking from students' written journals using a coding scheme based on the work of Mezirow. International Journal of Lifelong Education, 18(1), 18–30.

  • Kember, D., McKay, J., Sinclair, K., & Wong, F. K. Y. (2008). A four-category scheme for coding and assessing the level of reflection in written work. Assessment & Evaluation in Higher Education, 33(4), 369–379.

    Google Scholar 

  • Kim, Y., Almond, R., & Shute, V. (2016). Applying evidence-centered design for the development of game-based assessments in physics playground. International Journal of Testing, 16(2), 142–163.

    Google Scholar 

  • Kinnebrew, J. S., Gauch, B. C., Segedy, J. R., & Biswas, G. (2015). Studying student use of self-regulated learning tools in an open-ended learning environment. In Proceedings of the 20th international conference on artificial intelligence in education (pp. 185–194). Cham: Springer.

    Google Scholar 

  • Kostopoulos, G., Stamatis, K., & Kotsiantis, S. (2019). Multiview learning for early prognosis of academic performance: A case study. IEEE Transactions on Learning Technologies, 12(2), 212–224.

    Google Scholar 

  • Kovanović, V., Joksimović, S., Mirriahi, N., Blaine, E., Gašević, D., Siemens, G. , & Dawson, S. (2018). Understand students’ self-reflections through learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 389–398).

  • Lai, G., & Calandra, B. (2010). Examining the effects of computer-based scaffolds on novice teachers’ reflective journal writing. Educational Technology Research and Development, 58(4), 421–437.

    Google Scholar 

  • Lester, J., Spires, H., Nietfeld, J., Minogue, J., Mott, B., & Lobene, E. (2014). Designing game-based learning environments for elementary science education: A narrative-centered learning perspective. Information Sciences, 264, 4–18.

    Google Scholar 

  • Lykourentzou, I., Giannoukos, I., Mpardis, G., Nikolopoulos, V., & Loumos, V. (2009). Early and dynamic student achievement prediction in e-learning courses using neural networks. Journal of the American Society for Information Science and Technology, 60(2), 372–380.

    Google Scholar 

  • Mao, Y., Zhi, R., Khoshnevisan, F., Price, T., Barnes, M., and Chi, M. (2019). One minute is enough: Early prediction of student success and event-level difficulty during novice programming tasks. In Proceedings of the 12th International Conference on Educational Data Mining (pp. 119–128).

  • Min, W., Baikadi, A., Mott, B., Rowe, J., Liu, B., Ha, E. Y., & Lester, J. (2016). A generalized multidimensional evaluation framework for player goal recognition. In Proceedings of the 12th Artificial Intelligence and Interactive Digital Entertainment Conference (pp. 197–203).

  • Min, W., Frankosky, M. H., Mott, B. W., Rowe, J. P., Smith, A., Wiebe, E., Boyer, K. E., & Lester, J. C. (2020). DeepStealth: Game-based learning stealth assessment with deep neural networks. IEEE Transactions on Learning Technologies, 13(2), 312–325.

  • Min, W., Mott, B.; Rowe, J., Taylor, R., Wiebe, E., Boyer, K., and Lester, J. (2017a). Multimodal goal recognition in open-world digital games. In Proceedings of the 13th Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (pp. 80–86).

  • Min, W., Frankosky, M. H., Mott, B. W., Wiebe, E. N., Boyer, K. E., & Lester, J. C. (2017b). Inducing stealth assessors from game interaction data. In Proceedings of the Eighteenth International Conference on Artificial Intelligence in Education, (pp. 212–223).

  • Meluso, A., Zheng, M., Spires, H. A., & Lester, J. (2012). Enhancing 5th graders’ science content knowledge and self-efficacy through game-based learning. Computers & Education, 59(2), 497–504.

    Google Scholar 

  • Minott, M. A., (2008). Valli’s typology of reflection and the analysis of pre-service teachers’ reflective journals. Australian Journal of Teacher Education, 33(5), 55–65.

  • Nietfeld, J., Shores, L., & Hoffmann, K. (2014). Self-regulation and gender within a game-based learning environment. Journal of Educational Psychology, 106(4), 1–13.

    Google Scholar 

  • Olivé, D. M., Huynh, D. Q., Reynolds, M., Dougiamas, M., & Damyon, W. (2019). A quest for a one-size-fits-all neural network: Early prediction of students at risk in online courses. IEEE Transactions on Learning Technologies, 12(2), 171–183.

    Google Scholar 

  • Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825–2830.

    MathSciNet  MATH  Google Scholar 

  • Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011). Measuring emotions in students’ learning and performance: The achievement emotions questionnaire (AEQ). Contemporary Educational Psychology, 36(1), 36–48.

    Google Scholar 

  • Pennington, J., Socher, R., & Manning, C. (2014, October). GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (pp. 1532–1543).

  • Pesare, E., Roselli, T., Corriero, N., & Rossano, V. (2016). Game-based learning and gamification to promote engagement and motivation in medical learning contexts. Smart Learning Environments, 3(5), 1–21.

    Google Scholar 

  • Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2227–2237).

  • Piech, C., Spencer, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L., & Sohl-Dickstein, J. (2015). Deep knowledge tracing. Advances in Neural Information Processing Systems, 505–513.

  • Plass, J. L., Mayer, R. E., & Homer, B. D. (Eds.). (2020). Handbook of game-based learning. MIT Press.

  • Poldner, E., Van der Schaaf, M., Simons, P. R.-J., Van Tartwijk, J., & Wijngaards, G. (2014). Assessing student teachers’ reflective writing through quantitative content analysis. European Journal of Teacher Education, 37(3), 348–373.

    Google Scholar 

  • Polyzou, A., & Karypis, G. (2019). Feature extraction for next-term prediction of poor student performance. IEEE Transactions on Learning Technologies, 12(2), 237–248.

    Google Scholar 

  • Prilla, M., & Renner, B. (2014). Supporting collaborative reflection at work: A comparative case analysis. In Proceedings of the 18th international conference on supporting group work (pp. 182–193). New York: ACM Press.

    Google Scholar 

  • Qian, M., & Clark, K. (2016). Game-based learning and 21st century skills: A review of recent research. Computers in Human Behavior, 63, 50–58.

    Google Scholar 

  • Rogers, R. (2001). Reflection in higher education: A concept analysis. Innovative Higher Education, 26(1), 37–57.

    Google Scholar 

  • Rowe, J., Shores, L., Mott, B., & Lester, J. (2011). Integrating learning, problem solving, and engagement in narrative-centered learning environments. International Journal of Artificial Intelligence in Education, 21(1–2), 115–133.

    Google Scholar 

  • Sabourin, J., Mott, B., & Lester, J. (2013a). Discovering behavior patterns of self-regulated learners in an inquiry-based learning environment. In Proceedings of the 16th international conference on artificial intelligence in education (pp. 209–218). Berlin, Heidelberg: Springer.

    Google Scholar 

  • Sabourin, J., Mott, B., & Lester, J. (2013b). Utilizing dynamic Bayes nets to improve early prediction models of self-regulated learning. In Proceedings of the 21st International Conference on User Modeling, Adaptation, and Personalization (pp. 228–241).

  • San Pedro, M., Baker, R., Gowda, S., & Hetternan, N. (2013). Towards an understanding of affect and knowledge from student interaction with an intelligent tutoring system. In Proceedings of the 16th International Conference on Artificial Intelligence in Education, 41–50.

  • Schön, D. A. (2017). The reflective practitioner: How professionals think in action. Routledge.

  • Schunk, D. H., & Greene, J. A. (2018). Historical, contemporary, and future perspectives on self-regulated learning and performance. Handbook of Self-Regulation of Learning and Performance, 2, 1–16.

    Google Scholar 

  • Shen, D., Wang, G., Wang, W., Min, M., Su, Q., Zhang, Y., Li, C., Henao, R., and Carin, L. (2018). Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, (pp. 440–450).

  • Shute, V. (2011). Stealth assessment in computer-based games to support learning. Computer Games and Instruction, 55(2), 503–524.

    Google Scholar 

  • Shute, V., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106–117.

    Google Scholar 

  • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929–1958.

    MathSciNet  MATH  Google Scholar 

  • Taub, M., Mudrick, N., Bradbury, A. E., & Azevedo, R. (in press). Self-regulation, self-explanation, and reflection in game-based learning. In J. Plass, B. Horner, & R. Mayer (Eds.), Handbook of game-based learning. Boston: MIT Press.

  • Thorpe, K. (2004). Reflective learning journals: From concept to practice. Reflective Practice, 5(3), 327–343.

    Google Scholar 

  • Tsatsou, D., Vretos, N., & Daras, P. (2019). Adaptive game-based learning in multi-agent educational settings. Journal of Computers in Education, 6(2), 215–239.

    Google Scholar 

  • Ullmann, T. (2011). An architecture for the automated detection of textual indicators of reflection. In Proceedings of the 1st European Workshop on Awareness and Reflection in Learning Networks, 138–151.

  • Ullmann, T. (2015). Automated detection of reflection in texts: A machine learning based approach. PhD Thesis.

  • Ullmann, T. (2019). Automated analysis of reflection in writing: Validating machine learning approaches. International Journal of Artificial Intelligence in Education, 29(2), 217–257.

    Google Scholar 

  • Van den Boom, G., Paas, F., Van Merrienboer, J. J. G., & Van Gog, T. (2004). Reflection prompts and tutor feedback in a web-based learning environment: Effects on students' self-regulated learning competence. Computers in Human Behavior, 20, 551–567.

    Google Scholar 

  • Van den Boom, G., Paas, F., & Van Merrienboer, J. J. G. (2007). Effects of elicited reflections combined with tutor or peer feedback on self-regulated learning and learning outcomes. Learning and Instruction, 17(5), 532–548.

    Google Scholar 

  • Wang, L., Sy, A., Liu, L., & Piech, C. (2017). Learning to represent student knowledge on programming exercises using deep learning. In Proceedings of the 10th International Conference on Educational Data Mining (pp. 324–329).

  • Wieting, J., Bansal, M., Gimpel, K., and Livescu, K. (2016). Towards universal paraphrastic sentence embeddings. In Proceedings of the 4th International Conference on Learning Representations (pp. 1–19).

  • Winchell, M., Mozer, M., Lan, A., Grimaldi, P., & Pashler, H. (2018). Can textbook annotations serve as an early predictor of student learning? In Proceedings of the 11th International Conference on Educational Data Mining (pp. 431–437).

  • Winne, P., & Azevedo, R. (2014). Metacognition. In R.K. Sawyer (2nd Eds.), The Cambridge handbook of the learning sciences, (pp. 63–87). New York, NY.

  • Winne, P., & Hadwin, A. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. Graesser (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). Mahwah: Erlbaum.

    Google Scholar 

  • Wong, F., Kember, D., Chung, L., & CertEd, L. (1995). Assessing the level of student reflection from reflective journals. Journal of Advanced Nursing, 22(1), 48–57.

    Google Scholar 

  • Wu, M., Mosse, M., Goodman, N., & Piech, C. (2019). Zero shot learning for code education: Rubric sampling with deep learning inference. In Proceedings of the 33rd International Conference of the Association for Advancement of Artificial Intelligence (pp. 782–790).

  • Xiong, X., Zhao, S., van Inwegen, E., & Beck, J. (2016). Going deeper with deep knowledge tracing. In Proceedings of the 9th International Educational Data Mining (pp. 545–550).

Download references

Acknowledgements

This research was supported by funding from the National Science Foundation (NSF) under Grant DRL-1661202. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.

Availability of Data and Material

Data and materials created for this research are available upon request. Please direct all inquiries to the corresponding author.

Funding

This research was supported by funding from the National Science Foundation (NSF) under Grant DRL-1661202.

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Michael Geden, Andrew Emerson, Dan Carpenter, and Jonathan Rowe. The first draft of the manuscript was written by Michael Geden, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Andrew Emerson.

Ethics declarations

Conflicts of Interest/Competing Interests

No potential conflicts of interest.

Code Availability

Code created for this research is available upon request. Please direct all inquiries to the corresponding author.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Geden, M., Emerson, A., Carpenter, D. et al. Predictive Student Modeling in Game-Based Learning Environments with Word Embedding Representations of Reflection. Int J Artif Intell Educ 31, 1–23 (2021). https://doi.org/10.1007/s40593-020-00220-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40593-020-00220-4

Keywords

Navigation