skip to main content
10.1145/3576050.3576101acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
research-article
Public Access

Multimodal Predictive Student Modeling with Multi-Task Transfer Learning

Published: 13 March 2023 Publication History

Abstract

Game-based learning environments have the distinctive capacity to promote learning experiences that are both engaging and effective. Recent advances in sensor-based technologies (e.g., facial expression analysis and eye gaze tracking) and natural language processing have introduced the opportunity to leverage multimodal data streams for learning analytics. Learning analytics and student modeling informed by multimodal data captured during students’ interactions with game-based learning environments hold significant promise for designing effective learning environments that detect unproductive student behaviors and provide adaptive support for students during learning. Learning analytics frameworks that can accurately predict student learning outcomes early in students’ interactions hold considerable promise for enabling environments to dynamically adapt to individual student needs. In this paper, we investigate a multimodal, multi-task predictive student modeling framework for game-based learning environments. The framework is evaluated on two datasets of game-based learning interactions from two student populations (n=61 and n=118) who interacted with two versions of a game-based learning environment for microbiology education. The framework leverages available multimodal data channels from the datasets to simultaneously predict student post-test performance and interest. In addition to inducing models for each dataset individually, this work investigates the ability to use information learned from one source dataset to improve models based on another target dataset (i.e., transfer learning using pre-trained models). Results from a series of ablation experiments indicate the differences in predictive capacity among a combination of modalities including gameplay, eye gaze, facial expressions, and reflection text for predicting the two target variables. In addition, multi-task models were able to improve predictive performance compared to single-task baselines for one target variable, but not both. Lastly, transfer learning showed promise in improving predictive capacity in both datasets.

References

[1]
Pengcheng An, Kenneth Holstein, Bernice d'Anjou, Berry Eggen, and Saskia Bakker. 2020. The TA Framework: Designing Real-time Teaching Augmentation for K-12 Classrooms. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3313831
[2]
Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E. Mete, Eda Okur, Sidney K. D'Mello, and Asli Arslan Esme. 2019. Investigating the Impact of a Real-time, Multimodal Student Engagement Analytics Technology in Authentic Classrooms. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 304, 1–12. https://doi.org/10.1145/3290605.3300534
[3]
Roger Azevedo and Dragan Gašević. 2019. Analyzing multimodal multichannel data about self-regulated learning with advanced learning technologies: Issues and challenges. Computers in Human Behavior, 96, 207–210. https://doi.org/10.1016/j.chb.2019.03.025
[4]
Paulo Blikstein and Marcelo Worsley. 2016. Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238. https://doi.org/10.18608/jla.2016.32.11
[5]
Ritwick Chaudhry, Harvineet Singh, Pradeep Dogga, and Shiv Kumar Saini. 2018. Modeling Hint-Taking Behavior and Knowledge State of Students with Multi-Task Learning. In Proceedings of the 11th International Conference on Educational Data Mining, 21-31.
[6]
Sidney K. D'Mello and Arthur Graesser. 2010. Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User Adapted Interaction, 20, 147–187.
[7]
Paul Ekman and Erika Rosenberg. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS). New York, NY: Oxford University Press.
[8]
Andrew Emerson, Elizabeth B. Cloude, Roger Azevedo, and James Lester. 2020. Multimodal learning analytics for game‐based learning. British Journal of Educational Technology, 51(5), 1505-1526.
[9]
Andrew Emerson, Nathan Henderson, Jonathan Rowe, Wookhee Min, Seung Lee, James Minogue, and James Lester. 2020. Early Prediction of Visitor Engagement in Science Museums with Multimodal Learning Analytics. In Proceedings of the 2020 International Conference on Multimodal Interaction. Association for Computing Machinery, New York, NY, USA, 107–116. https://doi.org/10.1145/3382507.3418890.
[10]
Michael Geden, Andrew Emerson, Dan Carpenter, Jonathan Rowe, Roger Azevedo, and James Lester. 2021. Predictive student modeling in game-based learning environments with word embedding representations of reflection. International Journal of Artificial Intelligence in Education, 31, 1-23.
[11]
Michael Geden, Andrew Emerson, Jonathan Rowe, Roger Azevedo, and James Lester. 2020. Predictive student modeling in educational games with multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 34(01), 654-661. https://doi.org/10.1609/aaai.v34i01.5406
[12]
Nathan Henderson, Wookhee Min, Andrew Emerson, Jonathan Rowe, Seung Lee, James Minogue, and James Lester. 2021. Early Prediction of Museum Visitor Engagement with Multimodal Adversarial Domain Adaptation. In Proceedings of the 14th International Conference on Educational Data Mining, 93-104.
[13]
Nathan Henderson, Wookhee Min, Jonathan Rowe, and James Lester. 2020. Enhancing Affect Detection in Game-Based Learning Environments with Multimodal Conditional Generative Modeling. Proceedings of the 2020 International Conference on Multimodal Interaction. Association for Computing Machinery, New York, NY, USA, 134–143. https://doi.org/10.1145/3382507.3418892
[14]
Nathan Henderson, Wookhee Min, Jonathan Rowe, and James Lester. 2021. Enhancing Multimodal Affect Recognition with Multi-Task Affective Dynamics Modeling. In Proceedings of the 9th International Conference on Affective Computing and Intelligent Interaction (ACII), 1-8. IEEE.
[15]
iMotions. 2016. Attention Tool, 6.2. Boston, MA: iMotions Inc.
[16]
G. Tanner Jackson and Danielle S. McNamara. 2013. Motivation and performance in a game-based intelligent tutoring system. Journal of Educational Psychology, 105(4), 1036-1049.
[17]
Shamya Karumbaiah, Ryan Baker, and Valeria Shute. 2018. Predicting quitting in students playing a learning game. In Proceedings of the 12th International Conference on Educational Data Mining, 167–176.
[18]
Angelika Kasparova, Oya Celiktutan, and Mutlu Cukurova. 2020. Inferring Student Engagement in Collaborative Problem Solving from Visual Cues. In Companion Publication of the 2020 International Conference on Multimodal Interaction (ICMI '20 Companion). Association for Computing Machinery, New York, NY, USA, 177–181. https://doi.org/10.1145/3395035.3425961
[19]
Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7482–7491. IEEE Computer Society.
[20]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
[21]
Y. Alex Kolchinski, Sherry Ruan, Dan Schwartz, and Emma Brunskill. 2018. Adaptive natural-language targeting for student feedback. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (L@S '18). Association for Computing Machinery, New York, NY, USA, Article 26, 1–4. https://doi.org/10.1145/3231644.3231684
[22]
Mei-Jen Kuo. 2007. How does an online game-based learning environment promote students' intrinsic motivation for learning natural science and how does it affect their learning outcomes? In 2007 First IEEE International Workshop on Digital Game and Intelligent Toy Enhanced Learning, 135-142. IEEE.
[23]
James C. Lester, Eun Y. Ha, Seung Y. Lee, Bradford W. Mott, Jonathan P. Rowe, and Jennifer L. Sabourin. 2013. Serious games get smart: Intelligent game-based learning environments. AI Magazine, 34(4), 31-45. https://doi.org/10.1609/aimag.v34i4.2488
[24]
Ran Liu, John C. Stamper, and Jodi Davenport. 2018. A novel method for the in-depth multimodal analysis of student learning trajectories in intelligent tutoring systems. Journal of Learning Analytics, 5(1), 41-54. https://doi.org/10.18608/jla.2018.51.4
[25]
Ye Mao, Farzaneh Khoshnevisan, Thomas Price, Tiffany Barnes, and Min Chi. 2022. Cross-Lingual Adversarial Domain Adaptation for Novice Programming. In Proceedings of the AAAI Conference on Artificial Intelligence.
[26]
Lynn McAlpine, Cynthia Weston, Catherine Beauchamp, C. Wiseman, and Jacinthe Beauchamp. 1999. Building a metacognitive model of reflection. Higher education, 37(2), 105-131.
[27]
Wookhee Min, Alok Baikadi, Bradford Mott, Jonathan Rowe, Barry Liu, Eun Young Ha, and James Lester. 2016. A generalized multidimensional evaluation framework for player goal recognition. In Proceedings of the 12th International Conference on Artificial Intelligence and Interactive Digital Entertainment.
[28]
Wookhee Min, Megan Frankosky, Bradford W. Mott, Jonathan P. Rowe, Andy Smith, Eric Wiebe, Kristy Elizabeth Boyer, and James C. Lester. 2020. DeepStealth: Game-based learning stealth assessment with deep neural networks. IEEE Transactions on Learning Technologies, 13(2), 312-325.
[29]
Wookhee Min, Randall Spain, Jason D. Saville, Bradford Mott, Keith Brawner, Joan Johnston, and James Lester. 2021. Multidimensional team communication modeling for adaptive team training: A hybrid deep learning and graphical modeling framework. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence in Education (pp. 293-305). Springer, Cham.
[30]
Xavier Ochoa and Marcelo Worsley. 2016. Augmenting learning analytics with multimodal sensory data. Journal of Learning Analytics, 3, 213–219.
[31]
Sharon Oviatt, Joseph Grafsgaard, Lei Chen, and Xavier Ochoa. 2018. Multimodal learning analytics: assessing learners' mental state during the process of learning. The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition - Volume 2. Association for Computing Machinery and Morgan & Claypool, 331–374. https://doi.org/10.1145/3107990.3108003
[32]
Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), 1345-1359.
[33]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
[34]
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 1532– 1543.
[35]
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2227–2237.
[36]
Jan L. Plass, Richard E. Mayer, and Bruce D. Homer. 2020. Handbook of game-based learning. Cambridge, MA: MIT Press.
[37]
Konstantinos Pliakos, Seang-Hwane Joo, Jung Yeon Park, Frederik Cornillie, Celine Vens, and Wilm Van den Noortgate. 2019. Integrating machine learning into item response theory for addressing the cold start problem in adaptive learning systems. Computers & Education, 137, 91-103.
[38]
Luis P. Prieto, Kshitij Sharma, Pierre Dillenbourg, and María Jesús. 2016. Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (LAK '16). Association for Computing Machinery, New York, NY, USA, 148–157. https://doi.org/10.1145/2883851.2883927
[39]
Ramkumar Rajendran, Anurag Kumar, Kelly E. Carter, Daniel T. Levin, and Gautam Biswas. 2018. Predicting Learning by Analyzing Eye-Gaze Data of Reading Behavior. In Proceedings of the 12th International Conference on Educational Data Mining, 455-461.
[40]
Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372-422.
[41]
Richard M. Ryan. 1982. Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology, 43, 450–461.
[42]
Richard M. Ryan and Edward L. Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68–78.
[43]
Kshitij Sharma, Zacharoula Papamitsiou, Jennifer K. Olsen, and Michail Giannakos. 2020. Predicting learners' effortful behaviour in adaptive assessment using multimodal data. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (LAK '20). Association for Computing Machinery, New York, NY, USA, 480–489. https://doi.org/10.1145/3375462.3375498
[44]
Samuel Spaulding, Jocelyn Shen, Haewon Park, and Cynthia Breazeal. 2021. Towards Transferrable Personalized Student Models in Educational Games. In Proceedings of the 20th International Conference on Autonomous Agents and Multi Agent Systems, 1245-1253.
[45]
Yu Su, Qingwen Liu, Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Chris Ding, Si Wei, and Guoping Hu. 2018. Exercise-enhanced sequential modeling for student performance prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, 32(01), 2435-2443.
[46]
Chaohong Tan and Zhenhua Ling. 2019. Multi-Classification Model for Spoken Language Understanding. In 2019 International Conference on Multimodal Interaction (ICMI '19). Association for Computing Machinery, New York, NY, USA, 526–530. https://doi.org/10.1145/3340555.3356099
[47]
Michelle Taub, Nicholas Mudrick, Amanda E. Bradbury, and Roger Azevedo. 2019. Self-regulation, self-explanation, and reflection in game-based learning. In J. Plass, B. Horner, & R. Mayer (Eds.), Handbook of game-based learning, 239-262. Boston, MA: MIT Press.
[48]
Michelle Taub, Robert Sawyer, Andy Smith, Jonathan Rowe, Roger Azevedo, and James Lester. 2020. The agency effect: The impact of student agency on learning, emotions, and problem-solving behaviors in a game-based learning environment. Computers & Education, 147, 103781.
[49]
Güray Tonguç and Betul Ozaydın Ozkara. 2020. Automatic recognition of student emotions from facial expressions during a lecture. Computers & Education, 148, 103797.
[50]
Christos Troussas, Akrivi Krouska, and Cleo Sgouropoulou. 2020. Collaboration and fuzzy-modeled personalization for mobile game-based learning in higher education. Computers & Education, 144, 103698.
[51]
Gerard Van den Boom, Fred Paas, and Jeroen JG VanMerrienboer. 2007. Effects of elicited reflections combined with tutor or peer feedback on self-regulated learning and learning outcomes. Learning and Instruction, 17(5), 532–548.
[52]
Marcelo Worsley. 2018. (Dis)engagement matters: identifying efficacious learning practices with multimodal learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge. Association for Computing Machinery, New York, NY, USA, 365–369.

Cited By

View all
  • (2025)Gaze-Based Prediction of Students’ Math Difficulties - A Time Dynamic Machine Learning Approach to Enable Early Individual AssistanceInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00447-5Online publication date: 22-Jan-2025
  • (2023)Supporting Adolescent Engagement with Artificial Intelligence–Driven Digital Health Behavior Change InterventionsJournal of Medical Internet Research10.2196/4030625(e40306)Online publication date: 24-May-2023

Index Terms

  1. Multimodal Predictive Student Modeling with Multi-Task Transfer Learning

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference
      March 2023
      692 pages
      ISBN:9781450398657
      DOI:10.1145/3576050
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 13 March 2023

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Game-Based Learning
      2. Multimodal Learning Analytics
      3. Predictive Student Modeling

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Funding Sources

      Conference

      LAK 2023

      Acceptance Rates

      Overall Acceptance Rate 236 of 782 submissions, 30%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)254
      • Downloads (Last 6 weeks)21
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Gaze-Based Prediction of Students’ Math Difficulties - A Time Dynamic Machine Learning Approach to Enable Early Individual AssistanceInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00447-5Online publication date: 22-Jan-2025
      • (2023)Supporting Adolescent Engagement with Artificial Intelligence–Driven Digital Health Behavior Change InterventionsJournal of Medical Internet Research10.2196/4030625(e40306)Online publication date: 24-May-2023

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Login options

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media