skip to main content
10.1145/3576050.3576079acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
research-article
Public Access

Effects of Modalities in Detecting Behavioral Engagement in Collaborative Game-Based Learning

Published: 13 March 2023 Publication History

Abstract

Collaborative game-based learning environments have significant potential for creating effective and engaging group learning experiences. These environments offer rich interactions between small groups of students by embedding collaborative problem solving within immersive virtual worlds. Students often share information, ask questions, negotiate, and construct explanations between themselves towards solving a common goal. However, students sometimes disengage from the learning activities, and due to the nature of collaboration, their disengagement can propagate and negatively impact others within the group. From a teacher's perspective, it can be challenging to identify disengaged students within different groups in a classroom as they need to spend a significant amount of time orchestrating the classroom. Prior work has explored automated frameworks for identifying behavioral disengagement. However, most prior work relies on a single modality for identifying disengagement. In this work, we investigate the effects of using multiple modalities to detect disengagement behaviors of students in a collaborative game-based learning environment. For that, we utilized facial video recordings and group chat messages of 26 middle school students while they were interacting with Crystal Island: EcoJourneys, a game-based learning environment for ecosystem science. Our study shows that the predictive accuracy of a unimodal model heavily relies on the modality of the ground truth, whereas multimodal models surpass the unimodal models, trading resources for accuracy. Our findings can benefit future researchers in designing behavioral engagement detection frameworks for assisting teachers in using collaborative game-based learning within their classrooms.

References

[1]
Atezaz Ahmad, Jan Schneider, Dai Griffiths, Daniel Biedermann, Daniel Schiffner, Wolfgang Greller, and Hendrik Drachsler. 2022. Connecting the dots–A literature review on learning analytics indicators from a learning design perspective. Journal of Computer Assisted Learning (2022).
[2]
Laura K Allen, Caitlin Mills, Matthew E Jacovina, Scott Crossley, Sidney D'mello, and Danielle S McNamara. 2016. Investigating boredom and engagement during writing using multiple sources of information: the essay, the writer, and keystrokes. In Proceedings of the 6th International Conference on Learning Analytics & Knowledge, 114–123.
[3]
Sinem Aslan, Sinem Emine Mete, Eda Okur, Ece Oktay, Nese Alyuz, Utku Ergin Genc, David Stanhill, and Asli Arslan Esme. 2017. Human expert labeling process (HELP): Towards a reliable higher-order user state labeling process and tool to assess student engagement. Educational Technology (2017), 53–59. Retrieved from https://www.jstor.org/stable/44430540
[4]
Ryan SJd Baker, Sidney K D'Mello, Ma Mercedes T Rodrigo, and Arthur C Graesser. 2010. Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68, 4 (2010), 223–241.
[5]
Robert Bixler and Sidney D'Mello. 2016. Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Modeling and User-Adapted Interaction 26, 1 (2016), 33–68.
[6]
Dan Carpenter, Andrew Emerson, Bradford W Mott, Asmalina Saleh, Krista D Glazewski, Cindy E Hmelo-Silver, and James C Lester. 2020. Detecting off-task behavior from student dialogue in game-based collaborative learning. In Proceedings of the 21st International Conference on Artificial Intelligence in Education, Springer, 55–66.
[7]
Juanjuan Chen, Minhong Wang, Paul A Kirschner, and Chin-Chung Tsai. 2018. The role of collaboration, computer use, learning environments, and supporting strategies in CSCL: A meta-analysis. Review of Educational Research 88, 6 (2018), 799–843.
[8]
Sidney D'Mello and Art Graesser. 2007. Monitoring affective trajectories during complex learning. In Proceedings of the Annual Meeting of the Cognitive Science Society, 203–208. Retrieved from https://escholarship.org/uc/item/6p18v65q
[9]
Sidney K D'mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR) 47, 3 (2015), 1–36.
[10]
Sidney S D'Mello, Patrick Chipman, and Art Graesser. 2007. Posture as a predictor of learner's affective engagement. In Proceedings of the Annual Meeting of the Cognitive Science Society. Retrieved from https://escholarship.org/uc/item/7hs9v2hr
[11]
Ryan S J d Baker, Gregory R Moore, Angela Z Wagner, Jessica Kalka, Aatish Salvi, Michael Karabinos, Colin A Ashe, and David Yaron. 2011. The dynamics between student affect and behavior occurring outside of educational software. In Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, Springer, 14–24.
[12]
Fahmid Morshed Fahid, Halim Acosta, Seung Lee, Dan Carpenter, Bradford Mott, Haesol Bae, Asmalina Saleh, Thomas Brush, Krista Glazewski, Cindy E Hmelo-Silver, and James Lester. 2022. Multimodal behavioral disengagement detection for collaborative game-based learning. In Proceedings of the 23rd International Conference on Artificial Intelligence in Education, Springer, 218–221.
[13]
Simon Greipl, Korbinian Moeller, and Manuel Ninaus. 2020. Potential and limits of game-based learning. International Journal of Technology Enhanced Learning 12, 4 (2020), 363–389.
[14]
J C Huizenga, G T M Ten Dam, J M Voogt, and W F Admiraal. 2017. Teacher perceptions of the value of game-based learning in secondary education. Computers & Education 110, (2017), 105–115.
[15]
Heisawn Jeong, Cindy E Hmelo-Silver, and Kihyun Jo. 2019. Ten years of computer-supported collaborative learning: A meta-analysis of CSCL in STEM education during 2005–2014. Educational Research Review 28, (2019), 100284.
[16]
Ângelo Magno de Jesus and Ismar Frango Silveira. 2019. A collaborative game-based learning framework to improve computational thinking skills. In Proceedings of the 9th International Conference on Virtual Reality and Visualization, IEEE, 161–166.
[17]
Kyungbin Kwon, Ying-Hsiu Liu, and LaShaune P Johnson. 2014. Group regulation and social-emotional interactions observed in computer supported collaborative learning: Comparison between good vs. poor collaborators. Computers & Education 78, (2014), 185–200.
[18]
Ha Le, Jeroen Janssen, and Theo Wubbels. 2018. Collaborative learning practices: teacher and student perceived obstacles to effective student collaboration. Cambridge Journal of Education 48, 1 (2018), 103–122.
[19]
Taeckyung Lee, Dain Kim, Sooyoung Park, Dongwhi Kim, and Sung-Ju Lee. 2022. Predicting mind-wandering with facial videos in online lectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2104–2113.
[20]
Gulipari Maimaiti, Chengyuan Jia, and Khe Foon Hew. 2021. Student disengagement in web-based videoconferencing supported online learning: an activity theory perspective. Interactive Learning Environments (2021), 1–20.
[21]
Jonna Malmberg, Sanna Järvelä, Jukka Holappa, Eetu Haataja, Xiaohua Huang, and Antti Siipo. 2019. Going beyond what is visible: What multichannel data can reveal about interaction in the context of collaborative learning? Computers in Human Behavior 96, (2019), 235–245.
[22]
Su Mu, Meng Cui, and Xiaodi Huang. 2020. Multimodal data fusion in learning analytics: A systematic review. Sensors 20, 23 (2020), 6856.
[23]
Stefanos Nikiforos, Spyros Tzanavaris, and Katia-Lida Kermanidis. 2020. Virtual learning communities (VLCs) rethinking: influence on behavior modification—bullying detection through machine learning and natural language processing. Journal of Computers in Education 7, 4 (2020), 531–551.
[24]
Zachary A Pardos, Ryan S J D Baker, Maria O C Z San Pedro, Sujith M Gowda, and Supreeth M Gowda. 2013. Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In Proceedings of the third international conference on learning analytics and knowledge, 117–124.
[25]
Kyungjin Park, Hyunwoo Sohn, Bradford Mott, Wookhee Min, Asmalina Saleh, Krista Glazewski, Cindy Hmelo-Silver, and James Lester. 2021. Detecting disruptive talk in student chat-based discussion within collaborative game-based learning environments. In Proceedings of the 11th International Learning Analytics and Knowledge Conference, 405–415.
[26]
Luis P Prieto, Kshitij Sharma, and Pierre Dillenbourg. 2015. Studying teacher orchestration load in technology-enhanced classrooms. In European Conference on Technology Enhanced Learning, Springer, 268–281.
[27]
Asmalina Saleh, Chen Feng, Haesol Bae, Cindy E Hmelo-Silver, K Glazewski, Seung Lee, Bradford Mott, and James Lester. 2021. Negotiating accountability and epistemic stances in middle-school collaborative discourse. In Proceedings of the 14th International Conference on Computer-Supported Collaborative Learning, 197–200.
[28]
Yusuf Can Semerci and Dionysis Goularas. 2021. Evaluation of students’ flow state in an e-learning environment through activity and performance using deep learning techniques. Journal of Educational Computing Research 59, 5 (2021), 960–987.
[29]
Kshitij Sharma and Michail Giannakos. 2020. Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology 0, 0 (2020), 1–35.
[30]
Kshitij Sharma and Michail Giannakos. 2020. Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology 51, 5 (2020), 1450–1484.
[31]
Kshitij Sharma, Zacharoula Papamitsiou, Jennifer K Olsen, and Michail Giannakos. 2020. Predicting learners’ effortful behaviour in adaptive assessment using multimodal data. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge, 480–489.
[32]
Aixin Sun, Ee-Peng Lim, and Ying Liu. 2009. On strategies for imbalanced text classification using SVM: A comparative study. Decision Support Systems 48, 1 (2009), 191–201.
[33]
Chinchu Thomas and Dinesh Babu Jayagopi. 2017. Predicting student engagement in classrooms using facial behavioral cues. In Proceedings of the 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education, 33–40.
[34]
Jacob Whitehill, Zewelanji Serpell, Yi-Ching Lin, Aysha Foster, and Javier R. Movellan. 2014. The faces of engagement: Automatic recognition of student engagementfrom facial expressions. IEEE Transactions on Affective Computing 5, 1 (2014), 86–98.
[35]
Zi Yang Wong and Gregory Arief D Liem. 2021. Student engagement: Current state of the construct, conceptual refinement, and future research directions. Educational Psychology Review 34 (2021), 107–138.
[36]
Marcelo Worsley. 2018. (Dis) engagement matters: Identifying efficacious learning practices with multimodal learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, 365–369.
[37]
Marcelo Worsley and Paulo Blikstein. 2018. A multimodal analysis of making. International Journal of Artificial Intelligence in Education 28, 3 (2018), 385–419.
[38]
Jialing Zeng, Sophie Parks, and Junjie Shang. 2020. To learn scientifically, effectively, and enjoyably: A review of educational games. Human Behavior and Emerging Technologies 2, 2 (2020), 186–195.

Cited By

View all
  • (2024)Pedagogical memes: a creative and effective tool for teaching STEM subjectsInternational Journal of Mathematical Education in Science and Technology10.1080/0020739X.2024.2328818(1-31)Online publication date: 26-Mar-2024
  • (2024)Gamification Based Collaborative Learning: The Impact of Rewards on Student MotivationTowards a Hybrid, Flexible and Socially Engaged Higher Education10.1007/978-3-031-51979-6_13(124-130)Online publication date: 1-Feb-2024
  • (2023)Exploring the Impact of Collaborative Behavior on Digital Collaborative Game-based Learning2023 8th International Conference on Multimedia Communication Technologies (ICMCT)10.1109/ICMCT60483.2023.00017(57-61)Online publication date: 4-Aug-2023

Index Terms

  1. Effects of Modalities in Detecting Behavioral Engagement in Collaborative Game-Based Learning

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference
        March 2023
        692 pages
        ISBN:9781450398657
        DOI:10.1145/3576050
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 13 March 2023

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Behavioral engagement
        2. Collaborative game-based learning
        3. K-12 education
        4. Multimodal learning analytics

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        Conference

        LAK 2023

        Acceptance Rates

        Overall Acceptance Rate 236 of 782 submissions, 30%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)254
        • Downloads (Last 6 weeks)28
        Reflects downloads up to 16 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Pedagogical memes: a creative and effective tool for teaching STEM subjectsInternational Journal of Mathematical Education in Science and Technology10.1080/0020739X.2024.2328818(1-31)Online publication date: 26-Mar-2024
        • (2024)Gamification Based Collaborative Learning: The Impact of Rewards on Student MotivationTowards a Hybrid, Flexible and Socially Engaged Higher Education10.1007/978-3-031-51979-6_13(124-130)Online publication date: 1-Feb-2024
        • (2023)Exploring the Impact of Collaborative Behavior on Digital Collaborative Game-based Learning2023 8th International Conference on Multimedia Communication Technologies (ICMCT)10.1109/ICMCT60483.2023.00017(57-61)Online publication date: 4-Aug-2023

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Login options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media