skip to main content
10.1145/3382507.3418890acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Early Prediction of Visitor Engagement in Science Museums with Multimodal Learning Analytics

Published: 22 October 2020 Publication History

Abstract

Modeling visitor engagement is a key challenge in informal learning environments, such as museums and science centers. Devising predictive models of visitor engagement that accurately forecast salient features of visitor behavior, such as dwell time, holds significant potential for enabling adaptive learning environments and visitor analytics for museums and science centers. In this paper, we introduce a multimodal early prediction approach to modeling visitor engagement with interactive science museum exhibits. We utilize multimodal sensor data including eye gaze, facial expression, posture, and interaction log data captured during visitor interactions with an interactive museum exhibit for environmental science education, to induce predictive models of visitor dwell time. We investigate machine learning techniques (random forest, support vector machine, Lasso regression, gradient boosting trees, and multi-layer perceptron) to induce multimodal predictive models of visitor engagement with data from 85 museum visitors. Results from a series of ablation experiments suggest that incorporating additional modalities into predictive models of visitor engagement improves model accuracy. In addition, the models show improved predictive performance over time, demonstrating that increasingly accurate predictions of visitor dwell time can be achieved as more evidence becomes available from visitor interactions with interactive science museum exhibits. These findings highlight the efficacy of multimodal data for modeling museum exhibit visitor engagement.

Supplementary Material

MP4 File (3382507.3418890.mp4)
Presentation video.

References

[1]
Pengcheng An, Kenneth Holstein, Bernice d'Anjou, Berry Eggen, and Saskia Bakker. 2020. The TA Framework: Designing real-time teaching augmentation for K-12 classrooms. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems 1--17.
[2]
Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E. Mete, Eda Okur, Sidney K. D'Mello, and Asli Arslan Esme. 2019. Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--12.
[3]
Silvia Bach, Ulla Richardson, Daniel Brandeis, Ernst Martin, and Silvia Brem. 2013. Print-specific multimodal brain activation in kindergarten improves prediction of reading skills in second grade. Neuroimage 82, (2013), 605--615.
[4]
Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). 1--10.
[5]
Paulo Blikstein, and Marcelo Worsley. 2016. Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. J. of Learn. Analytics 3, 2 (2016). 220--238.
[6]
Florian Block, James Hammerman, Michael Horn, Amy Spiegel, Jonathan Christiansen, Brenda Phillips, Judy Diamond, E. Margaret Evans, and Chia Shen. 2015. Fluid grouping: Quantifying group engagement around interactive tabletop exhibits in the wild. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 867--876.
[7]
Nigel Bosch, Huili Chen, Sidney D'Mello, Ryan Baker, and Valeria Shute (2015, November). Accuracy vs. availability heuristic in multimodal affect detection in the wild. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. 267--274.
[8]
Nigel Bosch, Sidney K. D'Mello, Ryan S. Baker, Jaclyn Ocumpaugh, Valerie Shute, Matthew Ventura, Lubin Wang, and Weinan Zhao. 2016. Detecting student emotions in computer-enabled classrooms. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. 4125--4129.
[9]
Angelo Cafaro, Johannes Wagner, Tobias Baur, Soumia Dermouche, Mercedes Torres Torres, Catherine Pelachaud, Elisabeth André, and Michel Valstar. 2017. The NoXi database: Multimodal recordings of mediated novice-expert interactions. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. 350--359.
[10]
Cheng Chang, Cheng Zhang, Lei Chen, and Yang Liu. 2018. An ensemble model using face and body tracking for engagement detection. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 616--622.
[11]
Sidney D'Mello and Arthur Graesser. 2010. Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction 20, 2 (2010), 147--187.
[12]
Sidney D'Mello, Andrew Olney, Nathan Blanchard, Borhan Samei, Xiaoyi Sun, Brooke Ward, and Sean Kelly. 2015. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. In Proceedings of the 17th International Conference on Multimodal Interaction. 557--566.
[13]
Jeanine DeFalco, Jonathan Rowe, Luc Paquette, Vasiliki Georgoulas-Sherry, Keith Brawner, Bradford Mott, Ryan Baker, and James Lester. 2018. Detecting and addressing frustration in a serious game for military training. International Journal of Artificial Intelligence in Education 28, 2 (2018), 152--193.
[14]
Abhinav Dhall, Amanjot Kaur, Roland Goecke, and Tom Gedeon. 2018. Emotiw 2018: Audio-video, student engagement and group-level affect prediction. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 653--656.
[15]
Judy Diamond, Michael Horn, and David H. Uttal. 2016. Practical evaluation guide: Tools for museums and other informal educational settings. Rowman & Littlefield.
[16]
Eyal Dim and Tsvi Kuflik. 2014. Automatic detection of social behavior of museum visitor pairs. ACM Transactions on Interactive Intelligent Systems (TiiS) 4, 4 (2014). 1--30.
[17]
Vanessa Echeverría, Allan Avendaño, Katherine Chiluiza, Aníbal Vásquez, and Xavier Ochoa. 2014. Presentation skills estimation based on video and Kinect data analysis. In Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge. 53--60.
[18]
Lucca Eloy, Angela Stewart, Mary Jean Amon, Caroline Reinhardt, Amanda Michaels, Chen Sun, Valerie Shute, Nicholas Duran, and Sidney D'Mello. 2019. Modeling team-level multimodal dynamics during multiparty collaboration. In Proceedings of the 21st International Conference on Multimodal Interaction. 244258.
[19]
Andrew Emerson, Nathan Henderson, Jonathan Rowe, Wookhee Min, Seung Lee, James Minogue, and James Lester. 2020. Investigating visitor engagement in interactive science museum exhibits with multimodal Bayesian hierarchical models. In Proceedings of the 21st International Conference on Artificial Intelligence in Education. 165--176.
[20]
John H. Falk and Lynn D. Dierking. 2018. Learning from museums. Rowman & Littlefield.
[21]
Joseph Grafsgaard, Kristy Boyer, Eric Wiebe, and James Lester. 2012. Analyzing posture and affect in task-oriented tutoring. In Proceedings of the 25th International Florida Artificial Intelligence Research Society Conference. 438--443.
[22]
Lisa Halverson and Charles Graham. 2019. Learner engagement in blended learning environments: A conceptual framework. Online Learning 23, 2 (2019). 145--178.
[23]
Nathan Henderson, Jonathan Rowe, Luc Paquette, Ryan Baker, and James Lester. 2020. Improving affect detection in game-based learning with multimodal data fusion. In Proceedings of the 21st International Conference on Artificial Intelligence in Education. 228--239.
[24]
George Hein. 2009. Learning science in informal environments: People, places, and pursuits. Museums and Social Issues 4, 1 (2009). 113--124.
[25]
Karina Huang, Tonya Bryant, and Bertrand Schneider. 2019. Identifying collaborative learning states using unsupervised machine learning on eyetracking, physiological and motion sensor data. In Proceedings of the 12th International Conference on Educational Data Mining. 323--329.
[26]
Karen Knutson, Mandela Lyon, Kevin Crowley, and Lauren Giarratani. 2016. Flexible interventions to increase family engagement at natural history museum dioramas. Curator: The Museum Journal 59, 4 (2016). 339--352.
[27]
Tsvi Kuflik and Eyal Dim. 2013. Early detection of pairs of visitors by using a museum triage. In Proceedings of the Annual Conference of Museums and the Web. 113--124.
[28]
Chad Lane, Dan Noren, Daniel Auerbach, Mike Birch, and William Swartout. 2011. Intelligent tutoring goes to the museum in the big city: A pedagogical agent for informal science education. In Proceedings of the 15th International Conference on Artificial Intelligence in Education. 155--162.
[29]
Duri Long, Tom McKlin, Anna Weisling, William Martin, Hannah Guthrie, and Brian Magerko. 2019. Trajectories of physical engagement and expression in a co-creative museum installation. In Proceedings of the 2019 ACM SIGCHI Conference on Creativity and Cognition. 246--257.
[30]
Wookhee Min, Alok Baikadi, Bradford Mott, Jonathan Rowe, Barry Liu, Eun Young Ha, and James Lester. 2016. A generalized multidimensional evaluation framework for player goal recognition. In Proceedings of the 12th Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. 197203.
[31]
Wookhee Min, Bradford Mott, Jonathan Rowe, Robert Taylor, Eric Wiebe, Kristy Boyer, and James Lester. 2017. Multimodal goal recognition in open-world digital games. In Proceedings of the 13th Artificial Intelligence and Interactive Digital Entertainment Conference. 80--86.
[32]
Hamed Monkaresi, Nigel Bosch, Rafael Calvo, and Sidney D'Mello. 2016. Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Trans. on Affect. Comp. 8, 1 (2016). 15--28.
[33]
Xavier Ochoa. 2017. Multimodal learning analytics. The Handbook of Learning Analytics 1 (2017). 129--141.
[34]
Sharon Oviatt. 2018. Ten opportunities and challenges for advancing studentcentered multimodal learning analytics. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 87--94.
[35]
Sharon Oviatt, Joseph Grafsgaard, Lei Chen, and Xavier Ochoa. 2018. Multimodal learning analytics: Assessing learners' mental state during the process of learning. The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition 2, (2018). 331374.
[36]
Luis Prieto, Kshitij Sharma, Pierre Dillenbourg, and María Jesús. 2016. Teaching analytics: Towards automatic extraction of orchestration graphs using wearable sensors. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. 148--157.
[37]
Mirko Raca, and Pierre Dillenbourg. 2013. System for assessing classroom attention. In Proceedings of the Third International Conference on Learning Analytics and Knowledge. 265--269.
[38]
Mirko Raca, Roland Tormey, and Pierre Dillenbourg. 2014. Sleepers' lag-study on motion and attention. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge. 36--43.
[39]
Keith Rayner, Xingshan Li, Carrick C. Williams, Kyle R. Cave, and Arnold D. Well. 2007. Eye movements during information processing tasks: Individual differences and cultural effects. Vision Research 47, 21 (2007). 2714--2726.
[40]
Joseph Reilly, Milan Ravenell, and Bertrand Schneider. 2018. Exploring collaboration using motion sensors and multi-modal learning analytics. In Proceedings of the 11th International Conference on Educational Data Mining. 333339.
[41]
Jonathan P. Rowe, Eleni V. Lobene, Bradford W. Mott, and James C. Lester. 2017. Play in the museum: Design and development of a game-based learning exhibit for informal science education. International Journal of Gaming and ComputerMediated Simulations 9, 3 (2017). 96--113.
[42]
Robert Sawyer, Andy Smith, Jonathan Rowe, Roger Azevedo, and James Lester. 2017. Enhancing student models in game-based learning with facial expression recognition. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. 192--201.
[43]
Kshitij Sharma, Zacharoula Papamitsiou, and Michail Giannakos. 2019. Building pipelines for educational data using AI and multimodal analytics: A 'grey'box' approach. British Journal of Educational Technology 50, 6 (2019). 3004--3031.
[44]
Kshitij Sharma, Zacharoula Papamitsiou, Jennifer K. Olsen, and Michail Giannakos. 2020. Predicting learners' effortful behaviour in adaptive assessment using multimodal data. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge. 480--489.
[45]
Michelle Taub, Robert Sawyer, Andy Smith, Jonathan Rowe, Roger Azevedo, and James Lester. 2020. The agency effect: The impact of student agency on learning, emotions, and problem-solving behaviors in a game-based learning environment. Computers & Education 147, (2020).
[46]
Chinchu Thomas, Nitin Nair, and Dinesh Babu Jayagopi. 2018. Predicting engagement intensity in the wild using temporal convolutional network. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 604--610.
[47]
Jacob Whitehill, Zewelanji Serpell, Yi-Ching Lin, Aysha Foster, and Javier R. Movellan. 2014. The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing 5, 1 (2014). 86--98.
[48]
Joseph Wiggins, Mayank Kulkarni, Wookhee Min, Bradford Mott, Kristy Boyer, Eric Wiebe, and James Lester. 2018. Affect-based early prediction of player mental demand and engagement for educational games. In Proceedings of the 14th Artificial Intelligence and Interactive Digital Entertainment Conference. 243249.
[49]
Marcelo Worsley, Stefan Scherer, Louis-Philippe Morency, and Paulo Blikstein. 2015. Exploring behavior representation for learning analytics. In Proceedings of the 2015 ACM International Conference on Multimodal Interaction. 251--258.
[50]
Liang-Chih Yu, C. W. Lee, H. I. Pan, Chih-Yueh Chou, Po-Yao Chao, Z. H. Chen, S. F. Tseng, C. L. Chan, and K. Robert Lai. 2018. Improving early prediction of academic failure using sentiment analysis on self-evaluated comments. Journal of Computer Assisted Learning 34, 4 (2018). 358--365.

Cited By

View all
  • (2024)SEMPI: A Database for Understanding Social Engagement in Video-Mediated Multiparty InteractionProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685752(546-555)Online publication date: 4-Nov-2024
  • (2024)Linger: Extending Museum Experiences to Post-Visit PhasesAdjunct Proceedings of the 2024 Nordic Conference on Human-Computer Interaction10.1145/3677045.3685436(1-5)Online publication date: 13-Oct-2024
  • (2024)A systematic review of digital transformation technologies in museum exhibitionComputers in Human Behavior10.1016/j.chb.2024.108407161(108407)Online publication date: Dec-2024
  • Show More Cited By

Index Terms

  1. Early Prediction of Visitor Engagement in Science Museums with Multimodal Learning Analytics

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction
      October 2020
      920 pages
      ISBN:9781450375818
      DOI:10.1145/3382507
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 22 October 2020

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. early prediction
      2. multimodal learning analytics
      3. museum-based learning
      4. visitor modeling

      Qualifiers

      • Research-article

      Conference

      ICMI '20
      Sponsor:
      ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
      October 25 - 29, 2020
      Virtual Event, Netherlands

      Acceptance Rates

      Overall Acceptance Rate 453 of 1,080 submissions, 42%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)108
      • Downloads (Last 6 weeks)10
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)SEMPI: A Database for Understanding Social Engagement in Video-Mediated Multiparty InteractionProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685752(546-555)Online publication date: 4-Nov-2024
      • (2024)Linger: Extending Museum Experiences to Post-Visit PhasesAdjunct Proceedings of the 2024 Nordic Conference on Human-Computer Interaction10.1145/3677045.3685436(1-5)Online publication date: 13-Oct-2024
      • (2024)A systematic review of digital transformation technologies in museum exhibitionComputers in Human Behavior10.1016/j.chb.2024.108407161(108407)Online publication date: Dec-2024
      • (2023)Supporting Adolescent Engagement with Artificial Intelligence–Driven Digital Health Behavior Change InterventionsJournal of Medical Internet Research10.2196/4030625(e40306)Online publication date: 24-May-2023
      • (2023)Real-time Multi-CNN-based Emotion Recognition System for Evaluating Museum Visitors’ SatisfactionJournal on Computing and Cultural Heritage 10.1145/363112317:1(1-18)Online publication date: 30-Oct-2023
      • (2023)Social Visual Behavior Analytics for Autism Therapy of Children Based on Automated Mutual Gaze DetectionProceedings of the 8th ACM/IEEE International Conference on Connected Health: Applications, Systems and Engineering Technologies10.1145/3580252.3586976(11-21)Online publication date: 21-Jun-2023
      • (2023)Multimodal Predictive Student Modeling with Multi-Task Transfer LearningLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576101(333-344)Online publication date: 13-Mar-2023
      • (2023)Positive Artificial Intelligence in Education (P-AIED): A RoadmapInternational Journal of Artificial Intelligence in Education10.1007/s40593-023-00357-y34:3(732-792)Online publication date: 3-Aug-2023
      • (2022)Effects of learner control design in an AR-based exhibit on visitors’ museum learningPLOS ONE10.1371/journal.pone.027482617:10(e0274826)Online publication date: 18-Oct-2022
      • (2022)Predicting User Confidence in Video Recordings with Spatio-Temporal Multimodal AnalyticsCompanion Publication of the 2022 International Conference on Multimodal Interaction10.1145/3536220.3558007(98-104)Online publication date: 7-Nov-2022
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media