skip to main content
10.1145/3448139.3448201acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
short-paper

Multimodal Data Collection Made Easy: The EZ-MMLA Toolkit: A data collection website that provides educators and researchers with easy access to multimodal data streams.

Published: 12 April 2021 Publication History

Abstract

While Multimodal Learning Analytics (MMLA) is becoming a popular methodology in the LAK community, most educational researchers still rely on traditional instruments for capturing learning processes (e.g., click-stream, log data, self-reports, qualitative observations). MMLA has the potential to complement and enrich traditional measures of learning by providing high frequency data on learners’ behavior, cognition and affects. However, there is currently no easy-to-use toolkit for recording multimodal data streams. Existing methodologies rely on the use of physical sensors and custom-written code for accessing sensor data. In this paper, we present the EZ-MMLA toolkit. This toolkit was implemented as a website that provides easy access to the latest machine learning algorithms for collecting a variety of data streams from webcams: attention (eye-tracking), physiological states (heart rate), body posture (skeletal data), hand gestures, emotions (from facial expressions and speech), and lower-level computer vision algorithms (e.g., fiducial / color tracking). This toolkit can run from any browser and does not require special hardware or programming experience. We compare this toolkit with traditional methods and describe a case study where the EZ-MMLA toolkit was used in a classroom context. We conclude by discussing other applications of this toolkit, potential limitations, and future steps.

References

[1]
Ahuja, K., Kim, D., Xhakaj, F., Varga, V., Xie, A., Zhang, S. & Agarwal, Y. (2019). EduSense: Practical classroom sensing at Scale. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(3), 1-26
[2]
Anderson, J.R. (2002), Spanning seven orders of magnitude: a challenge for cognitive modeling. Cognitive Science, 26: 85-112.
[3]
Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238.
[4]
Cukurova, M., Giannakos, M., & Martinez‐Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology.
[5]
Di Mitri, Daniele & Schneider, Jan & Specht, Marcus & Drachsler, Hendrik. (2019). Multimodal Pipeline: A generic approach for handling multimodal data for supporting learning.
[6]
Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, William T. Freeman. (2012). Eulerian Video Magnification for Revealing Subtle Changes in the World ACM Transactions on Graphics, Volume 31, Number 4 (Proc. SIGGRAPH),
[7]
iMotions (2013). iMotions Biometric Research Platform. Available at: https://imotions.com/.
[8]
Martinez-Maldonado, R., Mangaroska, K., Schulte, J., Elliott, D., Axisa, C., & Shum, S. B. (2020). Teacher Tracking with Integrity: What Indoor Positioning Can Reveal About Instructional Proxemics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1), 1-27.
[9]
Ocumpaugh, J., Baker, R.S.J.D., Rodrigo, M.A. (2012). Quantitative Field Observation (QFOs) Baker-Rodrigo Observation Method Protocol (BROMP) 1.0 Training Manual version 1.0 (October 17, 2012)
[10]
Saquib, N., Bose, A., George, D., & Kamvar, S. (2018). Sensei: Sensing Educational Interaction. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(4), 1-27.
[11]
Schneider, B., & Blikstein, P. (2014). Unraveling Students’ Interaction Around a Tangible Interface using Gesture Recognition. In J. Stamper, Z. Pardos, M. Mavrikis, B. Mclauren (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (EDM’14), (pp.320-323).
[12]
Schneider, B., Reilly, J. & Radu, I. (2020). Lowering Barriers for Accessing Sensor Data in Education: Lessons Learned from Teaching Multimodal Learning Analytics to Educators. Journal for STEM Educ Res 3, 91–124 . https://doi.org/10.1007/s41979-020-00027-x
[13]
Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015). Augmenting the senses: a review on sensor-based learning support. Sensors, 15(2), 4097-4133.
[14]
Sharma, Kshitij & Giannakos, Michail. (2020). Multimodal data capabilities for learning: What can multimodal data tell us about learning?. British Journal of Educational Technology. 10.1111/bjet.12993.
[15]
Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., & André, E. (2013). The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In Proceedings of the 21st ACM international conference on Multimedia (pp. 831-834).
[16]
Worsley, Marcelo & Blikstein, Paulo. (2011). What's an Expert? Using Learning Analytics to Identify Emergent Markers of Expertise through Automated Speech, Sentiment and Sketch Analysis. 235-240.
[17]
Zhang (2018): Zhang, Y., Qin, F., Liu, B., Qi, X., Zhao, Y., & Zhang, D. (2018). Wearable neurophysiological recordings in middle-school classroom correlate with students’ academic performance. Frontiers in Human Neuroscience, 12, 1–8

Cited By

View all
  • (2024)Your body tells how you engage in collaboration: Machine‐detected body movements as indicators of engagement in collaborative math knowledge buildingBritish Journal of Educational Technology10.1111/bjet.1347355:5(1950-1973)Online publication date: 10-May-2024
  • (2023)ModBand: Design of a Modular Headband for Multimodal Data Collection and InferenceAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586182.3616682(1-3)Online publication date: 29-Oct-2023
  • (2023)ChimeraPy: A Scientific Distributed Streaming Framework for Real-time Multimodal Data Retrieval and Processing2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386382(201-206)Online publication date: 15-Dec-2023
  • Show More Cited By
  1. Multimodal Data Collection Made Easy: The EZ-MMLA Toolkit: A data collection website that provides educators and researchers with easy access to multimodal data streams.

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    LAK21: LAK21: 11th International Learning Analytics and Knowledge Conference
    April 2021
    645 pages
    ISBN:9781450389358
    DOI:10.1145/3448139
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 April 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Computer Visions
    2. Data Collection Toolkit
    3. Multimodal Analytics

    Qualifiers

    • Short-paper
    • Research
    • Refereed limited

    Conference

    LAK21

    Acceptance Rates

    Overall Acceptance Rate 236 of 782 submissions, 30%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)103
    • Downloads (Last 6 weeks)7
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Your body tells how you engage in collaboration: Machine‐detected body movements as indicators of engagement in collaborative math knowledge buildingBritish Journal of Educational Technology10.1111/bjet.1347355:5(1950-1973)Online publication date: 10-May-2024
    • (2023)ModBand: Design of a Modular Headband for Multimodal Data Collection and InferenceAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586182.3616682(1-3)Online publication date: 29-Oct-2023
    • (2023)ChimeraPy: A Scientific Distributed Streaming Framework for Real-time Multimodal Data Retrieval and Processing2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386382(201-206)Online publication date: 15-Dec-2023
    • (2023)A Semantic Enhanced Course Recommender System via Knowledge Graphs for Limited User Information ScenariosSN Computer Science10.1007/s42979-023-02399-45:1Online publication date: 19-Dec-2023
    • (2023)Editorial: Nine elements for robust collaborative learning analytics: A constructive collaborative critiqueInternational Journal of Computer-Supported Collaborative Learning10.1007/s11412-023-09389-x18:1(1-9)Online publication date: 13-Mar-2023
    • (2022)Visualizing Collaboration in Teamwork: A Multimodal Learning Analytics Platform for Non-Verbal CommunicationApplied Sciences10.3390/app1215749912:15(7499)Online publication date: 26-Jul-2022
    • (2022)Toward capturing divergent collaboration in makerspaces using motion sensorsInformation and Learning Sciences10.1108/ILS-08-2020-0182123:5/6(276-297)Online publication date: 29-Mar-2022
    • (2022)Front-end deep learning web apps development and deployment: a reviewApplied Intelligence10.1007/s10489-022-04278-653:12(15923-15945)Online publication date: 30-Nov-2022
    • (2021)Investigate the Effects of Background Music on Visual Cognitive Tasks Using Multimodal Learning Analytics2021 International Conference on Advanced Learning Technologies (ICALT)10.1109/ICALT52272.2021.00141(445-447)Online publication date: Jul-2021

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media