EmoMA-Net: A Novel Model for Emotion Recognition Using Hybrid Multimodal Neural Networks in Adaptive Educational Systems
Pages 65 - 71
Abstract
This study presents an Emotion Recognition Multi-Attention Model (EmoMA-Net), a novel multimodal neural network aimed at enhancing real-time emotion recognition in educational environments. By leveraging the WESAD dataset, our model combines Convolutional Neural Networks (CNN), a Time Series Memory System (TSMS), and a Multi-Attention Mechanism to analyze diverse physiological signals, such as heart rate variability (HRV) and electroencephalogram (EEG). Unlike traditional emotion recognition methods reliant on subjective self-reports, our model delivers objective and accurate predictions of student stress levels through multimodal physiological data collected from wearable sensors. Achieving accuracy up to 99.66%, it facilitates adaptive educational systems to provide real-time feedback to educators, enabling prompt adjustments to teaching strategies. This advancement represents forward in emotion prediction technology, contributing to more responsive and adaptive educational experiences based on real-time emotional insights.
References
[1]
A Evangelos Triantafillou, Andreas Pomportsis, and Stavros Demetriadis. 2003a. The design and the formative evaluation of an adaptive educational system based on cognitive styles. Computers & Education 41, 1 (August 2003), 87–103.
[2]
Valerie J. Shute and Diego Zapata-Rivera. 2012. Adaptive Educational Systems. Adaptive Technologies for Training and Education (February 2012), 7–27.
[3]
Krithika, L. B., & GG, L. P. (2016). Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia Computer Science, 85, 767-776.
[4]
Broekens, J., & Brinkman, W.-P. (2013). Affectbutton: A method for reliable and valid affective self-report. International Journal of Human-Computer Studies, 71(6), 641–667.
[5]
Eisenbarth, H. (2008). Assessment of emotional detachment in psychopathy via self-report and an emotion detection task (Doctoral dissertation, Universität Würzburg).
[6]
Balconi, M., & Bortolotti, A. (2012). Detection of the facial expression of emotion and self-report measures in empathic situations are influenced by sensorimotor circuit inhibition by low-frequency rtms. Brain Stimulation, 5(3), 330–336.
[7]
Simonov, P. V. (2013). The emotional brain: physiology, neuroanatomy, psychology, and emotion. Springer Science & Business Media.
[8]
Siam, A. I., Soliman, N. F., Algarni, A. D., Abd El-Samie, F. E., & Sedik, A. (2022). Deploying machine learning techniques for human emotion detection. Computational Intelligence and Neuroscience, 2022, 1–16.
[9]
Binali, H., Wu, C., & Potdar, V. (2010). Computational approaches for emotion detection in text. 4th IEEE International Conference on Digital Ecosystems and Technologies.
[10]
Rim, B., Sung, N.-J., Min, S., & Hong, M. (2020). Deep learning in Physiological Signal Data: A survey. Sensors, 20(4), 969.
[11]
W. -L. Zheng, W. Liu, Y. Lu, B. -L. Lu and A. Cichocki, "EmotionMeter: A Multimodal Framework for Recognizing Human Emotions," in IEEE Transactions on Cybernetics, vol. 49, no. 3, pp. 1110-1122, March 2019.
[12]
Taurah, S. P., Bhoyedhur, J., & Sungkur, R. K. (2020). Emotion-based adaptive learning systems. In Machine Learning for Networking: Second IFIP TC 6 International Conference, MLN 2019, Paris, France, December 3–5, 2019, Revised Selected Papers 2 (pp. 273-286). Springer International Publishing.
[13]
Hwang, G.-J., Sung, H.-Y., Chang, S.-C., & Huang, X.-C. (2020). A fuzzy expert system-based adaptive learning approach to improving students’ learning performances by considering affective and cognitive factors. Computers and Education: Artificial Intelligence, 1, 100003.
[14]
Li, Q.; Ren, Y.; Wei, T.; Wang, C.; Liu, Z.; Yue, J. A Learning Attention Monitoring System via Photoplethysmogram Using Wearable Wrist Devices. In Artificial Intelligence Supported Educational Technologies; Advances in Analytics for Learning and Teaching; Pinkwart, N., Liu, S., Eds.; Springer International Publishing: Cham, Germany, 2020; pp. 133–150. ISBN 978-3-030-41099-5.
[15]
Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. 2018. Introducing WESAD, a multimodal dataset for wearable stress and affect detection. Proceedings of the 20th ACM International Conference on Multimodal Interaction (October 2018).
[16]
Soukaina Ennouamani and Zouhir Mahani. 2017. An overview of adaptive e-learning systems. 2017 Eighth International Conference on Intelligent Computing and Information Systems (ICICIS) (December 2017).
[17]
Tongshuai Song, Guanming Lu, and Jingjie Yan. 2020. Emotion recognition based on physiological signals using convolution neural networks. Proceedings of the 2020 12th International Conference on Machine Learning and Computing (February 2020).
[18]
J. Wang, X. Qiao, C. Liu, X. Wang, Y. Liu, L. Yao, H. Zhang, Automated ECG classification using a non-local convolutional block attention module, Comput. Methods Programs Biomed. 203 (2021) 106006,.
[19]
R. Ge, T. Shen, Y. Zhou, C. Liu, L. Zhang, B. Yang, Y. Yan, J.-L. Coatrieux,Y. Chen, Convolutional squeeze-and-excitation network for ECG arrhythmia detection, Artif. Intell. Med. 121 (2021) 102181,.
[20]
Tianqi Fan et al. 2023a. A new deep convolutional neural network incorporating attentional mechanisms for ECG emotion recognition. Computers in Biology and Medicine 159 (June 2023), 106938.
[21]
Liu, Y., Sun, G., Qiu, Y., Zhang, L., Chhatkuli, A., & Van Gool, L. (2021). Transformer in convolutional neural networks. arXiv preprint arXiv:2106.03180, 3.
[22]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (November 15, 1997), 1735–1780.
[23]
T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, Z. Cui, MPED: A multi-modal physiological emotion database for discrete emotion recognition, IEEE Access 7 (2019) 12177–12191,.
[24]
S. Hu, W. Cai, T. Gao, M. Wang, A hybrid transformer model for obstructive sleep apnea detection based on self-attention mechanism using single-lead ECG, IEEE Trans. Instrum. Meas. 71 (2022) 1–11,.
[25]
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018a. CBAM: Convolutional Block Attention Module. (July 2018). Retrieved July 24, 2024 from https://arxiv.org/abs/1807.06521
[26]
Liu, Y., Kang, J., Li, Y., & Ji, B. (2021). A network intrusion detection method based on CNN and CBAM. IEEE INFOCOM 2021 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).
[27]
Burcu F. Darst, Kristen C. Malecki, and Corinne D. Engelman. 2018. Using recursive feature elimination in random forest to account for correlated variables in high dimensional data. BMC Genetics 19, S1 (September 2018).
[28]
Pritam Sarkar and Ali Etemad. 2022b. Self-supervised ECG Representation Learning for emotion recognition. IEEE Transactions on Affective Computing 13, 3 (July 2022), 1541–1554.
[29]
Tianqi Fan et al. 2023. A new deep convolutional neural network incorporating attentional mechanisms for ECG emotion recognition. Computers in Biology and Medicine 159 (June 2023), 106938.
[30]
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021). Review of Deep Learning: Concepts, CNN Architectures, challenges, applications, Future Directions. Journal of Big Data, 8(1).
[31]
Hu, R., Chen, J., & Zhou, L. (2022). A transformer-based deep neural network for arrhythmia detection using continuous ECG signals. Computers in Biology and Medicine, 144, 105325.
[32]
Maximilian Beck et al. 2024. XLSTM: Extended Long Short-term memory. (May 2024). Retrieved July 24, 2024 from https://arxiv.org/abs/2405.04517
[33]
Sriram Kumar, P., Govarthan, P. K., Gadda, A. A., Ganapathy, N., & Ronickom, J. F. (2024). Deep learning-based automated emotion recognition using multi modal physiological signals and time-frequency methods. IEEE Transactions on Instrumentation and Measurement, 1–1.
Index Terms
- EmoMA-Net: A Novel Model for Emotion Recognition Using Hybrid Multimodal Neural Networks in Adaptive Educational Systems
Recommendations
A novel facial emotion recognition method for stress inference of facial nerve paralysis patients
Highlights- We built an emotional face video dataset from facial nerve paralysis patients.
- The emotions of facial nerve paralysis patients are recognizable via face images.
- Transfer learning helps conquer the problem of limited data size in ...
AbstractFacial nerve paralysis results in muscle weakness or complete paralysis on one side of the face. Patients suffer from difficulties in speech, mastication and emotional expression, impacting their quality of life by causing anxiety and depression. ...
Multimodal Emotion Recognition via Convolutional Neural Networks: Comparison of different strategies on two multimodal datasets
AbstractThe aim of this paper is to investigate emotion recognition using a multimodal approach that exploits convolutional neural networks (CNNs) with multiple input. Multimodal approaches allow different modalities to cooperate in order to achieve ...
Graphical abstractDisplay Omitted
Highlights- Emotion recognition through multimodal architectures.
- Comparison of early to late fusion of 1-input models.
- Synchronization of video and audio channels.
Comments
Information & Contributors
Information
Published In
September 2024
111 pages
ISBN:9798400716980
DOI:10.1145/3704289
Copyright © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 January 2025
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Xi'an Jiaotong-Liverpool University
Conference
ICBDE 2024
ICBDE 2024: 2024 the 7th International Conference on Big Data and Education
September 24 - 26, 2024
Oxford, United Kingdom
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 21Total Downloads
- Downloads (Last 12 months)21
- Downloads (Last 6 weeks)21
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in