Abstract
It is crucial to evaluate learning outcomes by identifying the cognitive state of the learner during the learning process. Studies utilizing Electroencephalography (EEG) and other peripheral physiological signals, combined with deep learning models, have demonstrated improved performance in cognitive state recognition. These studies have primarily focused on unimodal data, which are vulnerable to various types of noise, making it difficult to fully capture and represent cognitive states. Leveraging the complementarity between multimodal physiological signals can mitigate the impact of anomalies in unimodal data, thereby improving the accuracy and stability of cognitive state recognition. Therefore, this study proposes a multimodal physiological signal feature representation fusion model based on multi-level attention (PSFMMA). The model aims to integrate multimodal physiological signals to identify learners’ cognitive states with greater stability and accuracy. PSFMMA first extracts the temporal features of physiological signals by multiplexing the embedding layer. Subsequently, it generates signal representation vectors by further extracting semantic features through a signal feature mapping layer and enhancing important features with designed attention modules. Finally, the model employs an attention mechanism based on different signal representation vectors to fuse multimodal information for identifying learners’ cognitive states. This study designs various learning activities and collects electroencephalography (EEG), electrodermal activity (EDA), and photoplethysmography (PPG) data from 22 participants engaging in these activities to create the Based on Learning Activities Collection (BLAC) dataset. The proposed model was evaluated on the BLAC dataset, achieving an identification accuracy of 96.32 ± 0.32%. The results demonstrate that the model can effectively recognize learners’ cognitive states. Furthermore, the model’s performance was validated on the publicly available emotion classification dataset DEAP, attaining an accuracy of 99.15 ± 0.12%. The source code is available at https://github.com/chengshudaxuesheng/PSFMMA.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Availability of Data and Materials
The datasets are not publicly available due to protect the privacy of study participants. Specifc inquiries regarding the data may be directed to the corresponding author.
References
Elkerdawy M, Elhalaby M, Hassan A, Maher M, Shawky D, Badawi A (2020) Building cognitive profiles of learners using eeg. In: 2020 11th International Conference on Information and Communication Systems (ICICS), IEEE, pp 027–032
Jamil N, Belkacem AN, Lakas A (2023) On enhancing students’ cognitive abilities in online learning using brain activity and eye movements. Educ Inf Technol 28(4):4363–4397
Liu X, Tan PN, Liu L, Simske SJ (2017) Automated classification of eeg signals for predicting students’ cognitive state during learning. In: Proceedings of the international conference on web intelligence, pp 442–450
Gerjets P, Walter C, Rosenstiel W, Bogdan M, Zander TO (2014) Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain-computer interface approach. Front Neurosci 8:385
Chen S, Zhang J (2008) The adaptive learning system based on learning style and cognitive state. In: 2008 International symposium on knowledge acquisition and modeling, IEEE, pp 302–306
Bhattacharya S, Roy S, Chowdhury S (2018) A neural network-based intelligent cognitive state recognizer for confidence-based e-learning system. Neural Comput Appl 29:205–219
D’Mello S, Olney A, Williams C, Hays P (2012) Gaze tutor: a gaze-reactive intelligent tutoring system. Int J Hum Comput Stud 70(5):377–398
Caram CA, Davis PB (2005) Inviting student engagement with questioning. Kappa Delta Pi Record 42(1):18–23
Khedher AB, Jraidi I, Frasson C (2019) Tracking students’ mental engagement using eeg signals during an interaction with a virtual learning environment. J Intell Learn Syst Appl 11(01):1–14
Liu S, Liu S, Liu Z, Peng X, Yang Z (2022) Automated detection of emotional and cognitive engagement in mooc discussions to predict learning achievement. Comput Educ 181:104461
Du X, Zhang L, Hung JL, Li H, Tang H, Dai M (2022) Analyzing the effects of instructional strategies on students’ on-task status from aspects of their learning behaviors and cognitive factors. J Comput Higher Educ:1–28
Charlton SG (2019) Measurement of cognitive states in test and evaluation. In: Handbook of human factors testing and evaluation. CRC Press, pp 97–126
Ciolacu M, Tehrani AF, Binder L, Svasta PM (2018) Education 4.0-artificial intelligence assisted higher education: early recognition system with machine learning to support students’ success. In: 2018 IEEE 24th International Symposium for Design and Technology in Electronic Packaging (SIITME), IEEE, pp 23–30
Dewan M, Murshed M, Lin F (2019) Engagement detection in online learning: a review. Smart Learn Environ 6(1):1–20
Dirican AC, Göktürk M (2011) Psychophysiological measures of human cognitive states applied in human computer interaction. Procedia Comput Sci 3:1361–1367
Lohani M, Payne BR, Strayer DL (2019) A review of psychophysiological measures to assess cognitive states in real-world driving. Front Hum Neurosci 13:57
Borghini G, Aricò P, Graziani I, Salinari S, Sun Y, Taya F et al (2016) Quantitative assessment of the training improvement in a motor-cognitive task by using eeg, ecg and eog signals. Brain Topogr 29:149–161
Dahal N, Nandagopal N, Nafalski A, Nedic Z (2011) Modeling of cognition using eeg: a review and a new approach. In: TENCON 2011-2011 IEEE region 10 conference, IEEE, pp 1045–1049
Zhou Y, Huang S, Xu Z, Wang P, Wu X, Zhang D (2021) Cognitive workload recognition using eeg signals and machine learning: a review. IEEE Trans Cogn Develop Syst
Katmah R, Al-Shargie F, Tariq U, Babiloni F, Al-Mughairbi F, Al-Nashash H (2021) A review on mental stress assessment methods using eeg signals. Sensors 21(15):5043
Cheng Y, Cai Y, Chen H, Cai Z, Wu G, Huang J (2021) A cognitive level evaluation method based on a deep neural network for online learning: from a bloom’s taxonomy of cognition objectives perspective. Front Psychol 12:661235
Bhise PR, Kulkarni SB, Aldhaheri TA (2021) Survey on assessment of cognitive states during learning activities using brain computer interface based eeg. Int J Comput Appl 975:8887
Monteiro TG, Skourup C, Zhang H (2020) Optimizing cnn hyperparameters for mental fatigue assessment in demanding maritime operations. IEEE Access 8:40402–40412
Han SY, Kwak NS, Oh T, Lee SW (2020) Classification of pilots’ mental states using a multimodal deep learning network. Biocybernetics Biomed Eng 40(1):324–336
Li J, Wang Q (2022) Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation. Inform Fusion 79:229–247
Antonenko P, Paas F, Grabner R, Van Gog T (2010) Using electroencephalography to measure cognitive load. Educ Psychol Rev 22:425–438
Mills C, Fridman I, Soussou W, Waghray D, Olney AM, D’Mello SK (2017) Put your thinking cap on: detecting cognitive load using eeg during learning. In: Proceedings of the seventh international learning analytics & knowledge conference, pp 80–89
Zhou Y, Xu T, Cai Y, Wu X, Dong B (2017) Monitoring cognitive workload in online videos learning through an eeg-based brain-computer interface. In: Learning and Collaboration Technologies. Novel Learning Ecosystems: 4th International Conference, LCT 2017, Held as Part of HCI International 2017, Vancouver, BC, Canada, July 9-14, 2017, Proceedings, Part I 4, Springer, pp 64–73
Dey AK, Poddar B, Pramanik PKD, Debnath NC, Aljahdali S, Choudhury P (2020) Real-time learner classification using cognitive score. In: CATA, pp 264–276
Sinha A, Gavas R, Chatterjee D, Das R, Sinharay A (2015) Dynamic assessment of learners’ mental state for an improved learning experience. In: 2015 IEEE frontiers in education conference (FIE), IEEE, pp 1–9
Dutta S, Nandy A (2020) An extensive analysis on deep neural architecture for classification of subject-independent cognitive states. In: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD. Association for Computing Machinery, New York, NY, USA, CoDS COMAD 2020, p 180–184
Qayyum A, Khan MA, Mazher M, Suresh M (2018) Classification of eeg learning and resting states using 1d-convolutional neural network for cognitive load assesment. In: 2018 IEEE Student Conference on Research and Development (SCOReD), IEEE, pp 1–5
Pandey P, Miyapuram KP (2021) Brain2depth: Lightweight cnn model for classification of cognitive states from eeg recordings. In: Medical Image Understanding and Analysis: 25th Annual Conference, MIUA 2021, Oxford, United Kingdom, July 12–14, 2021, Proceedings 25, Springer, pp 394–407
Li P, Zhang Y, Liu S, Lin L, Zhang H, Tang T et al (2023) An eeg-based brain cognitive dynamic recognition network for representations of brain fatigue. Appl Soft Comput 146:110613
Kalra P, Sharma V (2023) Mental stress assessment using ppg signal a deep neural network approach. IETE J Res 69(2):879–885
Butkevičiūtė E, Michalkovič A, Bikulčienė L (2022) Ecg signal features classification for the mental fatigue recognition. Mathematics 10(18):3395
Alreshidi I, Moulitsas I, Jenkins KW (2023) Multimodal approach for pilot mental state detection based on eeg. Sensors 23(17):7350
Gao N, Shao W, Rahaman MS, Salim FD (2020) n-gage: predicting in-class emotional, behavioural and cognitive engagement in the wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4(3):1–26
Markova V, Ganchev T, Kalinkov K, Markov M (2021) Detection of acute stress caused by cognitive tasks based on physiological signals. Bulletin Electrical Eng Inform 10(5):2539–2547
Li Y, Li K, Wang S, Li Y, Wen D, et al (2022) Towards safer flights: A multi-modality fusion technology-based cognitive load recognition framework. In: 2022 IEEE 4th International Conference on Civil Aviation Safety and Information Technology (ICCASIT), IEEE, pp 525–530
Aygun A, Nguyen T, Haga Z, Aeron S, Scheutz M (2022) Investigating methods for cognitive workload estimation for assistive robots. Sensors 22(18):6834
Huang J, Liu Y, Peng X (2022) Recognition of driver’s mental workload based on physiological signals, a comparative study. Biomed Signal Process Control 71:103094
Najafi TA, Affanni A, Rinaldo R, Zontone P (2023) Drivers’ mental engagement analysis using multi-sensor fusion approaches based on deep convolutional neural networks. Sensors 23(17):7346
Awais M, Raza M, Singh N, Bashir K, Manzoor U, Islam SU et al (2020) Lstm-based emotion detection using physiological signals: Iot framework for healthcare and distance learning in covid-19. IEEE Internet Things J 8(23):16863–16871
Pham TD (2021) Time-frequency time-space lstm for robust classification of physiological signals. Sci Rep 11(1):6936
Alhagry S, Fahmy AA, El-Khoribi RA (2017) Emotion recognition based on eeg using lstm recurrent neural network. Int J Advan Comput Sci Appl 8(10)
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 770–778
Craik A, He Y, Contreras-Vidal JL (2019) Deep learning for electroencephalogram (eeg) classification tasks: a review. J Neural Eng 16(3):031001
Yang H, Han J, Min K (2019) A multi-column cnn model for emotion recognition from eeg signals. Sensors 19(21):4736
Cao B, Niu H, Hao J, Wang G (2022) Building eeg-based cad object selection intention discrimination model using convolutional neural network (cnn). Adv Eng Inform 52:101548
Lu Y, Wang H, Feng N, Jiang D, Wei C (2022) Online interaction method of mobile robot based on single-channel eeg signal and end-to-end cnn with residual block model. Adv Eng Inform 52:101595
Cui F, Wang R, Ding W, Chen Y, Huang L (2022) A novel de-cnn-bilstm multi-fusion model for eeg emotion recognition. Mathematics 10(4):582
Wang J, Cheng S, Tian J, Gao Y (2023) A 2d cnn-lstm hybrid algorithm using time series segments of eeg data for motor imagery classification. Biomed Signal Process Control 83:104627
Schmidt P, Dürichen R, Reiss A, Van Laerhoven K, Plötz T (2019) Multi-target affect detection in the wild: an exploratory study. In: Proceedings of the 2019 ACM international symposium on wearable computers, pp 211–219
Vijayakumar S, Flynn R, Corcoran P, Murray N (2022) Cnn-based emotion recognition from multimodal peripheral physiological signals. Proceedings of the IMX 22
Singh G, Phukan OC, Kumar R (2023) Stress recognition with multi-modal sensing using bootstrapped ensemble deep learning model. Expert Syst:e13239
Ganapathy N, Veeranki YR, Swaminathan R (2020) Convolutional neural network based emotion classification using electrodermal activity signals and time-frequency features. Expert Syst Appl 159:113571
Shakerian S, Habibnezhad M, Ojha A, Lee G, Liu Y, Jebelli H et al (2021) Assessing occupational risk of heat stress at construction: a worker-centric wearable sensor-based approach. Saf Sci 142:105395
Can YS, Ersoy C (2023) Smart affect monitoring with wearables in the wild: An unobtrusive mood-aware emotion recognition system. IEEE Trans Affect Comput 14(4):2851–2863
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al (2017) Attention is all you need. Advan Neural Inform Process Syst 30
Song Jg (2021) Ufo-vit: High performance linear vision transformer without softmax. arXiv:2109.14382
Koelstra S, Muhl C, Soleymani M, Lee JS, Yazdani A, Ebrahimi T et al (2011) Deap: A database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31
Chi MT, Adams J, Bogusch EB, Bruchok C, Kang S, Lancaster M et al (2018) Translating the icap theory of cognitive engagement into practice. Cognitive science 42(6):1777–1832
Bloom BS, Krathwohl DR (2020) Taxonomy of educational objectives: The classification of educational goals. Book 1, Cognitive domain. longman
Chi MT, Wylie R (2014) The icap framework: Linking cognitive engagement to active learning outcomes. Educational psychologist 49(4):219–243
Groß-Mlynek L, Graf T, Harring M, GabrielBusse K, Feldhoff T (2022) Cognitive activation in a close-up view: Triggers of high cognitive activity in students during group work phases. In: Frontiers in Education, Frontiers Media SA, p 873340
Cahya MD, Zubaidah S, Munzil M (2024) Exploring students’ creative thinking skills when learning biology through the reading concept mapping team quiz (remap-tq). KnE Social Sciences pp 70–82
Wu X, Zheng WL, Li Z, Lu BL (2022) Investigating eeg-based functional connectivity patterns for multimodal emotion recognition. J Neural Eng 19(1):016012
Zhang Y, Cheng C, Wang S, Xia T (2022) Emotion recognition using heterogeneous convolutional neural networks combined with multimodal factorized bilinear pooling. Biomed Signal Process Control 77:103877
Liao J, Zhong Q, Zhu Y, Cai D (2020) Multimodal physiological signal emotion recognition based on convolutional recurrent neural network. In: IOP conference series: materials science and engineering, IOP Publishing, p 032005
Ma J, Tang H, Zheng WL, Lu BL (2019) Emotion recognition using multimodal residual lstm network. In: Proceedings of the 27th ACM international conference on multimedia, pp 176–183
Chen J, Liu Y, Xue W, Hu K, Lin W (2022) Multimodal eeg emotion recognition based on the attention recurrent graph convolutional network. Information 13(11):550
Li Q, Liu Y, Yan F, Zhang Q, Liu C (2023) Emotion recognition based on multiple physiological signals. Biomed Signal Process Control 85:104989
Funding
This research was supported by National Natural Science Foundation of China (NSFC) for the Project “A Study on the Perception and Attribution Analysis of Learners’ Higher-Order Thinking Activities” (No. 62177023).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Ethics Approval and Consent to Participate
The study protocol and data collection procedures were reviewed and approved by the local ethics committee. Informed consent was obtained from all individuals participating in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, Y., Li, Y., He, X. et al. Learner’s cognitive state recognition based on multimodal physiological signal fusion. Appl Intell 55, 127 (2025). https://doi.org/10.1007/s10489-024-05958-1
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-05958-1