Skip to main content

Advertisement

Learner’s cognitive state recognition based on multimodal physiological signal fusion

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

It is crucial to evaluate learning outcomes by identifying the cognitive state of the learner during the learning process. Studies utilizing Electroencephalography (EEG) and other peripheral physiological signals, combined with deep learning models, have demonstrated improved performance in cognitive state recognition. These studies have primarily focused on unimodal data, which are vulnerable to various types of noise, making it difficult to fully capture and represent cognitive states. Leveraging the complementarity between multimodal physiological signals can mitigate the impact of anomalies in unimodal data, thereby improving the accuracy and stability of cognitive state recognition. Therefore, this study proposes a multimodal physiological signal feature representation fusion model based on multi-level attention (PSFMMA). The model aims to integrate multimodal physiological signals to identify learners’ cognitive states with greater stability and accuracy. PSFMMA first extracts the temporal features of physiological signals by multiplexing the embedding layer. Subsequently, it generates signal representation vectors by further extracting semantic features through a signal feature mapping layer and enhancing important features with designed attention modules. Finally, the model employs an attention mechanism based on different signal representation vectors to fuse multimodal information for identifying learners’ cognitive states. This study designs various learning activities and collects electroencephalography (EEG), electrodermal activity (EDA), and photoplethysmography (PPG) data from 22 participants engaging in these activities to create the Based on Learning Activities Collection (BLAC) dataset. The proposed model was evaluated on the BLAC dataset, achieving an identification accuracy of 96.32 ± 0.32%. The results demonstrate that the model can effectively recognize learners’ cognitive states. Furthermore, the model’s performance was validated on the publicly available emotion classification dataset DEAP, attaining an accuracy of 99.15 ± 0.12%. The source code is available at https://github.com/chengshudaxuesheng/PSFMMA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of Data and Materials

The datasets are not publicly available due to protect the privacy of study participants. Specifc inquiries regarding the data may be directed to the corresponding author.

References

  1. Elkerdawy M, Elhalaby M, Hassan A, Maher M, Shawky D, Badawi A (2020) Building cognitive profiles of learners using eeg. In: 2020 11th International Conference on Information and Communication Systems (ICICS), IEEE, pp 027–032

  2. Jamil N, Belkacem AN, Lakas A (2023) On enhancing students’ cognitive abilities in online learning using brain activity and eye movements. Educ Inf Technol 28(4):4363–4397

    MATH  Google Scholar 

  3. Liu X, Tan PN, Liu L, Simske SJ (2017) Automated classification of eeg signals for predicting students’ cognitive state during learning. In: Proceedings of the international conference on web intelligence, pp 442–450

  4. Gerjets P, Walter C, Rosenstiel W, Bogdan M, Zander TO (2014) Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain-computer interface approach. Front Neurosci 8:385

    MATH  Google Scholar 

  5. Chen S, Zhang J (2008) The adaptive learning system based on learning style and cognitive state. In: 2008 International symposium on knowledge acquisition and modeling, IEEE, pp 302–306

  6. Bhattacharya S, Roy S, Chowdhury S (2018) A neural network-based intelligent cognitive state recognizer for confidence-based e-learning system. Neural Comput Appl 29:205–219

    MATH  Google Scholar 

  7. D’Mello S, Olney A, Williams C, Hays P (2012) Gaze tutor: a gaze-reactive intelligent tutoring system. Int J Hum Comput Stud 70(5):377–398

    Google Scholar 

  8. Caram CA, Davis PB (2005) Inviting student engagement with questioning. Kappa Delta Pi Record 42(1):18–23

    Google Scholar 

  9. Khedher AB, Jraidi I, Frasson C (2019) Tracking students’ mental engagement using eeg signals during an interaction with a virtual learning environment. J Intell Learn Syst Appl 11(01):1–14

    Google Scholar 

  10. Liu S, Liu S, Liu Z, Peng X, Yang Z (2022) Automated detection of emotional and cognitive engagement in mooc discussions to predict learning achievement. Comput Educ 181:104461

    MATH  Google Scholar 

  11. Du X, Zhang L, Hung JL, Li H, Tang H, Dai M (2022) Analyzing the effects of instructional strategies on students’ on-task status from aspects of their learning behaviors and cognitive factors. J Comput Higher Educ:1–28

  12. Charlton SG (2019) Measurement of cognitive states in test and evaluation. In: Handbook of human factors testing and evaluation. CRC Press, pp 97–126

  13. Ciolacu M, Tehrani AF, Binder L, Svasta PM (2018) Education 4.0-artificial intelligence assisted higher education: early recognition system with machine learning to support students’ success. In: 2018 IEEE 24th International Symposium for Design and Technology in Electronic Packaging (SIITME), IEEE, pp 23–30

  14. Dewan M, Murshed M, Lin F (2019) Engagement detection in online learning: a review. Smart Learn Environ 6(1):1–20

    MATH  Google Scholar 

  15. Dirican AC, Göktürk M (2011) Psychophysiological measures of human cognitive states applied in human computer interaction. Procedia Comput Sci 3:1361–1367

    MATH  Google Scholar 

  16. Lohani M, Payne BR, Strayer DL (2019) A review of psychophysiological measures to assess cognitive states in real-world driving. Front Hum Neurosci 13:57

    MATH  Google Scholar 

  17. Borghini G, Aricò P, Graziani I, Salinari S, Sun Y, Taya F et al (2016) Quantitative assessment of the training improvement in a motor-cognitive task by using eeg, ecg and eog signals. Brain Topogr 29:149–161

    Google Scholar 

  18. Dahal N, Nandagopal N, Nafalski A, Nedic Z (2011) Modeling of cognition using eeg: a review and a new approach. In: TENCON 2011-2011 IEEE region 10 conference, IEEE, pp 1045–1049

  19. Zhou Y, Huang S, Xu Z, Wang P, Wu X, Zhang D (2021) Cognitive workload recognition using eeg signals and machine learning: a review. IEEE Trans Cogn Develop Syst

  20. Katmah R, Al-Shargie F, Tariq U, Babiloni F, Al-Mughairbi F, Al-Nashash H (2021) A review on mental stress assessment methods using eeg signals. Sensors 21(15):5043

    MATH  Google Scholar 

  21. Cheng Y, Cai Y, Chen H, Cai Z, Wu G, Huang J (2021) A cognitive level evaluation method based on a deep neural network for online learning: from a bloom’s taxonomy of cognition objectives perspective. Front Psychol 12:661235

    MATH  Google Scholar 

  22. Bhise PR, Kulkarni SB, Aldhaheri TA (2021) Survey on assessment of cognitive states during learning activities using brain computer interface based eeg. Int J Comput Appl 975:8887

    MATH  Google Scholar 

  23. Monteiro TG, Skourup C, Zhang H (2020) Optimizing cnn hyperparameters for mental fatigue assessment in demanding maritime operations. IEEE Access 8:40402–40412

    MATH  Google Scholar 

  24. Han SY, Kwak NS, Oh T, Lee SW (2020) Classification of pilots’ mental states using a multimodal deep learning network. Biocybernetics Biomed Eng 40(1):324–336

    MATH  Google Scholar 

  25. Li J, Wang Q (2022) Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation. Inform Fusion 79:229–247

    MATH  Google Scholar 

  26. Antonenko P, Paas F, Grabner R, Van Gog T (2010) Using electroencephalography to measure cognitive load. Educ Psychol Rev 22:425–438

    Google Scholar 

  27. Mills C, Fridman I, Soussou W, Waghray D, Olney AM, D’Mello SK (2017) Put your thinking cap on: detecting cognitive load using eeg during learning. In: Proceedings of the seventh international learning analytics & knowledge conference, pp 80–89

  28. Zhou Y, Xu T, Cai Y, Wu X, Dong B (2017) Monitoring cognitive workload in online videos learning through an eeg-based brain-computer interface. In: Learning and Collaboration Technologies. Novel Learning Ecosystems: 4th International Conference, LCT 2017, Held as Part of HCI International 2017, Vancouver, BC, Canada, July 9-14, 2017, Proceedings, Part I 4, Springer, pp 64–73

  29. Dey AK, Poddar B, Pramanik PKD, Debnath NC, Aljahdali S, Choudhury P (2020) Real-time learner classification using cognitive score. In: CATA, pp 264–276

  30. Sinha A, Gavas R, Chatterjee D, Das R, Sinharay A (2015) Dynamic assessment of learners’ mental state for an improved learning experience. In: 2015 IEEE frontiers in education conference (FIE), IEEE, pp 1–9

  31. Dutta S, Nandy A (2020) An extensive analysis on deep neural architecture for classification of subject-independent cognitive states. In: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD. Association for Computing Machinery, New York, NY, USA, CoDS COMAD 2020, p 180–184

  32. Qayyum A, Khan MA, Mazher M, Suresh M (2018) Classification of eeg learning and resting states using 1d-convolutional neural network for cognitive load assesment. In: 2018 IEEE Student Conference on Research and Development (SCOReD), IEEE, pp 1–5

  33. Pandey P, Miyapuram KP (2021) Brain2depth: Lightweight cnn model for classification of cognitive states from eeg recordings. In: Medical Image Understanding and Analysis: 25th Annual Conference, MIUA 2021, Oxford, United Kingdom, July 12–14, 2021, Proceedings 25, Springer, pp 394–407

  34. Li P, Zhang Y, Liu S, Lin L, Zhang H, Tang T et al (2023) An eeg-based brain cognitive dynamic recognition network for representations of brain fatigue. Appl Soft Comput 146:110613

    MATH  Google Scholar 

  35. Kalra P, Sharma V (2023) Mental stress assessment using ppg signal a deep neural network approach. IETE J Res 69(2):879–885

    MATH  Google Scholar 

  36. Butkevičiūtė E, Michalkovič A, Bikulčienė L (2022) Ecg signal features classification for the mental fatigue recognition. Mathematics 10(18):3395

    MATH  Google Scholar 

  37. Alreshidi I, Moulitsas I, Jenkins KW (2023) Multimodal approach for pilot mental state detection based on eeg. Sensors 23(17):7350

    MATH  Google Scholar 

  38. Gao N, Shao W, Rahaman MS, Salim FD (2020) n-gage: predicting in-class emotional, behavioural and cognitive engagement in the wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4(3):1–26

    MATH  Google Scholar 

  39. Markova V, Ganchev T, Kalinkov K, Markov M (2021) Detection of acute stress caused by cognitive tasks based on physiological signals. Bulletin Electrical Eng Inform 10(5):2539–2547

    MATH  Google Scholar 

  40. Li Y, Li K, Wang S, Li Y, Wen D, et al (2022) Towards safer flights: A multi-modality fusion technology-based cognitive load recognition framework. In: 2022 IEEE 4th International Conference on Civil Aviation Safety and Information Technology (ICCASIT), IEEE, pp 525–530

  41. Aygun A, Nguyen T, Haga Z, Aeron S, Scheutz M (2022) Investigating methods for cognitive workload estimation for assistive robots. Sensors 22(18):6834

    MATH  Google Scholar 

  42. Huang J, Liu Y, Peng X (2022) Recognition of driver’s mental workload based on physiological signals, a comparative study. Biomed Signal Process Control 71:103094

    MATH  Google Scholar 

  43. Najafi TA, Affanni A, Rinaldo R, Zontone P (2023) Drivers’ mental engagement analysis using multi-sensor fusion approaches based on deep convolutional neural networks. Sensors 23(17):7346

    MATH  Google Scholar 

  44. Awais M, Raza M, Singh N, Bashir K, Manzoor U, Islam SU et al (2020) Lstm-based emotion detection using physiological signals: Iot framework for healthcare and distance learning in covid-19. IEEE Internet Things J 8(23):16863–16871

    Google Scholar 

  45. Pham TD (2021) Time-frequency time-space lstm for robust classification of physiological signals. Sci Rep 11(1):6936

    MATH  Google Scholar 

  46. Alhagry S, Fahmy AA, El-Khoribi RA (2017) Emotion recognition based on eeg using lstm recurrent neural network. Int J Advan Comput Sci Appl 8(10)

  47. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 770–778

  48. Craik A, He Y, Contreras-Vidal JL (2019) Deep learning for electroencephalogram (eeg) classification tasks: a review. J Neural Eng 16(3):031001

    MATH  Google Scholar 

  49. Yang H, Han J, Min K (2019) A multi-column cnn model for emotion recognition from eeg signals. Sensors 19(21):4736

    MATH  Google Scholar 

  50. Cao B, Niu H, Hao J, Wang G (2022) Building eeg-based cad object selection intention discrimination model using convolutional neural network (cnn). Adv Eng Inform 52:101548

    MATH  Google Scholar 

  51. Lu Y, Wang H, Feng N, Jiang D, Wei C (2022) Online interaction method of mobile robot based on single-channel eeg signal and end-to-end cnn with residual block model. Adv Eng Inform 52:101595

    MATH  Google Scholar 

  52. Cui F, Wang R, Ding W, Chen Y, Huang L (2022) A novel de-cnn-bilstm multi-fusion model for eeg emotion recognition. Mathematics 10(4):582

    MATH  Google Scholar 

  53. Wang J, Cheng S, Tian J, Gao Y (2023) A 2d cnn-lstm hybrid algorithm using time series segments of eeg data for motor imagery classification. Biomed Signal Process Control 83:104627

    Google Scholar 

  54. Schmidt P, Dürichen R, Reiss A, Van Laerhoven K, Plötz T (2019) Multi-target affect detection in the wild: an exploratory study. In: Proceedings of the 2019 ACM international symposium on wearable computers, pp 211–219

  55. Vijayakumar S, Flynn R, Corcoran P, Murray N (2022) Cnn-based emotion recognition from multimodal peripheral physiological signals. Proceedings of the IMX 22

  56. Singh G, Phukan OC, Kumar R (2023) Stress recognition with multi-modal sensing using bootstrapped ensemble deep learning model. Expert Syst:e13239

  57. Ganapathy N, Veeranki YR, Swaminathan R (2020) Convolutional neural network based emotion classification using electrodermal activity signals and time-frequency features. Expert Syst Appl 159:113571

    Google Scholar 

  58. Shakerian S, Habibnezhad M, Ojha A, Lee G, Liu Y, Jebelli H et al (2021) Assessing occupational risk of heat stress at construction: a worker-centric wearable sensor-based approach. Saf Sci 142:105395

  59. Can YS, Ersoy C (2023) Smart affect monitoring with wearables in the wild: An unobtrusive mood-aware emotion recognition system. IEEE Trans Affect Comput 14(4):2851–2863

    MATH  Google Scholar 

  60. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al (2017) Attention is all you need. Advan Neural Inform Process Syst 30

  61. Song Jg (2021) Ufo-vit: High performance linear vision transformer without softmax. arXiv:2109.14382

  62. Koelstra S, Muhl C, Soleymani M, Lee JS, Yazdani A, Ebrahimi T et al (2011) Deap: A database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Google Scholar 

  63. Chi MT, Adams J, Bogusch EB, Bruchok C, Kang S, Lancaster M et al (2018) Translating the icap theory of cognitive engagement into practice. Cognitive science 42(6):1777–1832

    Google Scholar 

  64. Bloom BS, Krathwohl DR (2020) Taxonomy of educational objectives: The classification of educational goals. Book 1, Cognitive domain. longman

  65. Chi MT, Wylie R (2014) The icap framework: Linking cognitive engagement to active learning outcomes. Educational psychologist 49(4):219–243

    MATH  Google Scholar 

  66. Groß-Mlynek L, Graf T, Harring M, GabrielBusse K, Feldhoff T (2022) Cognitive activation in a close-up view: Triggers of high cognitive activity in students during group work phases. In: Frontiers in Education, Frontiers Media SA, p 873340

  67. Cahya MD, Zubaidah S, Munzil M (2024) Exploring students’ creative thinking skills when learning biology through the reading concept mapping team quiz (remap-tq). KnE Social Sciences pp 70–82

  68. Wu X, Zheng WL, Li Z, Lu BL (2022) Investigating eeg-based functional connectivity patterns for multimodal emotion recognition. J Neural Eng 19(1):016012

    Google Scholar 

  69. Zhang Y, Cheng C, Wang S, Xia T (2022) Emotion recognition using heterogeneous convolutional neural networks combined with multimodal factorized bilinear pooling. Biomed Signal Process Control 77:103877

    Google Scholar 

  70. Liao J, Zhong Q, Zhu Y, Cai D (2020) Multimodal physiological signal emotion recognition based on convolutional recurrent neural network. In: IOP conference series: materials science and engineering, IOP Publishing, p 032005

  71. Ma J, Tang H, Zheng WL, Lu BL (2019) Emotion recognition using multimodal residual lstm network. In: Proceedings of the 27th ACM international conference on multimedia, pp 176–183

  72. Chen J, Liu Y, Xue W, Hu K, Lin W (2022) Multimodal eeg emotion recognition based on the attention recurrent graph convolutional network. Information 13(11):550

    MATH  Google Scholar 

  73. Li Q, Liu Y, Yan F, Zhang Q, Liu C (2023) Emotion recognition based on multiple physiological signals. Biomed Signal Process Control 85:104989

    MATH  Google Scholar 

Download references

Funding

This research was supported by National Natural Science Foundation of China (NSFC) for the Project “A Study on the Perception and Attribution Analysis of Learners’ Higher-Order Thinking Activities” (No. 62177023).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xiuling He.

Ethics declarations

Conflict of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Ethics Approval and Consent to Participate

The study protocol and data collection procedures were reviewed and approved by the local ethics committee. Informed consent was obtained from all individuals participating in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Li, Y., He, X. et al. Learner’s cognitive state recognition based on multimodal physiological signal fusion. Appl Intell 55, 127 (2025). https://doi.org/10.1007/s10489-024-05958-1

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05958-1

Keywords