Skip to main content

Advertisement

Springer Nature Link
Log in

Enhancing the feasibility of cognitive load recognition in remote learning using physiological measures and an adaptive feature recalibration convolutional neural network

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

The precise assessment of cognitive load during a learning phase is an important pathway to improving students’ learning efficiency and performance. Physiological measures make it possible to continuously monitor learners’ cognitive load in remote learning during the COVID-19 outbreak. However, maintaining a good balance between performance and computational cost is still a major challenge in advancing cognitive load recognition technology to real-world applications. This paper introduced an adaptive feature recalibration (AFR) convolutional neural network to overcome this challenge by capturing the most discriminative physiological features (EEG and eye-tracking). The results revealed that the optimal average classification accuracy of the feature combination obtained by the AFR method reached 95.56% with only 60 feature dimensions. Additionally, compared with the best result of the conventional correlation-based feature selection (CFS) method, the introduced AFR algorithm achieved higher accuracy and cheaper computational cost, as well as a 2.06% improvement in accuracy and a 51.21% reduction in feature dimension, which is more in line with the requirements of low delay and real-time performance in practical BCI applications.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Paas F, Renkl A, Sweller J (2004) Cognitive load theory: instructional implications of the interaction between information structures and cognitive architecture. Instr Sci 32:1–8

    Article  Google Scholar 

  2. Sweller J (2010) Element interactivity and intrinsic, extraneous, and germane cognitive load. Educ Psychol Rev 22:123–138. https://doi.org/10.1007/s10648-010-9128-5

    Article  Google Scholar 

  3. Sweller J, van Merrienboer JJG, Paas FGWC (1998) Cognitive architecture and instructional design. Educ Psychol Rev 10:251–296. https://doi.org/10.1023/A:1022193728205

    Article  Google Scholar 

  4. Paas F, Tuovinen JE, Tabbers H, Van Gerven PWM (2003) Cognitive load measurement as a means to advance cognitive load theory. Educ Psychol 38:63–71. https://doi.org/10.1207/S15326985EP3801_8

    Article  Google Scholar 

  5. Baddeley A (1992) Working memory: the interface between memory and cognition. J Cogn Neurosci 4:281–288. https://doi.org/10.1162/jocn.1992.4.3.281

    Article  PubMed  CAS  Google Scholar 

  6. Frey J, Mühl C, Lotte F, Hatchet M (2013) Review of the use of electroencephalography as an evaluation method for human-computer interaction[J]. arXiv preprint arXiv:1311.2222. https://doi.org/10.48550/arXiv.1311.2222

  7. Tran TQ, Boring RL, Dudenhoeffer DD, Hallbert BP, Keller MD, Anderson TM (2007) Advantages and disadvantages of physiological assessment for next generation control room design. In: 2007 IEEE 8th human factors and power plants and HPRCT 13th Annual Meeting. pp. 259–263. https://doi.org/10.1109/HFPP.2007.4413216

  8. Gevins A, Smith ME (2000) Neurophysiological measures of working memory and individual differences in cognitive ability and cognitive style. Cereb Cortex 10:829–839. https://doi.org/10.1093/cercor/10.9.829

    Article  PubMed  CAS  Google Scholar 

  9. Murata A (2005) An attempt to evaluate mental workload using wavelet transform of EEG. Hum Factors 47:498–508. https://doi.org/10.1518/001872005774860096

    Article  PubMed  Google Scholar 

  10. Başar E, Başar-Eroglu C, Karakaş S, Schürmann M (2001) Gamma, alpha, delta, and theta oscillations govern cognitive processes. Int J Psychophysiol 39:241–248. https://doi.org/10.1016/S0167-8760(00)00145-8

    Article  PubMed  Google Scholar 

  11. Gevins A (1997) High-resolution EEG mapping of cortical activation related to working memory: effects of task difficulty, type of processing, and practice. Cereb Cortex 7:374–385. https://doi.org/10.1093/cercor/7.4.374

    Article  PubMed  CAS  Google Scholar 

  12. Herweg NA, Solomon EA, Kahana MJ (2020) Theta oscillations in human memory. Trends Cogn Sci 24:208–227. https://doi.org/10.1016/j.tics.2019.12.006

    Article  PubMed  PubMed Central  Google Scholar 

  13. Sauseng P, Griesmayr B, Freunberger R, Klimesch W (2010) Control mechanisms in working memory: a possible function of EEG theta oscillations. Neurosci Biobehav Rev 34:1015–1022. https://doi.org/10.1016/j.neubiorev.2009.12.006

    Article  PubMed  Google Scholar 

  14. Schacter DL (1977) EEG theta waves and psychological phenomena: a review and analysis. Biol Psychol 5:47–82. https://doi.org/10.1016/0301-0511(77)90028-X

    Article  PubMed  CAS  Google Scholar 

  15. Sterman MB, Mann CA, Kaiser DA, Suyenobu BY (1994) Multiband topographic EEG analysis of a simulated visuomotor aviation task. Int J Psychophysiol 16:49–56. https://doi.org/10.1016/0167-8760(94)90041-8

    Article  PubMed  CAS  Google Scholar 

  16. Chen W, Zhuang J, Yu W, Wang Z (2009) Measuring complexity using FuzzyEn, ApEn, and SampEn. Med Eng Phys 31:61–68. https://doi.org/10.1016/j.medengphy.2008.04.005

    Article  PubMed  Google Scholar 

  17. Gribkov D, Gribkova V (2000) Learning dynamics from nonstationary time series: analysis of electroencephalograms. Phys Rev E 61:6538–6545. https://doi.org/10.1103/PhysRevE.61.6538

    Article  CAS  Google Scholar 

  18. Pincus SM (1991) Approximate entropy as a measure of system complexity. Proc Nat Acad Sci 88(6):2297–2301. https://doi.org/10.1073/pnas.88.6.2297

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  19. Skaramagkas V, Giannakakis G, Ktistakis E, Manousos D, Karatzanis I, Tachos N, Tripoliti EE, Marias K, Fotiadis DI, Tsiknakis M (2021) Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev Biomed Eng 1–1. https://doi.org/10.1109/RBME.2021.3066072

  20. Kahneman D, Beatty J (1966) Pupil diameter and load on memory. Science. https://doi.org/10.1126/science.154.3756.1583

    Article  PubMed  Google Scholar 

  21. van der Wel P, van Steenbergen H (2018) Pupil dilation as an index of effort in cognitive control tasks: a review. Psychon Bull Rev 25:2005–2015. https://doi.org/10.3758/s13423-018-1432-y

    Article  PubMed  PubMed Central  Google Scholar 

  22. Borys, M., Tokovarov, M., Wawrzyk, M., Wesolowska, K., Plechawska-Wojcik, M., Dmytruk, R., Kaczorowska, M., 2017. An analysis of eye-tracking and electroencephalography data for cognitive load measurement during arithmetic tasks, in: 2017 10th International Symposium on Advanced Topics in Electrical Engineering (ATEE). IEEE, Bucharest, Romania, pp. 287–292. https://doi.org/10.1109/ATEE.2017.7905130

  23. Scharinger C, Schüler A, Gerjets P (2020) Using eye-tracking and EEG to study the mental processing demands during learning of text-picture combinations. Int J Psychophysiol 158:201–214. https://doi.org/10.1016/j.ijpsycho.2020.09.014

    Article  PubMed  Google Scholar 

  24. Pandey P, Miyapuram KP (2021) Brain2depth: lightweight CNN model for classification of cognitive states from EEG recordings. In: Annual conference on medical image understanding and analysis. Springer, Cham, pp 394–407. https://doi.org/10.1007/978-3-030-80432-9_30

  25. Michalak K, Kwaśnicka H (2006) Correlation-based feature selection strategy in classification problems. Int J Appl Math Comput Sci 16:503–511

    Google Scholar 

  26. Wang X-W, Nie D, Lu B-L (2014) Emotional state classification from EEG data using machine learning approach. Neurocomputing 129:94–106. https://doi.org/10.1016/j.neucom.2013.06.046

    Article  Google Scholar 

  27. Nie D, Wang X-W, Shi L-C, Lu B-L (2011) EEG-based emotion recognition during watching movies, in: 2011 5th International IEEE/EMBS Conference on Neural Engineering. IEEE, Cancun, pp 667–670. https://doi.org/10.1109/NER.2011.5910636

  28. Guo M-H, Xu T-X, Liu J-J, Liu Z-N, Jiang P-T, Mu T-J, Zhang S-H, Martin RR, Cheng M-M, Hu S-M (2022) Attention mechanisms in computer vision: a survey. Comput Vis Media 8:331–368. https://doi.org/10.1007/s41095-022-0271-y

    Article  Google Scholar 

  29. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Salt Lake City, UT, pp 7132–7141

  30. Mursalin M, Zhang Y, Chen Y, Chawla NV (2017) Automated epileptic seizure detection using improved correlation-based feature selection with random forest classifier. Neurocomputing 241:204–214. https://doi.org/10.1016/j.neucom.2017.02.053

    Article  Google Scholar 

  31. Hoddes E, Zarcone V, Smythe H, Phillips R, Dement WC (1973) Quantification of sleepiness: a new approach. Psychophysiology 10:431–436. https://doi.org/10.1111/j.1469-8986.1973.tb00801.x

    Article  PubMed  CAS  Google Scholar 

  32. Mayer RE, Estrella G (2014) Benefits of emotional design in multimedia instruction. Learn Instr 33:12–18. https://doi.org/10.1016/j.learninstruc.2014.02.004

    Article  Google Scholar 

  33. Um E “Rachel,” Plass JL, Hayward EO, Homer BD (2012) Emotional design in multimedia learning. J. Educ. Psychol. 104:485–498. https://doi.org/10.1037/a0026609

    Article  Google Scholar 

  34. Kalyuga S, Chandler P, Sweller J (2000) Incorporating learner experience into the design of multimedia instruction. J Educ Psychol 92:126

    Article  Google Scholar 

  35. Kalyuga S, Sweller J (2005) Rapid dynamic assessment of expertise to improve the efficiency of adaptive e-learning. Educ Technol Res Dev 53:83–93

    Article  Google Scholar 

  36. Tuovinen JE, Paas F (2004) Exploring multidimensional approaches to the efficiency of instructional conditions. Instr Sci 32:133–152

    Article  Google Scholar 

  37. Herwig U, Satrapi P, Schönfeldt-Lecuona C (2003) Using the international 10–20 EEG system for positioning of transcranial magnetic stimulation. Brain Topogr 16:95–99. https://doi.org/10.1023/B:BRAT.0000006333.93597.9d

    Article  PubMed  Google Scholar 

  38. Duncan CC, Barry RJ, Connolly JF, Fischer C, Michie PT, Näätänen R, Polich J, Reinvang I, Van Petten C (2009) Event-related potentials in clinical research: guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400. Clin Neurophysiol 120:1883–1908. https://doi.org/10.1016/j.clinph.2009.07.045

    Article  PubMed  Google Scholar 

  39. Gratton G, Coles MGH, Donchin E (1983) A new method for off-line removal of ocular artifact. Electroencephalogr Clin Neurophysiol 55:468–484. https://doi.org/10.1016/0013-4694(83)90135-9

    Article  PubMed  CAS  Google Scholar 

  40. Pesonen M, Hämäläinen H, Krause CM (2007) Brain oscillatory 4–30 Hz responses during a visual n-back memory task with varying memory load. Brain Res 1138:171–177. https://doi.org/10.1016/j.brainres.2006.12.076

    Article  PubMed  CAS  Google Scholar 

  41. Aftanas LI, Lotova NV, Koshkarov VI, Pokrovskaja VL, Popov SA, Makhnev VP (1997) Non-linear analysis of emotion EEG: calculation of Kolmogorov entropy and the principal Lyapunov exponent. Neurosci Lett 226:13–16. https://doi.org/10.1016/S0304-3940(97)00232-2

    Article  PubMed  CAS  Google Scholar 

  42. Zhang C, Wang H, Fu R (2014) Automated detection of driver fatigue based on entropy and complexity measures. IEEE Trans Intell Transp Syst 15:168–177. https://doi.org/10.1109/TITS.2013.2275192

    Article  Google Scholar 

  43. Al-Angari HM, Sahakian AV (2007) Use of sample entropy approach to study heart rate variability in obstructive sleep apnea syndrome. IEEE Trans Biomed Eng 54:1900–1904. https://doi.org/10.1109/TBME.2006.889772

    Article  PubMed  Google Scholar 

  44. Mazher M, Abd Aziz A, Malik AS, Ullah Amin H (2017) An EEG-based cognitive load assessment in multimedia learning using feature extraction and partial directed coherence. IEEE Access 5:14819–14829. https://doi.org/10.1109/ACCESS.2017.2731784

    Article  Google Scholar 

  45. Richman JS, Lake DE, Moorman JR (2004) Sample entropy. In: Methods in enzymology, vol 384. Academic Press, pp 172–184. https://doi.org/10.1016/S0076-6879(04)84011-4

  46. Yentes JM, Hunt N, Schmid KK, Kaipust JP, McGrath D, Stergiou N (2013) The appropriate use of approximate entropy and sample entropy with short data sets. Ann Biomed Eng 41:349–365. https://doi.org/10.1007/s10439-012-0668-3

    Article  PubMed  Google Scholar 

  47. Kukker A, Sharma R (2019) EEG based epilepsy seizure analysis and classification methods: an overview. Int J Comput Sci Eng 7(8):328–346. https://doi.org/10.26438/ijcse/v7i8.328346

  48. Ocak H (2009) Automatic detection of epileptic seizures in EEG using discrete wavelet transform and approximate entropy. Expert Syst Appl 36:2027–2036. https://doi.org/10.1016/j.eswa.2007.12.065

    Article  Google Scholar 

  49. Subasi A (2007) EEG signal classification using wavelet feature extraction and a mixture of expert model. Expert Syst Appl 32:1084–1093. https://doi.org/10.1016/j.eswa.2006.02.005

    Article  Google Scholar 

  50. Daubechies I (1990) The wavelet transform, time-frequency localization and signal analysis. IEEE Trans Inf Theory 36:961–1005. https://doi.org/10.1109/18.57199

    Article  Google Scholar 

  51. Zhang J, Yin Z, Chen P, Nichele S (2020) Emotion recognition using multi-modal data and machine learning techniques: a tutorial and review. Inf Fusion 59:103–126. https://doi.org/10.1016/j.inffus.2020.01.011

    Article  Google Scholar 

  52. Kira K, Rendell LA (1992) A practical approach to feature selection. In: Sleeman D, Edwards P (eds) Machine learning proceedings 1992. Morgan Kaufmann, San Francisco, pp 249–256

    Chapter  Google Scholar 

  53. Hu J, Shen L, Albanie S, Sun G, Wu E (2020) Squeeze-and-Excitation Networks. IEEE Trans Pattern Anal Mach Intell 42(8):2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372

    Article  PubMed  Google Scholar 

  54. Hall MA (2000) Correlation-based feature selection of discrete and numeric class machine learning. (Working paper 00/08). University of Waikato, Department of Computer Science, Hamilton, New Zealand

  55. Hall MA (1999) Correlation-based feature selection for machine learning. The University of Waikato

  56. Noble WS (2006) What is a support vector machine? Nat Biotechnol 24:1565–1567. https://doi.org/10.1038/nbt1206-1565

    Article  PubMed  CAS  Google Scholar 

  57. Boser, B.E., Guyon, I.M., Vapnik, V.N., 1992. A training algorithm for optimal margin classifiers, in: Proceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT ’92. Association for Computing Machinery, New York, NY, USA, pp. 144–152. https://doi.org/10.1145/130385.130401

  58. Zhao Z-D, Lou Y-Y, Ni J-H, Zhang J (2009) RBF-SVM and its application on reliability evaluation of electric power system communication network. In: 2009 International Conference on Machine Learning and Cybernetics. pp 1188–1193

  59. Vapnik V (1999) The nature of statistical learning theory. Springer Science & Business Media

  60. Vapnik V (1998) The support vector method of function estimation. In: Suykens JAK, Vandewalle J (eds) Nonlinear modeling: advanced black-box techniques. Springer, Boston, pp 55–85

    Chapter  Google Scholar 

  61. Vapnik VN (1999) An overview of statistical learning theory. IEEE Trans Neural Netw 10:988–999. https://doi.org/10.1109/72.788640

    Article  PubMed  CAS  Google Scholar 

  62. Zammouri A, Ait Moussa A, Mebrouk Y (2018) Brain-computer interface for workload estimation: assessment of mental efforts in learning processes. Expert Syst Appl 112:138–147. https://doi.org/10.1016/j.eswa.2018.06.027

    Article  Google Scholar 

  63. Plass JL, Heidig S, Hayward EO, Homer BD, Um E (2014) Emotional design in multimedia learning: effects of shape and color on affect and learning. Learn Instr 29:128–140. https://doi.org/10.1016/j.learninstruc.2013.02.006

    Article  Google Scholar 

  64. Kruger JL, Doherty S (2016) Measuring cognitive load in the presence of educational video: towards a multimodal methodology. Aust J Educ Technol 32(6). https://doi.org/10.14742/ajet.3084

  65. Guo MH, Xu TX, Liu JJ, Liu ZN, Jiang PT, Mu TJ, Zhang SH, Martin RR, Cheng MM, Hu SM (2021) Attention mechanisms in computer vision: a survey. Comput Vis Media 8(3):331–368. https://doi.org/10.1007/s41095-022-0271-y

  66. Lu Y, Zheng WL, Li B, Lu BL (2015) Combining eye movements and EEG to enhance emotion recognition. In: Twenty-Fourth International Joint Conference on Artificial Intelligence. pp 1170–1176

  67. Eldele E, Chen Z, Liu C, Wu M, Kwoh C-K, Li X, Guan C (2021) An attention-based deep learning approach for sleep stage classification with single-channel EEG. IEEE Trans Neural Syst Rehabil Eng 29:809–818. https://doi.org/10.1109/TNSRE.2021.3076234

    Article  PubMed  Google Scholar 

Download references

Funding

This research was supported by the Humanities and Social Sciences Foundation of the Ministry of Education of China (No. 20YJA880034), the National Undergraduate Innovation and Entrepreneurship Training Program of China (No. 202011057019), the Zhejiang Xinmiao Talents Program (No. 2021R415006), and the Research Project of Department of Education of Zhejiang Province (No. Y202044360).

Author information

Authors and Affiliations

  1. School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou, China

    Chennan Wu, Yang Liu, Xiang Guo, Tianshui Zhu & Zongliang Bao

Authors
  1. Chennan Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yang Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xiang Guo
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Tianshui Zhu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Zongliang Bao
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yang Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, C., Liu, Y., Guo, X. et al. Enhancing the feasibility of cognitive load recognition in remote learning using physiological measures and an adaptive feature recalibration convolutional neural network. Med Biol Eng Comput 60, 3447–3460 (2022). https://doi.org/10.1007/s11517-022-02670-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-022-02670-5

Keywords