Abstract
Intelligent video monitoring and analysis enable correction of personalized learning behavior from a quantitative perspective. Unfortunately, such approaches can only suggest individual concentration level/status and thinking activity based on their appearance, which is subjective. Although resting-state electroencephalogram (EEG) has been recognized as an indicator which is closely related to people’s thinking activities, EEG based models have not yet provided sufficient solutions due to following reasons: (1) insufficient extraction of features due to single modality of input signal; (2) lack of attention to multiscale features caused by the convolutions with single kernel size. To address the issues above, we propose the following solutions. Firstly, a paradigm is designed for extracting resting-state EEG before class (pre-class) and after class (post-class) instead of video monitoring in class. Secondly, we propose a novel framework named multimodal and multiscale Convolutional Neural Network (\(\mathrm { M^{3}S \text{- }CNN}\)) for feature extraction, which consists of two modules: (1) a feature extraction module with Gramian Angular Summation Field (GASF) and Gramian Angular Difference Field (GADF), aiming to transform the EEG features into forms of pictures and (2) a multiscale module, aiming to improve feature extraction ability by grouping different convolution kernel sizes. Finally, a number of classifiers are employed for student status identification. \(\mathrm{M^{3}S\text{- }CNN}\) is evaluated using one private dataset for three classes, which is divided into training, validation and test sets using an 8:1:1 ratio. Experimental results demonstrate that \(\mathrm{M^{3}S\text{- }CNN}\) along with the classifier of Random Forest (RF) is superior to others with accuracy of 99.77%. This indicates the viability of the proposed model for identification of student status during class.
Y. Xu, J. Luo, J. Liang and S. Song—Contributed equally to the work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sethi, K., Jaiswal, V.: PSU-CNN: prediction of student understanding in the classroom through student facial images using convolutional neural network. Mater. Today: Proc. 62(5), 4957–4964 (2022)
Urhahne, D., Wijnia, L.: A review on the accuracy of teacher judgments. Educ. Res. Rev. 32, 100374 (2021)
Rajendran, V.G., Jayalalitha, S., Adalarasu, K.: EEG based evaluation of examination stress and test anxiety among college students. IRBM 43(5), 349–361 (2021)
Francisco-Vicencio, M.A., Góngora-Rivera, F., Ortiz-Jiménez, X., Martinez-Peon., D.: Sustained attention variation monitoring through EEG effective connectivity. Biomed. Signal Process. Control 76, 103650 (2022)
Bashiri, M., Mumtaz, W.; Malik, A.S., Waqar, K.: EEG-based brain connectivity analysis of working memory and attention. In: 2015 IEEE Student Symposium in Biomedical Engineering & Sciences (ISSBES), pp. 41–45 (2015)
Janssen, T.W.P., et al.: Long-term effects of theta/beta neurofeedback on EEG power spectra in children with attention deficit hyperactivity disorder. Clin. Neurophysiol. 131(6), 1332–1341 (2020)
Zhang, D.: The role of resting-state EEG localized activation and central nervous system arousal in executive function performance in children with Attention-Deficit/Hyperactivity Disorder. Clin. Neurophysiol. 129(6), 1192–1200 (2018)
Komarov, O., Ko, L.W., Jung, T.P.: Associations among emotional state, sleep quality, and resting-state eeg spectra: a longitudinal study in graduate students. IEEE Trans. Neural Syst. Rehabil. Eng. 28(4), 795–804 (2020)
Wang, J., Barstein, J., Ethridge, L.E., Mosconi, M.W., Takarae, Y., Sweeney, J.A.: Resting state EEG abnormalities in autism spectrum disorders. J. Neurodev. Disord. 5(1), 24 (2013)
Xia, L., Malik, A.S., Subhani, A.R.: A physiological signal-based method for early mental-stress detection. Biomed. Signal Process. Control 32(6), 2257–2264 (2019)
Xia, L., et al.: MuLHiTA: a novel multiclass classification framework with multibranch lstm and hierarchical temporal attention for early detection of mental stress. IEEE Trans. Neural Netw. Learn. Syst. (TNNLS), 1–13 (2022)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Assoc. Comput. Mach. 60(6), 84–90 (2017)
Liu, P., Qiu, X., Chen, X., Wu, S., Huang, X. J.: Multi-timescale long short-term memory neural network for modelling sentences and documents. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2326–2335. (2015)
Huang, J., Chen, B., Yao, B., He, W.: ECG arrhythmia classification using STFT-based spectrogram and convolutional neural network. IEEE Access 7, 92871–92880 (2019)
Thanaraj, K.P., Parvathavarthini, B., Tanik, U. J., Rajinikanth, V., Kadry, S., Kamalanand, K.: Implementation of deep neural networks to classify EEG signals using gramian angular summation field for epilepsy diagnosis. https://arxiv.org/abs/2003.04534. Accessed 8 Mar 2020
Ahmad, Z., Tabassum, A., Guan, L., Khan, N.M.: ECG heartbeat classification using multimodal fusion. IEEE Access 9, 100615–100626 (2021)
Wang, Z., Oates T.: Imaging time-series to improve classification and imputation. In: Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI), pp. 3939–3945 (2015)
Fan, X., Yao, Q., Cai, Y., Miao, F., Sun, F., Li, Y.: Multi-scaled fusion of deep convolutional neural networks for screening atrial fibrillation from single lead short ECG recordings. IEEE J. Biomed. Health Inf. 22(6), 1744–1753 (2018)
Zhou, K., Yang, Y., Cavallaro, A., Xiang, T.: Omni-scale feature learning for person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 370–3711. (2019)
Delorme, A., Makeig, S.: EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134(1), 9–21 (2004)
Andrey, P., Anokhin, N.B., Werner, L., Andrey, N., Friedrich, V.: Age increases brain complexity. Electroencephalogr. Clin. Neurophysiol. 99(1), 63–68 (1996)
Fell, J., Axmacher, N., Haupt, S.: From alpha to gamma: electrophysiological correlates of meditation-related states of consciousness. Med. Hypoth. 75(2), 218–224 (2010)
Acknowledgement
This work is supported by Beijing Natural Science Foundation (4202011), National Natural Science Foundation of China (61572076).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xu, Y. et al. (2023). \(\mathrm {M^{3}S \text{- }CNN}\): Resting-State EEG Based Multimodal and Multiscale Feature Extraction for Student Behavior Prediction in Class. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Communications in Computer and Information Science, vol 1792. Springer, Singapore. https://doi.org/10.1007/978-981-99-1642-9_44
Download citation
DOI: https://doi.org/10.1007/978-981-99-1642-9_44
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-1641-2
Online ISBN: 978-981-99-1642-9
eBook Packages: Computer ScienceComputer Science (R0)