Abstract
Human emotions are intricate psychological phenomena that reflect an individual’s current physiological and psychological state. Emotions have a pronounced influence on human behavior, cognition, communication, and decision-making. However, current emotion recognition methods often suffer from suboptimal performance and limited scalability in practical applications. To solve this problem, a novel electroencephalogram (EEG) emotion recognition network named VG-DOCoT is proposed, which is based on depthwise over-parameterized convolutional (DO-Conv), transformer, and variational automatic encoder-generative adversarial network (VAE-GAN) structures. Specifically, the differential entropy (DE) can be extracted from EEG signals to create mappings into the temporal, spatial, and frequency information in preprocessing. To enhance the training data, VAE-GAN is employed for data augmentation. A novel convolution module DO-Conv is used to replace the traditional convolution layer to improve the network. A transformer structure is introduced into the network framework to reveal the global dependencies from EEG signals. Using the proposed model, a binary classification on the DEAP dataset is carried out, which achieves an accuracy of 92.52% for arousal and 92.27% for valence. Next, a ternary classification is conducted on SEED, which classifies neutral, positive, and negative emotions; an impressive average prediction accuracy of 93.77% is obtained. The proposed method significantly improves the accuracy for EEG-based emotion recognition.
摘要
人类情绪是反映个体当前生理和心理状态的复杂心理现象. 情绪对人类的行为、认知、交流和决策产生了显著的影响. 然而, 目前的情绪识别方法在实际应用中往往存在性能不佳和可扩展性有限的问题. 为此, 我们提出一种新颖的脑电图(EEG)情绪识别框架VG-DOCoT, 它基于深度过参数化卷积(DO-Conv)、变换器和变分自编码器-生成对抗网络(VAE-GAN)结构. 具体来说, 在预处理中, 可以从EEG信号中提取微分熵(DE), 以映射到时间、空间和频率信息中. 为了增强训练数据, 采用VAE-GAN进行数据增强. 使用一种新颖的卷积模块DO-Conv替代传统的卷积层, 以提高网络性能. 在网络框架中引入了变换器结构, 以揭示EEG信号中的全局依赖性. 使用所提出的模型, 在DEAP数据集上进行了二分类任务仿真, 唤醒度和效价度的准确率分别达到92.52%和92.27%. 另外, 在SEED数据集上进行了三分类任务测试, 包括中性、积极和消极三种情绪, 获得的平均预测准确率为93.77%. 所提出的方法显著提高了脑电情绪识别的准确率.
This is a preview of subscription content, log in via an institution to check access.
Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
Aznan NKN, Atapour-Abarghouei A, Bonner S, et al., 2019. Simulating brain signals: creating synthetic EEG data via neural-based generative models for improved SSVEP classification. Int Joint Conf on Neural Networks, p.1–8. https://doi.org/10.1109/IJCNN.2019.8852227
Bahdanau D, Cho K, Bengio Y, 2015. Neural machine translation by jointly learning to align and translate. 3rd Int Conf on Learning Representations.
Bernat E, Bunce S, Shevrin H, 2001. Event-related brain potentials differentiate positive and negative mood adjectives during both supraliminal and subliminal visual processing. Int J Psychophysiol, 42(1): 11–34. https://doi.org/10.1016/S0167-8760(01)00133-7
Cao JM, Li YY, Sun MC, et al., 2022. Do-Conv: depthwise over-parameterized convolutional layer. IEEE Trans Image Process, 31: 3726–3736. https://doi.org/10.1109/TIP.2022.3175432
Chao H, Dong L, 2021. Emotion recognition using three-dimensional feature and convolutional neural network from multichannel EEG signals. IEEE Sens J, 21(2): 2024–2034. https://doi.org/10.1109/JSEN.2020.3020828
Cheng J, Chen MY, Li C, et al., 2021. Emotion recognition from multi-channel EEG via deep forest. IEEE J Biomed Health Inform, 25(2): 453–464. https://doi.org/10.1109/JBHI.2020.2995767
Goodfellow I, Pouget-Abadie J, Mirza M, et al., 2020. Generative adversarial networks. Commun ACM, 63(11): 139–144. https://doi.org/10.1145/3422622
Guo JY, Cai Q, An JP, et al., 2022. A Transformer based neural network for emotion recognition and visualizations of crucial EEG channels. Phys A Stat Mech Appl, 603: 127700. https://doi.org/10.1016/j.physa.2022.127700
Hu JF, Min JL, 2018. Automated detection of driver fatigue based on EEG signals using gradient boosting decision tree model. Cogn Neurodyn, 12(4): 431–440. https://doi.org/10.1007/s11571-018-9485-1
Jenke R, Peer A, Buss M, 2014. Feature extraction and selection for emotion recognition from EEG. IEEE Trans Affect Comput, 5(3): 327–339. https://doi.org/10.1109/TAFFC.2014.2339834
Kingma DP, Welling M, 2014. Auto-encoding variational Bayes. 2nd Int Conf on Learning Representations.
Koelstra S, Muhl C, Soleymani M, et al., 2012. DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput, 3(1): 18–31. https://doi.org/10.1109/T-AFFC.2011.15
Lan ZR, Sourina O, Wang LP, et al., 2016. Real-time EEG-based emotion monitoring using stable features. Vis Comput, 32(3): 347–358. https://doi.org/10.1007/s00371-015-1183-y
Lew WCL, Wang D, Shylouskaya K, et al., 2020. EEG-based emotion recognition using spatial-temporal representation via Bi-GRU. 42nd Annual Int Conf of the IEEE Engineering in Medicine & Biology Society, p.116–119. https://doi.org/10.1109/EMBC44109.2020.9176682
Li C, Lin XJ, Liu Y, et al., 2022. EEG-based emotion recognition via efficient convolutional neural network and contrastive learning. IEEE Sens J, 22(20): 19608–19619. https://doi.org/10.1109/JSEN.2022.3202209
Li JP, Zhang ZX, He HG, 2018. Hierarchical convolutional neural networks for EEG-based emotion recognition. Cogn Comput, 10(2): 368–380. https://doi.org/10.1007/s12559-017-9533-x
Li SJ, Li W, Xing ZJ, et al., 2022. A personality-guided affective brain-computer interface for implementation of emotional intelligence in machines. Front Inform Technol Electron Eng, 23(8): 1158–1173. https://doi.org/10.1631/FITEE.2100489
Li X, Song DW, Zhang P, et al., 2016. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. IEEE Int Conf on Bioinformatics and Biomedicine, p.352–359. https://doi.org/10.1109/BIBM.2016.7822545
Li X, Zhang YZ, Tiwari P, et al., 2022. EEG based emotion recognition: a tutorial and review. ACM Comput Surv, 55(4): 79. https://doi.org/10.1145/3524499
Lin YP, Wang CH, Wu TL, et al., 2009. EEG-based emotion recognition in music listening: a comparison of schemes for multiclass support vector machine. IEEE Int Conf on Acoustics, Speech and Signal Processing, p.489–492. https://doi.org/10.1109/ICASSP.2009.4959627
Liu YJ, Yu MJ, Zhao GZ, et al., 2018. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE Trans Affect Comput, 9(4): 550–562. https://doi.org/10.1109/TAFFC.2017.2660485
Liu YS, Sourina O, 2014. EEG-based subject-dependent emotion recognition algorithm using fractal dimension. IEEE Int Conf on Systems, Man, and Cybernetics, p.3166–3171. https://doi.org/10.1109/SMC.2014.6974415
Mohammadi Z, Frounchi J, Amiri M, 2017. Wavelet-based emotion recognition system using EEG signal. Neur Comput Appl, 28(8): 1985–1990. https://doi.org/10.1007/s00521-015-2149-8
Picard RW, 2000. Affective Computing. MIT Press, Cambridge, UK. https://doi.org/10.7551/mitpress/1140.001.0001
Salama ES, El-Khoribi RA, Shoman ME, et al., 2018. EEG-based emotion recognition using 3D convolutional neural networks. Int J Adv Comput Sci Appl, 9(8): 329–337. https://doi.org/10.14569/IJACSA.2018.090843
Song TF, Zheng WM, Song P, et al., 2020. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput, 11(3): 532–541. https://doi.org/10.1109/TAFFC.2018.2817622
Sorkhabi MM, 2014. Emotion detection from EEG signals with continuous wavelet analyzing. Am J Comput Res Repos, 2(4): 66–70. https://doi.org/10.12691/ajcrr-2-4-3
Stam CJ, 2005. Nonlinear dynamical analysis of EEG and MEG: review of an emerging field. Clin Neurophysiol, 116(10): 2266–2301. https://doi.org/10.1016/j.clinph.2005.06.011
Tang ZC, Li C, Wu JF, et al., 2019. Classification of EEG-based single-trial motor imagery tasks using a B-CSP method for BCI. Front Inform Technol Electron Eng, 20(8): 1087–1098. https://doi.org/10.1631/FITEE.1800083
Tao W, Li C, Song RC, et al., 2023. EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans Affect Comput, 14(1): 382–393. https://doi.org/10.1109/TAFFC.2020.3025777
Tripathi S, Acharya S, Sharma RD, et al., 2017. Using deep and convolutional neural networks for accurate emotion classification on DEAP data. Proc 31st AAAI Conf on Artificial Intelligence, p.4746–4752. https://doi.org/10.1609/aaai.v31i2.19105
Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. 31st Int Conf on Neural Information Processing Systems, p.6000–6010.
Vijayan AE, Sen D, Sudheer AP, 2015. EEG-based emotion recognition using statistical measures and auto-regressive modeling. IEEE Int Conf on Computational Intelligence & Communication Technology, p.587–591. https://doi.org/10.1109/CICT.2015.24
Wang Q, Sourina O, Nguyen MK, 2011. Fractal dimension based neurofeedback in serious games. Vis Comput, 27(4): 299–309. https://doi.org/10.1007/s00371-011-0551-5
Wang XW, Nie D, Lu BL, 2014. Emotional state classification from EEG data using machine learning approach. Neurocomputing, 129: 94–106. https://doi.org/10.1016/j.neucom.2013.06.046
Wei C, Chen LL, Song ZZ, et al., 2020. EEG-based emotion recognition using simple recurrent units network and ensemble learning. Biomed Signal Process Contr, 58: 101756. https://doi.org/10.1016/j.bspc.2019.101756
Yang BH, He LF, Lin L, et al., 2015. Fast removal of ocular artifacts from electroencephalogram signals using spatial constraint independent component analysis based recursive least squares in brain-computer interface. Front Inform Technol Electron Eng, 16(6): 486–496. https://doi.org/10.1631/FITEE.1400299
Yang Y, Gao Q, Song XL, et al., 2021. Facial expression and EEG fusion for investigating continuous emotions of deaf subjects. IEEE Sens J, 21(15): 16894–16903. https://doi.org/10.1109/JSEN.2021.3078087
Yang YL, Wu QF, Fu YZ, et al., 2018a. Continuous convolutional neural network with 3D input for EEG-based emotion recognition. 25th Int Conf on Neural Information Processing, p.433–443. https://doi.org/10.1007/978-3-030-04239-4_39
Yang YL, Wu QF, Qiu M, et al., 2018b. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. Int Joint Conf on Neural Networks, p.1–7. https://doi.org/10.1109/IJCNN.2018.8489331
Yang YX, Gao ZK, Wang XM, et al., 2018. A recurrence quantification analysis-based channel-frequency convolutional neural network for emotion recognition from EEG. Chaos, 28(8): 085724. https://doi.org/10.1063/1.5023857
Yin YQ, Zheng XW, Hu B, et al., 2021. EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Appl Soft Comput, 100: 106954. https://doi.org/10.1016/j.asoc.2020.106954
Zhang DL, Yao LN, Zhang X, et al., 2018. Cascade and parallel convolutional recurrent neural networks on EEG-based intention recognition for brain computer interface. Proc 32nd AAAI Conf on Artificial Intelligence, p.1703–1710. https://doi.org/10.1609/aaai.v32i1.11496
Zhang QQ, Liu Y, 2018. Improving brain computer interface performance by data augmentation with conditional deep convolutional generative adversarial networks. https://arxiv.org/abs/1806.07108
Zhang T, Zheng WM, Cui Z, et al., 2019. Spatial–temporal recurrent neural network for emotion recognition. IEEE Trans Cybern, 49(3): 839–847. https://doi.org/10.1109/TCYB.2017.2788081
Zheng WL, Lu BL, 2015. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev, 7(3): 162–175. https://doi.org/10.1109/TAMD.2015.2431497
Zhong XY, Gu Y, Luo YT, et al., 2023. Bi-hemisphere asymmetric attention network: recognizing emotion from EEG signals based on the transformer. Appl Intell, 53(12): 15278–15294. https://doi.org/10.1007/s10489-022-04228-2
Author information
Authors and Affiliations
Contributions
Yanping ZHU and Lei HUANG designed the research. Lei HUANG processed the data and analyzed the experimental results. Lei HUANG and Jixin CHEN made the charts. Lei HUANG and Yanping ZHU drafted the paper. Jixin CHEN and Jianan CHEN helped compile the paper. Yanping ZHU, Shenyun WANG, and Fayu WAN revised and finalized the paper.
Corresponding author
Ethics declarations
All the authors declare that they have no conflict of interest.
Additional information
Project supported by the National Key Research and Development Program of China (No. 2022YFE0122700) and the National Natural Science Foundation of China (No. 61971230)
Rights and permissions
About this article
Cite this article
Zhu, Y., Huang, L., Chen, J. et al. VG-DOCoT: a novel DO-Conv and transformer framework via VAE-GAN technique for EEG emotion recognition. Front Inform Technol Electron Eng 25, 1497–1514 (2024). https://doi.org/10.1631/FITEE.2300781
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2300781
Key words
- Emotion recognition
- Electroencephalogram (EEG)
- Depthwise over-parameterized convolutional (DO-Conv)
- Transformer
- Variational automatic encoder-generative adversarial network (VAE-GAN)