Skip to main content
SpringerLink
Log in

Emotion Recognition on EEG Signal Using ResNeXt Attention 2D-3D Convolution Neural Networks

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Emotion recognition based on electroencephalogram (EEG) is an important part of human–machine interaction. This paper used deep learning methods to extract EEG data features to achieve the classification of human emotional states. We proposed a emotion recognition method based on two-dimensional convolution neural networks and three-dimensional convolution neural networks, called ResNeXt Attention 2D–3D Convolutional Neural Networks (RA2–3DCNN). The split-convert-merge techniques, residual and attention mechanism are introduced into the shallow network to improve the accuracy of the model. Then, 3D CNN was used to integrate the frequency, spatial and temporal information from EEG signal. Herein, the pre-processed EEG time series data was reconstructed into two-dimensional EEG frames as the input of the model according to the original electrode position. The accuracy of the emotional classification of the RA2–3DCNN was demonstrated by extensive experiments on the DEAP dataset. The results showed that the recognition accuracy of the method on arousal and valence classification task was 97.19% and 97.58%, respectively. Our results proved the spatio-temporal effectiveness of the method for emotion classification. In addition, we experimentally verified the optimal cardinality of split-convert-merge techniques in emotion recognition task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Halac E et al (2021) Impaired theory of mind and emotion recognition in pediatric bipolar disorder: a systematic review and meta-analysis. J Psychiatr Res 138:246–255. https://doi.org/10.1016/j.jpsychires.2021.04.011

    Article  Google Scholar 

  2. Dong H, Chen D, Zhang L, Ke H, Li X (2021) Subject sensitive EEG discrimination with fast reconstructable CNN driven by reinforcement learning: A case study of ASD evaluation. Neurocomputing 449:136–145. https://doi.org/10.1016/j.neucom.2021.04.009

    Article  Google Scholar 

  3. De Nadai D, et al (2016) Enhancing safety of transport by road by on-line monitoring of driver emotions (in English). In: 2016 11th Systems of System Engineering Conference (Sose). IEEE. https://doi.org/10.1109/SYSOSE.2016.7542941

  4. Martínez A, Belmonte LM, García AS, Fernández-Caballero A, Morales R (2021) Facial emotion recognition from an unmanned flying social Robot for home care of dependent people. Electronics 10(7):1. https://doi.org/10.3390/electronics10070868

    Article  Google Scholar 

  5. Garcia-Cordero I et al (2021) Metacognition of emotion recognition across neurodegenerative diseases. Cortex 137:93–107. https://doi.org/10.1016/j.cortex.2020.12.023

    Article  Google Scholar 

  6. Huang X, Wang S-J, Liu X, Zhao G, Feng X, Pietikainen M (2019) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 10(1):32–47. https://doi.org/10.1109/taffc.2017.2713359

    Article  Google Scholar 

  7. Zhang ZX, Wu BW, Schuller B (2019) Attention-augmented end-to-end multi-task learning for emotion prediction from speech, (in English). In: 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 6705–6709. https://doi.org/10.1109/ICASSP.2019.8682896

  8. Zheng W (2017) Multichannel EEG-based emotion recognition via group sparse canonical correlation analysis. IEEE Trans Cogn Devel Syst 9(3):281–290. https://doi.org/10.1109/tcds.2016.2587290

    Article  Google Scholar 

  9. Agrafioti F, Hatzinakos D, Anderson AK (2012) ECG pattern analysis for emotion detection. IEEE Trans Affect Comput 3(1):102–115. https://doi.org/10.1109/t-affc.2011.28

    Article  Google Scholar 

  10. Bo C, Liu GJI (2008) Emotion recognition from surface EMG signal using wavelet transform and neural network. https://doi.org/10.1109/ICBBE.2008.670

  11. Samara A, Menezes MLR, Galway L (2016) Feature extraction for emotion recognition and modelling using neurophysiological data (in English). In: 2016 15th International conference on ubiquitous computing and communications and 2016 international symposium on cyberspace and security (IUCC-CSS), pp 138–144. https://doi.org/10.1109/Iucc-Css.2016.26

  12. Zheng X, Zhang M, Li T, Ji C, Hu B (2021) A novel consciousness emotion recognition method using ERP components and MMSE. J Neural Eng 18(4):1. https://doi.org/10.1088/1741-2552/abea62

    Article  Google Scholar 

  13. Zheng W-L, Zhu J-Y, Lu B-L (2019) Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput 10(3):417–429. https://doi.org/10.1109/taffc.2017.2712143

    Article  Google Scholar 

  14. Shi LC, Jiao YY, Lu BL (2013) Differential entropy feature for EEG-based vigilance estimation. Annu Int Conf IEEE Eng Med Biol Soc 2013:6627–6630. https://doi.org/10.1109/EMBC.2013.6611075

    Article  Google Scholar 

  15. Hadjidimitriou SK, Hadjileontiadis LJ (2012) Toward an EEG-based recognition of music liking using time-frequency analysis. IEEE Trans Biomed Eng 59(12):3498–3510. https://doi.org/10.1109/TBME.2012.2217495

    Article  Google Scholar 

  16. Khosrowabadi R, Quek HC, Wahab A, Kai KA (2010) EEG-based emotion recognition using self-organizing map for boundary detection. In: International Conference on Pattern Recognition

  17. Verma GK, Tiwary US (2014) Multimodal fusion framework: a multiresolution approach for emotion classification and recognition from physiological signals. Neuroimage 102(1):162–172. https://doi.org/10.1016/j.neuroimage.2013.11.007

    Article  Google Scholar 

  18. Alex M, Tariq U, Al-Shargie F, Mir HS, Nashash HA (2020) Discrimination of genuine and acted emotional expressions using EEG signal and machine learning. IEEE Access 8:191080–191089. https://doi.org/10.1109/access.2020.3032380

    Article  Google Scholar 

  19. Duan RN, Zhu JY, Lu BL (2013) Differential entropy feature for EEG-based emotion classification (in English). In: 2013 6th International IEEE/EMBS Conference on Neural Engineering (Ner), pp 81–84. https://doi.org/10.1109/NER.2013.6695876

  20. Atkinson J, Campos D (2016) Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Syst Appl 47:35–41. https://doi.org/10.1016/j.eswa.2015.10.049

    Article  Google Scholar 

  21. Gao Y, Wang X, Potter T, Zhang J, Zhang Y (2020) Single-trial EEG emotion recognition using Granger Causality/Transfer Entropy analysis. J Neurosci Methods 346:108904. https://doi.org/10.1016/j.jneumeth.2020.108904

    Article  Google Scholar 

  22. Zhang L, Chen D, Chen P, Li W, Li X (2021) Dual-CNN based multi-modal sleep scoring with temporal correlation driven fine-tuning. Neurocomputing 420:317–328. https://doi.org/10.1016/j.neucom.2020.08.020

    Article  Google Scholar 

  23. Liu NJ, Fang YC, Li L, Hou LM, Yang FL, Guo YK (2018) Multiple Feature Fusion for Automatic Emotion Recognition Using Eeg Signals (in English). In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 896–900. https://doi.org/10.1109/ICASSP.2018.8462518

  24. Alhalaseh R, Alasasfeh S (2020) Machine-learning-based emotion recognition system using EEG signals. Computers 9(4):1. https://doi.org/10.3390/computers9040095

    Article  Google Scholar 

  25. Yang Y, Wu Q, Fu Y, Chen X (2018) Continuous convolutional neural network with 3D input for EEG-based emotion recognition. In: Neural Information Processing (Lecture Notes in Computer Science. pp 433–443

  26. Yin Y, Zheng X, Hu B, Zhang Y, Cui X (2021) EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Appl Soft Comput 100:1. https://doi.org/10.1016/j.asoc.2020.106954

    Article  Google Scholar 

  27. Yan M, Meng J, Zhou C, Tu Z, Tan Y-P, Yuan J (2020) Detecting spatiotemporal irregularities in videos via a 3D convolutional autoencoder. J Vis Commun Image Represent 67:1. https://doi.org/10.1016/j.jvcir.2019.102747

    Article  Google Scholar 

  28. Maqsood R, Bajwa UI, Saleem G, Raza RH, Anwar MW (2021) Anomaly recognition from surveillance videos using 3D convolution neural network. Multimed Tools Appl 80(12):18693–18716. https://doi.org/10.1007/s11042-021-10570-3

    Article  Google Scholar 

  29. Salama ES, El-Khoribi RA, Shoman ME, Wahby MA (2018) EEG-based emotion recognition using 3D convolutional neural networks. Int J Adv Comput Sci Appl 9(8):1. https://doi.org/10.14569/ijacsa.2018.090843

    Article  Google Scholar 

  30. Wang Y, Huang Z, McCane B, Neo P (2018) EmotioNet: A 3-D Convolutional Neural Network for EEG-based Emotion Recognition. In: Presented at the 2018 international joint conference on neural networks (IJCNN)

  31. Salama ES, El-Khoribi RA, Shoman ME, Wahby Shalaby MA (2021) A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egypt Inf J 22(2):167–176. https://doi.org/10.1016/j.eij.2020.07.005

    Article  Google Scholar 

  32. Hu J, Shen L, Albanie S, Sun G, Wu E (2020) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42(8):2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372

    Article  Google Scholar 

  33. Xie SN, Girshick R, Dollar P, Tu ZW, He KM (2017) Aggregated residual transformations for deep neural networks (in English). In: 30th IEEE conference on computer vision and pattern recognition (CVPR 2017), pp 5987–5995. https://doi.org/10.1109/Cvpr.2017.634

  34. Hara K, Kataoka H, Satoh Y (2018) Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? In: Presented at the 2018 IEEE/CVF conference on computer vision and pattern recognition

  35. Abadi M, et al. (2016) TensorFlow: A system for large-scale machine learning (in English). Proceedings of Osdi'16: 12th Usenix symposium on operating systems design and implementation, pp 265–283

  36. Koelstra S et al (2012) DEAP: A database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31. https://doi.org/10.1109/t-affc.2011.15

    Article  Google Scholar 

  37. Wang X-W, Nie D, Lu B-L (2014) Emotional state classification from EEG data using machine learning approach. Neurocomputing 129:94–106. https://doi.org/10.1016/j.neucom.2013.06.046

    Article  Google Scholar 

  38. Shen F, Dai G, Lin G, Zhang J, Kong W, Zeng H (2020) EEG-based emotion recognition using 4D convolutional recurrent neural network. Cogn Neurodyn 14(6):815–828. https://doi.org/10.1007/s11571-020-09634-1

    Article  Google Scholar 

  39. Kwon YH, Shin SB, Kim SD (2018) Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system. Sensors (Basel) 18(5):1. https://doi.org/10.3390/s18051383

    Article  Google Scholar 

  40. Luo Y et al (2020) EEG-based emotion classification using spiking neural networks. IEEE Access 8:46007–46016. https://doi.org/10.1109/access.2020.2978163

    Article  Google Scholar 

  41. Yang YL, Wu QF, Qiu M, Wang YD, Chen XW (2018) Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network (in English). In: 2018 International joint conference on neural networks (IJCNN), pp 793–799. https://doi.org/10.1109/IJCNN.2018.8489331

  42. Chen J, Jiang D, Zhang Y, Zhang P (2020) Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset. Comput Commun 154:58–65. https://doi.org/10.1016/j.comcom.2020.02.051

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China [62072394, 62173291], the Natural Science Foundation of Hebei Province of China [F2021203019] and Hebei Key Laboratory Project [202250701010046].

Author information

Authors and Affiliations

  1. Hebei Key Laboratory of Information Transmission and Signal Processing, School of Information Science and Engineering, Yanshan University, Qinhuangdao, 066004, China

    Dong Cui, Hongyuan Xuan, Jing Liu & Guanghua Gu

  2. National Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, 100875, China

    Xiaoli Li

Authors
  1. Dong Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongyuan Xuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guanghua Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaoli Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guanghua Gu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cui, D., Xuan, H., Liu, J. et al. Emotion Recognition on EEG Signal Using ResNeXt Attention 2D-3D Convolution Neural Networks. Neural Process Lett 55, 5943–5957 (2023). https://doi.org/10.1007/s11063-022-11120-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-11120-0

Keywords

Search

Navigation