Skip to main content
SpringerLink
Log in

EEG-based emotion recognition with cascaded convolutional recurrent neural networks

  • Short Paper
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

In recent years, deep learning has gradually become a prevailing way in EEG-based emotion recognition research because it can extract features and classify emotions automatically. To fully exploit the underlying information in EEG signals, we propose an emotion recognition method based on cascaded convolutional recurrent neural networks. Firstly, the differential entropy features of each channel signal are transformed into four-dimensional structure data, which are able to contain temporal-spatial-frequency information integratively. Then, the cascaded VGG16 and long short-term memory (LSTM) networks are applied to learn the spatiotemporal information of the samples, and the hidden layer of the last node of LSTM is output to a linear transformation classifier to perform classification. On DEAP dataset, the proposed method gives out an average accuracy of 94.43% and 94.85% in arousal-based and valence-based classification, respectively. On SEED dataset, the method achieves average accuracy of 94.16%. Compared with the existing methods, our method demonstrates superior performances in emotion recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig.3
Fig.4
Fig. 5
Fig. 6
Fig.7
Fig.8
Fig.9
Fig.10
Fig.11

Similar content being viewed by others

Data availability

The datasets used during the current study are available from the corresponding author on reasonable request. The authors use the public datasets DEAP and SEED to achieve the experiments.

Code availability

Custom code.

References

  1. Li M, Xu H, Liu X, Lu S (2018) Emotion recognition from multi channel EEG signals using K-nearest neighbor classification. Technol Health Care 26:509–519

    Article  Google Scholar 

  2. Lin S, Xie J, Yang M, Li Z, Yang X (2018) A review of emotion recognition using physiological signals. Sensors 18(7):2074

    Article  Google Scholar 

  3. Dzedzickis A, Kaklauskas A, Bucinskas V (2020) Human emotion recognition: review of sensors and methods. Sensors 20(3):592

    Article  Google Scholar 

  4. Velchev, Y, Radeva S, Sokolov S, Radev D (2016) Automated estimation of human emotion from EEG using statistical features and SVM. In: Digital media industry and academic forum. Santorini, Greece, pp 40–42

  5. Wu CH, Kuo B C, Tzeng G H (2015) Investigation of time interval size effect on SVM model in emotion norm database. In: Asian conference on intelliegent information and database systems, vol 8, pp 121–128

  6. Babiloni F, Bianchi L, Semeraro F, Millan J, Mourinyo J (2001) Mahalanobis distance-based classifiers are able to recognize EEG patterns by using few EEG electrodes. Engineering in medicine and biology society, 2001. In: Proceedings of the 23rd annual international conference of the IEEE, vol 1, pp 651–654

  7. Ko KE, Yang HC, Sim KB (2009) Emotion recognition using EEG signals with relative power values and bayesian network. Int J Control Autom Syst 7(5):865–870

    Article  Google Scholar 

  8. Lotte F, Congedo M, Lécuyer A, Lamarche F, Arnaldi B (2007) A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng 4(2):R1–R13. https://doi.org/10.1088/1741-2560/4/2/R01

    Article  Google Scholar 

  9. Zheng WL, Lu BL (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7(3):1–1

    Google Scholar 

  10. Yang Y, Wu Q, Fu Y, Chen X (2018) Continuous convolutional neural network with 3D input for EEG-based emotion recognition. In: International conference on neural information processing. Springer, Cham, pp 433–443

  11. Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2019) Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 49(3):839–847

    Article  Google Scholar 

  12. Hwang S, Hong K, Son G, Byun H (2019) Learning CNN features from DE features for EEG-based emotion recognition. Pattern Anal Appl 23:1323–1335

    Article  Google Scholar 

  13. Yang H, Han J, Min K (2019) Multi-column CNN model for emotion recognition from EEG signals. Sensors 19(21):4736

    Article  Google Scholar 

  14. Song T, Zheng W, Song P, Cui Z (2018) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11(3):532–541

    Article  Google Scholar 

  15. Deng X, Yang S, Zhu J (2021) SFE-Net: EEG-based emotion recognition with symmetrical spatial feature extraction. In: Proceedings of the 29th ACM International conference on multimedia, Association for computing machinery, New York, NY, USA, pp 2391–2400

  16. Alhagry S, Aly A, El-Khoribi RA (2017) Emotion recognition based on EEG using LSTM recurrent neural network. Int J Adv Comput Sci Appl 8:355–358

    Google Scholar 

  17. Kan W, Li Y, Computer SO (2019) Emotion recognition from EEG signals by using LSTM recurrent neural networks. J Nanjing Univ (Natl Sci) 55(1):110–116

    Google Scholar 

  18. Anubhav A, Nath D, Singh M, Sethia D, Diksha K, Indu S (2020) An efficient approach to eeg-based emotion recognition using LSTM network. In: 2020 16th IEEE international colloquium on signal processing and its applications (CS- PA), pp 88–92

  19. Wang Y, Huang ZY, Mccane B, Neo P (2018) Emotionet: A 3-D convolutional neural network for EEG-based emotion recognition. In: 2018 International joint conference on neural networks (IJCNN), pp 1–7

  20. Li J, Zhang Z, He H (2018) Hierarchical convolutional neural networks for EEG-based emotion recognition. Cogn Comput 10:360–380

    Article  Google Scholar 

  21. Yang Y, Wu Q, Qiu M, Wang Y, Chen X (2018) Emotion recognition from multi-channel eeg through parallel convolutional recurrent neural network. In: 2018 International joint conference on neural networks (IJCNN), Brazil, pp 1–7

  22. Ding Y, Robinson N, Zeng Q, Chen D, Aung Phyo wai A, Lee TS (2020) TSception: A deep learning framework for emotion detection using EEG. In: 2020 International joint conference on neural networks (IJCNN), UK, pp 1–7

  23. Jca B, Dj A, Yz A, Pz B (2020) Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wear- able multi-channel headset. Comput Commun 154:58–65

    Article  Google Scholar 

  24. Ziyu J, Youfang L, Xiyang C, Haobin C, Haijun G, Jing W (2020) SST-EmotionNet: Spatial-spectral-temporal based attention 3D dense network for EEG emotion recognition. In: Proceedings of the 28th ACM international conference on multimedia 2020. Association for computing machinery, Newyork, pp 2909–2917

  25. Koelstra S, Muhl C, Soleymani M, Lee JS, Yazdani A, Ebrahimi T et al (2012) DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Article  Google Scholar 

  26. Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39(6):1161–1980

    Article  Google Scholar 

  27. Asghar MA, Fawad, Khan MJ, Amin Y, Akram A (2020) EEG-based emotion recognition for multi-channel fast empirical mode decomposition using VGG-16. In: 2020 International conference on engineering and emerging technologies (ICEET), Pakistan, pp 1–7

  28. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. https://doi.org/10.48550/arXiv.1409.1556

  29. Tripathi S, Acharya S, Sharma R D, Mittal S, Bhattacharya S (2017) Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: Proceedings of the AAAI conference on innovative applications, San Francisco, USA, pp 4746–4752

  30. Cimtay Y, Ekmekcioglu E (2020) Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors 20(7):2034

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (No. 62271181 Ming Meng, 62071161 Yuliang Ma, and 61971168 Yunyuan Gao).

Author information

Authors and Affiliations

  1. Institute of Intelligent Control and Robotics, Hangzhou Dianzi University, Hangzhou, 310018, China

    Ming Meng, Yu Zhang, Yuliang Ma & Yunyuan Gao

  2. Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou, 310018, China

    Wanzeng Kong

Authors
  1. Ming Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuliang Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunyuan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wanzeng Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ming Meng.

Ethics declarations

Conflict of interest

Conflict of interest the authors have no competing interests.

Ethics approval

The research has no ethics problem.

Consent to participate

The authors approve the participation.

Consent for publication

The authors approve the publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Meng, M., Zhang, Y., Ma, Y. et al. EEG-based emotion recognition with cascaded convolutional recurrent neural networks. Pattern Anal Applic 26, 783–795 (2023). https://doi.org/10.1007/s10044-023-01136-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-023-01136-0

Keywords

Search

Navigation