Skip to main content
SpringerLink
Log in

An attention-based hybrid deep learning model for EEG emotion recognition

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Emotion recognition based on electroencephalography (EEG) has received much attention in recent years, and there is more and more research on emotion recognition utilizing deep learning. It is difficult to extract more discriminative features for emotion recognition. To solve this problem, an attention-based hybrid deep learning model is proposed for EEG emotion recognition. The proposed approach extracts the critical feature information and achieves an excellent classification effect. To begin, the differential entropy features of EEG data are extracted and organized according to electrodeposition. Then, the convolutional encoder is used to encode the EEG signal and extract the spatial features, and the band attention mechanism is introduced to assign adaptive weights to different bands. Finally, a long short-term memory network is utilized to extract temporal features, and a time attention mechanism is used to obtain critical temporal information. The performance of the proposed model is analyzed on benchmark emotion databases such as DEAP and SEED for the classification task. The experimental results in terms of accuracy are 85.86% and 84.27% on DEAP datasets and 92.47% on SEED datasets. The experimental analysis shows that the proposed model can effectively recognize emotions and has a good classification performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Safayari, A., Bolhasani, H.: Depression diagnosis by deep learning using EEG signals: a systematic review. Med. Novel Technol. Devices 12, 100102 (2021)

    Article  Google Scholar 

  2. Ahern, G.L., Schwartz, G.E.: Differential lateralization for positive and negative emotion in the human brain: EEG spectral analysis. Neuropsychologia 23(6), 745–755 (1985)

    Article  Google Scholar 

  3. Zhang, Y., Cheng, C., Zhang, Y.: Multimodal emotion recognition based on manifold learning and convolution neural network. Multimed. Tools Appl. 81(23), 33253–33268 (2022)

    Article  Google Scholar 

  4. Broek, E.: Ubiquitous emotion-aware computing. Pers. Ubiquit. Comput. 17, 53–67 (2013)

    Article  Google Scholar 

  5. Zhu, J.Y., Zheng, W.L., Lu, B.L.: Cross-subject and cross-gender emotion classification from EEG. World Congr. Med. Phys. Biomed. Eng. 51, 1188–1191 (2015)

    Google Scholar 

  6. Zhuang, N., Zeng, Y., Tong, L., Zhang, C., Zhang, H., Yan, B.: Emotion recognition from EEG signals using multidimensional information in EMD domain. Biomed. Res. Int. 2017, 8317357 (2017)

    Article  Google Scholar 

  7. Kamble, K.S., Sengupta, J.: Ensemble machine learning-based affective computing for emotion recognition using dual-decomposed EEG signals. IEEE Sens. J. 22(3), 2496–2507 (2022)

    Article  Google Scholar 

  8. Radhakrishnan, M., Ramamurthy, K., Choudhury, K.K., Won, D., Manoharan, T.A.: Performance analysis of deep learning models for detection of autism spectrum disorder from EEG signals. Traitement du Signal 38(3), 853–863 (2021)

    Article  Google Scholar 

  9. Song, T.F., Zheng, W.M., Song, P., Cui, Z.: EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans. Affect. Comput. 11(3), 532–541 (2018)

    Article  Google Scholar 

  10. Wen, Z.Y., Xu, R.F., Du, J.C.: A novel convolutional neural networks for emotion recognition based on EEG signal. In: International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), pp. 672–677. IEEE (2017)

  11. Liu, Y., Ding, Y., Li, C., Cheng, J., Song, R., Wan, F., Chen, X.: Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Comput. Biol. Med. 123, 103927 (2020)

    Article  Google Scholar 

  12. Zhang, Y., Cheng, C., Wang, S., Xia, T.: Emotion recognition using heterogeneous convolutional neural networks combined with multimodal factorized bilinear pooling. Biomed. Signal Process. Control 77, 103877 (2022)

    Article  Google Scholar 

  13. Zhang, Y., Cheng, C., Zhang, Y.D.: Multimodal emotion recognition using a hierarchical fusion convolutional neural network. IEEE Access 9, 7943–7951 (2021)

    Article  Google Scholar 

  14. Zheng, W.L., Guo, H.T., Lu, B.L.: Revealing critical channels and frequency bands for emotion recognition from EEG with deep belief network. In: 7th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 154–157. IEEE (2015)

  15. Mathur, N., Gupta, A., Jaswal, S., Verma, R.: Deep learning helps EEG signals predict different stages of visual processing in the human brain. Biomed. Signal Process. Control 70, 102996 (2021)

    Article  Google Scholar 

  16. Li, J.P., Zhang, Z.X., He, H.G.: Hierarchical convolutional neural networks for EEG-based emotion recognition. Cogn. Comput. 10, 368–380 (2018)

    Article  Google Scholar 

  17. Moon, S., Jang, S., Lee, J.: Convolutional neural network approach for EEG-based emotion recognition using brain connectivity and its spatial information. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2556–2560, IEEE (2018)

  18. Xing, X.F., Li, Z.Q., Xu, T.Y., Shu, L., Hu, B., Xu, X.M.: SAE+LSTM: a new framework for emotion recognition from multi-channel EEG. Front Neurorobot. 13, 37 (2019)

    Article  Google Scholar 

  19. Ma, J.X., Tang, H, Zheng, W.L., Lu, B.L.: Emotion recognition using multimodal residual LSTM network. In: 27th (ACM) International Conference on Multimedia, pp. 176–183. ACM (2019)

  20. Feng, L., Cheng, C., Zhao, M., Deng, H., Zhang, Y.: EEG-based emotion recognition using spatial–temporal graph convolutional LSTM with attention mechanism. IEEE J. Biomed. Health Inform. 26(11), 5406–5417 (2022)

    Article  Google Scholar 

  21. Chen, J.X., Jiang, D.M., Zhang, Y.N.: A hierarchical bidirectional GRU model with attention for EEG-based emotion classification. IEEE Access 7, 118530–118540 (2019)

    Article  Google Scholar 

  22. Lan, Y.T., Liu, W., Lu, B.L.: Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–6. IEEE (2020)

  23. Yang, Y.L., Wu, Q.F., Fu, Y.Z., Chen, X.W.: Continuous convolutional neural network with 3D input for EEG-based emotion recognition. In: International Conference on Neural Information Processing, pp. 433–443. Springer (2018)

  24. Shen, F.Y., Dai, G.J., Lin, G., Zhang, J.H., Kong, W.Z., Zeng, H.: EEG-based emotion recognition using 4D convolutional recurrent neural network. Cogn. Comput. 14, 815–828 (2020)

    Google Scholar 

  25. Yang, Z.C., Yang, D.Y., Dyer, C., He, X.D., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489. ACL (2016)

  26. Koelstra, S., Muhl, C., Soleymani, M., Lee, J.: DEAP: a database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)

    Article  Google Scholar 

  27. Zheng, W.L., Lu, B.L.: Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 7(3), 162–175 (2015)

    Article  Google Scholar 

  28. Topic, A., Russo, M.: Emotion recognition based on EEG feature maps through deep learning network. Eng. Sci. Technol. Int. J. 24(6), 1442–1454 (2021)

    Google Scholar 

  29. Zhong, Q.H., Zhu, Y.S., Cai, D.L., Xiao, L.W., Zhang, H.: Electroencephalogram access for emotion recognition based on a deep hybrid network. Front. Hum. Neurosci. 14, 589001 (2020)

    Article  Google Scholar 

  30. Sharma, R., Pachori, R.B., Sircar, P.: Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomed. Signal Process. Control 58, 101867 (2020)

    Article  Google Scholar 

  31. Wang, F., Wu, S.C., Zhang, W.W., Xu, Z.F., Zhang, Y.H., Wu, C.D., Coleman, S.: Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia 146, 107506 (2020)

    Article  Google Scholar 

  32. Du, X., Ma, C., Zhang, G., Li, J., Lai, Y.K., Zhao, G., Deng, X., Liu, Y., Wang, H.: An efficient LSTM network for emotion recognition from multichannel EEG signals. IEEE Trans. Affect. Comput. 13(3), 1528–1540 (2022)

    Article  Google Scholar 

  33. Iyer, A., Das, S.S., Teotia, R., Maheshwari, S., Sharma, R.R.: CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings. Multimed. Tools Appl. (2022). https://doi.org/10.1007/s11042-022-12310-7

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61772252, the Natural Science Foundation of Liaoning Province of China under Grant 2019-MS-216, and the Scientific Research Foundation of the Education Department of Liaoning Province (No. LJKZ0965).

Author information

Authors and Affiliations

  1. School of Computer and Information Technology, Liaoning Normal University, Dalian, 116081, China

    Yong Zhang, Yidie Zhang & Shuai Wang

  2. School of Information Engineering, Huzhou University, Huzhou, 313000, China

    Yong Zhang

Authors
  1. Yong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yidie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yong Zhang.

Ethics declarations

Conflict of interest

The authors declare that there are no financial or personal relationships with other people or organizations that could inappropriately influence this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Zhang, Y. & Wang, S. An attention-based hybrid deep learning model for EEG emotion recognition. SIViP 17, 2305–2313 (2023). https://doi.org/10.1007/s11760-022-02447-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02447-1

Keywords

Search

Navigation