Skip to main content
SpringerLink
Log in

Multi-domain fusion deep graph convolution neural network for EEG emotion recognition

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Electroencephalogram (EEG)-based emotion recognition has become a hot research field, with the most attention given to decoding three basic types of emotional states (i.e., positive, negative, and neutral) by EEG. Traditional EEG emotion recognition is a single feature input mode that cannot cover multiple feature information. For brain function networks, the feature extraction process is very cumbersome. There will also be individual differences in the characteristics of different subjects. Therefore, the theory of graph convolution and brain function connection is introduced into this research, and the multi-domain fusion features input deep graph convolution neural network (MdGCNN) is proposed in this paper. Pearson correlation is used to determine the adjacency matrix. The Sortpooling layer is employed as a bridge between the graph convolution neural layer and the normal neural network layer and sorts the node features in a consistent order. Based on analyzing the characteristics of a single electrode, the brain topology structure features are automatically extracted. Taking MdGCNN as the basic model and considering the method of minimizing the feature distance between the source and the target domain, we propose a transfer learning (TL) emotion recognition model for cross-subject called MdGCNN-TL. Meanwhile, MdGCNN-TL is extended to traverse the target domain of a single subject in a two-to-one domain form. According to the idea of principal component analysis, the transfer model with a high recognition effect is determined with the degree of subject correlation (SC), and MdGCNN-TL is upgraded to MdGCNN-TL-SC. Experimental analysis on the SEED dataset is performed to evaluate the proposed models. Further validation of the model is implemented on the DEAP dataset. The results show that the proposed model has achieved better performance in EEG emotion recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Wang C, Hu J, Liu K, Jia Q, Chen J, Yang K, Feng, J (2021) Eeg-based emotion recognition fusing spacial-frequency domain features and data-driven spectrogram-like features, In: International Symposium on Bioinformatics Research and Applications. Springer, , pp. 460–470

  2. Zheng W-L, Zhu J-Y, Lu B-L (2019) Identifying stable patterns over time for emotion recognition from eeg. IEEE Trans. Affective Comput. 10(3):417–429

    Article  Google Scholar 

  3. Rahim A, Sagheer A, Nadeem K, Dar MN, Rahim A, Akram U (2019) Emotion charting using real-time monitoring of physiological signals, In: 2019 International Conference on Robotics and Automation in Industry (ICRAI), , pp. 1–5

  4. Acharya D, Jain R, Panigrahi SS, Sahni R, Jain S, Deshmukh SP, Bhardwaj A(2020) Multi-class emotion classification using eeg signals, In: International Advanced Computing Conference. Springer, , pp. 474–491

  5. Zhang S, Hu B, Li T, Zheng X (2018) A study on emotion recognition based on hierarchical adaboost multi-class algorithm, In: International Conference on Algorithms and Architectures for Parallel Processing. Springer, , pp. 105–113

  6. Sarıkaya MA, İnce G (2017) Emotion recognition from eeg signals through one electrode device. In: 2017 25th Signal Processing and Communications Applications Conference (SIU), pp. 1–4

  7. Chen X, Zeng W, Shi Y, Deng J, Ma Y (2019) Intrinsic prior knowledge driven cica fmri data analysis for emotion recognition classification. IEEE Access 7:59944–59950

    Article  Google Scholar 

  8. Leal A, Lopes R, Arriaga P, Esteves F (2014) The brain mapping of emotion in human faces: clinical aplication in epilepsy, In: 2014 IEEE International Symposium on Medical Measurements and Applications (MeMeA), pp. 1–4

  9. Chen X, Tao X, Wang FL, Xie H(2021) Global research on artificial intelligence-enhanced human electroencephalogram analysis, Neural Computing and Applications, pp. 1–39,

  10. Geethanjali P, Mohan YK, Sen J (2012)Time domain feature extraction and classification of eeg data for brain computer interface, In: 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery, pp. 1136–1139

  11. Roberge J-P, Rispal S, Wong T, Duchaine V (2016)Unsupervised feature learning for classifying dynamic tactile events using sparse coding, In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 2675–2681

  12. Duan R-N, Zhu J-Y, Lu B-L (2013) Differential entropy feature for eeg-based emotion classification,” In: 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 81–84

  13. Sakai M (2013) Kernel nonnegative matrix factorization with constraint increasing the discriminability of two classes for the eeg feature extraction, In: 2013 International Conference on Signal-Image Technology Internet-Based Systems, pp. 966–970

  14. Ren F, Dong Y, Wang W(2018) Emotion recognition based on physiological signals using brain asymmetry index and echo state network, Neural Computing and Applications, pp. 1–11

  15. Mohammadi Z, Frounchi J, Amiri M (2015) Wavelet-based emotion recognition system using eeg signal. Neural Comput Appl 28:1985–1990

    Article  Google Scholar 

  16. Popov A, Zhukov M, Kanaykin A, and Bodilovsky O (2010)Identification of epileptiform complexes in electroencephalogram using adapted continuous wavelet transform, In: 2010 International Conference on Modern Problems of Radio Engineering, Telecommunications and Computer Science (TCSET), pp. 132–132

  17. Yao H, He H, Wang S, Xie Z (2019)Eeg-based emotion recognition using multi-scale window deep forest, In: 2019 IEEE Symposium Series on Computational Intelligence (SSCI) pp. 381–386

  18. Feng G, Hao L, Nuo G(2019) Feature extraction algorithm based on csp and wavelet packet for motor imagery eeg signals, In: 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP),, pp. 798–802

  19. Alakuş TB,. Türkoğlu (2019) Eeg-based emotion estimation with different deep learning models, In: 2019 4th International Conference on Computer Science and Engineering (UBMK), pp. 33–37

  20. Cao G, Ma Y, Meng X, Gao Y, Meng M (2019) Emotion recognition based on cnn, In: 2019 Chinese Control Conference (CCC), pp. 8627–8630

  21. Chao H, Liu Y (2020) Emotion recognition from multi-channel eeg signals by exploiting the deep belief-conditional random field framework. IEEE Access 8:33002–33012

    Article  Google Scholar 

  22. Acharya D, Goel S, Bhardwaj H, Sakalle A, Bhardwaj A (2020) A long short term memory deep learning network for the classification of negative emotions using eeg signals, In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8

  23. Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2019) Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans Cybernet 49(3):839–847

    Article  Google Scholar 

  24. Zheng W-L, Liu W, Lu Y, Lu B-L, Cichocki A (2019) Emotionmeter: a multimodal framework for recognizing human emotions. IEEE Trans. Cybernet. 49(3):1110–1122

    Article  Google Scholar 

  25. Cheah KH, Nisar H, Yap VV, Lee C-Y (2019) Convolutional neural networks for classification of music-listening eeg: comparing 1d convolutional kernels with 2d kernels and cerebral laterality of musical influence. Neural Comput. Appl. 32:8867–8891

    Article  Google Scholar 

  26. Xue B, Lv Z, Xue J (2020)Feature transfer learning in eeg-based emotion recognition, In: 2020 Chinese Automation Congress (CAC), pp. 3608–3611

  27. Zhang W, Wang F, Jiang Y, Xu Z, Wu S, Zhang Y (2019) Cross-subject eeg-based emotion recognition with deep domain confusion, In: International conference on intelligent robotics and applications. Springer, pp. 558–570

  28. Cai X, Wu Z, Zhong K, Su B, Dai D, Meng H, (2021) Unsupervised cross-lingual speech emotion recognition using domain adversarial neural network, In: 2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP), pp. 1–5

  29. Li Y, Zheng W, Zong Y, Cui Z, Zhang T, Zhou X (2021) A bi-hemisphere domain adversarial neural network model for eeg emotion recognition. IEEE Trans Affect Comput 12(2):494–504

    Article  Google Scholar 

  30. Cheah KH, Nisar H, Yap VV, Lee C-Y (2019) Convolutional neural networks for classification of music-listening eeg: comparing 1d convolutional kernels with 2d kernels and cerebral laterality of musical influence. Neural Comput Appl 32:8867–8891

    Article  Google Scholar 

  31. Organization (2004) Development and function of complex brain networks. Trends in Cognitive Sciences, 8(9): 418–425,

  32. Lee Y-Y, Hsieh S (2014) Classifying different emotional states by means of eeg-based functional connectivity patterns. PloS one 9(4):e95415

    Article  Google Scholar 

  33. Wang Z, Tong Y, Heng X (2019) Phase-locking value based graph convolutional neural networks for emotion recognition. IEEE Access 7:93711–93722

    Article  Google Scholar 

  34. Song T, Zheng W, Liu S, Zong Y, Cui Z, Li Y (2021) Graph-embedded convolutional neural network for image-based eeg emotion recognition, IEEE Transactions on Emerging Topics in Computing, pp. 1–1,

  35. Song T, Zheng W, Song P, Cui Z (2020) Eeg emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11(3):532–541

    Article  Google Scholar 

  36. Zhong P, Wang D, Miao C (2020) Eeg-based emotion recognition using regularized graph neural networks, IEEE Transactions on Affective Computing, pp. 1–1

  37. Chin T-J, Suter D (2007) Incremental kernel principal component analysis. IEEE Trans Image Process 16(6):1662–1674

    Article  MathSciNet  Google Scholar 

  38. W.-L. Zheng, H.-T. Guo, and B.-L. Lu, (2015) Revealing critical channels and frequency bands for emotion recognition from eeg with deep belief network, In: 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 154–157

  39. Li J, Zhang Z, He H (2016)Implementation of eeg emotion recognition system based on hierarchical convolutional neural networks, In: International Conference on Brain Inspired Cognitive Systems. Springer, pp. 22–33

  40. Yang B, Han X, Tang J, (2017) Three class emotions recognition based on deep learning using staked autoencoder, In: 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1–5

  41. Gao Z, Wang X, Yang Y, Li Y, Ma K, Chen G (2020) A channel-fused dense convolutional network for eeg-based emotion recognition, IEEE Transactions on Cognitive and Developmental Systems, pp. 1–1

  42. Li H, Jin Y-M, Zheng W-L, Lu B-L(2018) Cross-subject emotion recognition using deep adaptation networks, In: International conference on neural information processing. Springer, pp. 403–413

  43. Yan K, Kou L, Zhang D (2018) Learning domain-invariant subspace using domain features and independence maximization. IEEE Trans Cybernet 48(1):288–299

    Article  Google Scholar 

  44. Lan Z, Sourina O, Wang L, Scherer R, Müller-Putz GR (2019) Domain adaptation techniques for eeg-based emotion recognition: a comparative study on two public datasets. IEEE Trans Cognitive and Develop Syst 11(1):85–94

    Article  Google Scholar 

  45. Li X, Zhao Z, Song D, Zhang Y, Niu C, Zhang J, Huo J, Li J (2019) Variational autoencoder based latent factor decoding of multichannel eeg for emotion recognition, In: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 684–687

  46. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2011) Deap: A database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Article  Google Scholar 

  47. Lan Z, Sourina O, Wang L, Scherer R, Müller-Putz GR (2018) Domain adaptation techniques for eeg-based emotion recognition: a comparative study on two public datasets. IEEE Trans Cognitive and Develop Syst 11(1):85–94

    Article  Google Scholar 

  48. Wichakam I, Vateekul P, (2014) An evaluation of feature extraction in eeg-based emotion prediction with support vector machines, In: (2014) 11th international joint conference on computer science and software engineering (JCSSE). IEEE pp. 106–110

  49. Atkinson J, Campos D (2016) Improving bci-based emotion recognition by combining eeg feature selection and kernel classifiers. Exp Syst Appl 47:35–41

    Article  Google Scholar 

  50. Ren F, Dong Y, Wang W (2019) Emotion recognition based on physiological signals using brain asymmetry index and echo state network. Neural Comput Appl 31(9):4491–4501

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Foundation of National Natural Science Foundation of China under Grant: 61973065, 52075531, the Fundamental Research Funds for the Central Universities of China under Grant: N2104008, the Central Government Guides the Local Science And Technology Development Special Fund: 2021JH6/10500129, and the Innovative Talents Support Program of Liaoning Provincial Universities under LR2020047.

Author information

Authors and Affiliations

  1. College of Information Science and Engineering, Northeastern University, Shenyang, 110169, China

    Jinying Bi

  2. Faculty of Robot Science and Engineering, Northeastern University, Shenyang, 110819, China

    Fei Wang, Xin Yan, Jingyu Ping & Yongzhao Wen

Authors
  1. Jinying Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jingyu Ping
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yongzhao Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fei Wang.

Ethics declarations

Conflict of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bi, J., Wang, F., Yan, X. et al. Multi-domain fusion deep graph convolution neural network for EEG emotion recognition. Neural Comput & Applic 34, 22241–22255 (2022). https://doi.org/10.1007/s00521-022-07643-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07643-1

Keywords

Search

Navigation