Skip to main content
SpringerLink
Log in

Instance-representation transfer method based on joint distribution and deep adaptation for EEG emotion recognition

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Electroencephalogram (EEG) emotion recognition technology is essential for improving human–computer interaction. However, the practical application of emotion recognition technology is limited due to the variety of subjects and sessions. Transfer learning has been applied to address this issue and has received extensive research and application. Studies mainly concentrate on either instance transfer or representation transfer methods. This paper proposes an emotion recognition method called Joint Distributed Instances Represent Transfer (JD-IRT), which includes two core components: Joint Distribution Deep Adaptation (JDDA) and Instance-Representation Transfer (I-RT). Specifically, JDDA is different from common representation transfer methods in transfer learning. It bridges the discrepancies of marginal and conditional distributions simultaneously and combines multiple adaptive layers and kernels for deep domain adaptation. On the other hand, I-RT utilizes instance transfer to select source domain data for better representation transfer. We performed experiments and compared them with other representative methods in the SEED, SEED-IV, and SEED-V datasets. In cross-subject experiments, our approach achieved an average accuracy of 83.21% in SEED, 52.12% in SEED-IV, and 60.17% in SEED-V. Similarly, in cross-session experiments, the accuracy was 91.29% in SEED, 59.02% in SEED-IV, and 65.91% in SEED-V. These results demonstrate the improvement in the accuracy of EEG emotion recognition using the proposed approach.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Dolan RJ (2002) Emotion, cognition, and behavior[J]. Science 298(5596):1191–1194

    Article  CAS  PubMed  Google Scholar 

  2. Li X, Hu B, Zhu T et al (2009) Towards affective learning with an EEG feedback approach[C]//Proceedings of the first ACM international workshop on Multimedia technologies for distance learning. 33–38

  3. Chen YF, Cui YL, Wang SX (2017) Review of emotion recognition based on physiological signals[J]. Syst Simul Technol 13(1):1–5

    Google Scholar 

  4. Pan SJ, Yang Q (2010) A survey on transfer learning[J]. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  5. Zhuang F, Qi Z, Duan K et al (2020) A comprehensive survey on transfer learning[J]. Proc IEEE 109(1):43–76

    Article  Google Scholar 

  6. Li J, Chen H, Cai T (2020) Foit: Fast online instance transfer for improved EEG emotion recognition[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE. 2618–2625

  7. Huang J, Gretton A, Borgwardt K et al (2007) Correcting sample selection bias by unlabeled data[J]. Adv Neural Inf Proces Syst 19:601–608

  8. Tan Q, Deng H, Yang P (2012) Kernel mean matching with a large margin[C]//Advanced Data Mining and Applications: 8th International Conference, ADMA 2012, Nanjing, China, December 15-18, 2012. Proceedings 8. Springer Berlin Heidelberg, 223–234

  9. Miao YQ, Araujo R, Kamel MS (2012) Cross-domain facial expression recognition using supervised kernel mean matching[C]//2012 11th International Conference on Machine Learning and Applications. IEEE 2:326–332

  10. Sugiyama M, Nakajima S, Kashima H et al (2007) Direct importance estimation with model selection and its application to covariate shift adaptation[C]. In: Proceedings of the 20th International Conference on Neural Information Processing Systems, pp 1433–1440

  11. Li J, Qiu S, Shen YY et al (2019) Multisource transfer learning for cross-subject EEG emotion recognition[J]. IEEE Trans Cybernet 50(7):3281–3293

    Google Scholar 

  12. Pan SJ, Tsang IW, Kwok JT et al (2010) Domain adaptation via transfer component analysis[J]. IEEE Trans Neural Networks 22(2):199–210

    Article  PubMed  Google Scholar 

  13. Gretton A, Borgwardt K, Rasch M et al (2006) A kernel method for the two-sample-problem[C]. In: Proceedings of the 19th International Conference on Neural Information Processing Systems, pp 513–520

  14. Pan SJ, Kwok JT, Yang Q (2008) Transfer learning via dimensionality reduction[C]. In: Proceedings of the 23rd National Conference on Artificial Intelligence, vol 2, pp 677–682

  15. Borgwardt KM, Gretton A, Rasch MJ et al (2006) Integrating structured biological data by Kernel Maximum Mean Discrepancy[J]. Bioinformatics 22(14):e49–e57

  16. Xiao D, Huang Y, Zhao L et al (2019) Domain adaptive motor fault diagnosis using deep transfer learning[J]. IEEE Access 7:80937–80949

    Article  Google Scholar 

  17. Long M, Wang J, Ding G et al (2013) Transfer feature learning with joint distribution adaptation[C]//Proceedings of the IEEE international conference on computer vision. 2200–2207

  18. Tzeng E, Hoffman J, Zhang N et al (2014) Deep domain confusion: maximizing for domain invariance[J]. arXiv preprint arXiv:1412.3474

  19. Gretton A, Sriperumbudur B, Sejdinovic D et al (2012) Optimal kernel choice for large-scale two-sample tests[C]. In: 26th Annual Conference on Neural Information Processing Systems 2012, NIPS 2012, pp 1205–1213

  20. Ganin Y, Ustinova E, Ajakan H et al (2016) Domain-adversarial training of neural networks[J]. J Mach Learn Res 17(1):2096–2030

    Google Scholar 

  21. Yosinski J, Clune J, Bengio Y et al (2014) How transferable are features in deep neural networks? [J] Adv Neural Inf Proces Syst 3320–3328

  22. Donahue J, Jia Y, Vinyals O et al (2014) Decaf: a deep convolutional activation feature for generic visual recognition[C]. Proceedings of the 31st International Conference on Machine Learning, PMLR 32(1):647–655

  23. Long M, Cao Y, Wang J et al (2015) Learning transferable features with deep adaptation networks[C]. In: International Conference on Machine Learning, pp 97–105

  24. Wang F, Zhang W, Xu Z et al (2021) A deep multi-source adaptation transfer network for cross-subject electroencephalogram emotion recognition[J]. Neural Comput Appl 33:9061–9073

    Article  Google Scholar 

  25. Chen H, Jin M, Li Z et al (2021) MS-MDA: multisource marginal distribution adaptation for cross-subject and cross-session EEG emotion recognition[J]. Front Neurosci 15:778488

    Article  PubMed  PubMed Central  Google Scholar 

  26. Li Y, Wang D, Zhang H et al (2016) Changes of brain connectivity in the primary motor cortex after subcortical stroke: a multimodal magnetic resonance imaging study[J]. Medicine 95(6):e2579

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Zhang T, Guo L, Li K et al (2011) Predicting functional brain ROIs via fiber shape models[C]//Medical Image Computing and Computer-Assisted Intervention–MICCAI 2011: 14th International Conference, Toronto, Canada, September 18-22, 2011, Proceedings, Part II 14. Springer, Berlin, 42-49

  28. Zheng WL, Lu BL (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Trans Auton Ment Dev 7(3):162–175

    Article  Google Scholar 

  29. Sun Q, Chattopadhyay R, Panchanathan S et al (2011) A two-stage weighting framework for multi-source domain adaptation[C]. In: Proceedings of the 24th International Conference on Neural Information Processing Systems, pp 505–513

  30. Zhong E, Fan W, Peng J et al (2009) Cross domain distribution adaptation via kernel mapping[C]//Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 1027–1036

  31. Quanz B, Huan J, Mishra M (2012) Knowledge transfer with low-quality data: A feature extraction issue[J]. IEEE Trans Knowl Data Eng 24(10):1789–1802

    Article  Google Scholar 

  32. Duan RN, Zhu JY, Lu BL (2013) Differential entropy feature for EEG-based emotion classification[C]//2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE 81–84

  33. Zheng WL, Liu W, Lu Y et al (2018) Emotionmeter: a multimodal framework for recognizing human emotions[J]. IEEE Trans Cybernet 49(3):1110–1122

    Article  Google Scholar 

  34. Liu W, Qiu JL, Zheng WL et al (2021) Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition[J]. IEEE Trans Cognit Dev Syst 14(2):715–729

    Article  Google Scholar 

  35. Van der Maaten L, Hinton G (2008) Visualizing data using t-sne[J]. J Mach Learn Res 9:2579–2605

Download references

Funding

The work was supported by the Key Research and Development Project of Zhejiang Province (2020C04009) and the Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province (2020E10010).

Author information

Authors and Affiliations

  1. School of Automation, Hangzhou Dianzi University, Hangzhou, 310000, China

    Lei Zhu, Fei Yu, Aiai Huang & Nanjiao Ying

  2. Center for Drug Inspection of Zhejiang Province, Hangzhou, 310000, China

    Nanjiao Ying

  3. School of Computer Science, Hangzhou Dianzi University, Hangzhou, 310000, China

    Jianhai Zhang

  4. Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou, 310000, China

    Jianhai Zhang

Authors
  1. Lei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fei Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aiai Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nanjiao Ying
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jianhai Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Zhu.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, L., Yu, F., Huang, A. et al. Instance-representation transfer method based on joint distribution and deep adaptation for EEG emotion recognition. Med Biol Eng Comput 62, 479–493 (2024). https://doi.org/10.1007/s11517-023-02956-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02956-2

Keywords

Search

Navigation