Skip to main content
SpringerLink
Log in

Multi-channel EEG-based emotion recognition in the presence of noisy labels

  • Research Paper
  • Special Focus on Brain Machine Interfaces and Applications
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

A large number of deep learning classification methods for emotion recognition tasks based on electroencephalogram (EEG) have achieved excellent performance, and it is implicitly assumed that all labels are correct. However, humans have natural bias, subjectiveness, and inconsistencies in their judgment, which would lead to noisy labels for the EEG emotion state. To this end, we propose a framework for multi-channel EEG-based emotion recognition in the presence of noisy labels. The proposed noisy labels classification method is based on the capsule network using a joint optimization strategy (JO-CapsNet) until convergence. Specifically, the network parameters are updated based on the loss function of the capsule network, and the pseudo label is updated by predicting the existence possibility of the class label based on the output of the capsule network. In this way, the alternate updating strategy can promote each other to correct the noisy labels. Experimental results demonstrate the advantage of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Zhang T, Wang X H, Xu X M, et al. GCB-Net: graph convolutional broad network and its application in emotion recognition. IEEE Trans Affect Comput, 2019. doi: https://doi.org/10.1109/TAFFC.2019.2937768

  2. Shojaeilangari S, Yau W Y, Nandakumar K, et al. Robust representation and recognition of facial emotions using extreme sparse learning. IEEE Trans Image Process, 2015, 24: 2140–2152

    Article  MathSciNet  MATH  Google Scholar 

  3. Castellano G, Villalba S D, Camurri A. Recognising human emotions from body movement and gesture dynamics. In: Proceedings of International Conference on Affective Computing and Intelligent Interaction, 2007. 71–82

  4. Vu H A, Yamazaki Y, Dong F, et al. Emotion recognition based on human gesture and speech information using RT middleware. In: Proceedings of IEEE International Conference on Fuzzy Systems, 2011. 787–791

  5. Razak A, Yusof M H M, Komiya R. Towards automatic recognition of emotion in speech. In: Proceedings of the 3rd IEEE International Symposium on Signal Processing and Information Technology, 2003. 548–551

  6. Guo H W, Huang Y S, Lin C H, et al. Heart rate variability signal features for emotion recognition by using principal component analysis and support vectors machine. In: Proceedings of the 16th International Conference on Bioinformatics and Bioengineering, 2016. 274–277

  7. Silva D C, Vinhas V, Reis L P, et al. Biometric emotion assessment and feedback in an immersive digital environment. Int J Soc Robot, 2009, 1: 307–317

    Article  Google Scholar 

  8. Liu M Y, Fan D, Zhang X H, et al. Human emotion recognition based on galvanic skin response signal feature selection and SVM. In: Proceedings of International Conference on Smart City and Systems Engineering, 2016. 157–160 9 Yang G, Yang S. Emotion recognition of electromyography based on support vector machine. In: Proceedings of the 3rd International Symposium on Intelligent Information Technology and Security Informatics, 2010. 298–301

  9. Chen X, Li C, Liu A P, et al. Toward open-world electroencephalogram decoding via deep learning: a comprehensive survey. IEEE Signal Process Mag, 2022, 39: 117–134

    Article  Google Scholar 

  10. Cheng J, Chen M Y, Li C, et al. Emotion recognition from multi-channel EEG via deep forest. IEEE J Biomed Health Inform, 2021, 25: 453–464

    Article  Google Scholar 

  11. Li C, Tao W, Cheng J, et al. Robust multichannel EEG compressed sensing in the presence of mixed noise. IEEE Sens J, 2019, 19: 10574–10583

    Article  Google Scholar 

  12. Adolphs R, Tranel D, Damasio A R. Dissociable neural systems for recognizing emotions. Brain Cognition, 2003, 52: 61–69

    Article  Google Scholar 

  13. Li C, Wang B, Zhang S L, et al. Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism. Comput Biol Med, 2022, 143: 105303

    Article  Google Scholar 

  14. Nie D, Wang X W, Shi L C, et al. EEG-based emotion recognition during watching movies. In: Proceedings of the 5th International IEEE/EMBS Conference on Neural Engineering, 2011. 667–670

  15. Li C, Zhang Z Z, Song R C, et al. EEG-based emotion recognition via neural architecture search. IEEE Trans Affect Comput, 2021. doi: https://doi.org/10.1109/TAFFC.2021.3130387

  16. Tao W, Li C, Song R C, et al. EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans Affect Comput, 2020. doi: https://doi.org/10.1109/TAFFC.2020.3025777

  17. Li M, Lu B L. Emotion classification based on gamma-band EEG. In: Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2009. 1223–1226

  18. Patil A, Deshmukh C, Panat A. Feature extraction of EEG for emotion recognition using Hjorth features and higher order crossings. In: Proceedings of Conference on Advances in Signal Processing, 2016. 429–434

  19. Shi L C, Jiao Y Y, Lu B L. Differential entropy feature for EEG-based vigilance estimation. In: Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2013. 6627–6630

  20. Duan R N, Zhu J Y, Lu B L. Differential entropy feature for EEG-based emotion classification. In: Proceedings of the 6th International IEEE/EMBS Conference on Neural Engineering, 2013. 81–84

  21. Yang Y L, Wu Q F, Fu Y Z, et al. Continuous convolutional neural network with 3D input for EEG-based emotion recognition. In: Proceedings of International Conference on Neural Information Processing, 2018. 433–443

  22. Song T F, Zheng W M, Song P, et al. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput, 2020, 11: 532–541

    Article  Google Scholar 

  23. Alhagry S, Fahmy A A, El-Khoribi R A. Emotion recognition based on EEG using LSTM recurrent neural network. Emotion, 2017, 8: 355–358

    Google Scholar 

  24. Dose H, Møller J S, Iversen H K, et al. An end-to-end deep learning approach to MI-EEG signal classification for BCIs. Expert Syst Appl, 2018, 114: 532–542

    Article  Google Scholar 

  25. Fahimi F, Zhang Z, Goh W B, et al. Inter-subject transfer learning with an end-to-end deep convolutional neural network for EEG-based BCI. J Neural Eng, 2019, 16: 026007

    Article  Google Scholar 

  26. Porbadnigk A K, Görnitz N, Sannelli C, et al. When brain and behavior disagree: tackling systematic label noise in EEG data with machine learning. In: Proceedings of International Winter Workshop on Brain-Computer Interface, 2014. 1–4

  27. Fayek H M, Lech M, Cavedon L. Modeling subjectiveness in emotion recognition with deep neural networks: ensembles vs soft labels. In: Proceedings of International Joint Conference on Neural Networks, 2016. 566–570

  28. Koelstra S, Muhl C, Soleymani M, et al. DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput, 2012, 3: 18–31

    Article  Google Scholar 

  29. Chen X, Xu X Y, Liu A P, et al. Removal of muscle artifacts from the EEG: a review and recommendations. IEEE Sens J, 2019, 19: 5353–5368

    Article  Google Scholar 

  30. Tripathi S, Acharya S, Sharma R D, et al. Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence, 2017. 4746–4752

  31. Yang Y L, Wu Q F, Qiu M, et al. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. In: Proceedings of International Joint Conference on Neural Networks, 2018. 1–7

  32. Wu X, He R, Sun Z N, et al. A light CNN for deep face representation with noisy labels. IEEE Trans Inform Forensic Secur, 2018, 13: 2884–2896

    Article  Google Scholar 

  33. Jiang J J, Ma J Y, Wang Z, et al. Hyperspectral image classification in the presence of noisy labels. IEEE Trans Geosci Remote Sens, 2019, 57: 851–865

    Article  Google Scholar 

  34. Karimi D, Dou H, Warfield S K, et al. Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. Med Image Anal, 2020, 65: 101759

    Article  Google Scholar 

  35. Frenay B, Verleysen M. Classification in the presence of label noise: a survey. IEEE Trans Neural Netw Learn Syst, 2014, 25: 845–869

    Article  MATH  Google Scholar 

  36. Zhu X Q, Wu X D. Class noise vs. attribute noise: a quantitative study. Artif Intell Rev, 2004, 22: 177–210

    Article  MATH  Google Scholar 

  37. Ringeval F, Eyben F, Kroupi E, et al. Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data. Pattern Recogn Lett, 2015, 66: 22–30

    Article  Google Scholar 

  38. Zhong P X, Wang D, Miao C Y. EEG-based emotion recognition using regularized graph neural networks. IEEE Trans Affect Comput, 2020. doi: https://doi.org/10.1109/TAFFC.2020.2994159

  39. Hinton G E, Krizhevsky A, Wang S D. Transforming auto-encoders. In: Proceedings of International conference on artificial neural networks, 2011. 44–51

  40. Sabour S, Frosst N, Hinton G E. Dynamic routing between capsules. In: Proceedings of Advances in Neural Information Processing Systems, 2017. 3856–3866

  41. Liu Y, Ding Y F, Li C, et al. Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Comput Biol Med, 2020, 123: 103927

    Article  Google Scholar 

  42. Yin J H, Li S, Zhu H M, et al. Hyperspectral image classification using CapsNet with well-initialized shallow layers. IEEE Geosci Remote Sens Lett, 2019, 16: 1095–1099

    Article  Google Scholar 

  43. Turan M A T, Erzin E. Monitoring infant’s emotional cry in domestic environments using the capsule network architecture. In: Proceedings of Interspeech, 2018. 132–136

  44. Wang Y Q, Sun A X, Huang M L, et al. Aspect-level sentiment analysis using as-capsules. In: Proceedings of the World Wide Web Conference, 2019. 2033–2044

  45. Afshar P, Mohammadi A, Plataniotis K N. Brain tumor type classification via capsule networks. In: Proceedings of the 25th IEEE International Conference on Image Processing, 2018. 3129–3133

  46. Haffari G R, Sarkar A. Analysis of semi-supervised learning with the Yarowsky algorithm. 2012. ArXiv:1206.5240

  47. Lee D H. Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: Proceedings of Workshop on Challenges in Representation Learning, 2013

  48. Zhu X J. Semi-supervised learning literature survey. Computer Sci TR 1530, 2008

  49. Du C D, Du C Y, Wang H, et al. Semi-supervised deep generative modelling of incomplete multi-modality emotional data. In: Proceedings of the 26th ACM International Conference on Multimedia, 2018. 108–116

  50. Reed S, Lee H, Anguelov D, et al. Training deep neural networks on noisy labels with bootstrapping. 2014. ArXiv:1412.6596

  51. Moore A W. Cross-validation for Detecting and Preventing Overfitting. Pittsburgh: School of Computer Science Carneigie Mellon University, 2001

    Google Scholar 

  52. Suykens J A K, Vandewalle J. Least squares support vector machine classifiers. Neural Process Lett, 1999, 9: 293–300

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant Nos. 61922075, 41901350, 32150017, 62176081, 62171176), National Defense Basic Scientific Research Program of China (Grant No. JCKY2019548B001), Fundamental Research Funds for the Central Universities (Grant Nos. JZ2021HGTB0078, JZ2021HGPA0061, PA2021KCPY0051), USTC Research Funds of the Double First-Class Initiative (Grant No. KY2100000123), Provincial Natural Science Foundation of Anhui (Grant No. 2008085QF285), and Anhui Key Project of Research and Development Plan (Grant No. 202104d07020005).

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Hefei University of Technology, Hefei, 230009, China

    Chang Li, Yimeng Hou, Rencheng Song, Juan Cheng & Yu Liu

  2. Anhui Province Key Laboratory of Measuring Theory and Precision Instrument, Hefei University of Technology, Hefei, 230009, China

    Chang Li

  3. Epilepsy Center, Department of Neurosurgery, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, 230001, China

    Xun Chen

  4. School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei, 230027, China

    Xun Chen

  5. USTC IAT-Huami Joint Laboratory for Brain-Machine Intelligence, Institute of Advanced Technology, University of Science and Technology of China, Hefei, 230088, China

    Xun Chen

Authors
  1. Chang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yimeng Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rencheng Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juan Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xun Chen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, C., Hou, Y., Song, R. et al. Multi-channel EEG-based emotion recognition in the presence of noisy labels. Sci. China Inf. Sci. 65, 140405 (2022). https://doi.org/10.1007/s11432-021-3439-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-021-3439-2

Keywords

Search

Navigation