Skip to main content

Advertisement

Springer Nature Link
Log in

Multi-modal emotion identification fusing facial expression and EEG

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Aiming at solving the matter of low accuracy of emotion identification way in traditional facial expression images, this paper presents a new way of multi-modal emotion identification. Based on the modal information of facial expression, the method designs a multi-level convolutional neural network(CNN) model for facial expression emotion identification. Based on electroencephalograhpy (EEG) information modes, the method creates a stacked bidirectional LSTM(Bi-LSTM) model for emotion identification. At the decision level, D-S evidence theory is used to fuse the emotion identification results. The consequences show that the identification accuracy of multi-mode decision-level information fusion method in the two dimensions of valence and arousal in DEAP dataset reaches 95.30% and 94.94% respectively. The multi-level CNN feature extraction model proposed in this paper has a identification accuracy of 96.36% and 73.00% on CK+ dataset and Fer2013 dataset respectively. Compared with other ways, our way improves the accuracy of emotion identification

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The data used in this paper is publicly available in a public repository.

References

  1. Acharya D, Jain R, Panigrahi SS, Sahni R, Jain S, Deshmukh SP, Bhardwaj A (2021) Multi-class emotion classification using EEG signals. In: International Advanced Computing Conference, pp 474–491. https://doi.org/10.1007/978-981-16-0401-0_38

    Chapter  Google Scholar 

  2. Alchalabi B, Faubert J, Labbé D (2021) A multi-modal modified feedback self-paced BCI to control the gait of an avatar. J Neural Eng 18:056005. https://doi.org/10.1088/1741-2552/abee51

    Article  Google Scholar 

  3. Bao-Liang L, Yaqian Z, Wei-Long Z (2021) A survey of affective brain-computer interface. Chin J Intell Sci Technol 01:36–48

    Google Scholar 

  4. Bhatti UA, Huang M, Wu D, Yu Z, Mehmood A, Han H (2019) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enterp Inf Syst 13(3):329–351. https://doi.org/10.1080/17517575.2018.1557256

    Article  Google Scholar 

  5. Bishop CM (2016) Pattern recognition and machine learning (Information Science and Statistics). Springer-Verlag New York, Inc.

  6. De Nadai S et al (2016) Enhancing safety of transport by road by on-line monitoring of driver emotions. In: 2016 11th system of systems engineering conference (SoSE), pp 1–4. https://doi.org/10.1109/SYSOSE.2016.7542941

    Chapter  Google Scholar 

  7. Dempster AP (1967) Upper and lower probabilities induced by a multi-valued mapping. Ann Math Stat 38(2):325–339. https://doi.org/10.1214/aoms/1177698950

    Article  MATH  Google Scholar 

  8. Do LN, Yang HJ, Nguyen HD et al (2021) Deep neural network-based fusion model for emotion recognition using visual data. J Supercomput 77:10773–10790. https://doi.org/10.1007/s11227-021-03690-y

    Article  Google Scholar 

  9. Duan RN, Zhu JY, Lu BL (2013) Differential entropy feature for EEG-based emotion classification. In: 2013 6th international IEEE/EMBS conference on neural engineering (NER), pp 81–84. https://doi.org/10.1109/NER.2013.6695876

    Chapter  Google Scholar 

  10. Fridman L, Stolerman A, Acharya S, Brennan P, Juola P, Greenstadt R, Kam M (2015) Multi-modal decision fusion for continuous authentication. Comput Electr Eng 41:142–156. https://doi.org/10.1016/j.compeleceng.2014.10.018

    Article  Google Scholar 

  11. Guo R, Li S, He L, Gao W, Qi H, Owens G (2013) Pervasive and unobtrusive emotion sensing for human mental health. In: 2013 7th international conference on pervasive computing technologies for healthcare and workshops, pp 436–439. https://doi.org/10.4108/icst.pervasivehealth.2013.252133

    Chapter  Google Scholar 

  12. Hao C, Liang D, Yongli L et al (2019) Emotion recognition from multiband EEG signals using CapsNet. Sensors 19(9):2212. https://doi.org/10.3390/s19092212

    Article  Google Scholar 

  13. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  14. Kaggle. Fer2013 dataset. Accessed: 2018–11–10. https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data

  15. Khorrami P, Paine TL, Huang TS (2015) Do deep neural networks learn facial action units when doing expression recognition? In: 2015 IEEE international conference on computer vision workshop (ICCVW), pp 19–27. https://doi.org/10.1109/ICCVW.2015.12

    Chapter  Google Scholar 

  16. Koelstra S et al (2012) DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31. https://doi.org/10.1109/T-AFFC.2011.15

    Article  Google Scholar 

  17. Lei Z (2017) Driver's mental status assessment based on fusion of visual information and EEG. Dissertation, Northeastern University

  18. Liao JX, Zhong QH, Zhu YS et al (2020) Multimodal physiological signal emotion recognition based on convolutional recurrent neural network. IOP Conf Ser Mater Sci Eng 782:032005

    Article  Google Scholar 

  19. Liu S, Xu W, Zhao L et al (2021) 3DCANN: a Spatio-temporal convolution attention neural network for EEG emotion recognition. IEEE J Biomed Health Inform PP:1. https://doi.org/10.1109/JBHI.2021.3083525

    Article  Google Scholar 

  20. Lucey P, Cohn JF, Kanade T et al (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition - workshops, pp 94–101. https://doi.org/10.1109/CVPRW.2010.5543262

    Chapter  Google Scholar 

  21. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE winter conference on applications of computer vision (WACV), pp 1–10. https://doi.org/10.1109/WACV.2016.7477450

    Chapter  Google Scholar 

  22. Morris JD (1995) Observations SAM: the self-assessment manikin- an efficient cross-cultural measurement of emotional response. J Advert Res 35:63–68

    Google Scholar 

  23. Pusarla N, Singh A, Tripathi S (2020) Ensemble algorithms for EEG based emotion recognition. In: 2020 national conference on communications (NCC), pp 1–4. https://doi.org/10.1109/NCC48643.2020.9056002

    Chapter  Google Scholar 

  24. Qiaohong C, Zeyuan Y, Qi S, Yubo J (2020) Speech emotion identification based on attentional mechanism and LSTM. J Zhejiang Sci-Tech Univ 06:815–822

    Google Scholar 

  25. Schoneveld L, Othmani A, Abdelkawy H (2021) Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recogn Lett (2). https://doi.org/10.48550/arXiv.2103.09154

  26. Shafer G (1976) A mathematical theory of evidence. Technometrics 20(1). https://doi.org/10.1515/9780691214696

  27. Simard PY, Steinkraus D, Platt JC (2003) Best practices for convolutional neural networks applied to visual document analysis. In: 7th international conference on document analysis and identification (ICDAR 2003), pp 958–963. https://doi.org/10.1109/ICDAR.2003.1227801

    Chapter  Google Scholar 

  28. Song J (2021) Research on multi-modal emotion recognition combining face image and EEG signal. Dissertation, Taiyuan University of Technology

  29. Sunny M, Maswood MMS, Alharbi A (2020) Deep learning-based stock price prediction using LSTM and Bi-directional LSTM model. In: 2020 2nd novel intelligent and leading emerging sciences conference (NILES), pp 87–92. https://doi.org/10.1109/NILES50944.2020.9257950

    Chapter  Google Scholar 

  30. Tang H, Liu W, Zheng WL et al (2017) Multimodal emotion recognition using deep neural networks. In: International conference on neural information processing. Springer, Cham

    Google Scholar 

  31. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57:137–154. https://doi.org/10.1023/B:VISI.0000013087.49260.fb

    Article  Google Scholar 

  32. Waltz EL (1990) Multisensor data fusion. In: Artech House, Multisensor Data Fusion 1990:585–610

  33. Wang M, Huang Z, Li Y, Dong L, Pan H (2021) Maximum weight multi-modal information fusion algorithm of electroencephalographs and face images for emotion recognition. Comput Electr Eng 94:107319. https://doi.org/10.1016/j.compeleceng.2021.107319

    Article  Google Scholar 

  34. Wenfen L, Sihan C, Yong P, Wanzeng K (2021) Multi-modal physiological signal emotion identification based on 3D hierarchical convolution fusion. Chin J Intell Sci Technol 01:76–84

    Google Scholar 

  35. Yin Z, Wang Y, Liu L, Zhang W, Zhang J (2017) Cross-subject EEG feature selection for emotion recognition using transfer recursive feature elimination. Front Neurorobot 11:19. https://doi.org/10.3389/fnbot.2017.00019

    Article  Google Scholar 

  36. Yin Y, Zheng X, Hu B, Zhang Y, Cui X (2021) EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Appl Soft Comput 100:1568–4946. https://doi.org/10.1016/j.asoc.2020.106954

    Article  Google Scholar 

  37. Yongrui H, Yang J, Siyu L et al (2019) Combining facial expressions and electroencephalography to enhance emotion recognition. Future Internet 11(5):105. https://doi.org/10.3390/fi11050105

    Article  Google Scholar 

  38. Zhenyue Q, Jie W (2018) Visual saliency maps can apply to facial expression recognition. https://doi.org/10.48550/arXiv.1811.04544

  39. Ziyu D, Yuliang M, Yunyuan G, Qingshan S, Ming M, Jianhai Z (2021) A multi-scale convolutional kernel CNN for EEG emotion identification. Chin J Sens Actuators 04:496–503

    Google Scholar 

Download references

Acknowledgments

We want to thank to Dr. Jinhua Li for his guidance in the writing of this paper, who served as its scientific advisor. This work is supported by the Key Research and Development Plan-Major Scientific and Technological Innovation Projects of ShanDong Province (2019JZZY020101).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Qingdao University, Qingdao, Shandong, China

    Yongzhen Wu

  2. Qingdao Engineering Research Center of Ship-Building Intelligent Precision Measurement, Qingdao, Shandong, China

    Jinhua Li

Authors
  1. Yongzhen Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jinhua Li
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Jinhua Li.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Y., Li, J. Multi-modal emotion identification fusing facial expression and EEG. Multimed Tools Appl 82, 10901–10919 (2023). https://doi.org/10.1007/s11042-022-13711-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13711-4

Keywords