Skip to main content
SpringerLink
Log in

CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings

  • 1215: Multimodal Interaction and IoT Applications
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Emotion is a significant parameter in daily life and is considered an important factor for human interactions. The human-machine interactions and their advanced stages like humanoid robots essentially require emotional investigation. This paper proposes a novel method for human emotion recognition using electroencephalogram (EEG) signals. We have considered three emotions namely neutral, positive, and negative. These EEG signals are separated into five frequency bands according to EEG rhythms and the differential entropy is computed over the different frequency band components. The convolution neural network (CNN) and long short-term memory (LSTM) based hybrid model is developed for accurate emotion detection. Further, the extracted features are fed to all three models for emotion recognition. Finally, an ensemble model combines the predictions of all three models. The proposed approach is validated on two datasets namely SEED and DEAP for EEG based emotion analysis. The developed method achieved 97.16% accuracy on SEED dataset for emotion classification. The experimental results indicate that the proposed approach is effective and yields better performance than the compared methods for EEG-based emotion analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Agarap AF (2019) Deep learning using rectified linear units (relu)

  2. Anastassiou GA (2011) Multivariate hyperbolic tangent neural network approximation. Comput Math Applic 61(4):809–821

    Article  MATH  Google Scholar 

  3. Bos DO, et al. (2006) Eeg-based emotion recognition. The Influence of Visual and Auditory Stimuli 56(3):1–17

    Google Scholar 

  4. Chen Z, Cao F, Hu J (2015) Approximation by network operators with logistic activation functions. Appl Math Comput 256:565–571

    MATH  Google Scholar 

  5. Cheng B, Liu G (2008) Emotion recognition from surface emg signal using wavelet transform and neural network. In: Proceedings of the 2nd international conference on bioinformatics and biomedical engineering (ICBBE), pp 1363–1366

  6. Cui H, Liu A, Zhang X, Chen X, Wang K, Chen X (2020) Eeg-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl-Based Syst 205:106243

    Article  Google Scholar 

  7. Dash M, Liu H (1997) Feature selection for classification. Intell Data Anal 1(1–4):131–156

    Article  Google Scholar 

  8. Duan RN, Zhu J, Lu B (2013) Differential entropy feature for EEG-based emotion classification. In: 6th International IEEE/EMBS conference on neural engineering (NER), pp 81–84. IEEE

  9. Glowinski D, Camurri A, Volpe G, Dael N, Scherer K (2008) Technique for automatic emotion recognition by body gesture analysis. In: 2008 IEEE Computer society conference on computer vision and pattern recognition workshops, pp 1–6. IEEE

  10. Greff K, Srivastava RK, Koutnik J, Steunebrink BR, Schmidhuber J (2017) Lstm: a search space odyssey. IEEE Trans Neural Netw Learn Syst 28 (10):2222–2232

    Article  Google Scholar 

  11. Grozea C, Voinescu CD, Fazli S (2011) Bristle-sensors—low-cost flexible passive dry eeg electrodes for neurofeedback and bci applications. J Neur Eng 8(2):025008

    Article  Google Scholar 

  12. Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J, Chen T (2018) Recent advances in convolutional neural networks. Pattern Recogn 77:354–377

    Article  Google Scholar 

  13. Huang YJ, Wu CY, Wong AMK, Lin BS (2014) Novel active comb-shaped dry electrode for eeg measurement in hairy site. IEEE Trans Biomed Eng 62(1):256–263

    Article  Google Scholar 

  14. Hwang S, Hong K, Son G, Byun H (2020) Learning cnn features from de features for eeg-based emotion recognition. Pattern Anal Applic 23 (3):1323–1335

    Article  Google Scholar 

  15. Kurbalija V, Ivanović M, Radovanović M, Geler Z, Dai W, Zhao W (2018) Emotion perception and recognition: an exploration of cultural differences and similarities. Cogn Syst Res 52:103–116

    Article  Google Scholar 

  16. Lan Z, Sourina O, Wang L, Scherer R, Müller-Putz GR (2019) Domain adaptation techniques for eeg-based emotion recognition: a comparative study on two public datasets. IEEE Trans Cogn Develop Syst 11(1):85–94

    Article  Google Scholar 

  17. Li H, Jin YM, Zheng W, Lu B (2018) Cross-subject emotion recognition using deep adaptation networks. In: Neural information processing, pp 403–413. Springer International Publishing

  18. Li M, Lu B (2009) Emotion classification based on gamma-band eeg. In: 2009 Annual international conference of the IEEE engineering in medicine and biology society, pp 1223–1226. IEEE

  19. Li P, Liu H, Si Y, Li C, Li F, Zhu X, Huang X, Zeng Y, Yao D, Zhang Y, Xu P (2019) Eeg based emotion recognition by combining functional connectivity network and local activations. IEEE Trans Biomed Eng 66 (10):2869–2881

    Article  Google Scholar 

  20. Li X, Song D, Zhang P, Zhang Y, Hou Y, Hu B (2018) Exploring eeg features in cross-subject emotion recognition. Front Neurosci 12:162

    Article  Google Scholar 

  21. Li Y, Zheng W, Zong Y, Cui Z, Zhang T, Zhou X (2018) A bi-hemisphere domain adversarial neural network model for eeg emotion recognition. IEEE Transactions on Affective Computing

  22. Liu J, Wu G, Luo Y, Qiu S, Yang S, Li W, Bi Y (2020) Eeg-based emotion classification using a deep neural network and sparse autoencoder. Front Syst Neurosci 14:43

    Article  Google Scholar 

  23. Liu NH, Chiang CY, Hsu HM (2013) Improving driver alertness through music selection using a mobile eeg to detect brainwaves. Sensors 13(7):8199–8221

    Article  Google Scholar 

  24. Liu W, Wen Y, Yu Z, Yang M (2017) Large-margin softmax loss for convolutional neural networks

  25. Liu W, Zheng W, Lu BL (2016) Multimodal emotion recognition using multimodal deep learning

  26. Manli S, Song Z, Jiang X, Pan J, Pang Y (2016) Learning pooling for convolutional neural network. Neurocomputing, 224

  27. Mariooryad S, Busso C (2014) Compensating for speaker or lexical variabilities in speech for emotion recognition. Speech Comm 57:1–12

    Article  Google Scholar 

  28. Mathersul D, Williams LM, Hopkinson PJ, Kemp AH (2008) Investigating models of affect: relationships among eeg alpha asymmetry, depression, and anxiety. Emotion 8(4):560

    Article  Google Scholar 

  29. Park S, Kwak N (2017) Analysis on the dropout effect in convolutional neural networks, pp 189–204

  30. Piana S, Staglianò A, Odone F, Camurri A (2016) Adaptive body gesture representation for automatic emotion recognition. ACM Trans Interact Intell Syst (TiiS) 6(1):1–31

    Article  Google Scholar 

  31. Qiu JL, Liu W, Lu B (2018) Multi-view emotion recognition using deep canonical correlation analysis. In: Neural information processing, pp 221–231. Springer International Publishing

  32. Rodriguez JD, Perez A, Lozano JA (2010) Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Trans Pattern Anal Mach Intell 32(3):569–575

    Article  Google Scholar 

  33. Sammler D, Grigutsch M, Fritz T, Koelsch S (2007) Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music. Psychophysiology 44(2):293–304

    Article  Google Scholar 

  34. Sarkar P, Etemad A (2020) Self-supervised ecg representation learning for emotion recognition. IEEE Transactions on Affective Computing

  35. Sauvet F, Bougard C, Coroenne M, Lely L, Van Beers P, Elbaz M, Guillard M, Leger D, Chennaoui M (2014) In-flight automatic detection of vigilance states using a single eeg channel. IEEE Trans Biomed Eng 61 (12):2840–2847

    Article  Google Scholar 

  36. Sharma R, Sahu SS, Upadhyay A, Sharma RR, Sahoo AK (2021) Sleep stage classification using DWT and dispersion entropy applied on EEG signals. In: Computer-aided design and diagnosis methods for biomedical applications, pp 35–56. CRC Press

  37. Sharma RR, Pachori RB (2017) Time-frequency representation using IEVDHM-HT with application to classification of epileptic EEG signals. IET Sci Measur Technol 12(1):72–82

    Article  Google Scholar 

  38. Sharma S, Sharma RR (2022) Variational mode decomposition based finger flexion movement detection using ECoG signals. In: Artificial intelligence-based brain-computer interface, pp 101–119. Elsevier

  39. Shu L, Xie J, Yang M, Li Z, Li Z, Liao D, Xu X, Yang X (2018) A review of emotion recognition using physiological signals. Sensors 18 (7):2074

    Article  Google Scholar 

  40. Song T, Zheng W, Song P, Cui Z (2020) Eeg emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11(3):532–541

    Article  Google Scholar 

  41. Subasi A, Gursoy MI (2010) Eeg signal classification using pca, ica, lda and support vector machines. Exp Syst Applic 37(12):8659–8666

    Article  Google Scholar 

  42. Tabar YR, Halici U (2016) A novel deep learning approach for classification of EEG motor imagery signals. J Neur Eng 14(1):016003

    Article  Google Scholar 

  43. Tahir MA, Kittler J, Bouridane A (2012) Multilabel classification using heterogeneous ensemble of multi-label classifiers. Pattern Recogn Lett 33 (5):513–523

    Article  Google Scholar 

  44. Tang H, Liu W, Zheng W, Lu B (2017) Multimodal emotion recognition using deep neural networks. In: Neural information processing, pp 811–819. Springer International Publishing

  45. Teuwen J, Moriakov N (2020) Chapter 20 - convolutional neural networks. In: Handbook of medical image computing and computer assisted intervention, the Elsevier and MICCAI society book series, pp 481–501. Academic Press

  46. Tripathi S, Acharya S, Sharma RD, Mittal S, Bhattacharya S (2017) Using deep and convolutional neural networks for accurate emotion classification on deap dataset. In: Proceedings of the thirty-first AAAI conference on artificial intelligence, pp 4746–4752. AAAI Press

  47. van Laarhoven T (2017) L2 regularization versus batch and weight normalization

  48. Wang YX, Ramanan D, Hebert M (2019) Meta-learning to detect rare objects. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV)

  49. Wang Z, Tong Y, Heng X (2019) Phase-locking value based graph convolutional neural networks for emotion recognition. IEEE Access 7:93711–93722

    Article  Google Scholar 

  50. Webb GI, Zheng Z (2004) Multistrategy ensemble learning: reducing error by combining ensemble learning techniques. IEEE Trans Knowl Data Eng 16 (8):980–991

    Article  Google Scholar 

  51. Young AW, Rowland D, Calder AJ, Etcoff NL, Seth A, Perrett DI (1997) Facial expression megamix: tests of dimensional and category accounts of emotion recognition. Cognition 63(3):271–313

    Article  Google Scholar 

  52. Zhang D, Yao L, Zhang X, Wang S, Chen W, Boots R, Benatallah B (2018) Cascade and parallel convolutional recurrent neural networks on eeg-based intention recognition for brain computer interface. In: Proceedings of the thirty-second AAAI conference on artificial intelligence, pp 1703–1710. AAAI Press

  53. Zheng W, Zhu J, Peng Y, Lu B (2014) EEG-based emotion classification using deep belief networks. In: 2014 IEEE International conference on multimedia and expo (ICME), pp 1–6

  54. Zheng W, Lu B (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7(3):162–175

    Article  Google Scholar 

  55. Zheng W, Zhu J, Lu B (2017) Identifying stable patterns over time for emotion recognition from eeg. IEEE Trans Affect Comput 10(3):417–429

    Article  Google Scholar 

  56. Zhong P, Wang D, Miao C (2020) EEG-based emotion recognition using regularized graph neural networks. IEEE Transactions on Affective Computing

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Electronics Engineering, Birla Institute of Technology and Science, Pilani, 333031, India

    Abhishek Iyer, Srimit Sritik Das & Reva Teotia

  2. School of Electronics Engineering (SENSE), Vellore Institute of Technology (VIT), Chennai, 600127, India

    Shishir Maheshwari

  3. Department of Electronics Engineering, Defence Institute of Advanced Technology, Pune, 411025, India

    Rishi Raj Sharma

Authors
  1. Abhishek Iyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Srimit Sritik Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Reva Teotia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shishir Maheshwari
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rishi Raj Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rishi Raj Sharma.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Iyer, A., Das, S.S., Teotia, R. et al. CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings. Multimed Tools Appl 82, 4883–4896 (2023). https://doi.org/10.1007/s11042-022-12310-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12310-7

Keywords

Search

Navigation