Skip to main content

Advertisement

Springer Nature Link
Log in

Emotion recognition in EEG signals using the continuous wavelet transform and CNNs

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Emotions are mental states associated with changes that influence people’s behavior, thinking, and health. Emotional changes can also appear in the organs and tissues of the human body as electrical potential differences gathered as biosignals in datasets. This work proposes the classification of emotions in electroencephalographical signals, transforming these discrete signals into a time-scale representation by spectral analysis. Our approach uses the wavelet transform to obtain scalogram images of electroencephalographic signals, treating these images as the scaled distribution of energy associated with a sign. Feature extraction from the scalograms is performed using convolutional neural networks (CNNs), leading to the proposal of two classification models. The threshold values in primitive emotions define one model of four emotions and the second of eight. The data augmentation technique increases the dataset size to compensate for the extra classes added in the second CNN model. The classification results were evaluated using different performance metrics and compared with related works in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Data availability

The dataset used in this work was acquired from https://www.eecs.qmul.ac.uk/mmv/datasets/deap/.

References

  1. Ekman P, Friesen W (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Malor Books, Katihar

    Google Scholar 

  2. Verma G, Tiwary U (2017) Affect representation and recognition in 3D continuous valence-arousal-dominance space. Multimed Tools Appl 76:2159–2183

    Article  Google Scholar 

  3. Kantz H, Kurths J, Mayer-Kress G (2012) Nonlinear analysis of physiological data. Springer Science & Business Media, Berlin

    MATH  Google Scholar 

  4. Tarnowski P, Kołodziej M, Majkowski A, Rak R (2017) Emotion recognition using facial expressions. Procedia Comput Sci 108:1175–1184. https://doi.org/10.1016/j.procs.2017.05.025

    Article  Google Scholar 

  5. Mehendale N (2020) Facial emotion recognition using convolutional neural networks (FERC). SN Appl Sci. https://doi.org/10.1007/s42452-020-2234-1

    Article  Google Scholar 

  6. Madupu R, Kothapalli C, Yarra V, Harika S & Basha C (2020) Automatic human emotion recognition system using facial expressions with convolution neural network. 2020 4th international conference on electronics, communication and aerospace technology (ICECA), pp 1179–1183

  7. Mustaqeem SK (2021) MLT-DNet: speech emotion recognition using 1D dilated CNN based on multi-learning trick approach. Expert Syst Appl 167:114117. https://doi.org/10.1016/j.eswa.2020.114177

    Article  Google Scholar 

  8. Uddin M, Nilsson E (2020) Emotion recognition using speech and neural structured learning to facilitate edge intelligence. Eng Appl Arti Intell 94:103775

    Article  Google Scholar 

  9. Wang K, Su G, Liu L, Wang S (2020) Wavelet packet analysis for speaker-independent emotion recognition. Neurocomputing 398:257–264

    Article  Google Scholar 

  10. Zhang X, Yao L, Wang X, Monaghan J, McAlpine D, Zhang Y (2021) A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers. J Neural Eng 18:031002

    Article  Google Scholar 

  11. Cai S, Li H, Wu Q, Liu J, Zhang Y (2022) Motor imagery decoding in the presence of distraction using graph sequence neural networks. IEEE Trans Neural Syst Rehabil Eng 30:1716–1726

    Article  Google Scholar 

  12. Zhang Y, Zhou T, Wu W, Xie H, Zhu H, Zhou G, Cichocki A (2022) Improving EEG decoding via clustering-based multitask feature learning. IEEE Trans Neural Netw Learn Syst 33:3587–3597

    Article  Google Scholar 

  13. Campion J, Javed A, Sartorius N, Marmot M (2020) Addressing the public mental health challenge of COVID-19. Lancet Psychiatry 7:657–659

    Article  Google Scholar 

  14. Winkler P, Formanek T, Mlada K, Kagstrom A, Mohrova Z, Mohr P, Csemy L (2019) Increase in prevalence of current mental disorders in the context of COVID-19: analysis of repeated nationwide cross-sectional surveys. Epidemiol Psychiatric Sci 29:1–8

    Google Scholar 

  15. Andalibi, N. & Buss, J (2020) The human in emotion recognition on social media: attitudes, outcomes, risks. InProceedings Of The 2020 CHI conference On human factors in computing systems

  16. Zhang Y, Zhang S, Ji X (2018) EEG-based classification of emotions using empirical mode decomposition and autoregressive model. Multimed Tools Appl 77:26697–26710

    Article  Google Scholar 

  17. Zhuang N, Zeng Y, Tong L, Zhang C, Zhang H, Yan B (2017) Emotion recognition from EEG signals using multidimensional information in EMD domain. BioMed Res Int 2017:9

    Article  Google Scholar 

  18. Lin Y, Wang C, Jung T, Wu T, Jeng S, Duann J, Chen J (2010) EEG-based emotion Recognition in music listening. IEEE Trans Biomed Eng 57:1798–1806

    Article  Google Scholar 

  19. Liu, Y. & Sourina, O (2014) EEG-based subject-dependent emotion recognition algorithm using fractal dimension. In2014 IEEE international conference on systems, man, and cybernetics (SMC)

  20. Taran S, Bajaj V (2019) Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method. Comput Methods Progr Biomed 173:157–165

    Article  Google Scholar 

  21. Koelstra S, Mühl C, Soleymani M, Lee J, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: a database for emotion analysis; Using physiological signals. IEEE Trans Affect Comput 3:18–31

    Article  Google Scholar 

  22. Yang K, Tong L, Shu J, Zhuang N, Yan B, Zeng Y (2020) High gamma band EEG closely related to emotion: evidence from functional network. Front Human Neurosci 14:89

    Article  Google Scholar 

  23. Norwood M, Lakhani A, Maujean A, Zeeman H, Creux O, Kendall E (2019) Brain activity, underlying mood and the environment: a systematic review. J Environ Psychol 65:101321

    Article  Google Scholar 

  24. Khare S, Bajaj V (2020) Time–frequency representation and convolutional neural network-based emotion recognition. IEEE Trans Neural Netw Learn Syst 32:2901–2909

    Article  Google Scholar 

  25. Gannouni S, Aledaily A, Belwafi K, Aboalsamh H (2020) Adaptive emotion detection using the valence-arousal-dominance model and EEG brain rhythmic activity changes in relevant brain lobes. IEEE Access 8:67444–67455

    Article  Google Scholar 

  26. Garg, A., Kapoor, A., Bedi, A. & Sunkaria, R (2019). Merged LSTM Model for emotion classification using EEG signals. In: international conference on data science and engineering, ICDSE 2019. pp 139–143

  27. Cimtay Y, Ekmekcioglu E (2020) Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors (Switz) 20:1–20

    Google Scholar 

  28. Garg D, Verma G (2020) Emotion recognition in valence-arousal space from multi-channel EEG data and wavelet based deep learning framework. Procedia Comput Sci 171:857–867

    Article  Google Scholar 

  29. Salama, E., El-Khoribi, R., Shoman, M. & Shalaby, M(2021) A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egypt Inform J 22:167–176

  30. Zheng W, Liu W, Lu Y, Lu B, Cichocki A. EmotionMeter (2018) A Multimodal framework for recognizing human emotions. IEEE Trans Cybern 49:1110–1122

    Article  Google Scholar 

  31. Cimtay Y, Ekmekcioglu E (2020) Loughborough University EEG based Emotion Recognition Dataset. https://www.dropbox.com/s/xlh2orv6mgweehq/LUMED_EEG.zip?dl=0

  32. Wang X, Nie D, Lu B (2011) EEG-based emotion recognition using frequency domain features and support vector machines. Lect Notes Comput Sci (Incl Subser Lect Notes Artif Intell Lect Notes Bioinform) 7062:734–743

    Google Scholar 

  33. Alazrai R, Homoud R, Alwanni H, Daoud M (2018) EEG-based emotion recognition using quadratic time-frequency distribution. Sensors (Switz) 18:1–32

    Google Scholar 

  34. Murugappan M, Ramachandran N, Sazali Y (2010) Classification of human emotion from EEG using discrete wavelet transform. J Biomed Sci Eng 3:390–396

    Article  Google Scholar 

  35. Yang J, Huang X, Wu H, Yang X (2020) EEG-based emotion classification based on bidirectional long short-term memory network. Procedia Comput Sci 174:491–504. https://doi.org/10.1016/j.procs.2020.06.117

    Article  Google Scholar 

  36. Akin M (2002) Comparison of wavelet transform and FFT methods in the analysis of EEG signals. J Med Syst 26:241–247

    Article  Google Scholar 

  37. Guo T, Wu C, Qu D (2004) Wavelet transform theory and its application progress: a review. Inf Control 33:67–71

    Google Scholar 

  38. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V & Rabinovich A (2015) Going deeper with convolutions. InProceedings of the IEEE computer society conference on computer vision and pattern recognition

  39. Türk Ö, Özerdem M (2019) Epilepsy detection by using scalogram based convolutional neural network from EEG signals. Brain Sci 9:115

    Article  Google Scholar 

  40. Lilly J, Olhede S (2010) On the analytic wavelet transform. IEEE Trans Inf Theory 56:4135–4156

    Article  MathSciNet  MATH  Google Scholar 

  41. Sakalle A, Tomar P, Bhardwaj H, Acharya D, Bhardwaj A (2021) A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system. Expert Syst Appl 173:114516

    Article  Google Scholar 

  42. Fourati R, Ammar B, Sanchez-Medina J, Alimi A (2022) Unsupervised learning in reservoir computing for EEG-based emotion recognition. IEEE Trans Affect Comput 13:972–984

    Article  Google Scholar 

  43. Nakisa B, Rastgoo M, Rakotonirainy A, Maire F, Chandran V (2018) Long short term memory hyperparameter optimization for a neural network based emotion recognition framework. IEEE Access 6:49325–49338

    Article  Google Scholar 

  44. Zheng W, Zhu J, Lu B (2019) Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput 10:417–429

    Article  Google Scholar 

  45. Xin L, Xiao-Qi S, Xiao-Ying Q & Xiao-Feng S (2016) Relevance vector machine based EEG emotion recognition. In2016 sixth international conference on instrumentation & measurement, computer, communication and control (IMCCC)

  46. Ali M, Mosa A, Machot F & Kyamakya, K (2016) EEG-based emotion recognition approach for e-healthcare applications. In2016 eighth international conference on Ubiquitous and future networks (ICUFN)

Download references

Funding

The research leading to these results was funded by scholarship No. CVU 1007303 granted by “Consejo Nacional de Ciencia y Tecnología (CONACyT, Mexico).” All the authors are grateful to the University of Guanajuato.

Author information

Authors and Affiliations

  1. Digital Signal Processing Laboratory, Electronic Engineering Department, DICIS, Universidad de Guanajuato, Carr. Salamanca-Valle de Santiago Km. 3.5 + 1.8 Km., 36885, Salamanca, Guanajuato, Mexico

    Oscar Almanza-Conejo, Dora Luz Almanza-Ojeda, Jose Luis Contreras-Hernandez & Mario Alberto Ibarra-Manzano

Authors
  1. Oscar Almanza-Conejo
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Dora Luz Almanza-Ojeda
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jose Luis Contreras-Hernandez
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Mario Alberto Ibarra-Manzano
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by OA-C, MAI-M, DLA-O and JLC-H. OA-C wrote the first draft of the manuscript, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Mario Alberto Ibarra-Manzano.

Ethics declarations

Conflict of interest

Authors have no conflict of interest/competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Almanza-Conejo, O., Almanza-Ojeda, D.L., Contreras-Hernandez, J.L. et al. Emotion recognition in EEG signals using the continuous wavelet transform and CNNs. Neural Comput & Applic 35, 1409–1422 (2023). https://doi.org/10.1007/s00521-022-07843-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07843-9

Keywords

Profiles

  1. Oscar Almanza-Conejo View author profile
  2. Mario Alberto Ibarra-Manzano View author profile