Abstract
Background/Introduction
The rapid advancement of computer technologies, along with the significant role of emotions in daily life, has driven interest in intelligent emotion recognition systems. Electroencephalography (EEG) serves as a prominent objective tool in affective computing. However, effectively integrating multichannel EEG spatial and temporal information remains a critical challenge. This study introduces a novel emotion recognition model grounded in cognitive and biological principles, emphasizing the importance of spatiotemporal dynamics in emotional processing.
Methods
In this research, brain frequency bands were extracted through wavelet analysis, and the signals within predefined time windows were quantified. These features were then concatenated across distinct brain channels to create a comprehensive matrix representing spatiotemporal brain information. The matrix was characterized using both the summation of matrix cells and the highest singular value to optimize computational costs during classification. The resulting attributes were input into a classification module for emotion detection.
Results
Experimental results on the Database for Emotion Analysis using Physiological Signals (DEAP) achieved a maximum accuracy of 89.55%.
Conclusions
This work introduces a novel approach to analyzing and classifying EEG signals elicited by various emotional stimuli, demonstrating that the proposed model is competitive with the state-of-the-art classification schemes, thereby paving the way for future development of a robust spatiotemporal-based EEG emotion recognition system.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
“The DEAP dataset analyzed in this experiment is available at http://www.eecs.qmul.ac.uk/mmv/datasets/deap/index.html and can be assessed upon approval.”
References
Al-Nafjan A, Hosny M, Al-Ohali Y, Al-Wabil A. Review and classification of emotion recognition based on EEG Brain-Computer Interface System Research: a systematic review. Appl Sci. 2017;7(12):1239. https://doi.org/10.3390/app7121239.
Alhagry S, Fahmy AA, El-Khoribi RA. Emotion Recognition based on EEG using LSTM recurrent neural network. Int J Adv Comput Sci Appl. 2017;8(10):355–8. https://doi.org/10.14569/IJACSA.2017.081046.
Arjun AS, Rajpoot MR, Panicker. Subject independent emotion recognition using EEG signals employing attention driven neural networks. Biomed Signal Proces Control. 2022;75:103547. https://doi.org/10.1016/j.bspc.2022.103547.
Ayrulu-Erdem B, Barshan B. Leg Motion classification with Artificial neural networks using Wavelet-based features of gyroscope signals. Sensors. 2011;11(2):1721–43. https://doi.org/10.3390/s110201721.
Bender P, Zakutna D, Disch S, Marcano L, Venero A, D., and, Honecker D. Using the singular value decomposition to extract 2D correlation functions from scattering patterns. Acta Crystallogr Found Adv. 2019;75:766–71. https://doi.org/10.1107/S205327331900891X.
Bhattacharyya A, Tripathy RK, Garg L, Pachori RB. A Novel Multivariate-Multiscale Approach for Computing EEG spectral and temporal complexity for human emotion recognition. IEEE Sens J. 2021;21(3):3579–91. https://doi.org/10.1109/JSEN.2020.3027181.
Bulagang AF, Weng Ng G, Mountstephens J, Teo J. A review of recent approaches for emotion classification using electrocardiography and electrodermography signals. Inf Med Unlocked. 2020;20:100363. https://doi.org/10.1016/j.imu.2020.100363.
Burle B, Spieser L, Roger C, Casini L, Hasbroucq T, Vidal F. Spatial and temporal resolutions of EEG: is it really black and white? A scalp current density view. Int J Psychophysiol. 2015;97(3):210–20. https://doi.org/10.1016/j.ijpsycho.2015.05.004.
Calvo RA, D’Mello S. Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput. 2010;1(1):18–37. https://doi.org/10.1109/T-AFFC.2010.1.
Cattani C. Haar wavelets based technique in evolution problems. Proc. Estonian Acad. Sci. Phys. Math. 2004; 53(1): 45–63. https://doi.org/10.3176/phys.math.2004.1.04
Chai MT, Amin HU, Izhar LI, Saad MNM, Abdul Rahman M, Malik AS, Tang TB. Exploring EEG Effective Connectivity Network in estimating influence of color on emotion and memory. Front Neuroinform. 2019;13:66. https://doi.org/10.3389/fninf.2019.00066.
Cui G, Li X, Touyama H. Emotion recognition based on group phase locking value using convolutional neural network. Sci Rep. 2023;13:3769. https://doi.org/10.1038/s41598-023-30458-6.
Dixon ML. Cognitive control, emotional value, and the lateral prefrontal cortex. Front Psychol. 2015;6:758. https://doi.org/10.3389/fpsyg.2015.00758.
Dremin IM, Ivanov OV, Nechitailo VA. Wavelets and their uses. Phys Usp. 2001;44(5):447–78. https://doi.org/10.1070/PU2001v044n05ABEH000918.
Egger M, Ley M, Hanke S. Emotion Recognition from Physiological Signal Analysis: a review. Electro Notes Theor Comput Sci. 2019;343:35–55. https://doi.org/10.1016/j.entcs.2019.04.009.
Feng H, Golshan HM, Mahoor MH. A wavelet-based approach to emotion classification using EDA signals. Expert Syst Appl. 2018;112:77–86. https://doi.org/10.1016/j.eswa.2018.06.014.
Gao X, Huang W, Liu Y, Zhang Y, Zhang J, Li C, et al. A novel robust student’s t-based Granger causality for EEG based brain network analysis. Biomed Signal Process Control. 2023;80(1):104321. https://doi.org/10.1016/j.bspc.2022.104321.
Ghodousi M, Pousson JE, Voicikas A, Bernhofs V, Pipinis E, Tarailis P, Burmistrova L, Lin Y-P. Griškova-Bulanova, I. EEG Connectivity during active emotional musical performance. Sensors. 2022;22:4064. https://doi.org/10.3390/s22114064.
Ghosh D, Sengupta R, Sanyal S, Banerjee A. Emotions from Hindustani classical music: an EEG based study including neural hysteresis. Musicality of human brain through fractal analytics. Singapore: Springer; 2018. pp. 49–72. https://doi.org/10.1007/978-981-10-6511-8_3.
Ginebreda A, Sabater-Liesa L, Barcelo D. Quantification of ecological complexity and resilience from multivariate biological metrics datasets using singular value decomposition entropy. MethodsX. 2019;6:1668–76. https://doi.org/10.1016/j.mex.2019.07.020.
Gonuguntla V, Kim JH. EEG-Based functional connectivity representation using phase locking value for Brain Network based applications. Annu Int Conf IEEE Eng Med Biol Soc. 2020;2020:2853–6. https://doi.org/10.1109/EMBC44109.2020.9175397.
Goshvarpour A, Goshvarpour A. A novel feature level fusion for HRV classification using correntropy and Cauchy-Schwarz divergence. J Med Syst. 2018a;42:109. https://doi.org/10.1007/s10916-018-0961-2.
Goshvarpour A, Goshvarpour A. Poincaré’s section analysis for PPG-based automatic emotion recognition. Chaos Soliton Fract. 2018b;114:400–7. https://doi.org/10.1016/j.chaos.2018.07.035.
Goshvarpour A, Goshvarpour A. EEG spectral powers and source localization in depressing, sad, and fun music videos focusing on gender differences. Cogn Neurodyn. 2019;13(2):161–73. https://doi.org/10.1007/s11571-018-9516-y.
Goshvarpour A, Goshvarpour A. A novel approach for EEG electrode selection in automated emotion recognition based on lagged Poincare’s indices and sLORETA. Cogn Comput. 2020a;12:602–18. https://doi.org/10.1007/s12559-019-09699-z.
Goshvarpour A, Goshvarpour A. Evaluation of novel entropy-based complex wavelet sub-bands measures of PPG in an emotion recognition system. J Med Biol Eng. 2020b;40:451–61. https://doi.org/10.1007/s40846-020-00526-7.
Goshvarpour A, Goshvarpour A. The potential of photoplethysmogram and galvanic skin response in emotion recognition using nonlinear features. Phys Eng Sci Med. 2020c;43:119–34. https://doi.org/10.1007/s13246-019-00825-7.
Goshvarpour A, Goshvarpour A. (2022a) Novel high-dimensional phase space features for EEG emotion recognition. SIViP. https://doi.org/10.1007/s11760-022-02248-6
Goshvarpour A, Goshvarpour A. Innovative Poincare’s plot asymmetry descriptors for EEG emotion recognition. Cogn Neurodyn. 2022b;16:545–59. https://doi.org/10.1007/s11571-021-09735-5.
Goshvarpour A, Goshvarpour A. Lemniscate of Bernoulli’s map quantifiers: innovative measures for EEG emotion recognition. Cogn Neurodyn. 2023b. https://doi.org/10.1007/s11571-023-09968-6.
Goshvarpour A, Goshvarpour A. Emotion Recognition using a Novel Granger Causality Quantifier and Combined electrodes of EEG. Brain Sci. 2023;13(5):759. https://doi.org/10.3390/brainsci13050759.
Guillemot V, Beaton D, Gloaguen A, Löfstedt T, Levine B, Raymond N, et al. A constrained singular value decomposition method that integrates sparsity and orthogonality. PLoS ONE. 2019;14:e0211463. https://doi.org/10.1371/journal.pone.0211463.
Guo T, Zhang T, Lim E, López-Benítez M, Ma F, Yu L. A review of Wavelet Analysis and its applications: challenges and opportunities. IEEE Access. 2022;10:58869–903. https://doi.org/10.1109/ACCESS.2022.3179517.
Guzel Aydin S, Kaya T, Guler H. Wavelet-based study of valence-arousal model of emotions on EEG signals with Lab-VIEW. Brain Inf. 2016;3(2):109–17. https://doi.org/10.1007/s40708-016-0031-9.
Haddad AE, Najafizadeh L, Society. (EMBC), (Milan: IEEE), 558–561. https://doi.org/10.1109/EMBC.2015.7318423
Hjorth B. EEG analysis based on time domain properties. Electroencephalogr Clin Neurophysiol. 1970. https://doi.org/10.1016/0013-4694(70)90143-4.
Hou HR, Zhang XN, Meng QH. Odor-induced emotion recognition based on average frequency band division of EEG signals. J Neurosci Methods. 2020;334:108599. https://doi.org/10.1016/j.jneumeth.2020.108599.
Houssein EH, Hammad A, Ali AA. Human emotion recognition from EEG-based brain–computer interface using machine learning: a comprehensive review. Neural Comput Applic. 2022;34:12527–57. https://doi.org/10.1007/s00521-022-07292-4.
Hu W, Zhang Z, Zhao H, et al. EEG microstate correlates of emotion dynamics and stimulation content during video watching. Cereb Cortex. 2023;33(3):523–42. https://doi.org/10.1093/cercor/bhac082.
Jenke R, Peer A, Buss M. Feature extraction and selection for emotion recognition from EEG. IEEE Trans Affect Comput. 2014;5(3):327–39. https://doi.org/10.1109/TAFFC.2014.2339834.
Kang Q, Li F, Gao J. Exploring the functional Brain Network of Deception in source-level EEG via partial mutual information. Electronics. 2023;12(7):1633. https://doi.org/10.3390/electronics12071633.
Khalili Z, Moradi MH. (2009) Emotion recognition system using brain and peripheral signals: using correlation dimension to improve the results of EEG, in Proceedings of the 2009 International Joint Conference on Neural Networks (New York, NY: IEEE Press), 1571–1575. https://doi.org/10.1109/IJCNN.2009.5178854
Khazaei E, Mohammadzade H. Temporal analysis of functional brain connectivity for eeg-based emotion recognition. arXiv Preprint arXiv. 2021;211212380. https://doi.org/10.48550/arXiv.2112.12380.
Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput. 2012;3(1):18–31. https://doi.org/10.1109/T-AFFC.2011.15.
Lan Z, Sourina O, Wang L, Scherer R, Müller-Putz GR. Domain adaptation techniques for EEG-based emotion recognition: a comparative study on two public datasets. IEEE Trans Cogn Dev Syst. 2019;11(1):85–94. https://doi.org/10.1109/TCDS.2018.2826840.
Li X, Song D, Zhang P, Zhang Y, Hou Y, Hu B. Exploring EEG features in cross-subject emotion recognition. Front NeuroSci. 2018;12:294333. https://doi.org/10.3389/fnins.2018.00162.
Li W, Zhang Z, Song A. Physiological-signal-based emotion recognition: an odyssey from methodology to philosophy. Measurement. 2021;172:108747. https://doi.org/10.1016/j.measurement.2020.108747.
Liang Z, Zhou R, Zhang L, Li L, Huang G, Zhang Z, Ishii S. EEGFuseNet: Hybrid Unsupervised Deep feature characterization and Fusion for High-Dimensional EEG with an application to emotion recognition. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2021;29:1913–25. https://doi.org/10.1109/TNSRE.2021.3111689.
Lin O, Liu G-Y, Yang J-M, Du Y-Z. (2015) Neurophysiological markers of identifying regret by 64 channels EEG signal. 12th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), 18–20 Dec. 2015, Chengdu, China, 395–399. https://doi.org/10.1109/ICCWAMTIP.2015.7494017
Lindfield G, Penny J. Chapter 8 - Analyzing Data Using Discrete Transforms, in Numerical Methods (Fourth Edition). Academic Press. 2019; pp. 383–431. https://doi.org/10.1016/B978-0-12-812256-3.00017-8
Maffei A, Angrilli A. Spontaneous blink rate as an index of attention and emotion during film clips viewing. Physiol Behav. 2019;204:256–63. https://doi.org/10.1016/j.physbeh.2019.02.037.
Mirabella G. The weight of emotions in Decision-Making: how fearful and happy facial Stimuli modulate Action readiness of goal-Directed actions. Front Psychol. 2018;9:1334. https://doi.org/10.3389/fpsyg.2018.01334.
Mitchell J. Emotion and attention. Philos Stud. 2023;180:73–99. https://doi.org/10.1007/s11098-022-01876-5.
Murias M, Swanson JM, Srinivasan R. Functional connectivity of frontal cortex in healthy and ADHD children reflected in EEG coherence. Cereb Cortex. 2007;17(8):1788–99. https://doi.org/10.1093/cercor/bhl089.
Murugappan M, Zheng BS, Khairunizam W. Recurrent quantification analysis-based emotion classification in Stroke using Electroencephalogram signals. Arab J Sci Eng. 2021;46:9573–88. https://doi.org/10.1007/s13369-021-05369-1.
Naser DS, Saha G. Influence of music liking on EEG based emotion recognition. Biomed Signal Process Control. 2021;64:102251. https://doi.org/10.1016/j.bspc.2020.102251.
Nawaz R, Hwa Cheah K, Nisar H, Yap VV. Comparison of different feature extraction methods for EEG-based emotion recognition. Biocybern Biomed Eng. 2020;40(3):910–26. https://doi.org/10.1016/j.bbe.2020.04.005.
Nigam HK, Srivastava HM. Filtering of Audio signals using Discrete Wavelet transforms. Mathematics. 2023;11(19):4117. https://doi.org/10.3390/math11194117.
Özerdem MS, Polat H. Emotion recognition based on EEG features in movie clips with channel selection. Brain Inf. 2017;4(4):241–52. https://doi.org/10.1007/s40708-017-0069-3.
Pandey P, Seeja K. Subject independent emotion recognition system for people with facial deformity: an EEG based approach. J Amb Intel Hum Comp. 2021;12(2):2311–20. https://doi.org/10.1007/s12652-020-02338-8.
Pandey P, Seeja K. Subject independent emotion recognition from EEG using VMD and deep learning. J King Saud Univ-Comput Inf Sci. 2022;34(5):1730–8. https://doi.org/10.1016/j.jksuci.2019.11.003.
Pane ES, Wibawa AD, Purnomo MH. Improving the accuracy of EEG emotion recognition by combining valence lateralization and ensemble learning with tuning parameters. Cogn Process. 2019;20(4):405–17. https://doi.org/10.1007/s10339-019-00924-z.
Patel PR, Annavarapu RN. EEG-based human emotion recognition using entropy as a feature extraction measure. Brain Inf. 2021;8:20. https://doi.org/10.1186/s40708-021-00141-5.
Prasad R, Tarai S, Bit A. Investigation of frequency components embedded in EEG recordings underlying neuronal mechanism of cognitive control and attentional functions. Cogn Neurodyn. 2022. https://doi.org/10.1007/s11571-022-09888-x.
Rafiee J, Rafiee MA, Prause N, Schoen MP. Wavelet basis functions in biomedical signal processing. Expert Syst Appl. 2011;38(5):6190–201. https://doi.org/10.1016/j.eswa.2010.11.050.
Rayatdoost S, Soleymani M, IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP). Cross-corpus EEG-based emotion recognition. 2018 (2018), Aalborg, Denmark, pp. 1–6. https://doi.org/10.1109/MLSP.2018.8517037
Salama ES, El-Khoribi RA, Shoman ME, Shalaby MAW. EEG-based emotion recognition using 3D convolutional neural networks. Int J Adv Comput Sci Appl. 2018;9(8):329–37. https://doi.org/10.14569/IJACSA.2018.090843.
Salankar N, Mishra P, Garg L. Emotion recognition from EEG signals using empirical mode decomposition and second-order difference plot. Biomed Signal Process Control. 2021;65:102389. https://doi.org/10.1016/j.bspc.2020.102389.
Sanyal S, Banerjee A, Basu M, Nag S, Ghosh D, Karmakar S. Do musical notes correlate with emotions? A neuro-acoustical study with Indian classical music. Proc Mtgs Acoust. 2020;42(1):035005. https://doi.org/10.1121/2.0001397.
Shahid A, Kamel N, Malik AS. (2014). Singular values as a detector of epileptic seizures in EEG signals, in Proceedings of the 2014 5th International Conference on Intelligent and Advanced Systems (ICIAS), (Kuala Lumpur: IEEE), 1–5. https://doi.org/10.1109/ICIAS.2014.6869459
Sheng W, Li X. Multi-task learning for Gait-based Identity Recognition and emotion recognition using attention enhanced temporal graph Convolutional Network. Pattern Recognit. 2021;114:107868. https://doi.org/10.1016/j.patcog.2021.107868.
Siddharth, Jung TP, Sejnowski TJ. Utilizing Deep Learning towards Multi-modal Bio-sensing and Vision-Based Affective Computing. IEEE Trans Affect Comput. 2022;13(1):96–107. https://doi.org/10.1109/TAFFC.2019.2916015.
Silva R, Salvador G, Bota P, Fred A, da Silva HP. Impact of sampling rate and interpolation on photoplethysmography and electrodermal activity signals’ waveform morphology and feature extraction. Neural Comput Applic. 2022. https://doi.org/10.1007/s00521-022-07212-6.
Tarai S, Bit A. (2021). Neurocognitive Perspectives of Prosocial and Positive Emotional Behaviours: Theory to Application. IOP Publishing. https://doi.org/10.1088/978-0-7503-3383-2
Tarai S, Qurratul QA, Ratre V, et al. Neurocognitive functions of prosocial and unsocial incongruency information during language comprehension: evidence from time–frequency analysis of EEG signals. Med Biol Eng Comput. 2022;60:1033–53. https://doi.org/10.1007/s11517-022-02528-w.
Tenke CE, Kayser J. Generator localization by current source density (CSD): implications of volume conduction and field closure at intracranial and scalp resolutions. Clin Neurophysiol. 2012;123(12):2328–45. https://doi.org/10.1016/j.clinph.2012.06.005.
Tuncer T, Dogan S, Subasi A. A new fractal pattern feature generation function based emotion recognition method using EEG. Chaos Soliton Fract. 2021;144:110671. https://doi.org/10.1016/j.chaos.2021.110671.
van Berkum JJA. (2019) Language Comprehension and Emotion: Where Are the Interfaces, and Who Cares? in Greig I. de Zubicaray, and Niels O. Schiller, editors, The Oxford Handbook of Neurolinguistics, Oxford Handbooks (2019; online edn, Oxford Academic, 14 Mar. 2019), https://doi.org/10.1093/oxfordhb/9780190672027.013.29
Wang X, Chen X, Cao C. Human emotion recognition by optimally fusing facial expression and speech feature. Signal Process Image Commun. 2020;84:115831. https://doi.org/10.1016/j.image.2020.115831.
Wang X, Wanniarachchi H, Wu A, Liu H. Combination of Group Singular Value Decomposition and eLORETA identifies human EEG networks and responses to Transcranial Photobiomodulation. Front Hum Neurosci. 2022;16:853909. https://doi.org/10.3389/fnhum.2022.853909.
Yao Q, Gu H, Wang S, Li X. A feature-fused Convolutional Neural Network for Emotion Recognition from Multichannel EEG signals. IEEE Sens J. 2022;22(12):11954–64. https://doi.org/10.1109/JSEN.2022.3172133.
Yin Z, Liu L, Chen J, Zhao B, Wang Y. Locally robust EEG feature selection for individual-independent emotion recognition. Expert Syst Appl. 2020;162:113768. https://doi.org/10.1016/j.eswa.2020.113768.
Zhang J, Zhang X, Chen G, Huang L, Sun Y. EEG emotion recognition based on cross-frequency granger causality feature extraction and fusion in the left and right hemispheres. Front Neurosci. 2022;16:974673. https://doi.org/10.3389/fnins.2022.974673.
Zheng WL, Lu BL. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev. 2015;7:162–75. https://doi.org/10.1109/TAMD.2015.2431497.
Zheng WL, Zhu JY, Lu BL. Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput. 2016;10(3):417–29. https://doi.org/10.1109/TAFFC.2017.2712143.
Funding
“This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.”
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
“The author declares no conflict of interest.”
Ethical Approval
“This article examined EEG signals of the DEAP (Koelstra et al. 2012), freely available in the public domain. This article contains no studies with human participants performed by any authors.”
Informed Consent
“Not applicable.”
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Goshvarpour, A., Goshvarpour, A. Cognitive-Inspired Spectral Spatiotemporal Analysis for Emotion Recognition Utilizing Electroencephalography Signals. Cogn Comput 17, 2 (2025). https://doi.org/10.1007/s12559-024-10361-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s12559-024-10361-6