Abstract
Emotion recognition using electroencephalography (EEG) is becoming an interesting topic among researchers. It has made a remarkable entry in the domain of biomedical, smart environment, brain-computer interface (BCI), communication, security, and safe driving. In the past decade, several studies have been published that viewed emotion recognition tasks in a variety of manners. Multiple algorithms have been developed to accurately capture the EEG signal and identify the emotions from such EEG signals. The advent of artificial intelligence (AI) has changed the landscape of every application including emotion recognition. Two categories of AI-based algorithms such as machine learning and deep learning algorithms are becoming popular in the emotion recognition domain. This narrative review is an attempt to provide deep insight into the AI-based techniques, their role in EEG-based emotion recognition, and their potential future possibilities in accurate emotion identification. Furthermore, this review also provides an overview of the several important topics in emotion recognition such as emotion paradigms, EEG and its processing, and the public databases.
Similar content being viewed by others
References
Abhang PA, Gawali B, Mehrotra SC (2016) Introduction to EEG-and speech-based emotion recognition. Academic Press
Alhagry S, Fahmy AA, El-Khoribi RA (2017) Emotion recognition based on EEG using LSTM recurrent neural network. Emotion 8:355–358
Alickovic E, Babic Z (2015) The effect of denoising on classification of ECG signals. In: 2015 XXV International Conference on Information, Communication and Automation Technologies (ICAT), pp. 1–6
Bajaj V, Taran S, Sengur A (2018) Emotion classification using flexible analytic wavelet transform for electroencephalogram signals. Health Inf Sci Syst 6:12
Bălan O, Moise G, Petrescu L, Moldoveanu A, Leordeanu M, Moldoveanu F (2020) Emotion classification based on biophysical signals and machine learning techniques. Symmetry 12:21
Bhattacharyya A, Pachori RB (2017) A multivariate approach for patient-specific EEG seizure detection using empirical wavelet transform. IEEE Trans Biomed Eng 64:2003–2015
Bhattacharyya A, Singh L, Pachori RB (2018) Fourier–Bessel series expansion based empirical wavelet transform for analysis of non-stationary signals. Digit Signal Process 78:185–196
Bhattacharyya A, Sharma M, Pachori RB, Sircar P, Acharya UR (2018) A novel approach for automated detection of focal EEG signals using empirical wavelet transform. Neural Comput & Applic 29:47–57
Bhattacharyya A, Tripathy RK, Garg L, Pachori RB (2020) A novel multivariate-multiscale approach for computing EEG spectral and temporal complexity for human emotion recognition. IEEE Sensors J 21:3579–3591
Bigirimana AD, Siddique N, Coyle D (2016) A hybrid ICA-wavelet transform for automated artefact removal in EEG-based emotion recognition. In: 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 004429–004434
Chen T, He T, Benesty M, Khotilovich V, Tang Y, Cho H (2015) Xgboost: extreme gradient boosting. R package version 0.4–2, vol. 1, pp. 1–4
Chen J, Zhang P, Mao Z, Huang Y, Jiang D, Zhang Y (2019) Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks. IEEE Access 7:44317–44328
Cheng Y, Liu G-Y, Zhang H-L (2010) The research of EMG signal in emotion recognition based on TS and SBS algorithm. In: The 3rd International Conference on Information Sciences and Interaction Sciences, pp. 363–366
Cheng J, Chen M, Li C, Liu Y, Song R, Liu A et al (2020) Emotion recognition from multi-channel eeg via deep forest. IEEE J Biomed Health Inform 25:453–464
Cui H, Liu A, Zhang X, Chen X, Wang K, Chen X (2020) EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl-Based Syst 205:106243
Demir F, Sobahi N, Siuly S, Sengur A (2021) Exploring deep learning features for automatic classification of human emotion using EEG rhythms. IEEE Sensors J 21(13):14923–14930
Ekman P (1992) An argument for basic emotions. Cognit Emot 6:169–200
Ekman P, Friesen WV (2003) Unmasking the face: A guide to recognizing emotions from facial clues, vol 10. Ishk
Frantzidis CA, Bratsas C, Papadelis CL, Konstantinidis E, Pappas C, Bamidis PD (2010) Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli. IEEE Trans Inf Technol Biomed 14:589–597
Gao Z, Li Y, Yang Y, Wang X, Dong N, Chiang H-D (2020) A GPSO-optimized convolutional neural networks for EEG-based emotion recognition. Neurocomputing 380:225–235
García HF, Álvarez MA, Orozco ÁA (2016) Gaussian process dynamical models for multimodal affect recognition. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 850–853
Garg A, Chaturvedi V, Kaur AB, Varshney V, Parashar A (2022) Machine learning model for mapping of music mood and human emotion based on physiological signals. Multimed Tools Appl:1–41
Gicić A, Subasi A (2019) Credit scoring for a microcredit data set using the synthetic minority oversampling technique and ensemble classifiers. Expert Syst 36:e12363
Grühn D, Scheibe S (2008) Age-related differences in valence and arousal ratings of pictures from the international affective picture system (IAPS): do ratings become more extreme with age? Behav Res Methods 40:512–521
Gupta V, Chopda MD, Pachori RB (2018) Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals. IEEE Sensors J 19:2266–2274
Hill NJ, Lal TN, Schröder M, Hinterberger T, Widman G, Elger CE et al (2006) Classifying event-related desynchronization in EEG, ECoG and MEG signals. In: Joint Pattern Recognition Symposium, pp. 404–413
Hjorth B (1970) EEG analysis based on time domain properties. Electroencephalogr Clin Neurophysiol 29:306–310
Huang NE, Shen Z, Long SR, Wu MC, Shih HH, Zheng Q, Yen NC, Tung CC, Liu HH (1998) The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences. 454:903–995
Huang J, Xu X, Zhang T (2017) Emotion classification using deep neural networks and emotional patches. In: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 958–962
Jenke R, Peer A, Buss M (2014) Feature extraction and selection for emotion recognition from EEG. IEEE Trans Affect Comput 5:327–339
Jerritta S, Murugappan M, Wan K, Yaacob S (2014) Emotion recognition from facial EMG signals using higher order statistics and principal component analysis. J Chin Inst Eng 37:385–394
Jirayucharoensak S, Pan-Ngum S, Israsena P (2014) EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci World J 2014:1–10
Kamble KS, Sengupta J (2021) Ensemble machine learning-based affective computing for emotion recognition using dual-decomposed EEG signals. IEEE Sensors J 22:2496–2507
Katsigiannis S, Ramzan N (2017) DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform 22:98–107
Khare SK, Bajaj V (2020) Constrained based tunable Q wavelet transform for efficient decomposition of EEG signals. Appl Acoust 163:107234
Khare SK, Bajaj V (2020) An evolutionary optimized variational mode decomposition for emotion recognition. IEEE Sensors J 21:2035–2042
Khare SK, Bajaj V (2020) Time-frequency representation and convolutional neural network-based emotion recognition. IEEE Trans Neural Netw Learn Syst 32:2901–2909
Khosrowabadi R, bin Abdul Rahman AW (2010) Classification of EEG correlates on emotion using features from Gaussian mixtures of EEG spectrogram. In: Proceeding of the 3rd International Conference on Information and Communication Technology for the Moslem World (ICT4M) 2010, pp. E102-E107
Khosrowabadi R, Quek HC, Wahab A, Ang KK (2010) EEG-based emotion recognition using self-organizing map for boundary detection. In: 2010 20th International Conference on Pattern Recognition, pp. 4242–4245
Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30:2067–2083
Koelstra S, Patras I (2013) Fusion of facial expressions and EEG for implicit affective tagging. Image Vis Comput 31:164–174
Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T et al (2011) Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3:18–31
Kolodyazhniy V, Kreibig SD, Gross JJ, Roth WT, Wilhelm FH (2011) An affective computing approach to physiological emotion specificity: toward subject-independent and stimulus-independent classification of film-induced emotions. Psychophysiology 48:908–922
Krishna AH, Sri AB, Priyanka KYVS, Taran S, Bajaj V (2018) Emotion classification using EEG signals based on tunable-Q wavelet transform. IET Sci Meas Technol 13:375–380
Kroupi E, Yazdani A, Ebrahimi T (2011) EEG correlates of different emotional states elicited during watching music videos. In: International Conference on Affective Computing and Intelligent Interaction: Fourth International Conference, ACII, Memphis, TN, USA, October 9–12, 2011, Proceedings, Part II pp. 457–466. Springer Berlin, Heidelberg
Lakhan P, Banluesombatkul N, Changniam V, Dhithijaiyratn R, Leelaarporn P, Boonchieng E, Hompoonsup S, Wilaiprasitporn T (2019) Consumer grade brain sensing for emotion recognition. IEEE Sensors J 19(21):9896–9907
Lan Z, Sourina O, Wang L, Liu Y (2016) Real-time EEG-based emotion monitoring using stable features. Vis Comput 32:347–358
Lemm S, Blankertz B, Curio G, Muller K-R (2005) Spatio-spectral filters for improving the classification of single trial EEG. IEEE Trans Biomed Eng 52:1541–1548
Li H, Qing C, Xu X, Zhang T (2017) A novel DE-PCCM feature for EEG-based emotion recognition. In: 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), pp. 389–393
Li T-H, Liu W, Zheng W-L, Lu B-L (2019) Classification of five emotions from EEG and eye movement signals: Discrimination ability and stability over time. In: 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 607–610
Li R, Ren C, Zhang X, Hu B (2022) A novel ensemble learning method using multiple objective particle swarm optimization for subject-independent EEG-based emotion recognition. Comput Biol Med 140:105080
Liu Y, Sourina O (2013) Real-time fractal-based valence level recognition from EEG. In: Transactions on computational science XVIII. ed: Springer, pp 101–120
Liu Y, Sourina O (2013) EEG databases for emotion recognition. In: 2013 international conference on cyberworlds, pp. 302–309
Liu W, Zheng W-L, Lu B-L (2016) Emotion recognition using multimodal deep learning. In: International conference on neural information processing, pp. 521–529
Liu Y-J, Yu M, Zhao G, Song J, Ge Y, Shi Y (2017) Real-time movie-induced discrete emotion recognition from EEG signals. IEEE Trans Affect Comput 9:550–562
Liu J, Su Y, Liu Y (2017) Multi-modal emotion recognition with temporal-band attention based on LSTM-RNN. In: Pacific Rim Conference on Multimedia, pp. 194–204
Liu Y, Ding Y, Li C, Cheng J, Song R, Wan F, Chen X (2020) Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Comput Biol Med 123:103927
Liu W, Qiu J-L, Zheng W-L, Lu B-L (2021) Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Trans Cogn Develop Syst 14:715–729
Maheshwari D, Ghosh S, Tripathy R, Sharma M, Acharya UR (2021) Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals. Comput Biol Med 134:104428
Mahmoud S, Elamien M, Zaatar A, ElTinai M (2014) Low noise low power instrumentation amplifier for biomedical application
Mehrabian A (1996) Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Curr Psychol 14:261–292
Murugappan M, Murugappan S (2013) Human emotion recognition through short time Electroencephalogram (EEG) signals using Fast Fourier Transform (FFT). In: 2013 IEEE 9th International Colloquium on Signal Processing and its Applications, pp. 289–294
Murugappan M, Ramachandran N, Sazali Y (2010) Classification of human emotion from EEG using discrete wavelet transform. J Biomed Sci Eng 3:390–396
Murugappan M, Nagarajan R, Yaacob S (2011) Combining spatial filtering and wavelet transform for classifying human emotions using EEG signals. J Med Biol Eng 31:45–51
Nakisa B, Rastgoo MN, Tjondronegoro D, Chandran V (2018) Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors. Expert Syst Appl 93:143–155
Nie D, Wang X-W, Shi L-C, Lu B-L (2011) EEG-based emotion recognition during watching movies. In: 2011 5th International IEEE/EMBS Conference on Neural Engineering, pp. 667–670
Onton JA, Makeig S (2009) High-frequency broadband modulation of electroencephalographic spectra. Front Hum Neurosci 3:61
Paithane A, Bormane D (2014) Analysis of nonlinear and non-stationary signal to extract the features using Hilbert Huang transform. In: 2014 IEEE International Conference on Computational Intelligence and Computing Research, pp. 1–4
Pandeya YR, Lee J (2021) Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimed Tools Appl 80:2887–2905
Patel R, Janawadkar MP, Sengottuvel S, Gireesan K, Radhakrishnan TS (2016) Suppression of eye-blink associated artifact using single channel EEG data by combining cross-correlation with empirical mode decomposition. IEEE Sensors J 16:6947–6954
Petrantonakis PC, Hadjileontiadis LJ (2010) Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis. IEEE Trans Affect Comput 1:81–97
Plutchik R, Kellerman H (1980) Theories of emotion. Academic Press
Rached TS, Perkusich A (2013) Emotion recognition based on brain-computer interface systems. Brain-computer interface systems-Recent progress and future prospects, pp. 253–270
Rázuri JG, Sundgren D, Rahmani R, Cardenas AM (2013) Automatic emotion recognition through facial expression analysis in merged images based on an artificial neural network. In: 2013 12th Mexican International Conference on Artificial Intelligence, pp. 85–96
Reuderink B, Mühl C, Poel M (2013) Valence, arousal and dominance in the EEG during game play. Int J Auton Adapt Commun Syst 6:45–62
Salama ES, El-Khoribi RA, Shoman ME, Shalaby MAW (2018) EEG-based emotion recognition using 3D convolutional neural networks. Int J Adv Comput Sci Appl 9:329–337
Salari S, Ansarian A, Atrianfar H (2018) Robust emotion classification using neural network models. In: 2018 6th Iranian Joint Congress on Fuzzy and Intelligent Systems (CFIS), pp. 190–194
Sangnark S, Autthasan P, Ponglertnapakorn P, Chalekarn P, Sudhawiyangkul T, Trakulruangroj M, Songsermsawad S, Assabumrungrat R, Amplod S, Ounjai K, Wilaiprasitporn T (2021) Revealing preference in popular music through familiarity and brain response. IEEE Sensors J 21:14931–14940
Sharma LD, Bhattacharyya A (2021) A computerized approach for automatic human emotion recognition using sliding mode singular Spectrum analysis. IEEE Sensors J 21:26931–26940
Shin D, Shin D, Shin D (2017) Development of emotion recognition interface using complex EEG/ECG bio-signal for interactive contents. Multimed Tools Appl 76:11449–11470
Shirke B, Wong J, Libut JC, George K, Oh SJ (2020) Brain-IoT based Emotion Recognition System. In: 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0991–0995
Siddharth S, Jung T-P, Sejnowski TJ (2019) Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Trans Affect Comput
Siuly S, Khare SK, Bajaj V, Wang H, Zhang Y (2020) A computerized method for automatic detection of schizophrenia using EEG signals. IEEE Trans Neural Syst Rehabilitation Eng 28:2390–2400
Soleymani M, Lichtenauer J, Pun T, Pantic M (2011) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3:42–55
Soleymani M, Pantic M, Pun T (2011) Multimodal emotion recognition in response to videos. IEEE Trans Affect Comput 3:211–223
Song T, Zheng W, Song P, Cui Z (2018) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11:532–541
Song T, Zheng W, Song P, Cui Z (2018) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput
Song T, Liu S, Zheng W, Zong Y, Cui Z, Li Y, Zhou X (2021) Variational instance-adaptive graph for EEG emotion recognition. IEEE Trans Affect Comput
Subasi A, Tuncer T, Dogan S, Tanko D, Sakoglu U (2021) EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier. Biomed Signal Process Control 68:102648
Tao W, Li C, Song R, Cheng J, Liu Y, Wan F, Chen X (2020) EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans Affect Comput
Taran S, Bajaj V (2019) Motor imagery tasks-based EEG signals classification using tunable-Q wavelet transform. Neural Comput & Applic 31:6925–6932
Taran S, Bajaj V, Sharma D, Siuly S, Sengur A (2018) Features based on analytic IMF for classifying motor imagery EEG signals in BCI applications. Measurement 116:68–76
Torres E, Hernández-Álvarez M, Yoo SG (2020) EEG-Based BCI Emotion Recognition Using the Stock-Emotion Dataset. In: The International Conference on Advances in Emerging Trends and Technologies, pp. 226–235
Tuncer T, Dogan S, Subasi A (2021) A new fractal pattern feature generation function based emotion recognition method using EEG. Chaos Solit Fractals 144:110671
Valenza G, Lanata A, Scilingo EP (2012) The role of nonlinear dynamics in affective valence and arousal recognition. IEEE Trans Affect Comput 3:237–249
Verma GK, Tiwary US (2014) Multimodal fusion framework: a multiresolution approach for emotion classification and recognition from physiological signals. NeuroImage 102:162–172
Wang Y, Mo J (2013) Emotion feature selection from physiological signals using tabu search. In: 2013 25th Chinese Control and Decision Conference (CCDC), pp. 3148–3150
Wang J, Wang M (2021) Review of the emotional feature extraction and classification using EEG signals. Cognitive Robotics 1:29–40
Wang X-W, Nie D, Lu B-L (2011) EEG-based emotion recognition using frequency domain features and support vector machines. In: International conference on neural information processing, pp. 734–743
Wang X-W, Nie D, Lu B-L (2014) Emotional state classification from EEG data using machine learning approach. Neurocomputing 129:94–106
Wilson GF, Russell CA (2003) Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum Factors 45:635–644
Wu M, Wan T, Wan X, Du Y, She J (2017) Fast, accurate localization of epileptic seizure onset zones based on detection of high-frequency oscillations using improved wavelet transform and matching pursuit methods. Neural Comput 29:194–219
Yang B, Han X, Tang J (2017) Three class emotions recognition based on deep learning using staked autoencoder. In: 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1–5
Yohanes RE, Ser W, Huang G-B (2012) Discrete Wavelet Transform coefficients for emotion recognition from EEG signals. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2251–2254
Zhang J, Chen M, Hu S, Cao Y, Kozma R (2016) PNN for EEG-based Emotion Recognition. In: 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 002319–002323
Zhao S, Li Z, Cui R, Kang Y, Sun F, Song R (2017) Brain–machine interfacing-based teleoperation of multiple coordinated Mobile robots. IEEE Trans Ind Electron 64:5161–5170
Zheng W (2016) Multichannel EEG-based emotion recognition via group sparse canonical correlation analysis. IEEE Trans Cogn Dev Syst 9:281–290
Zheng W-L, Lu B-L (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7:162–175
Zheng W-L, Lu B-L (2017) A multimodal approach to estimating vigilance using EEG and forehead EOG. J Neural Eng 14:026017
Zheng W-L, Zhu J-Y, Peng Y, Lu B-L (2014) EEG-based emotion classification using deep belief networks. In: 2014 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6
Zheng W-L, Zhu J-Y, Lu B-L (2017) Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors do not have any conflicts of interests/ competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Kamble, K., Sengupta, J. A comprehensive survey on emotion recognition based on electroencephalograph (EEG) signals. Multimed Tools Appl 82, 27269–27304 (2023). https://doi.org/10.1007/s11042-023-14489-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-14489-9