Emotion recognition using empirical mode decomposition and approximation entropy

doi:10.1016/j.compeleceng.2018.09.022

Computers & Electrical Engineering

Volume 72, November 2018, Pages 383-392

https://doi.org/10.1016/j.compeleceng.2018.09.022 Get rights and content

Abstract

Automatic human emotion recognition is a key technology for human-machine interaction. In this paper, we propose an electroencephalogram (EEG) feature extraction method that leverages empirical mode decomposition and Approximation Entropy. In our proposed method, Empirical Mode Decomposition (EMD) is used to process EEG signals after data processing and obtains several intrinsic eigenmode functions. The Approximation Entropy (ApEn) of the first four Intrinsic Mode Functions (IMFs) is computed, which is used as the features from EEG signals for learning and recognition. An integration of Deep Belief Network and Support Vector Machine is devised for classification, which takes the eigenvectors from the extracted feature to identify four principal human emotions, namely happy, calm, sad, and fear. Experiments are conducted with EEG data acquired with a 16-lead device. Our experimental results demonstrate that the proposed method achieves an improved accuracy that is highly competitive to the state-of-the-art methods. The average accuracy is 83.34%, and the best accuracy reaches 87.32%.

Introduction

Automatic human emotion recognition is a key technology for human-machine interaction [1], [2]. Many research on emotion recognition relies on data from images, audio, and videos [3], [4], [5]. Discrete models [6] and dimensional models [7] are proposed to describe emotional states. Among a variety of data, physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), and electromyography (EMG) signals have been employed for emotion recognition [8]. EEG signal is closely correlated with brain activities and is more promising for recognizing emotional states [9], [10], [11].

Most recently, Hu et al. [12] proposed a classification method that combines Correlation-based Feature Selection (CFS) and a k-nearest-neighbor (KNN) algorithm for attention recognition. Lin et al. [4] used Support Vector Machine (SVM) to classify the emotional states based on EEG into four categories and found that the frontal and temporal lobes of the brain are the main areas of emotion generation, and the average classification accuracy of emotion achieves 82.29%. Goyal et al. [13] described the acquisition of EEG signals on frontal electrodes from five subjects for the classification of emotions. Gonuguntla et al. [14] analyzed the network mechanisms related to human emotion based on synchronization measure phase-locking value in EEG to formulate the emotion-specific brain functional network.

In addition to the development of classification methods, signal processing techniques have been studied. Empirical mode decomposition methods based on the Hilbert-Huang Transform (HHT) have been explored in the field of signal processing to improve the recognition performance [15], [16]. HHT includes empirical mode decomposition (EMD) and Hilbert transformation. It decomposes a signal into several approximate cosine waves and looks into their periods and amplitudes, which effectively suppresses noise and obtain the time-frequency characteristics of the signal. Such methods achieved greater results when dealing with non-stationary signals.

Despite the advancements in emotion recognition using EEG signals, there is much room to improve. This paper integrates EMD and Approximation Entropy (ApEn) and proposes an EEG feature extraction method (namely EMD-Approximation Entropy, in short E-ApEn) for feature extraction. This combined feature extraction method reduces the complexity of feature extraction. Using the emotion recognition model and integrating Deep Belief Network (DBN) and Support Vector Machine to get the feature vectors for training and classification, It is expected that a higher rate of emotion recognition can be achieved.

The rest of this paper is organized as follows: Section 2 presents our proposed method for emotion recognition using EEG signals. The section starts with an overview of the framework followed by the feature extraction method that integrates EMD and ApEn. A DBN-SVM is discussed, which makes multi-class decisions for emotion recognition. Section 3 discusses our experimental results and the section includes a description of data acquisition and preprocessing, as well as a comparison study with the state-of-the-art methods. Section 4 concludes this paper with a summary of our work and a highlight.

Section snippets

Framework of the proposed method

The framework of our proposed method is shown in Fig. 1. The process of establishing the emotion recognition model using EMD decomposition and approximate entropy consists of the following four steps:

1.
The signal is preprocessed to get signal clips of a fixed size and an independent component analysis (ICA) is used to suppress noise.
2.
For each attribute of a signal clip, EMD is used for decomposition, and the approximate entropy of the first 4 IMFs of the decomposed signal is calculated.
3.
Select the

Human subjects and acquisition device

Ten healthy subjects participated in the experiment (5 men, 5 women, and ages range 20–25). All participants had normal hearing and vision and no mental disorders. We use a 16-lead Emotiv brainwave instrument (14 of which are EEG acquisition channels and 2 of which are reference electrodes) at a frequency of 128 Hz.

Emotional induction

To induce the emotional states of human subjects, we used movie clips as experimental materials following the strategy in [21]. We select 20 videos from 200 videos prepared for the

Conclusion

In order to find out the correlation between emotional states and EEG signals and improve the recognition rate of emotion, an emotion recognition model using empirical mode decomposition and approximation entropy is proposed. In this method, empirical mode decomposition is performed to the EEG signals after intercepting valid duration data segments and noise suppression by ICA. Subsequently, the approximate entropy of the EEG signals is calculated. An integration of DBN and SVM is devised for

Acknowledgments

This work supported by The Key Program of the National Natural Science Foundation of China (Grant No. 61432004); The National Natural Science Foundation of China (Grant No. 61474035); The National Scholarship Foundation of China (Grant No. 201706695016); The fund of Affective Computing and Advanced Intelligent Machine Anhui Province Key Laboratory (Grant No. ACAIM180101).

Tian Chen received the B.E. M.E. and the Ph.D. degrees from Hefei University of Technology, China, in 1996, 2002 and 2011, respectively. She is an associate professor of the School of Computer Science and Information Engineering at the Hefei University of Technology, China. Her current research interests include affective computing, artificial intelligence, and design for test.

References (21)

Y. Liu et al.
Conditional convolutional neural network enhanced random forest for facial expression recognition
Pattern Recognit
(2018)
R.W. Picard et al.
Toward machine emotional intelligence: analysis of affective physiological state
IEEE Trans Pattern Anal Mach Intell
(2001)
S. Chu et al.
Environmental sound recognition using mp-based features
IEEE International Conference on Acoustics, Speech and Signal Processing
(2008)
A. Savran et al.
Emotion detection in the loop from brain signals and facial images
Proceedings of the eNTERFACE 2006 Workshop
(2006)
Y. Lin et al.
Eeg-based emotion recognition in music listening
IEEE Trans Biomed Eng
(2010)
P. Ekman
An argument for basic emotions
Cognition & Emotion
(1992)
A. Mehrabian
Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament
Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues
(1996)
S. Koelstra et al.
Deap: a database for emotion analysis; using physiological signals
IEEE Trans Affect Comput
(2012)
L.I. Aftanas et al.
Analysis of evoked eeg synchronization and desynchronization in conditions of emotional activation in humans: temporal and topographic characteristics
Neurosci Behav Physiol
(2004)
S.K. Hadjidimitriou et al.
Toward an eeg-based recognition of music liking using time-frequency analysis
IEEE Trans Biomed Eng
(2012)

There are more references available in the full text version of this article.

Cited by (71)

ICaps-ResLSTM: Improved capsule network and residual LSTM for EEG emotion recognition
2024, Biomedical Signal Processing and Control
Electroencephalography (EEG) emotion recognition is an important task for brain–computer interfaces. The time, frequency, and spatial domains of EEG signals have been widely studied. However, these methods often ignore the spatial and temporal correlations in dual modules, resulting in insufficient emotional representations. In this paper, a dual module EEG emotion recognition method based on an improved capsule network and residual Long-Short Term Memory (ResLSTM) is proposed. Using an improved capsule network as the spatial module is more advantageous in learning specific EEG spatial representations. The ResLSTM of the temporal module inherits the information flow from the upper spatial module and conducts complementary learning of the spatiotemporal dual module features through residual connections, thus obtaining more discriminative EEG features and ultimately boosting the classification capabilities of the model. The average accuracy of arousal, valence, and dominance on the DEAP dataset reached 98.06%, 97.94%, and 98.15%, respectively. The DREAMER dataset’s average accuracy of arousal, valence, and dominance reached 94.97%, 94.71%, and 94.96%, respectively. The results of our experiments indicate that our method outperforms state-of-the-art approaches.
EEG emotion recognition using attention-based convolutional transformer neural network
2023, Biomedical Signal Processing and Control
EEG-based emotion recognition has become an important task in affective computing and intelligent interaction. However, how to effectively combine the spatial, spectral, and temporal distinguishable information of EEG signals to achieve better emotion recognition performance is still a challenge. In this paper, we propose a novel attention-based convolutional transformer neural network (ACTNN), which effectively integrates the crucial spatial, spectral, and temporal information of EEG signals, and cascades convolutional neural network and transformer in a new way for emotion recognition task. We first organized EEG signals into spatial–spectral–temporal representations. To enhance the distinguishability of features, spatial and spectral attention masks are learned for the representation of each time slice. Then, a convolutional module is used to extract local spatial and spectral features. Finally, we concatenate the features of all time slices, and feed them into the transformer-based temporal encoding layer to use multi-head self-attention for global feature awareness. The average recognition accuracy of the proposed ACTNN on two public datasets, namely SEED and SEED-IV, is 98.47% and 91.90% respectively, outperforming the state-of-the-art methods. Besides, to explore the underlying reasoning process of the model and its neuroscience relevance with emotion, we further visualize spatial and spectral attention masks. The attention weight distribution shows that the activities of prefrontal lobe and lateral temporal lobe of the brain, and the gamma band of EEG signals might be more related to human emotion. The proposed ACTNN can be employed as a promising framework for EEG emotion recognition.
Electroencephalogram channel selection based on pearson correlation coefficient for motor imagery-brain-computer interface
2023, Measurement: Sensors
Decryption of Motor Imagery (MI) activity from an Electroencephalogram (EEG) data is a significant part of the Brain-Computer Interface (BCI) technology that allows motor-disabled persons to connect with external devices. Channel selection, feature extraction, and classification are essential requirements for an effective BCI system. Non-stationary EEG data confuses designing EEG-based BCIs. In this study, the Pearson correlation coefficient (PCC) technique is employed for channel selection for EEG signals in the BCI system. It selects the most associated fourteen channels for the sensorimotor area of subject's brain. The popular signal processing technique wavelet packet decomposition (WPD) is employed for feature extraction. After that approximate entropy (ApEn) feature is calculated for selected channels. The proposed study is a novel scheme combining Pearson correlation coefficient-based channel selection technique and wavelet packet decomposition for classifying MI signals. Finally, extracted features are classified with the help of two benchmark techniques, Support Vector Machine (SVM) and K-Nearest Neighbors (K-NN) and achieve maximum accuracy of 91.66% and 90.33%, respectively. The proposed technique is examined on freely available EEG datasets BCI competition-IV-Dataset I to prove its superiority over previously reported approaches. Obtained experimental findings demonstrated advantages over previous methods in terms of classification accuracy.
Exploring the effect of climate risk on agricultural and food stock prices: Fresh evidence from EMD-Based variable-lag transfer entropy analysis
2023, Journal of Environmental Management
Climate has traditionally played an important role in the development of countries, owing to its inherent relationship with agricultural output and pricing. This study explores one such association between the most well-known climate anomaly, the El Niño34 Southern Oscillation, and international commodity prices of agriculture and food indexes. This study addresses the potentially causal effect of El Niño34 on international agricultural and food stock prices. To do so, we develop a novel approach: the empirical mode decomposition variable-lag transfer entropy (EMD-VL transfer entropy) by combining the variable-lag transfer entropy framework and the empirical mode decomposition. The evidence reveals the following major results. First, climate shocks affect global agricultural stock prices in the short-term. Second, significant transfer entropy from El Niño34 to food index appeared at mid- and long-term business cycles. Third, unidirectional causal effect from climate shocks to agricultural and food stock prices is more intense in the short business cycle attesting to the impact of climate shocks on the food market, which is especially visible in the short-term horizon. Finally, our proposed method exceeds the traditional variable-lag transfer entropy by detecting such causal interplay at various business cycles, which is useful for investors and policymakers.
Analysis of Wrist Pulse Signal: Emotions and Physical Pain
2022, IRBM
Citation Excerpt :
Various ways are present to compute the entropy (randomness) of a signal. Approximate entropy [57] quantifies the data in the time domain whereas spectral entropy [58] has been frequently used in the frequency domain. Band powers are widely used in literature for emotion recognition especially in EEG [59].
Pulse diagnosis (wrist pulse signal) is a well-known traditional technique used for a health examination. It has the potential to detect cardiac and non-cardiac diseases.
A study was conducted to investigate human emotions using wrist pulse signal assessment. The aim was to categorize anxiety, boredom, physical pain, and reference state by processing and analysis of acquired signals.
A protocol was designed to induce emotions. Data were acquired from 24 healthy volunteers. Signals were processed and further analyzed using paired t-test and Analysis of Variance (ANOVA). Machine learning algorithms, Linear Discriminant Function (LDF), Quadratic Discriminant Function (QDF), and Support Vector Machine using kernel Gaussian radial basis function (RBF-SVM) were used to evaluate significant features and classify the emotions.
Computing significant plus ranked features performed better over randomly selected features for pairwise emotion classification. Here, the QDF classifier outperforms LDF. Additionally, ANOVA validated the effectiveness of statistically prominent features to classify emotional states. Ratio_Pulse_Strength, total_power, Spectral Entropy, and meancd5 came out as the four most significant features to classify the emotion “Anxiety”, “Boredom”, “Pain”, and “Reference” with positive prediction rate of 100%, 73%, 100%, and 86% respectively using RBF-SVM in the user-independent model.
Previously, WPS has been used mainly to detect physical abnormality in the human body. The results endorse the potential of user-independent human emotion detection using a wrist pulse signal. The present work was focusing on a few emotional states. Results are encouraging and may be well applied to many more states.
A State of the Art Technology with Machine Intelligence in Human Emotion Recognition
2022, Journal of Interconnection Networks

View all citing articles on Scopus

Sihang Ju received his B.E. degree in Computer Science and Technology from Hefei University of Technology, China, in 2016. He has been a postgraduate student since 2016. His research interest is affective computing based on EEG.

Xiaohui Yuan received the Ph.D. degree in computer science from Tulane University in 2004. He is an Associate Professor at the University of North Texas. His research interests include computer vision and artificial intelligence. He is a recipient of Ralph E. Powe Junior Faculty award and a senior member of IEEE. He published over 130 papers in journals and conferences.

Mohamed Elhoseny received the Ph.D. in Computer and Information from Mansoura University. He is an Assistant Professor at the Faculty of Computers and Information, Mansoura University, Egypt. His research interests include sensor and ad-hoc networks, Internet of things, data security, machine learning, and optimization. He published over 90 papers in journals, conferences, and books, and edited 3 books.

Fuji Ren received the B.E. and M.E. degrees from the Beijing University of Posts and Telecommunications, in 1982 and 1985, respectively, and the Ph.D. degree from Hokkaido University, Japan, in 1991. He is a Professor with the Faculty of Engineering, University of Tokushima, Tokushima, Japan. His current research interests include information science, artificial intelligence, language understanding, and affective computing.

Mingyan Fan received his B.E. degree in Computer Science and Technology from Hefei University of Technology, China, in 2016. He has been a postgraduate student since 2017. His research interest is wearable computing.

Zhangang Chen received his B.E. and M.S. degree in Computer Science and Technology from Hefei University of Technology, China, in 2015 and 2018 respectively. His research interest is affective computing based on EEG.

^☆: Reviews processed and recommended for publication to the Editor-in-Chief by Guest Editor Dr. Guanglong Du.

View full text

Emotion recognition using empirical mode decomposition and approximation entropy☆

Abstract

Introduction

Section snippets

Framework of the proposed method

Human subjects and acquisition device

Emotional induction

Conclusion

Acknowledgments

Pattern Recognit

Toward machine emotional intelligence: analysis of affective physiological state

IEEE Trans Pattern Anal Mach Intell

Environmental sound recognition using mp-based features

IEEE International Conference on Acoustics, Speech and Signal Processing

Emotion detection in the loop from brain signals and facial images

Proceedings of the eNTERFACE 2006 Workshop

Eeg-based emotion recognition in music listening

IEEE Trans Biomed Eng

An argument for basic emotions

Cognition & Emotion

Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament

Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues

Deap: a database for emotion analysis; using physiological signals

IEEE Trans Affect Comput

Analysis of evoked eeg synchronization and desynchronization in conditions of emotional activation in humans: temporal and topographic characteristics

Neurosci Behav Physiol

Toward an eeg-based recognition of music liking using time-frequency analysis

IEEE Trans Biomed Eng