Relevance vector classifier decision fusion and EEG graph-theoretic features for automatic affective state characterization
Introduction
Burgeoning research in the field of passive brain–computer interfaces (BCIs) has allowed for information about an individual׳s affective state to be measured continuously and effortlessly. Such so-called “affective BCIs” can play a significant role across a number of applications, including but not limited to: neuromarketing, serious gaming, quality-of-experience perception modelling, and more recently, affective video tagging/indexing [1], [2], [3], [4], [5], [6]. The amount of uploaded video content to online repositories is growing exponentially (e.g., on Youtube, an average 100 h of videos are uploaded per minute and over six billion hours of videos are watched per month), thus rendering manual tagging impossible and the video retrieval process ineffective. Affective BCIs, on the other hand, can allow for automated video tagging “on-the-fly” thus making video searches more effective, personal, and improve user׳s experience, with the technology.
Towards this end, it becomes necessary to extract implicit emotional tags for videos by assessing users׳ affective states while experiencing those videos. One approach towards recognizing users׳ affective states is to monitor their central nervous system activity using neuro-imaging tools, such as electroencephalography (EEG). EEG is a brain imaging modality that is well suited for this task as it is non-invasive and has very good temporal resolution (milliseconds). Spectral power features derived from several frequency bands, such as delta (δ: 1–4 Hz), theta (θ: 4–8 Hz), alpha (α: 8–13 Hz), beta (β: 13–30 Hz) and gamma (γ: 30–50 Hz), have been widely used as neurophysiological correlates of emotional activity [7]. Typically, alpha-power asymmetry is used as an indicator of emotional states. The measure is computed from spectral power differences between symmetric electrode pairs placed over the two hemispheres of the brain [8], [9], [10]. Spectral features from other sub-bands from specific regions of the brain have also been reported to encode affective states, such as parietal theta-power [11], right parietal alpha-power [10], parietal beta-power asymmetry [12] and parietal gamma-power [13], [14]. These studies provide evidence towards the feasibility of using features derived from EEG for affective state characterization.
The next step, however, is to develop classifiers to solve the emotion classification problem using the identified features. Previous studies have leveraged spectral power features together with support vector machines (SVM) to classify users׳ anger, joy, sadness and pleasantness with a classification accuracy of around 82% [15]. SVMs, in fact, are among the most widely used state-of-the-art emotion classification techniques [16], [17]. These have also been used with discrete wavelet transform (DWT) features, achieving an accuracy of around 81% [18]. In [19], in turn, authors used relative power changes for EEG along with a Bayesian network to predict emotional states. In [20], authors reported an accuracy of 73% for classifying emotional states using asymmetry features implemented with SVM classifier. Also, [21] reported an average accuracy of 63% using EEG time–frequency and mutual information features along with SVM classifier. Other studies which have used similar features and classifiers are reported in Table 1. In the table the following abbreviations are used: DT-CWPT: dual tree complex wavelet packet transform, SPF: spectral power features, AI: asymmetry index and CSP: common spatial pattern.
During the time course of watching a music video, however, several parts of the brain are activated to process and integrate the auditory and visual streams, as well as to evaluate emotional content via attentional and context updating mechanisms, which in turn can be influenced by different associative, imagery and memory processes [26], [27]. Generally, while experiencing a music video, the sensory information is processed in a segregated manner [28], [29], while the affective and cognitive processing mostly integrate processed information from sensory areas as well as among themselves [30]. In the context of brain networks, functional segregation refers to the network׳s ability to process specialized information in densely interconnected regions. On the other hand, functional integration refers to their ability to combine this specialized information from widely separated regions [31]. As can be seen, affective state characterization is comprised of a complex flow and interplay of information between brain regions that may span different EEG frequency bands. As such, it is expected that improved affective state characterization can be achieved by mapping the flow of information through the various neuronal networks involved and by computing their properties.
Having this said, it is hypothesized that changes in emotional content of the watched music clips would elicit changes in the information flow processes inside the brain, which, to some extent, would be captured in the recorded EEG activity by the electrodes placed over various brain regions. Thus, a functional connectivity analysis of the recorded data will reveal neural correlates of the viewers׳ affective states which can be used to solve the emotion classification problem. Generally, functional connectivity analysis describes the strength of the interactions between various regions of the brain. Commonly, graph-theory tools are used to characterize the topographical properties of the complex neuronal networks involved [32]. Using various neuroimaging modalities, recent studies have utilized graph-theoretical insights to analyze the emerging functional connectivity patterns, for e.g., emotions induced from visual stimuli [33], intelligence [34], and disease characterization [35]. For the present study, we propose using the graph-theoretical features to address the emotion classification problem using the well-known support vector machine (SVM) [36] and relevance vector machine (RVM) [37] classifiers. Moreover, we also propose using feature fusion and decision fusion techniques to improve the classification performance. It is hoped that the obtained insights can be used by the community to further advance affective BCI technologies and develop innovative applications, such as automated affective tagging/indexing of videos.
Section snippets
Database for emotion analysis using physiological signals (DEAP)
The pre-processed EEG and subjective data used in the present study were obtained from the publicly available “database for emotion analysis using physiological signals (DEAP)” [22]. Here, only a brief description of the data is presented; the interested reader is referred to [22] for more details. Thirty two healthy participants (50% females, average age=26.9 years) were recruited and consented to participate in the study. Thirty-two channel EEG data were recorded using a Biosemi ActiveTwo
Results
Table 3 reports the F1-score and classification accuracy for the classifier corresponding to the frequency sub-band with maximum F1-score chosen amongst the 10 individual classifiers for the arousal, valence, dominance and liking categories. The computed F1-scores were then tested for significance against a random chance level of 50% using an independent one-sample t-test, as proposed in [22]. It can be observed that graph, spectral power and AI features perform significantly better than chance
Discussion
From Table 3, it can be observed that graph, spectral power and AI features perform significantly better than chance in classifying users׳ emotional states, thus suggesting their utility in affect classification. Also, it can be said that the classifiers developed using the graph theoretical treatment of EEG data produced significantly () better classification metrics for emotional dimensions, specifically for valence and arousal, as compared to the traditionally used spectral power and
Conclusion
In this work, we have explored the use of EEG-based graph theoretical tools for affect recognition in audio–video stimuli. Several such metrics were shown to reliably discriminate between low and high levels of subjective affective dimensions. These findings suggest that objective affective characterisation is possible using the graph theoretical tools, and could lead to disruptive new technologies, such as affective BCIs. However, features which provide directional information and, are more
Acknowledgment
This work was funded by the Quebec Ministry of Economic Development, Innovation and Export (MDEIE – Ministere du Developpement Economique, de l׳Innovation et de l׳Exportation) (grant number MDEIE PSR-SIIRI-687) and the Natural Sciences and Engineering Research Council of Canada (NSERC) (grant number 402237).
Rishabh Gupta received his B.Tech degree from the VIT University, India, in 2011, and the M.Sc. degree from University of Warwick, U.K., in 2012, both in biomedical engineering. He is currently following the Ph.D. program in telecommunications at INRS in Montreal, Canada. His research interests include machine learning, signal processing, human factors engineering, and UX evaluation using physiological tools.
References (57)
- et al.
Issues and assumptions on the road from raw signals to metrics of frontal EEG asymmetry in emotion
Biol. Psychol.
(2004) Parietal electroencephalogram beta asymmetry and selective attention to angry facial expressions in healthy human subjects
Neurosci. Lett.
(2001)- et al.
Consciousness and arousal effects on emotional face processing as revealed by brain oscillations. A gamma band analysis
Int. J. Psychophysiol.
(2008) Processing of affective pictures modulates right-hemispheric gamma band EEG activity
Clin. Neurophysiol.
(1999)- et al.
Emotional state classification from EEG data using machine learning approach
Neurocomputing
(2014) - et al.
Analysis of positive and negative emotions in natural scene using brain activity and GIST
Neurocomputing
(2009) Short-term emotion assessment in a recall paradigm
Int. J. Hum.–Comput. Stud.
(2009)- et al.
Classification of emotions induced by music videos and correlation with participants׳ rating
Expert Syst. Appl.
(2014) - et al.
Non-linear dynamical coupling between different brain areas during evoked emotionsan EEG investigation
Biol. Psychol.
(1998) - et al.
Self-generated happy and sad emotions in low and highly hypnotizable persons during waking and hypnosislaterality and regional EEG activity differences
Int. J. Psychophysiol.
(1996)
Beyond the right hemispherebrain mechanisms mediating vocal emotional processing
Trends Cognit. Sci.
Complexity and coherencyintegrating information in the brain
Trends Cognit. Sci.
How does the metric choice affect brain functional connectivity networks?
Biomed. Signal Process. Control
Use of magnetoencephalography (MEG) to study functional brain networks in neurodegenerative disorders
J. Neurol. Sci.
Measuring emotionthe self-assessment manikin and the semantic differential
J. Behav. Therapy Exp. Psychiatr.
Cortical coordination dynamics and cognition
Trends Cognit. Sci.
A comparative study of synchrony measures for the early diagnosis of Alzheimer׳s disease based on EEG
NeuroImage
Neural correlates of emotional responses to musican EEG study
Neurosci. Lett.
The contribution of EEG coherence to the investigation of language
Brain Lang.
Complex network measures of brain connectivityuses and interpretations
NeuroImage
Global workspace theory of consciousnesstoward a cognitive neuroscience of human experience
Progr. Brain Res.
Automatic EEG signal classification for epilepsy diagnosis with Relevance Vector Machines
Expert Syst. Appl.
DEAPa database for emotion analysis; using physiological signals
IEEE Trans. Affect. Comput.
Affective brain–computer interfaces as enabling technology for responsive psychiatric stimulation
Brain–Comput. Interfaces
An optimized ERP brain–computer interface based on facial expression changes
J. Neural Eng.
A brain‐computerbrain–computer interface for potential non-verbal facial communication based on EEG signals related to specific emotions
Front. Neurosci
The changing face of P300 BCIs: a comparison of stimulus changes in a P300 BCI involving faces, emotion, and movement
PLoS One
Cited by (92)
A review on semi-supervised learning for EEG-based emotion recognition
2024, Information FusionDesign of subject independent 3D VAD emotion detection system using EEG signals and machine learning algorithms
2023, Biomedical Signal Processing and ControlStochastic weight averaging enhanced temporal convolution network for EEG-based emotion recognition
2023, Biomedical Signal Processing and ControlAutomated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals
2021, Computers in Biology and MedicineJoint Class Learning with Self Similarity Projection for EEG Emotion Recognition
2024, ACM International Conference Proceeding SeriesPS-VTS: particle swarm with visit table strategy for automated emotion recognition with EEG signals
2023, Health Information Science and Systems
Rishabh Gupta received his B.Tech degree from the VIT University, India, in 2011, and the M.Sc. degree from University of Warwick, U.K., in 2012, both in biomedical engineering. He is currently following the Ph.D. program in telecommunications at INRS in Montreal, Canada. His research interests include machine learning, signal processing, human factors engineering, and UX evaluation using physiological tools.
Khalil R. Laghari is currently serving as Senior Advisor UX at Alberta Health Services Canada. Previously, he was Postdoctoral research fellow at INRS Montreal Canada. His research interests include human factors engineering, usability, UX testing, and physiological evaluation of Multimedia and web, based on eye tracking and EEG tools.
Tiago H. Falk received the B.Sc. degree from the Federal University of Pernambuco, Brazil, in 2002, and the MSc and Ph.D. degrees from Queens University, Canada, in 2005 and 2008, respectively, all in electrical engineering. From 2009–2010 he was an NSERC Postdoctoral Fellow at Holland–Bloorview Kids Rehabilitation Hospital, affiliated with the University of Toronto. Since 2010, he has been with the Institut National de la Recherche Scientifique (INRS) in Montreal, Canada where he heads the Multimodal Signal Analysis and Enhancement (MuSAE) Laboratory. His research interests include multimedia/biomedical signal analysis and enhancement, pattern recognition, and their interplay in the development of biologically inspired technologies.