Elsevier

Neurocomputing

Volume 174, Part B, 22 January 2016, Pages 875-884
Neurocomputing

Relevance vector classifier decision fusion and EEG graph-theoretic features for automatic affective state characterization

https://doi.org/10.1016/j.neucom.2015.09.085Get rights and content

Abstract

Objective characterization of affective states during music clip watching could lead to disruptive new technologies, such as affective brain–computer interfaces, neuromarketing tools, and affective video tagging systems, to name a few. To date, the majority of existing systems have been developed based on analyzing electroencephalography (EEG) patterns in specific brain regions. With music videos, however, a complex interplay of information transfer exists between various brain regions. In this paper, we propose the use of EEG graph-theoretic analysis to characterize three emotional ratings: valence, arousal, and dominance, as well as the “liking” subjective rating. For characterization, graph-theoretic features were used to classify emotional states through support vector machine (SVM) and relevance vector machine (RVM) classifiers. Moreover, fusion schemes at feature and decision levels were also used to improve classification performance. In general, our study shows that the EEG graph-theoretic features are better suited for emotion classification than traditionally used EEG features such as, spectral power features (SPF) and asymmetry index (AI) features. The percentage increase in classification performance, represented by F1-scores, obtained using the proposed methodologies relative to the traditionally used SPF and AI features ranged from: Valence (7–9%), Arousal (3–8%), Dominance (5–6%) and Liking (4–7%). These findings suggest that an EEG graph-theoretical approach along with a robust classifier can better characterize human affective states evoked during music clip watching.

Introduction

Burgeoning research in the field of passive brain–computer interfaces (BCIs) has allowed for information about an individual׳s affective state to be measured continuously and effortlessly. Such so-called “affective BCIs” can play a significant role across a number of applications, including but not limited to: neuromarketing, serious gaming, quality-of-experience perception modelling, and more recently, affective video tagging/indexing [1], [2], [3], [4], [5], [6]. The amount of uploaded video content to online repositories is growing exponentially (e.g., on Youtube, an average 100 h of videos are uploaded per minute and over six billion hours of videos are watched per month), thus rendering manual tagging impossible and the video retrieval process ineffective. Affective BCIs, on the other hand, can allow for automated video tagging “on-the-fly” thus making video searches more effective, personal, and improve user׳s experience, with the technology.

Towards this end, it becomes necessary to extract implicit emotional tags for videos by assessing users׳ affective states while experiencing those videos. One approach towards recognizing users׳ affective states is to monitor their central nervous system activity using neuro-imaging tools, such as electroencephalography (EEG). EEG is a brain imaging modality that is well suited for this task as it is non-invasive and has very good temporal resolution (milliseconds). Spectral power features derived from several frequency bands, such as delta (δ: 1–4 Hz), theta (θ: 4–8 Hz), alpha (α: 8–13 Hz), beta (β: 13–30 Hz) and gamma (γ: 30–50 Hz), have been widely used as neurophysiological correlates of emotional activity [7]. Typically, alpha-power asymmetry is used as an indicator of emotional states. The measure is computed from spectral power differences between symmetric electrode pairs placed over the two hemispheres of the brain [8], [9], [10]. Spectral features from other sub-bands from specific regions of the brain have also been reported to encode affective states, such as parietal theta-power [11], right parietal alpha-power [10], parietal beta-power asymmetry [12] and parietal gamma-power [13], [14]. These studies provide evidence towards the feasibility of using features derived from EEG for affective state characterization.

The next step, however, is to develop classifiers to solve the emotion classification problem using the identified features. Previous studies have leveraged spectral power features together with support vector machines (SVM) to classify users׳ anger, joy, sadness and pleasantness with a classification accuracy of around 82% [15]. SVMs, in fact, are among the most widely used state-of-the-art emotion classification techniques [16], [17]. These have also been used with discrete wavelet transform (DWT) features, achieving an accuracy of around 81% [18]. In [19], in turn, authors used relative power changes for EEG along with a Bayesian network to predict emotional states. In [20], authors reported an accuracy of 73% for classifying emotional states using asymmetry features implemented with SVM classifier. Also, [21] reported an average accuracy of 63% using EEG time–frequency and mutual information features along with SVM classifier. Other studies which have used similar features and classifiers are reported in Table 1. In the table the following abbreviations are used: DT-CWPT: dual tree complex wavelet packet transform, SPF: spectral power features, AI: asymmetry index and CSP: common spatial pattern.

During the time course of watching a music video, however, several parts of the brain are activated to process and integrate the auditory and visual streams, as well as to evaluate emotional content via attentional and context updating mechanisms, which in turn can be influenced by different associative, imagery and memory processes [26], [27]. Generally, while experiencing a music video, the sensory information is processed in a segregated manner [28], [29], while the affective and cognitive processing mostly integrate processed information from sensory areas as well as among themselves [30]. In the context of brain networks, functional segregation refers to the network׳s ability to process specialized information in densely interconnected regions. On the other hand, functional integration refers to their ability to combine this specialized information from widely separated regions [31]. As can be seen, affective state characterization is comprised of a complex flow and interplay of information between brain regions that may span different EEG frequency bands. As such, it is expected that improved affective state characterization can be achieved by mapping the flow of information through the various neuronal networks involved and by computing their properties.

Having this said, it is hypothesized that changes in emotional content of the watched music clips would elicit changes in the information flow processes inside the brain, which, to some extent, would be captured in the recorded EEG activity by the electrodes placed over various brain regions. Thus, a functional connectivity analysis of the recorded data will reveal neural correlates of the viewers׳ affective states which can be used to solve the emotion classification problem. Generally, functional connectivity analysis describes the strength of the interactions between various regions of the brain. Commonly, graph-theory tools are used to characterize the topographical properties of the complex neuronal networks involved [32]. Using various neuroimaging modalities, recent studies have utilized graph-theoretical insights to analyze the emerging functional connectivity patterns, for e.g., emotions induced from visual stimuli [33], intelligence [34], and disease characterization [35]. For the present study, we propose using the graph-theoretical features to address the emotion classification problem using the well-known support vector machine (SVM) [36] and relevance vector machine (RVM) [37] classifiers. Moreover, we also propose using feature fusion and decision fusion techniques to improve the classification performance. It is hoped that the obtained insights can be used by the community to further advance affective BCI technologies and develop innovative applications, such as automated affective tagging/indexing of videos.

Section snippets

Database for emotion analysis using physiological signals (DEAP)

The pre-processed EEG and subjective data used in the present study were obtained from the publicly available “database for emotion analysis using physiological signals (DEAP)” [22]. Here, only a brief description of the data is presented; the interested reader is referred to [22] for more details. Thirty two healthy participants (50% females, average age=26.9 years) were recruited and consented to participate in the study. Thirty-two channel EEG data were recorded using a Biosemi ActiveTwo

Results

Table 3 reports the F1-score and classification accuracy for the classifier corresponding to the frequency sub-band with maximum F1-score chosen amongst the 10 individual classifiers for the arousal, valence, dominance and liking categories. The computed F1-scores were then tested for significance against a random chance level of 50% using an independent one-sample t-test, as proposed in [22]. It can be observed that graph, spectral power and AI features perform significantly better than chance

Discussion

From Table 3, it can be observed that graph, spectral power and AI features perform significantly better than chance in classifying users׳ emotional states, thus suggesting their utility in affect classification. Also, it can be said that the classifiers developed using the graph theoretical treatment of EEG data produced significantly (p<0.05) better classification metrics for emotional dimensions, specifically for valence and arousal, as compared to the traditionally used spectral power and

Conclusion

In this work, we have explored the use of EEG-based graph theoretical tools for affect recognition in audio–video stimuli. Several such metrics were shown to reliably discriminate between low and high levels of subjective affective dimensions. These findings suggest that objective affective characterisation is possible using the graph theoretical tools, and could lead to disruptive new technologies, such as affective BCIs. However, features which provide directional information and, are more

Acknowledgment

This work was funded by the Quebec Ministry of Economic Development, Innovation and Export (MDEIE – Ministere du Developpement Economique, de l׳Innovation et de l׳Exportation) (grant number MDEIE PSR-SIIRI-687) and the Natural Sciences and Engineering Research Council of Canada (NSERC) (grant number 402237).

Rishabh Gupta received his B.Tech degree from the VIT University, India, in 2011, and the M.Sc. degree from University of Warwick, U.K., in 2012, both in biomedical engineering. He is currently following the Ph.D. program in telecommunications at INRS in Montreal, Canada. His research interests include machine learning, signal processing, human factors engineering, and UX evaluation using physiological tools.

References (57)

  • A. Schirmer et al.

    Beyond the right hemispherebrain mechanisms mediating vocal emotional processing

    Trends Cognit. Sci.

    (2006)
  • G. Tononi et al.

    Complexity and coherencyintegrating information in the brain

    Trends Cognit. Sci.

    (1998)
  • C. Lithari

    How does the metric choice affect brain functional connectivity networks?

    Biomed. Signal Process. Control

    (2012)
  • C.J. Stam

    Use of magnetoencephalography (MEG) to study functional brain networks in neurodegenerative disorders

    J. Neurol. Sci.

    (2010)
  • M.M. Bradley et al.

    Measuring emotionthe self-assessment manikin and the semantic differential

    J. Behav. Therapy Exp. Psychiatr.

    (1994)
  • S.L. Bressler et al.

    Cortical coordination dynamics and cognition

    Trends Cognit. Sci.

    (2001)
  • J. Dauwels et al.

    A comparative study of synchrony measures for the early diagnosis of Alzheimer׳s disease based on EEG

    NeuroImage

    (2010)
  • I. Daly

    Neural correlates of emotional responses to musican EEG study

    Neurosci. Lett.

    (2014)
  • S. Weiss et al.

    The contribution of EEG coherence to the investigation of language

    Brain Lang.

    (2003)
  • M. Rubinov et al.

    Complex network measures of brain connectivityuses and interpretations

    NeuroImage

    (2010)
  • B.J. Baars

    Global workspace theory of consciousnesstoward a cognitive neuroscience of human experience

    Progr. Brain Res.

    (2005)
  • C.A.M. Lima et al.

    Automatic EEG signal classification for epilepsy diagnosis with Relevance Vector Machines

    Expert Syst. Appl.

    (2009)
  • S. Koelstra

    DEAPa database for emotion analysis; using physiological signals

    IEEE Trans. Affect. Comput.

    (2012)
  • J.-N. Antons, et al., Cognitive, affective, and experience correlates of speech quality perception in complex listening...
  • A.S. Widge et al.

    Affective brain–computer interfaces as enabling technology for responsive psychiatric stimulation

    Brain–Comput. Interfaces

    (2014)
  • J. Jin

    An optimized ERP brain–computer interface based on facial expression changes

    J. Neural Eng.

    (2014)
  • K. Kashihara

    A brain‐computerbrain–computer interface for potential non-verbal facial communication based on EEG signals related to specific emotions

    Front. Neurosci

    (2014)
  • J. Jin

    The changing face of P300 BCIs: a comparison of stimulus changes in a P300 BCI involving faces, emotion, and movement

    PLoS One

    (2012)
  • Cited by (92)

    View all citing articles on Scopus

    Rishabh Gupta received his B.Tech degree from the VIT University, India, in 2011, and the M.Sc. degree from University of Warwick, U.K., in 2012, both in biomedical engineering. He is currently following the Ph.D. program in telecommunications at INRS in Montreal, Canada. His research interests include machine learning, signal processing, human factors engineering, and UX evaluation using physiological tools.

    Khalil R. Laghari is currently serving as Senior Advisor UX at Alberta Health Services Canada. Previously, he was Postdoctoral research fellow at INRS Montreal Canada. His research interests include human factors engineering, usability, UX testing, and physiological evaluation of Multimedia and web, based on eye tracking and EEG tools.

    Tiago H. Falk received the B.Sc. degree from the Federal University of Pernambuco, Brazil, in 2002, and the MSc and Ph.D. degrees from Queens University, Canada, in 2005 and 2008, respectively, all in electrical engineering. From 2009–2010 he was an NSERC Postdoctoral Fellow at Holland–Bloorview Kids Rehabilitation Hospital, affiliated with the University of Toronto. Since 2010, he has been with the Institut National de la Recherche Scientifique (INRS) in Montreal, Canada where he heads the Multimodal Signal Analysis and Enhancement (MuSAE) Laboratory. His research interests include multimedia/biomedical signal analysis and enhancement, pattern recognition, and their interplay in the development of biologically inspired technologies.

    View full text