short-paper

Tracking liking state in brain activity while watching multiple movies

Authors:

Naoto Terasawa,

Sakriani Sakti,

Satoshi NakamuraAuthors Info & Claims

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

Pages 321 - 325

https://doi.org/10.1145/3136755.3136772

Published: 03 November 2017 Publication History

Abstract

Emotion is a valuable information in various applications ranging from human-computer interaction to automated multimedia content delivery. Conventional methods to recognize emotion were based on speech prosody cues, facial expression, and body language. However, this information may not appear when people watch a movie. In recent years, some studies have started to use electroencephalogram (EEG) signals in recognizing emotion. But, the EEG data were entirely analyzed in each scene of movies for emotion classification. Thus, the detailed information of emotional state changes cannot be extracted. In this study, we utilize EEG to track affective state during watching multiple movies. Experiments were done by measuring continuous liking state during watching three types of movies, and then constructing subject dependent emotional state tracking model. We used support vector machine (SVM) as a classifier, and support vector regression (SVR) for regression. As a result, the best classification accuracy was 77.6%, and the best regression model achieved 0.645 of correlation coefficient between actual liking state and predicted liking state. These results demonstrate that continuous emotional state can be predicted by our EEG-based method.

References

[1]

K. Anderson, P. W. McOwan, “A real-time automated system for the recognition of human facial expression," IEEE transactions on systems, man, and cybernetics, part B: Cybernetics, vol. 36, no. 1, Feb 2006, pp. 96-105.

Digital Library

[2]

J. Ang, R. Dhillon, A. Krupski, E. Shriberg, and A. Stolcke, “Prosody-based automatic detection of annoyance and frustration in human-computer dialog," International conference on spoken language processing, 2002, pp. 2037-2039.

[3]

R. W. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: analysis of affective physiological state," IEEE transactions on pattern analysis Tracking Liking State in Brain Activity... ICMI’17, November 13–17, 2017, Glasgow, UK and machine intelligence, vol. 23, no. 10, Oct 2001, pp. 1175-1191.

Digital Library

[4]

R. N. Khushaba, L. Greenacre, S. Kodagoda, J. Louviere, S. Burke, and G. Dissanayake, “Choice modeling and the brain: A study on the Electroencephalogram (EEG) of preference," Expert systems with applications, vol. 39, no. 16, Nov 2012, pp. 12378-12388.

Digital Library

[5]

G. L. Ahem, G. E. Schwartz, “Differential lateralization for positive and negative emotion in the human brain: EEG spectral analysis," Neuropsychologia, vol. 23, no. 6, 1985, pp. 745-555.

[6]

D. Nie, X. W. Wang, L. C. Shi, and B. L. Lu, “EEG-based emotion recognition during watching movies," Proceedings of the 5th international IEEE EMBS conference on neural engineering, May 2011, pp. 667-670.

[7]

M. M. Bradley and P. J. Lang, “Measuring emotion: The self-assessment manikin and the semantic differenctial," Journal of behavior therapy and experimental psychiatry, vol. 25, no. 1, Mar 1994, pp. 49-59.

[8]

J. P. Dmochowski, M. A. Bezdek, B. P. Abelson, J. S. Johnson, E. H. Schumacher, and L. C. Parra, “Audience preferences are predicted by temporal reliability of neural processing," Nature communications, vol. 5, no. 4567, July 2014.

[9]

S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, L. Patras, “DEAP: A database for emotion analysis using physiological signals," IEEE transactions on affective computing, vol. 3, no. 1, 2012, pp. 18-31.

Digital Library

[10]

M. Soleymani, J. Lichtenauer, T. Pun, M. Pantic, “A multimodal database for affect recognition and lmplicit tagging," IEEE transactions on affective computing, vol. 3, no. 1, 2012, pp. 42-55.

Digital Library

[11]

M. Angeliki, K. Athanassios, W. Yun and N. Shrikanth, “Tracking changes in continuous emotion states using body language and prosodic cues," IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011, pp. 2288―2291.

[12]

P. Petta, C. Pelachaud, and R. Cowie, “Emotion-oriented systems: The humaine handbook," 2011, pp. 215-244.

Digital Library

[13]

D. R. Peryam and F. J. Pilgrim, “Hedonic scale method of measuring food preferences," Food technology, vol. 11, Sept 1957, pp. 9-14.

[14]

W. Klimesch, “EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis," Brain research reviews, vol. 29, no. 2-3, Apr 1999, pp. 169-195.

[15]

W. J. Ray and H. W. Cole, “EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes," Science, vol. 228, May 1985, pp. 750-752.

[16]

R. D. Lane, E. M. Reiman, G. L. Ahern, G. E. Schwartz, and R. J. Davidson, “Neuroanatomical correlates of happiness, sadness, and disgust," The American journal of psychiatry, vol. 154, no. 7, Jul 1997, pp. 926-933.

[17]

A.C. North and D. J. Hargreaves, “Liking arousal potential, and the emotions expressed by music," Scandinavian journal of psychology, vol. 38, no. 1, Mar 1997, pp. 45-53.

[18]

H. Kang, Y. Nam, and S. Choi, “Composite common spatial pattern for subjectto-subject transfer," IEEE signal processing letters, vol. 16, no. 8, Aug 2009, pp. 683-686.

Cited By

Yang QFeng SZhao TKalantari S(2024)Design with myself: A brain–computer interface design tool that predicts live emotion to enhance metacognitive monitoring of designersInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103229185(103229)Online publication date: May-2024
https://doi.org/10.1016/j.ijhcs.2024.103229
Yang QFeng SZhao TKalantari S(2023)Co-Design with Myself: A Brain-Computer Interface Design Tool that Predicts Live Emotion to Enhance Metacognitive Monitoring of DesignersExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585701(1-8)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585701
Putze FPutze SSagehorn MMicek CSolovey E(2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
https://dl.acm.org/doi/10.1145/3490554
Show More Cited By

Index Terms

Tracking liking state in brain activity while watching multiple movies
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science

Recommendations

Discriminating the brain activities for brain---computer interface applications through the optimal allocation-based approach

The translation of brain activities into signals in brain---computer interface (BCI) systems requires a robust and accurate classification to develop a communication system for motor disabled people. In BCIs, motor imagery (MI) tasks generate brain ...
A hybrid steady-state visual evoked response-based brain-computer interface with MEG and EEG
Abstract
While recent developments in electroencephalogram (EEG)-based brain-computer interfaces (BCIs) have enabled a bridge between the brain and external devices with relatively high communication speed, there is still room for improvement. Notably, ...
EEG signals classification for brain computer interfaces based on Gaussian process classifier
ICICS'09: Proceedings of the 7th international conference on Information, communications and signal processing

Classification of electroencephalogram (EEG) is a crucial issue for EEG-based brain computer interface (BCI) system. In this paper, the performances of the Gaussian process classifier (GPC) for three different categories of EEG signals, i.e. steady ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

November 2017

676 pages

ISBN:9781450355438

DOI:10.1145/3136755

General Chairs:
Edward Lank
University of Waterloo, Canada
,
Alessandro Vinciarelli
University of Glasgow, UK
,
Program Chairs:
Eve Hoggan
Aarhus University, Denmark
,
Sriram Subramanian
University of Sussex, UK
,
Stephen A. Brewster
University of Glasgow, UK

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

JSPS KAKENHI

Conference

ICMI '17

Sponsor:

SIGCHI

ICMI '17: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 13 - 17, 2017

Glasgow, UK

Acceptance Rates

ICMI '17 Paper Acceptance Rate 65 of 149 submissions, 44%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
199
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang QFeng SZhao TKalantari S(2024)Design with myself: A brain–computer interface design tool that predicts live emotion to enhance metacognitive monitoring of designersInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103229185(103229)Online publication date: May-2024
https://doi.org/10.1016/j.ijhcs.2024.103229
Yang QFeng SZhao TKalantari S(2023)Co-Design with Myself: A Brain-Computer Interface Design Tool that Predicts Live Emotion to Enhance Metacognitive Monitoring of DesignersExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585701(1-8)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585701
Putze FPutze SSagehorn MMicek CSolovey E(2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
https://dl.acm.org/doi/10.1145/3490554
Katada SOkada S(2022)Biosignal-based user-independent recognition of emotion and personality with importance weightingMultimedia Tools and Applications10.1007/s11042-022-12711-881:21(30219-30241)Online publication date: 5-Apr-2022
https://doi.org/10.1007/s11042-022-12711-8
Wang BHuang LLiu JDeng ZFu M(2020)A Support Vector Regression-Based Integrated Navigation Method for Underwater VehiclesIEEE Sensors Journal10.1109/JSEN.2020.298599820:15(8875-8883)Online publication date: 1-Aug-2020
https://doi.org/10.1109/JSEN.2020.2985998

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten