short-paper

A Quality Adaptive Multimodal Affect Recognition System for User-Centric Multimedia Indexing

Authors:

Mojtaba Khomami Abadi,

Jesús Alejandro Cárdenes Cabré,

Fabio Morreale,

Nicu SebeAuthors Info & Claims

ICMR '16: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval

Pages 317 - 320

https://doi.org/10.1145/2911996.2912059

Published: 06 June 2016 Publication History

Abstract

The recent increase in interest for online multimedia streaming platforms has availed massive amounts of multimedia information that need to be indexed to be searchable and retrievable. User-centric implicit affective indexing employing emotion detection based on psycho-physiological signals, such as electrocardiography (ECG), galvanic skin response (GSR), electroencephalography (EEG) and face tracking, has recently gained attention. However, real world psycho-physiological signals obtained from wearable devices and facial trackers are contaminated by various noise sources that can result in spurious emotion detection. Therefore, in this paper we propose the development of psycho-physiological signal quality estimators for unimodal affect recognition systems. The presented systems perform adequately in classifying users affect however, they resulted in high failure rates due to rejection of bad quality samples. Thus, to reduce the affect recognition failure rate, a quality adaptive multimodal fusion scheme is proposed. The proposed scheme yields no failure, while at the same time classify the users' arousal/valence and liking with significantly above chance weighted F1-scores in a cross-user experiment. Another finding of this study is that head movements encode liking perception of users in response to music snippets. This work also includes the release of the employed dataset including psycho-physiological signals, their quality annotations, and users' affective self-assessments.

References

[1]

M. Esterman, B. J. Tamber-Rosenau, Y.-C. Chiu, and S. Yantis. Avoiding non-independence in fmri data analysis: leave one subject out. Neuroimage, 50(2):572--576, 2010.

[2]

R. Gupta, K. ur Rehman Laghari, and T. H. Falk. Relevance vector classifier decision fusion and EEG graph-theoretic features for automatic affective state characterization. Neurocomputing, 174(PB):875--884, Jan. 2016.

Digital Library

[3]

A. Hanjalic and L.-Q. Xu. Affective video content representation and modeling. IEEE Transactions on Multimedia, 7(1):143--154, 2005.

Digital Library

[4]

M. Khomami Abadi, S. M. Kia, R. Subramanian, P. Avesani, and N. Sebe. User-centric affective video tagging from MEG and peripheral physiological responses. In ACII, 2013.

Digital Library

[5]

M. Khomami Abadi, J. Staiano, A. Cappelletti, M. Zancanaro, and N. Sebe. Multimodal engagement classification for affective cinema. In ACII, 2013.

Digital Library

[6]

M. Khomami Abadi, R. Subramanian, S. M. Kia, P. Avesani, I. Patras, and N. Sebe. Decaf: MEG-based multimodal database for decoding affective physiological responses. IEEE Transactions on Affective Computing, 6(3):209--222, July 2015.

Digital Library

[7]

J. Kim and E. André. Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(12):2067--2083, Dec 2008.

Digital Library

[8]

S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras. Deap: A database for emotion analysis ;using physiological signals. IEEE Transactions on Affective Computing, 3(1):18--31, Jan 2012.

Digital Library

[9]

S. Koelstra and I. Patras. Fusion of facial expressions andEEG\ for implicit affective tagging. Image and Vision Computing, 31(2):164 -- 174, 2013.

Digital Library

[10]

F. Morreale, R. Masu, and A. De Angeli. Robin: an algorithmic composer for interactive scenarios. Proceedings of 10th Sound and Music Comp., 2013.

[11]

M. Pantic and A. Vinciarelli. Implicit human-centered tagging {social sciences}. IEEE Signal Processing Magazine, 26(6):173--180, November 2009.

[12]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825--2830, Nov. 2011.

Digital Library

[13]

T. Penzel, B. Kemp, G. Klosch, A. Schlogl, J. Hasan, A. Varri, and I. Korhonen. Acquisition of biomedical signals databases. IEEE Engineering in Medicine and Biology Magazine, 20(3):25--32, May 2001.

[14]

M.-K. Shan, F.-F. Kuo, M.-F. Chiang, and S.-Y. Lee. Emotion-based music recommendation by affinity discovery from film music. Expert Systems with Applications, 2009.

Digital Library

[15]

M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic. A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing, 3(1):42--55, Jan 2012.

Digital Library

[16]

X. Xiong and F. De la Torre. Supervised descent method and its applications to face alignment. In CVPR, 2013.

Digital Library

[17]

Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1):39--58, Jan 2009.

Digital Library

Cited By

Pan BHirota KJia ZDai Y(2023)A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methodsNeurocomputing10.1016/j.neucom.2023.126866561:COnline publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1016/j.neucom.2023.126866
Chen XXie HLi ZCheng GLeng MWang F(2023)Information fusion and artificial intelligence for smart healthcareInformation Processing and Management: an International Journal10.1016/j.ipm.2022.10311360:1Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.ipm.2022.103113
Jemioło PStorman DMamica MSzymkowski MŻabicka WWojtaszek-Główka MLigęza A(2022)Datasets for Automated Affect and Emotion Recognition from Cardiovascular Signals Using Artificial Intelligence— A Systematic ReviewSensors10.3390/s2207253822:7(2538)Online publication date: 25-Mar-2022
https://doi.org/10.3390/s22072538
Show More Cited By

Index Terms

Recommendations

Implicit Emotion Communication: EEG Classification and Haptic Feedback

Today, ubiquitous digital communication systems do not have an intuitive, natural way of communicating emotion, which, in turn, affects the degree to which humans can emotionally connect and interact with one another. To address this problem, a more ...
Bimodal HCI-related affect recognition
ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

Perhaps the most fundamental application of affective computing will be Human-Computer Interaction (HCI) in which the computer should have the ability to detect and track the user's affective states, and make corresponding feedback. The human multi-...
Emotion Recognition Using Physiological Signals
MIDI '15: Proceedings of the Mulitimedia, Interaction, Design and Innnovation

In this paper the problem of emotion recognition using physiological signals is presented. Firstly the problems with acquisition of physiological signals related to specific human emotions are described. It is not a trivial problem to elicit real ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMR '16: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval

June 2016

452 pages

ISBN:9781450343596

DOI:10.1145/2911996

General Chairs:
John R. Kender
Columbia University, USA
,
John R. Smith
IBM Research, USA
,
Program Chairs:
Jiebo Luo
University of Rochester, USA
,
Susanne Boll
University of Oldenburg, Germany
,
Winston Hsu
National Taiwan University, Taiwan

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Sensaura Inc.

Conference

ICMR'16

Sponsor:

SIGMM

ICMR'16: International Conference on Multimedia Retrieval

June 6 - 9, 2016

New York, New York, USA

Acceptance Rates

ICMR '16 Paper Acceptance Rate 20 of 120 submissions, 17%;

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
348
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)3

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pan BHirota KJia ZDai Y(2023)A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methodsNeurocomputing10.1016/j.neucom.2023.126866561:COnline publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1016/j.neucom.2023.126866
Chen XXie HLi ZCheng GLeng MWang F(2023)Information fusion and artificial intelligence for smart healthcareInformation Processing and Management: an International Journal10.1016/j.ipm.2022.10311360:1Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.ipm.2022.103113
Jemioło PStorman DMamica MSzymkowski MŻabicka WWojtaszek-Główka MLigęza A(2022)Datasets for Automated Affect and Emotion Recognition from Cardiovascular Signals Using Artificial Intelligence— A Systematic ReviewSensors10.3390/s2207253822:7(2538)Online publication date: 25-Mar-2022
https://doi.org/10.3390/s22072538
Apicella AArpaia PIsgro FMastrati GMoccaldi N(2022)A Survey on EEG-Based Solutions for Emotion Recognition With a Low Number of ChannelsIEEE Access10.1109/ACCESS.2022.321984410(117411-117428)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3219844
Saganowski SKomoszyńska JBehnke MPerz BKunc DKlich BKaczmarek ŁKazienko P(2022)Emognition dataset: emotion recognition with self-reports, facial expressions, and physiology using wearablesScientific Data10.1038/s41597-022-01262-09:1Online publication date: 7-Apr-2022
https://doi.org/10.1038/s41597-022-01262-0
Zhu YTong MJiang ZZhong STian Q(2022)Hybrid feature-based analysis of video’s affective content using protagonist detectionExpert Systems with Applications: An International Journal10.1016/j.eswa.2019.03.017128:C(316-326)Online publication date: 20-Apr-2022
https://dl.acm.org/doi/10.1016/j.eswa.2019.03.017
Leung CKuang NSung V(2021)Organizational Policy Learning and Evaluation Using Monte Carlo MethodsEncyclopedia of Organizational Knowledge, Administration, and Technology10.4018/978-1-7998-3473-1.ch016(195-208)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-3473-1.ch016
Qi FYang XXu CShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Zero-shot Video Emotion Recognition via Multimodal Protagonist-aware Transformer NetworkProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475647(1074-1083)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475647
Khan ARashid U(2021)A Relational Aggregated Disjoint Multimedia Search Results Approach using Semantics2021 International Conference on Artificial Intelligence (ICAI)10.1109/ICAI52203.2021.9445229(62-67)Online publication date: 5-Apr-2021
https://doi.org/10.1109/ICAI52203.2021.9445229
Muhammad GAlshehri FKarray FSaddik AAlsulaiman MFalk T(2021)A comprehensive survey on multimodal medical signals fusion for smart healthcare systemsInformation Fusion10.1016/j.inffus.2021.06.00776:C(355-375)Online publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1016/j.inffus.2021.06.007
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten