Multimodal Emotion Interaction and Visualization Platform
Pages 9368 - 9370
Abstract
In this paper, we present a multimodal emotion analysis platform, which can flexibly capture, detect and analyze the emotions of video object with multiple modalities under different situations, including offline and online application scenarios. This system can visualize the dynamic effects of different types of emotions from both multimodal and unimodal circumstances. The presented emotion analysis results show instant and time series states in both specific modality and multiple modalities. Our system fills the current research and application gaps in multimodal emotion analysis with an interactive interface. Notably, the constructed system can adaptively process pre-recorded video clips as well as collected real-world data with excellent practicality and interactivity.
References
[1]
Carlos Busso, Murtaza Bulut, ChiChun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42, 4, 335--339.
[2]
Dou Hu, Lingwei Wei, and Xiaoyong Huai. 2021. DialogueCRN: contextual reasoning networks for emotion recognition in conversations. In Proceedings of ACL, 7042--7052.
[3]
Yingjian Li, Zheng Zhang, Bingzhi Chen, Guangming Lu, and David Zhang. 2023. Deep margin-sensitive representation learning for cross-domain facial expression recognition. IEEE TMM, 25, 1359--1373.
[4]
Yong Li, Yuanzhi Wang, and Zhen Cui. 2023. Decoupled multimodal distilling for emotion recognition. In Proceedings of IEEE CVPR, 6631--6640.
[5]
Silero Team. 2021. Silero models: pre-trained enterprise-grade stt / tts models and benchmarks. https://github.com/snakers4/silero-models. (2021).
[6]
Silero Team. 2021. Silero vad: pre-trained enterprise-grade voice activity detec-tor (vad), number detector and language classifier. https://github.com/snakers 4/silero-vad. (2021).
[7]
Heqing Zou, Yuke Si, Chen Chen, Deepu Rajan, and Eng Siong Chng. 2022. Speech emotion recognition with co-attention based multi-level acoustic information. In IEEE ICASSP, 7367--7371
Index Terms
- Multimodal Emotion Interaction and Visualization Platform
Recommendations
Feature-Enhanced Multimodal Interaction model for emotion recognition in conversation
AbstractMultimodal Emotion Recognition in Conversation (MERC) aims to identify the emotional state of a speaker who expresses their opinions through text, vision, and audio information during conversations. MERC enables intelligent machines to exhibit ...
Highlights- The multimodal feature-enhanced module fully mines the comprehensive multimodal features, which include semantic and emotional features of text, vision, and audio.
- The incremental transformer dialogue module models the dialogue process,...
Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System
ACAI '22: Proceedings of the 2022 5th International Conference on Algorithms, Computing and Artificial IntelligenceEmotion recognition has been well researched in mono-modality in the past decade. However, people express their emotion or feelings naturally via more than one modalities like voice, facial expressions, text, and behaviors. In this paper, we propose a ...
On Multimodal Emotion Recognition for Human-Chatbot Interaction in the Wild
ICMI '24: Proceedings of the 26th International Conference on Multimodal InteractionThe field of natural language generation is swiftly evolving, giving rise to powerful conversational characters for use in different applications such as entertainment, education, and healthcare. A central aspect of these applications is providing ...
Comments
Information & Contributors
Information
Published In
October 2023
9913 pages
ISBN:9798400701085
DOI:10.1145/3581783
- General Chairs:
- Abdulmotaleb El Saddik,
- Tao Mei,
- Rita Cucchiara,
- Program Chairs:
- Marco Bertini,
- Diana Patricia Tobon Vallejo,
- Pradeep K. Atrey,
- M. Shamim Hossain
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2023
Check for updates
Author Tags
Qualifiers
- Demonstration
Funding Sources
- Guangdong Natural Science Foundation
- National Key Research and Development Program of China
- Shenzhen Science and Technology Program
Conference
MM '23
Sponsor:
MM '23: The 31st ACM International Conference on Multimedia
October 29 - November 3, 2023
Ottawa ON, Canada
Acceptance Rates
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 287Total Downloads
- Downloads (Last 12 months)187
- Downloads (Last 6 weeks)13
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in