Reproducibility Companion Paper of "MMSF: A Multimodal Sentiment-Fused Method to Recognize Video Speaking Style"
Authors: 
Fan Yu, Beibei Zhang, Yaqun Fang, Jia Bei, Tongwei Ren, Jiyi Li, Luca RossettoAuthors Info & Claims
Fan Yu, Beibei Zhang, Yaqun Fang, Jia Bei, Tongwei Ren, Jiyi Li, Luca RossettoAuthors Info & ClaimsPages 1232 - 1235
Abstract
To support the replication of "MMSF: A Multimodal Sentiment-Fused Method to Recognize Video Speaking Style", which was presented at ICMR'23, this companion paper provides the details of the artifacts. Speaking style recognition is aimed at recognizing the styles of conversations, which provides a fine-grained description about talking. In the original paper, we proposed a novel multimodal sentiment-fused method, MMSF, which extracts and integrates visual, audio and textual features of videos and introduced sentiment in MMSF with cross-attention mechanism to enhance the video feature to recognize speaking styles. In this paper, we explain the details of the implement code and the dataset used for experiments.
References
[1]
Edward Fish, Jon Weinbren, and Andrew Gilbert. 2022. Two-Stream Transformer Architecture for Long Form Video Understanding. In British Machine Vision Conference.
[2]
Md Mohaiminul Islam and Gedas Bertasius. 2022. Long movie clip classification with state-space video models. In European Conference on Computer Vision. Springer, 87--104.
[3]
Huisheng Mao, Ziqi Yuan, Hua Xu, Wenmeng Yu, Yihe Liu, and Kai Gao. 2022. M-sena: An integrated platform for multimodal sentiment analysis. arXiv preprint arXiv:2203.12441 (2022).
[4]
Chao-Yuan Wu and Philipp Krahenbuhl. 2021. Towards long-form video understanding. In IEEE Conference on Computer Vision and Pattern Recognition. 1884--1894.
[5]
Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Annual Meeting of the Association for Computational Linguistics. 3718--3727.
[6]
Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal Sentiment Analysis. In AAAI Conference on Artificial Intelligence, Vol. 35. 10790--10797.
[7]
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 (2016).
[8]
Beibei Zhang, Yaqun Fang, Fan Yu, Jia Bei, and Tongwei Ren. 2023. MMSF: A multimodal sentiment-fused method to recognize video speaking style. In ACM International Conference on Multimedia Retrieval. 289--297.
Index Terms
- Reproducibility Companion Paper of "MMSF: A Multimodal Sentiment-Fused Method to Recognize Video Speaking Style"
Recommendations
MMSF: A Multimodal Sentiment-Fused Method to Recognize Video Speaking Style
ICMR '23: Proceedings of the 2023 ACM International Conference on Multimedia RetrievalAs talking takes a large proportion of human lives, it is necessary to perform deeper understanding of human conversations. Speaking style recognition is aimed at recognizing the styles of conversations, which provides a fine-grained description about ...
Attention and Engagement Aware Multimodal Conversational Systems
ICMI '15: Proceedings of the 2015 ACM on International Conference on Multimodal InteractionDespite their ability to complete certain tasks, dialog systems still suffer from poor adaptation to users' engagement and attention. We observe human behaviors in different conversational settings to understand human communication dynamics and then ...
TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis
WWW '20: Proceedings of The Web Conference 2020Multimodal sentiment analysis is an important research area that predicts speaker’s sentiment tendency through features extracted from textual, visual and acoustic modalities. The central challenge is the fusion method of the multimodal information. A ...
Comments
Information & Contributors
Information
Published In
May 2024
1379 pages
ISBN:9798400706196
DOI:10.1145/3652583
- General Chairs:
- Cathal Gurrin,
- Rachada Kongkachandra,
- Klaus Schoeffmann,
- Program Chairs:
- Duc-Tien Dang-Nguyen,
- Luca Rossetto,
- Shin'ichi Satoh,
- Liting Zhou
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 June 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
- National Natural Science Foundation of China
- the Fundamental Research Funds for the Central Universities
- the Collaborative Innovation Center of Novel Software Technology and Industrialization
Conference
Acceptance Rates
Overall Acceptance Rate 254 of 830 submissions, 31%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 62Total Downloads
- Downloads (Last 12 months)62
- Downloads (Last 6 weeks)16
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in