We welcome you to the 5th Multimodal Sentiment Analysis Challenge and Workshop (MuSe 2024). MuSe 2024 is a half-day workshop in conjunction with ACM Multimedia 2024 in Melbourne, Australia. In its fifth edition, MuSe continues its tradition of offering novel multimodal sentiment analysis benchmark tasks within a unified challenge setting. This year, we invited participation in two distinct multimodal sub-challenges: 1) predicting 16 different social attributes of individuals and 2) detecting spontaneous, humorous communication in a cross-cultural context.
The challenge offers a stimulating environment for developing and evaluating novel approaches to multimedia processing and deep learning methods for automatic audio-visual and textual affect modeling, all within a common experimental framework. It is designed to appeal to various research communities, fostering interdisciplinary collaboration. Primarily, MuSe 2024 targets researchers in audio-visual emotion recognition, unimodal and multimodal representation learning, health informatics, and NLP-based emotion and sentiment analysis. All raw recordings, transcriptions, pre-extracted features, and baseline models are available on our website (www.muse-challenge.org).
Proceeding Downloads
The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition
- Shahin Amiriparian,
- Lukas Christ,
- Alexander Kathan,
- Maurice Gerczuk,
- Niklas Müller,
- Steffen Klug,
- Lukas Stappen,
- Andreas König,
- Erik Cambria,
- Björn W. Schuller,
- Simone Eulitz
The Multimodal Sentiment Analysis Challenge (MuSe) 2024 addresses two contemporary multimodal affect and sentiment analysis problems: In the Social Perception Sub-Challenge (MuSe-Perception), participants will predict 16 different social attributes of ...
MuSe '24: The 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception & Humor
The 5th Multimodal Sentiment Analysis Challenge (MuSe), a workshop in conjunction with ACM Multimedia '24, is focused on Multimodal Machine Learning in the domain of Affective Computing. Two different sub-challenges are proposed: Social Perception Sub-...
Modality Weights Based Fusion Model for Social Perception Prediction in Video, Audio, and Text
Social perception is a crucial psychological concept that explains how we understand and interpret others and their behaviors. It encompasses the complex process of discerning individual characteristics, intentions, and emotions, significantly ...
Larger Encoders, Smaller Regressors: Exploring Label Dimensionality Reduction and Multimodal Large Language Models as Feature Extractors for Predicting Social Perception
- Iván Martín-Fernández,
- Sergio Esteban-Romero,
- Jaime Bellver-Soler,
- Fernando Fernández-Martínez,
- Manuel Gil-Martín
Designing reliable automatic models for social perception can contribute to a better understanding of human behavior, enabling more trustworthy experiences in the multimedia on-line communication environment. However, predicting social attributes from ...
Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition
Automatic social perception recognition is a new task to mimic the measurement of human traits, which was previously done by humans via questionnaires. We evaluated unimodal and multimodal systems to predict agentive and communal traits from the LMU-ELP ...
Exploring Multimodal Approaches and Fusion Methods for CEO Social Attribute Prediction in 2024 MuSe-Perception
In this paper, we discuss the way we addressed the 2024 MuSe-Perception challenge, which aims to automatically recognize and quantify social attributes of CEOs using multimodal data. We investigated five different approaches: (1) optimizing a basic RNN ...
LLM-Driven Multimodal Fusion for Human Perception Analysis
- Sergio Esteban-Romero,
- Iván Martín-Fernández,
- Manuel Gil-Martín,
- David Griol-Barres,
- Zoraida Callejas-Carrión,
- Fernando Fernández-Martínez
The Multimodal Sentiment Analysis Challenge presents two distinct sub-challenges related to human perception characteristics. This paper focuses on the MUSE-PERCEPTION challenge, which aims to predict the perceptual attributes of CEOs from video data. ...
Social Perception Prediction for MuSe 2024: Joint Learning of Multiple Perceptions
- Zhuofan Wen,
- Hailiang Yao,
- Shun Chen,
- Haiyang Sun,
- Mingyu Xu,
- Licai Sun,
- Zheng Lian,
- Bin Liu,
- Fengyu Zhang,
- Siyuan Zhang,
- Jianhua Tao
In this paper, we present our unique method for the MuSe 2024 Perception sub-challenge. In the Perception sub-challenge, 21 labeled social perceptions data are given, 16 social perceptions are required to be predicted. Joint learning is crucial for our ...
Multimodal Humor Detection and Social Perception Prediction
The parallel audio-visual-text data contains vast amount of information. Thus it is essential to develop machine learning algorithms that can utilise them efficiently. In this work, we investigated unimodal and multimodal solutions for MuSe Humor and ...
AMTN: Attention-Enhanced Multimodal Temporal Network for Humor Detection
In this paper, we introduce the Attention-Enhanced Multimodal Temporal Network (AMTN) to address the MuSe 2024 Cross-Cultural Humor Detection Sub-Challenge (MuSe-Humor), which highlights the task of humor detection within a cross-cultural context, ...
DPP: A Dual-Phase Processing Method for Cross-Cultural Humor Detection
- Shun Chen,
- Hailiang Yao,
- Mingyu Xu,
- Zhuofan Wen,
- Haiyang Sun,
- Licai Sun,
- Zheng Lian,
- Bin Liu,
- Fengyu Zhang,
- Siyuan Zhang,
- Jianhua Tao
Multimodal humor detection has become an active research area in the field of artificial intelligence. This paper presents the solution for the MuSe-Humor sub-challenge of cross-cultural humor detection. The goal of the MuSe-Humor sub-challenge is to ...
Index Terms
- Proceedings of the 5th on Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor
Recommendations
MuSe '24: The 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception & Humor
MuSe'24: Proceedings of the 5th on Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and HumorThe 5th Multimodal Sentiment Analysis Challenge (MuSe), a workshop in conjunction with ACM Multimedia '24, is focused on Multimodal Machine Learning in the domain of Affective Computing. Two different sub-challenges are proposed: Social Perception Sub-...
Acceptance Rates
Year | Submitted | Accepted | Rate |
---|---|---|---|
MuSe' 22 | 17 | 14 | 82% |
Overall | 17 | 14 | 82% |