Modeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation | IEEE Journals & Magazine | IEEE Xplore

Modeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation


Abstract:

Approximating the uncertainty of an emotional AI agent is crucial for improving the reliability of such agents and facilitating human-in-the-loop solutions, especially in...Show More

Abstract:

Approximating the uncertainty of an emotional AI agent is crucial for improving the reliability of such agents and facilitating human-in-the-loop solutions, especially in critical scenarios. However, none of the existing systems for emotion recognition in conversation (ERC) has attempted to estimate the uncertainty of their predictions. In this article, we present HU-Dialogue, which models hierarchical uncertainty for the ERC task. We perturb contextual attention weight values with source-adaptive noises within each modality, as a regularization scheme to model context-level uncertainty and adapt the Bayesian deep learning method to the capsule-based prediction layer to model modality-level uncertainty. Furthermore, a weight-sharing triplet structure with conditional layer normalization is introduced to detect both invariance and equivariance among modalities for ERC. We provide a detailed empirical analysis for extensive experiments, which shows that our model outperforms previous state-of-the-art methods on three popular multimodal ERC datasets.
Published in: IEEE Transactions on Cybernetics ( Volume: 54, Issue: 1, January 2024)
Page(s): 187 - 198
Date of Publication: 12 July 2022

ISSN Information:

PubMed ID: 35820006

Funding Agency:


References

References is not available for this document.