Loading [MathJax]/extensions/MathMenu.js
A Novel Decoupled Prototype Completion Network for Incomplete Multimodal Emotion Recognition | IEEE Conference Publication | IEEE Xplore

A Novel Decoupled Prototype Completion Network for Incomplete Multimodal Emotion Recognition


Abstract:

Reconstructing missing modality based on available modalities is widely used to address inevitable modality-missing for Multimodal Emotion Recognition (MER). However, due...Show More

Abstract:

Reconstructing missing modality based on available modalities is widely used to address inevitable modality-missing for Multimodal Emotion Recognition (MER). However, due to explicit distribution gap across heterogeneous modalities, they fail to guarantee the consistency between the reconstructed data and the ground truth. To mitigate this problem, we propose a novel method to restore the missing modality using its weighted prototypes rather than other modalities. Specifically, prototypes of different classes of missing modality are used to encapsulate its representative knowledge. Then sample-to-prototype affinity measuring class similarity is used as weights to combine these prototypes for reconstruction, thereby effectively restoring the distribution-consistent modality. Furthermore, to improve the efficacy of prototype-based completion under seriously missing, we devise an adaptive knowledge distillation from the strong modality to the weaker ones. This reinforces the representation ability of weak modality features. Extensive experiments on CMU-MOSI and IEMOCAP datasets demonstrate the superiority of our method.
Date of Conference: 15-19 July 2024
Date Added to IEEE Xplore: 30 September 2024
ISBN Information:

ISSN Information:

Conference Location: Niagara Falls, ON, Canada

Funding Agency:


Contact IEEE to Subscribe

References

References is not available for this document.