Gated Multi-modal Fusion with Cross-modal Contrastive Learning for Video Question Answering

Abstract

Video Question Answering (VideoQA) is a challenging task that requires the model to understand the complex nature of video data and the variety of questions that can be asked about them. Existing approaches often suffer from the problem of ambiguous answer candidates with low relevance to the visual and auditory part of the video, which limits the performance of VideoQA systems. In this paper, we introduce a novel approach that leverages multi-modal fusion and cross-modal contrastive learning to utilize multi-modal information and enhance the relevance of answer candidates in VideoQA. First, we introduce a gated multi-modal fusion network that learns to combine different modalities such as visual and speech based on their relevance to the question to enrich the representations of video and improve the accuracy of finding the correct answer. Second, we introduce cross-modal contrastive learning to increase the similarity between positive example pairs (i.e., correct answers and corresponding video clips) while decreasing the similarity between negative example pairs (i.e., incorrect answers and unpaired video clips). Specifically, we use three-way contrastive learning between answer and video frame, answer and audio, answer and cross-modal features. Our proposed approach is evaluated on two benchmark audio-aware VideoQA datasets, including AVQA and Music-AVQA, and compared to several state-of-the-art methods. The results show that our approach significantly improves the performance of VideoQA, achieving new state-of-the-art results on these benchmarks.

C. Lyu and W. Li—Equal contribution.