Zero-Shot Video Moment Retrieval Using BLIP-Based Models

Abstract

Video Moment Retrieval (VMR) is a challenging task at the intersection of vision and language, with the goal to retrieve relevant moments from videos corresponding to natural language queries. State-of-the-art approaches for VMR often rely on large amounts of training data including frame-level saliency annotations, weakly supervised pre-training on speech captions, and signals from additional modalities such as audio, which can be limiting in practical scenarios. Moreover, most of these approaches make use of pre-trained spatio-temporal backbones for aggregating temporal features across multiple frames, which incurs significant training and inference costs. To address these limitations, we propose a zero-shot approach with sparse frame-sampling strategies that does not rely on additional modalities and performs well with feature extraction from just individual frames. Our approach uses Bootstrapped Language-Image Pre-training based models (BLIP/BLIP-2), which have been shown to be effective for various downstream vision-language tasks, even in zero-shot settings. We show that such models can be easily repurposed as effective, off-the-shelf feature extractors for VMR. On the QVHighlights benchmark for VMR, our approach outperforms both zero-shot approaches and supervised approaches (without saliency score annotations) by at least $25\%$ and $21\%$ respectively, on all metrics. Further, we also show that our approach is comparable to state-of-the-art supervised approaches trained on saliency score annotations and additional modalities, with a gap of at most $7\%$ across all metrics.