MoPE: Mixture of Pooling Experts Framework for Image-Text Retrieval

Abstract

Image-text retrieval is a fundamental and crucial task in the field of multimodal interaction, which assists internet users in retrieving the required visual and textual information conveniently. The dominant method for image-text retrieval aims to learn a visual semantic embedding space such that related visual and textual data are close to each other. Recent research focuses on designing sophisticated pooling strategies to better aggregate visual and textual features into holistic embeddings. However, existing methods often use the same pooling operator for the whole dataset, ignoring that samples with diverse intra-modality relationships require pooling operators trained with different parameters. To tackle this issue, we propose a novel Mixture of Pooling Experts (MoPE) framework, which combines multiple pooling operators to aggregate features for different data subsets. Specifically, we introduce a novel route gating strategy in combination with an aggregation expert module to dynamically learn diverse pooling experts for samples in different data subsets. Moreover, to fully exploit the intra-modality relationships, we develop a specialized router with a self-attention gate mechanism to direct each sample to the proper pooling expert. Extensive experiments conducted on two widely used benchmark datasets, namely Flickr30K and MS-COCO, demonstrate the superiority of our method over several state-of-the-art methods.