Adaptive Multi-granularity Aggregation Transformer for Image Captioning

Abstract

In image captioning, images often contain complex scenes where features at a single granularity level fail to capture all the visual information. For instance, grid features of an image provide spatial details but lack an understanding of semantic objects. Therefore, it is necessary to fuse the multi-granularity features of an image for a comprehensive representation. In this paper, we propose an adaptive multi-granularity aggregation transformer that integrates grid, region and global features of image. In contrast to previous approaches that rely on single-feature or two-feature representation, our approach integrates features of different granularity levels, which overcomes the incompleteness of traditional visual information characterization. Specifically, we construct an encoder with a multi-granularity feature enhancement module that explores intrinsic relationships between different features to reduce the redundancy of feature representation. We also design a multi-granularity feature adaptive fusion module to adjust the attention of features at different scales, enhancing cross-modal inference ability. Experiments on the MSCOCO dataset demonstrate that our model achieves superior performance, with a CIDEr score of 138.6 on the “Karpathy” split, surpassing the state-of-the-art fusion model by 2.5 points.