MGTANet: Multi-Scale Guided Token Attention Network for Image Captioning

Recent studies have shown that grid features can play a similar role to region features in visual-language tasks. At the same time, transformer and their variants have demonstrated outstanding performance in image captioning. However, due to the fine-grained nature of grid features, self-attention tends to overly concentrate on a very few adjacent visual features, failing to establish effective connections with global visual features. This results in the loss of object context information, thereby impairing model performance. To address this issue, this paper proposes a Multi-Scale Guided Token Attention Network (MGTANet). Specifically, we introduce a guided token self-attention (GTSA) mechanism in the encoder. By employing a mixed-scale extraction method, we generate "guided tokens" and feed them into the self-attention, thus reducing the number of grid features in subsequent processing. These guided tokens have the ability to capture features at different scales, effectively reducing the grid quantity required in the self-attention process, thereby reducing the computational complexity of the model while still preserving crucial global information. Additionally, we have developed an Adaptive Gated Attention (AGA) mechanism that operates on the decoder. It can flexibly combine global visual information with local detail features and integrate them into the decoder to guide the generation of image captions. To substantiate our model's effectiveness, we conducted thorough experiments and visualizations on the MS-COCO dataset. The results illustrate that under similar experimental conditions, MGTANet yields competitive quality compared to existing models.