GA-SATIC: Semi-autoregressive Transformer Image Captioning Based on Geometric Attention

Image captioning is a cross-domain task involving image and natural language processing. Most of the current models follow an encoder-decoder architecture, where the encoder takes image feature vectors as input, and the decoder uses an autoregressive or non-autoregressive approach for decoding. However, most models extract the feature vectors of an image through region proposals of the object detector, without considering the relative spatial relationship between objects. When decoding, the autoregressive decoding method is adopted, that is, the next word is generated based on the generated word, and the generation is performed step by step, which will lead to high delay in the inference process. To solve this problem, scholars have proposed a non-autoregressive approach, which speeds up inference by generating all words in parallel, but reduces the quality of generated captioning. Aiming at the above problems, this paper proposes a semi-autoregressive Transformer model based on geometric attention. The encoder integrates the relative spatial relationship between the detected objects through geometric attention and appearance attention, so as to enhance spatial awareness; the decoder adopts a semi-autoregressive decoding method, which is serialized globally and paralleled locally, enabling the model to achieve a better trade-off between decoding speed and captioning accuracy. Extensive experiments and ablation studies on the MSCOCO dataset have shown that the model achieves better performance compared to state-of-the-art models.