VD-SAN: Visual-Densely Semantic Attention Network for Image Caption Generation

Abstract

Recently, attribute has demonstrated its effectiveness in guiding image captioning system. However, most attributes based image captioning methods treat the attributes prediction task as a separate task and rely on a standalone stage to obtain the attributes for the given image, e.g., a pre-trained network like Fully Convolutional Neural Network (FCN) is usually adopted. Inherently, they ignore the correlation between the attribute prediction task and image representation extraction task, and at the same time increases the complexity of the image captioning system. In this paper, we aim to couple the attributes prediction stage and image representation extraction stage tightly and propose a novel and efficient image captioning framework called Visual-Densely Semantic Attention Network(VD-SAN). In particular, the whole captioning system consists of shared convolutional layers from Dense Convolutional Network (DenseNet), which are further split into a semantic attributes prediction branch and an image feature extraction branch, two semantic attention models, and a long short-term memory networks (LSTM) for caption generation. To evaluate the proposed architecture, we construct Flickr30K-ATT and MS-COCO-ATT datasets based on the original popular image caption datasets Flickr30K and MS COCO respectively, and each image from Flickr30K-ATT or MS-COCO-ATT is annotated with an attribute list in addition to the corresponding caption. Empirical results demonstrate that our captioning system can achieve significant improvements over state-of-the-art approaches.

Introduction

Image captioning, which attempts to generate captions (e.g., usually in English) to describe images or videos automatically, has become a hot topic due to recent advances in both computer vision and machine translation communities, especially on deep learning.

Recent work in this area adopts the famous encoder-decoder framework. The encoder-decoder framework was originally introduced for the sequence-to-sequence problem in machine translation society, where a recurrent neural network (RNN) is used to encode source sentence, and another RNN is employed to predict the target sentence. Such a framework has found applications in many areas like speech recognition [1], text recognition [2] etc. The sequence-to-sequence problem is also somehow analogous to image captioning problem. However, instead of translating a source sentence, image caption aims at “translating” the visual input into a sentence. Therefore, a convolutional neural network (CNN) is adopted to encode the image.

Most work uses the fully connected layer output of CNN as the encoded image representation, which will then be fed into the top RNN language model to inform it the image content, like [3], [4]. On the other hand, Xu et al. [3] propose to use the conv-layer output of CNN, and apply a spatial attention mechanism over the convolutional features. Further, Jia et al. [5] propose to feed semantic information extracted from the image into the model such that the generated caption are tightly coupled to the image content. He et al. [6] propose to exploit Part of Speech (PoS) tags as a guidance signal for caption generation. Until recently, many researchers have found that explicit visual attributes are also very effective in guiding the image caption generation process.

In this paper, we mainly focus on using explicit attributes to guide the image caption model. The work that bears the most similarity to our work is [7]. In their work, by extracting a set of visual attributes for a given image, and then feeding them into the RNN after using an attention model over the predicted semantic attributes, they achieve a big improvement. However, their method relies on an extra off-line attributes extraction stage, e.g., a separately trained fully convolutional network (FCN). Moreover, a standalone CNN is adopted to extract the image representations and thus increase the complexity of their captioning system. However, the attribute prediction task and caption task are highly related and should not be treated as separate tasks. By sharing features for the top attribute prediction sub-network and image representation extraction sub-network, we can jointly optimize the two tasks with multi-label classification loss and cross entropy loss applied on the Long short-term Memory (LSTM) respectively.

Further, we also collect two new datasets, named MS-COCO-ATT and Flickr30K-ATT. In essence, it is just MS COCO/Flickr30K dataset with attributes attached to each image besides its caption ¹.

In summary, our key contributions are as follows: (1) Based on MS COCO and Flickr30K dataset, we construct two datasets, i.e., MS-COCO-ATT and Flickr30K-ATT, where each image has an attribute list in addition to caption data. (2) We propose a novel framework for attribute-based image caption system, which couples the attributes prediction task and image representation extraction task for the captioning task more closely by sharing one base network. To the best of our knowledge, this is the first time for an attributes based image caption system that does not rely on rely on standalone systems like FCN to extract attributes. (3) We have done experiments on two image caption benchmarks and demonstrate its effectiveness. (4) We analyze the effects of different number of attributes on the caption performance, and show that with the increases of the attributes list length, the captioning performance converges.

The rest of this paper is organized as follows. Section 2 gives a brief review of related work on image caption. Section 3 introduces the details of our proposed method. Section 4 evaluates the proposed approach on two public benchmark datasets. Finally, Section 5 gives the conclusion.