Image captioning in Hindi language using transformer networks

Abstract

Neural encoder–decoder architectures have been used extensively for image captioning. Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) are popularly used in encoder and decoder models. Recurrent Neural Networks are popular architectures in natural language processing used for language modeling, but they are sequential in nature. The transformer model can solve this problem of sequential dependency by using an attention mechanism. Many works are available for image captioning in the English language, but models for generating Hindi captions are limited; hence, we have tried to fill this gap. We have created the Hindi dataset for image captioning by manually translating the popular MSCOCO dataset from English to Hindi. Experimental results show that our proposed model outperforms other models. The proposed model has attained the BLEU-1 score of 62.9, BLEU-2 score of 43.3, BLEU-3 score of 29.1, and BLEU4 score of 19.0.

Introduction

Generating well-formed sentences for an image is a challenging task. This is a complex problem as compared to image classification and image recognition. The caption of an image must recognize not only the objects in an image but also the relationships between each other as well as their attributes and activities; hence, apart from scene understanding, the language model is needed to express this semantic knowledge in natural language. It can be helpful for visually challenged people to interpret the content on the web. It helps people in organizing and navigating through a large amount of unstructured visual data. Recent advancements in object recognition and language modeling have made it possible to generate relevant captions of an image. Nowadays, image captioning is an essential research area in computer vision, natural language processing, and image processing. Despite the complex nature of this problem, plenty of works have been done to solve the image captioning problem. Recent advancements in deep neural network [1], [2] and development of large classification datasets like ImageNet [3] have made it possible to get better quality of generated captions using the CNN and RNN. In previous works on image captioning [4], [5], pre-trained CNN is used as an encoder, and its last hidden layer is used as input to decoder RNN. In the current study, we have also used similar encoder–decoder based architecture.

Transformers are a special kind of neural network proposed by authors of [6]. They have proposed an attention-based model to transform one input sequence to another with the help of encoder–decoder architecture. It differs from the conventional sequence to sequence model because it does not use a recurrent neural network. But there is no work in image captioning based on transformer-based models. Inspired by this, we have proposed to use transformer-based architecture for image captioning tasks.

Key contributions of this paper are as follows:

• We have created the Hindi dataset for image captioning as there was no dataset available in the Hindi language. We have used well known MSCOCO dataset; this dataset is translated into Hindi using Google translator. Further, google translated corpus is corrected by human annotators.

• We have developed a novel image captioning model using the transformer architecture [6]. We have used CNN as an encoder for feature extraction and the transformer model as a decoder.

The proposed image captioning architecture has attained a BLEU-1 score of 62.9, BLEU-2 score of 43.3, BLEU-3 score of 29.1, and BLEU4 score of 19.0. To establish our model’s efficacy, we have compared our results with four baselines models, as shown in Table 6. Results illustrate that the proposed method outperforms other methods. The organization of the paper is as follows:

Section 2 highlights the literature of image captioning. Motivation of our work is described in Section 3. Section 4 describes our proposed methodology. Experimental setup and results are shown in Section 5 and Section 6, respectively. Finally conclusion of the paper is presented in Section 7.