Image Annotation by Propagating Labels from Semantic Neighbourhoods

Abstract

Automatic image annotation aims at predicting a set of semantic labels for an image. Because of large annotation vocabulary, there exist large variations in the number of images corresponding to different labels (“class-imbalance”). Additionally, due to the limitations of human annotation, several images are not annotated with all the relevant labels (“incomplete-labelling”). These two issues affect the performance of most of the existing image annotation models. In this work, we propose 2-pass k-nearest neighbour (2PKNN) algorithm. It is a two-step variant of the classical k-nearest neighbour algorithm, that tries to address these issues in the image annotation task. The first step of 2PKNN uses “image-to-label” similarities, while the second step uses “image-to-image” similarities, thus combining the benefits of both. We also propose a metric learning framework over 2PKNN. This is done in a large margin set-up by generalizing a well-known (single-label) classification metric learning algorithm for multi-label data. In addition to the features provided by Guillaumin et al. (2009) that are used by almost all the recent image annotation methods, we benchmark using new features that include features extracted from a generic convolutional neural network model and those computed using modern encoding techniques. We also learn linear and kernelized cross-modal embeddings over different feature combinations to reduce semantic gap between visual features and textual labels. Extensive evaluations on four image annotation datasets (Corel-5K, ESP-Game, IAPR-TC12 and MIRFlickr-25K) demonstrate that our method achieves promising results, and establishes a new state-of-the-art on the prevailing image annotation datasets.

Appendix

In our conference paper Verma and Jawahar (2012), we proposed to learn a combined distance metric in both distance as well as feature spaces, denoted by $\mathbf {w}$ and $\mathbf {v}$ respectively. While the dimensionality of $\mathbf {w}$ is equal to the number of different features used to represent a sample, that of $\mathbf {v}$ is equal to the combined dimensionality of all the individual features. As a result, we found it was not easy to learn $\mathbf {v}$ using limited examples as the number of features increased, and the computational cost was also quite high. Hence, in this paper we have considered only the $\mathbf {w}$ metric. Empirically, we found that the performance obtained using only the $\mathbf {w}$ metric was usually comparable to that using both $\mathbf {w}$ and $\mathbf {v}$. E.g., in Table 10, we compare their performance using three feature combinations on the Corel-5K dataset. Here, we can observe that though we compromise a bit on the performance, the difference is not significant. More importantly, the training time improves by several orders of magnitude.

Table 10 Performance comparison in terms of F1, N+ and training time (in hours) between our previous metric learning approach (Verma and Jawahar 2012) that learns a combined metric in distance and feature spaces (2PKNN + ML (w + v)) with our current approach that learns a metric only in the distance space (2PKNN + ML(w)) on the Corel-5K dataset