A bag of constrained informative deep visual words for image retrieval

Highlights

• A new model for image retrieval

• Combination of deep features, information theory and constrained clustering

• Unsupervised clustering constraints built from mutual information

Abstract

In this paper, we propose a bag of constrained informative deep visual words (BoCIDVW) model for image retrieval. Informative patches from each image are first obtained using patch entropy values. Each such patch is represented by deep features extracted through VGG16-Net. Two sets of constraints, namely, the must-link (ML) and the cannot-link (CL), are obtained for each deep informative patch in an unsupervised manner from its mutual information values (with other patches). The patches are then quantized using the Linear-time Constrained Vector Quantization Error (LCVQE), a fast yet accurate constrained K-means algorithm. The resulting clusters, which we term constrained informative deep visual words, are employed to label each patch. Finally, a bag (histogram) of constrained informative visual words is developed for image retrieval. Experiments on three different publicly available datasets demonstrate the merit of the proposed formulation.

Introduction

Image retrieval [7] is a well studied problem in the pattern recognition community. Bag of Visual Words (BoVW) and its variants are effectively being used for image retrieval for quite some time [28]. In the basic BoVW model, the constituent patches in an image are first represented by hand-crafted features, like SURF [1] or SIFT [15]. These patches are then quantized in the feature space by the K-means algorithm [13]. Finally, each image patch is marked with the label of the nearest cluster (visual words) and an image is represented by a bag (histogram) of visual words. Some works are reported on improved BoVW models in connection with image retrieval. For example, Dimitrovski et al. have applied BoVW model with predictive clustering tree for improving image retrieval [10]. On the other hand, deep learning based approaches have become increasingly popular for solving the retrieval problem as well [12], [26].

In this paper, we propose a new patch based model for image retrieval using deep features, information theoretic measures and constrained clustering. All patches extracted from an image do not contain significant information. Essentially, the patches from more homogeneous regions carry less information. So, entropy is used to select informative patches, i.e., from object regions with higher entropy values. These patches are then represented by deep features through VGG16-Net. To develop the constraints for the patches in a supervised manner, one needs to store all the patch labels. This becomes a cumbersome process. In this paper, mutual information is employed to develop these clustering constraints in an unsupervised manner. Linear-time Constrained Vector Quantization Error (LCVQE) [23], a fast yet accurate constrained K-means algorithm is used to quantize the informative image patches. We term the resulting image representation model as Bag of Constrained Informative Deep Visual Words (BoCIDVW). For a preliminary version of this work, please see [19], where we developed a Bag of Constrained Visual Words (BoCVW) model. There are several key differences between the previous conference version and the current journal version. In the conference version, we used SURF features. In contrast, in the journal version, we have replaced SURF features by deep features. Secondly, we have used information theory (entropy, mutual information) at different stages of the solution. Furthermore, we have now added an algorithm, time-complexity analysis and more theoretical expositions. Finally, exhaustive comparisons with several existing and more recent approaches are included. Now, we summarize the contributions of this work:

• We use mean and standard deviation of entropy values of the patches appearing in an image to filter out the more informative patches. In a given image, we initially calculate entropy values of all the constituent patches and compute their mean and standard deviation. We filter out the informative patches, i.e., patches whose entropy values are greater than or equal to the mean entropy value plus standard deviation of the entropy values. These patches are then represented using deep features through the VGG16 network. So, we build deep informative patches.

• Mutual information is employed to develop two sets of clustering constraints (must-link and cannot-link) in an unsupervised manner. Linear-time Constrained Vector Quantization Error (LCVQE), a constrained K-means clustering algorithm is then applied to quantize the deep informative patches. Finally, a Bag of Constrained Informative Deep Visual Words (BoCIDVW) is developed for the purpose of image retrieval. This proposed model, a combination of deep features, information theoretic measures, and constrained clustering, is shown to yield very competitive results on the publicly available Coil-100, Oxford5K and Paris-6K datasets in terms of mean average precision (mAP) values.

The rest of the paper is organized as follows: in Section 2, we discuss the related work. In Section 3, we describe our method. In Section 4, we present the experimental results. The paper is concluded in Section 5 with directions of future research.