Unsupervised manifold learning using Reciprocal kNN Graphs in image re-ranking and rank aggregation tasks

Highlights

• Presentation of an unsupervised manifold learning algorithm using Reciprocal kNN Graphs

• Presentation of the Reciprocal kNN Graph ReRanking for improving the effectiveness of CBIR systems

• Description of how Reciprocal kNN Graph algorithm can be used for rank aggregation tasks

• Discussion about the computational complexity and the convergence of proposed algorithm

• Experimental evaluation considering different datasets, descriptors, and baselines

Abstract

In this paper, we present an unsupervised distance learning approach for improving the effectiveness of image retrieval tasks. We propose a Reciprocal kNN Graph algorithm that considers the relationships among ranked lists in the context of a k-reciprocal neighborhood. The similarity is propagated among neighbors considering the geometry of the dataset manifold. The proposed method can be used both for re-ranking and rank aggregation tasks. Unlike traditional diffusion process methods, which require matrix multiplication operations, our algorithm takes only a subset of ranked lists as input, presenting linear complexity in terms of computational and storage requirements. We conducted a large evaluation protocol involving shape, color, and texture descriptors, various datasets, and comparisons with other post-processing approaches. The re-ranking and rank aggregation algorithms yield better results in terms of effectiveness performance than various state-of-the-art algorithms recently proposed in the literature, achieving bull's eye and MAP scores of 100% on the well-known MPEG-7 shape dataset.

Introduction

The development of multimedia technologies for creating and sharing digital contents has triggered an exponential increase of image collections. Traditional search approaches based on image metadata can be unfeasible for large collections, since much human intervention is required for image annotation. Content-Based Image Retrieval (CBIR) systems have emerged as a promising alternative, aiming at retrieving the images that are the most similar to a given query.

The effectiveness of CBIR systems is very dependent on the distance measure adopted. Images are often modelled as high dimensional points in an Euclidean space, and the distances among them are usually measured by Euclidean distances. In this scenario, CBIR systems often consider only pairwise image analysis, that is, compute similarity measures considering only pairs of images, ignoring the information encoded in the relations among several images. On the contrary, the user perception considers the query specification and responses in a given context. In view of that, there has been significant research [44], [43], [13], [26], [14] on improving the distance measures in CBIR systems, replacing pairwise similarities by more global affinity measures that consider the relationships among images. The overall goal of these methods is to mimic the human behavior on judging the similarity among objects by taking into account the context of the search process. As previously observed [42], [40], an effective distance measure should describe the relationship between the query and retrieved objects in the context of the whole collection.

Therefore, how to capture and utilize the intrinsic manifold structure of a collection becomes a central problem in the vision and learning community [14]. A common recent approach is manifold learning, mainly based on non-linear dimensionality reduction techniques. The idea is to explicitly construct a new embedding space with a corresponding metric which is more faithful to the manifold structure and hence induces a better distance/similarity measure. The manifold learning algorithms are able to learn distances between data points that correspond to geodesic distances on the data manifold [44]. In other words, the new distances are estimated considering a walk along the geometric structure of the dataset.

In this paper, we propose an unsupervised learning algorithm based on Reciprocal kNN Graph. The proposed algorithm improves the effectiveness of image retrieval through re-ranking and rank aggregation tasks by taking into account the instrinsic the geometry of the dataset manifold. The capacity of considering the geometry of the dataset manifold is illustrated in Fig. 1, Fig. 2, Fig. 3. We illustrate the Two-Moon dataset, comparing the Euclidean distance with the proposed Reciprocal kNN Graph. One point is selected as a labeled point (marked with a triangle) in each moon. In the following, all other data points are assigned to the closest labeled point, determining their color. Fig. 1 illustrates the classification computed by the Euclidean distance. Fig. 2 illustrates the ideal classification (with points in red and blue) considering the dataset manifold. The Euclidean distance does not consider the geometry structure of the dataset. As it can be observed, the extremities of the moons are misclassified. Fig. 3 illustrates the distances learned by the Reciprocal kNN Graph, after only one iteration. We can observe that several points were corrected compared with the Euclidean distance. The arrows in Fig. 3 illustrates how the Reciprocal kNN Graph algorithm iteratively propagates the similarity along the dataset structure considering the connectivity of the data set: (i) the red points in the left and; (ii) the blue points in the right.

The Reciprocal kNN Graph is mainly based on the information encoded in the top positions of the ranked lists. Given a query image, the ranked lists represent a relevant source of contextual information, since they define relationships not only between pairs of images (as distance functions), but also among all the images in the ranked list. The modelling of the similarity information consists in the essential difference between the Reciprocal kNN Graph approach and existing diffusion-based algorithms: the Reciprocal kNN Graph is based only on the ranked lists, and therefore independent of any distance (or similarity) scores.

By analyzing the ranked lists, it is expected, for example, that similar images present reciprocal references at the beginning of their ranked lists. It is also expected that images ranked at the top positions of ranked lists are similar to each other. In this way, aiming at redefining the distance between two images, the Reciprocal kNN Graph uses both the reciprocal nearest neighbor references and the graph structure considering all references among images at top positions of ranked lists. This approach represents the main contribution of our method, since it enables exploiting the maximum contextual information available in the ranked lists with low computational efforts. Another contribution relies on the efficiency of the Reciprocal kNN Graph algorithm. Unlike other diffusion approaches based on matrices multiplication [3], [42], [44], which presents complexity of O(n³), our algorithm recomputes only the beginning of ranked lists with a constant size of elements, which presents computational and storage requirements of only O(n), where n represents the number of images in the collection.

We conducted a large evaluation protocol involving shape, color, and texture descriptors, different datasets and comparisons with other post-processing approaches. Experimental results demonstrate the effectiveness of our method. The re-ranking and rank aggregation algorithm yield better results in terms of effectiveness performance than various state-of-the-art algorithms.

This paper is organized as follows: Section 2 discusses related work; Section 3 discusses the definition of the image re-ranking problem; in Section 4, we present our Reciprocal kNN Graph algorithm. Section 5 presents the experimental evaluation and, finally, Section 6 draws on conclusions and presents future work.