Unsupervised manifold learning using Reciprocal kNN Graphs in image re-ranking and rank aggregation tasks☆
Introduction
The development of multimedia technologies for creating and sharing digital contents has triggered an exponential increase of image collections. Traditional search approaches based on image metadata can be unfeasible for large collections, since much human intervention is required for image annotation. Content-Based Image Retrieval (CBIR) systems have emerged as a promising alternative, aiming at retrieving the images that are the most similar to a given query.
The effectiveness of CBIR systems is very dependent on the distance measure adopted. Images are often modelled as high dimensional points in an Euclidean space, and the distances among them are usually measured by Euclidean distances. In this scenario, CBIR systems often consider only pairwise image analysis, that is, compute similarity measures considering only pairs of images, ignoring the information encoded in the relations among several images. On the contrary, the user perception considers the query specification and responses in a given context. In view of that, there has been significant research [44], [43], [13], [26], [14] on improving the distance measures in CBIR systems, replacing pairwise similarities by more global affinity measures that consider the relationships among images. The overall goal of these methods is to mimic the human behavior on judging the similarity among objects by taking into account the context of the search process. As previously observed [42], [40], an effective distance measure should describe the relationship between the query and retrieved objects in the context of the whole collection.
Therefore, how to capture and utilize the intrinsic manifold structure of a collection becomes a central problem in the vision and learning community [14]. A common recent approach is manifold learning, mainly based on non-linear dimensionality reduction techniques. The idea is to explicitly construct a new embedding space with a corresponding metric which is more faithful to the manifold structure and hence induces a better distance/similarity measure. The manifold learning algorithms are able to learn distances between data points that correspond to geodesic distances on the data manifold [44]. In other words, the new distances are estimated considering a walk along the geometric structure of the dataset.
In this paper, we propose an unsupervised learning algorithm based on Reciprocal kNN Graph. The proposed algorithm improves the effectiveness of image retrieval through re-ranking and rank aggregation tasks by taking into account the instrinsic the geometry of the dataset manifold. The capacity of considering the geometry of the dataset manifold is illustrated in Fig. 1, Fig. 2, Fig. 3. We illustrate the Two-Moon dataset, comparing the Euclidean distance with the proposed Reciprocal kNN Graph. One point is selected as a labeled point (marked with a triangle) in each moon. In the following, all other data points are assigned to the closest labeled point, determining their color. Fig. 1 illustrates the classification computed by the Euclidean distance. Fig. 2 illustrates the ideal classification (with points in red and blue) considering the dataset manifold. The Euclidean distance does not consider the geometry structure of the dataset. As it can be observed, the extremities of the moons are misclassified. Fig. 3 illustrates the distances learned by the Reciprocal kNN Graph, after only one iteration. We can observe that several points were corrected compared with the Euclidean distance. The arrows in Fig. 3 illustrates how the Reciprocal kNN Graph algorithm iteratively propagates the similarity along the dataset structure considering the connectivity of the data set: (i) the red points in the left and; (ii) the blue points in the right.
The Reciprocal kNN Graph is mainly based on the information encoded in the top positions of the ranked lists. Given a query image, the ranked lists represent a relevant source of contextual information, since they define relationships not only between pairs of images (as distance functions), but also among all the images in the ranked list. The modelling of the similarity information consists in the essential difference between the Reciprocal kNN Graph approach and existing diffusion-based algorithms: the Reciprocal kNN Graph is based only on the ranked lists, and therefore independent of any distance (or similarity) scores.
By analyzing the ranked lists, it is expected, for example, that similar images present reciprocal references at the beginning of their ranked lists. It is also expected that images ranked at the top positions of ranked lists are similar to each other. In this way, aiming at redefining the distance between two images, the Reciprocal kNN Graph uses both the reciprocal nearest neighbor references and the graph structure considering all references among images at top positions of ranked lists. This approach represents the main contribution of our method, since it enables exploiting the maximum contextual information available in the ranked lists with low computational efforts. Another contribution relies on the efficiency of the Reciprocal kNN Graph algorithm. Unlike other diffusion approaches based on matrices multiplication [3], [42], [44], which presents complexity of O(n3), our algorithm recomputes only the beginning of ranked lists with a constant size of elements, which presents computational and storage requirements of only O(n), where n represents the number of images in the collection.
We conducted a large evaluation protocol involving shape, color, and texture descriptors, different datasets and comparisons with other post-processing approaches. Experimental results demonstrate the effectiveness of our method. The re-ranking and rank aggregation algorithm yield better results in terms of effectiveness performance than various state-of-the-art algorithms.
This paper is organized as follows: Section 2 discusses related work; Section 3 discusses the definition of the image re-ranking problem; in Section 4, we present our Reciprocal kNN Graph algorithm. Section 5 presents the experimental evaluation and, finally, Section 6 draws on conclusions and presents future work.
Section snippets
Related work
Defining an effective distance measures consists in a key role in many multimedia applications, including classification and retrieval tasks. For example, choosing a good distance measure is often critical for building a content-based image retrieval (CBIR) system. In general, aiming at retrieving the most similar images to a given query image, CBIR systems compute a predefined distance measure between the query image and each collection image. Traditional distance measures that consider only
Problem formulation
Let be an image collection, where n is the number of images in the collection. Let be an image descriptor which defines a distance function between two images imgi and imgj as ρ(imgi, imgj). For simplicity and readability purposes, we use the notation ρ(i,j) for denoting the distance between images imgi and imgj.
Based on the distance function ρ, a ranked list τq can be computed in response to a query image imgq. Although the ranked lists contain distance information from
Reciprocal kNN Graph
In this section, we present the Reciprocal kNN Graph algorithm and its application in re-ranking and rank aggregation tasks. We also discuss convergence and efficiency aspects.
Experimental evaluation
This section demonstrates the effectiveness of the proposed re-ranking and rank aggregation methods in image retrieval tasks. A large set of experiments was conducted considering four datasets and nineteen CBIR descriptors, aiming at analyzing and comparing our method under several aspects.
Conclusions
In this work, we have presented a novel re-ranking and rank aggregation approach that exploits the Reciprocal kNN Graph for improving image retrieval tasks.
The main idea consists in analyzing the reciprocal references at top positions of ranked lists for performing re-ranking and rank aggregation tasks. The Reciprocal kNN Graph algorithm iteratively propagates the similarity along the dataset structure by taking into account intrinsic geometry of the dataset manifold.
We conducted a large set of
Acknowledgments
Authors thank AMD, FAEPEX, CAPES, FAPESP, and CNPq for financial support.
References (46)
- et al.
BAS: a perceptual shape descriptor based on the beam angle statistics
Pattern Recogn. Lett.
(2003) - et al.
Exploiting pairwise recommendation and clustering strategies for image re-ranking
Inf. Sci.
(2012) - et al.
Texture recognition and image retrieval using gradient indexing
J. Vis. Commun. Image Represent.
(2000) - et al.
Learning context-sensitive similarity by shortest path propagation
Pattern Recogn.
(2011) - et al.
BP-tree: an efficient index for similarity search in high-dimensional metric spaces
- et al.
Co-transduction for shape retrieval
- et al.
Shape matching and object recognition using shape contexts
IEEE Trans. Pattern Anal. Mach. Intell.
(2002) - et al.
Learning mid-level features for recognition
Textures: A Photographic Album for Artists and Designers
(1966)- et al.
Cedd: color and edge directivity descriptor: a compact descriptor for image indexing and retrieval
Fcth: fuzzy color and texture histogram—a low level feature for accurate image retrieval
Visual word ambiguity
IEEE Trans. Pattern Anal. Mach. Intell.
Articulation-invariant representation of non-planar shapes
Semi-supervised distance metric learning for collaborative image retrieval and clustering
ACM Trans. Multimed. Comput. Commun. Appl.
Image indexing using color correlograms
Accurate image search using the contextual dissimilarity measure
IEEE Trans. Pattern Anal. Mach. Intell.
Unsupervised metric learning by self-smoothing operator
Beyond pairwise shape similarity analysis
Color co-occurrence descriptors for querying-by-example
Shape descriptors for non-rigid shapes with a single closed contour
Shape classification using the inner-distance
IEEE Trans. Pattern Anal. Mach. Intell.
Balancing deformability and discriminability for shape matching
Object recognition from local scale-invariant features
Cited by (43)
A BFS-Tree of ranking references for unsupervised manifold learning
2021, Pattern RecognitionGraph-based selective rank fusion for unsupervised image retrieval
2020, Pattern Recognition LettersUnsupervised selective rank fusion for image retrieval tasks
2020, NeurocomputingA framework for speaker retrieval and identification through unsupervised learning
2019, Computer Speech and LanguageAn optimized unsupervised manifold learning algorithm for manycore architectures
2019, Information SciencesCitation Excerpt :However, for ensuring the complexity of O(n), various efficiency aspects, such as data structure, should be properly considered. Since the focus of [34] was the effectiveness evaluation, only a direct and non-optimized implementation of the method was initially devised. This section describes the algorithm used in [34] for computing the ReckNNmethod, while next sections present the proposed optimizations, a detailed discussion about efficiency aspects and the evaluation on manycore architectures.
Semi-supervised and active learning through Manifold Reciprocal kNN Graph for image retrieval
2019, NeurocomputingCitation Excerpt :Moreover, the rank-based approach requires low computational costs and is parallelizable widely, which makes it suitable for online responses and real-world applications. The Manifold Reciprocal kNN Graph [39] is an unsupervised manifold learning algorithm which propagates the similarity among neighbors by considering the geometry of the dataset manifold, with the aim to improve the effectiveness of retrieval tasks without the need of user intervention. The Reciprocal kNN Graph is mainly based on the relationships and information encoded in the top positions of the ranked lists, in the context of a k-reciprocal neighborhood.
- ☆
This paper has been recommended for acceptance by Thomas Brox.