Multi-class relevance feedback content-based image retrieval
Introduction
We consider the problem of retrieving images or video from a database. Entries in the database consist of feature measurements on q features representing image or video content, where t represents the transpose operator. Our goal is to retrieve images that are similar to a given query with feature vector .
Relevance feedback, originally developed in information retrieval [35], is a simple and appealing approach to this problem. The retrieval system finds a set of K nearest images to and then the user interacts with the system by marking the retrieved images as relevant or irrelevant. These marked images constitute training data. From the query and the training data, the system dynamically adjusts its retrieval mechanism, from which a new round of retrieval begins. The above process repeats until the user is satisfied with the results or the system cannot improve its performance from one iteration to the next. Relevance feedback retrieval has shown promise in a variety of image database applications [1], [3], [8], [11], [13], [14], [15], [16], [18], [19], [20], [21], [23], [25], [28], [29], [30], [31], [32], [33], [34], [36], [40], [42], [44], [46], [47], [48].
Relevance feedback retrieval provides an attractive framework upon which to develop a variety of adaptive content-based retrieval techniques. These techniques are capable of generating highly customized metrics for computing image similarity based on a discriminant as well as feature relevance analysis, as in [11], [19], [28], [30], [31], [44], [48], or shifting the input query (in addition to creating the customized metrics) toward relevant retrievals and, at the same time, away from irrelevant ones, as in [13], [15], [32], [34]. These techniques assume two (relevant and irrelevant) class relevance feedback that can be used to iteratively refine retrieval performance. While simple computationally, the assumption often becomes inadequate in providing sufficient information to help rapidly improve retrieval performance. All the information from classes other than relevant and irrelevant ones is lost. As a result, local data information cannot be fully exploited.
Fig. 1(a) illustrates the situation for a simple example. There are three classes in two dimensions that are normally distributed. The means of the classes are randomly generated within (−0.2, 0.2). Two (class one and two) of the classes have identical standard deviations 0.75 along the two dimensions. The third class has a standard deviation 0.25 along the horizontal axis and 1.0 along the vertical axis. There are 200 samples within each class. In this example, the first two classes are similar in that they follow similar distributions. Knowledge about one class reveals a great deal of information about the other.
Fig. 1(b) shows average retrieval precisions as a function of iteration obtained by the PFRL algorithm, a two-class relevance feedback technique described in [30], and a three-class relevance feedback technique to be described in this paper, respectively. Here retrieval precision for query class Q is defined aswhere R denotes the set of retrieved data and RQ the set of class Q data in the database. Note that the cardinality of R is set to 20 at each iteration. It can be seen that exploiting local data information helps improve retrieval performance. While the idea of multi-class relevance feedback may seem obvious, we could find few proposals along these lines in the literature.
In order to overcome the limitations associated with two-class relevance feedback, we propose in this paper a novel multi-class form of relevance feedback content-based retrieval to try to exploit local multi-class information. For a given query image, we estimate local feature relevance by approximating a χ2 distance with information provided by multi-class relevance feedback. Local feature relevance is then used to customize the retrieval metric to rank images. By exploiting multi-class information, our method is able to more accurately predict the local relevance of each feature dimension, thereby creating flexible metrics that better capture user perceived similarity. As a result, rapid performance improvement can be achieved in terms of higher precision with fewer iterations. We demonstrate our technique and compare it against other competing methods using both real and simulated data.
There is a subtle but critical difference between existing continuous 0–1 relevance feedback approach [15], [32] and the technique we propose here. Whereas the former uses continuous scores to compute a weighted distance matrix, the latter exploits multi-class (category) information. Providing categorical information places far less burden on the user than continuous one.
The rest of the paper is organized as follows. Section 2 describes related work addressing issues of relevance feedback in the context of image retrieval. Section 3 presents our adaptive multi-class relevance feedback approach to content-based image retrieval that better exploits local data densities over two-class relevance feedback techniques. Section 4 describes an efficient procedure for estimating local feature relevance, hence a flexible distance metric. Section 5 introduces Stein’s estimator that can be used to improve our feature relevance estimates. After that, Section 6 presents our adaptive multi-class relevance feedback algorithm. Section 7 presents experimental results demonstrating the efficacy of our technique using both real and simulated data. Section 8 provides a discussion on our proposal. Finally, Section 9 concludes this paper by pointing out possible extensions to the current work and future research directions.
Section snippets
Related work
There are numerous methods for learning flexible metrics from relevance feedback for image retrieval as well as classification [12], [10], [11], [13], [14], [15], [18], [20], [28], [30], [32], [33], [34], [44], [48]. MARS [34] is a content-based retrieval algorithm that makes use of two-class relevance feedback techniques developed in the field of information retrieval. In MARS, images are represented by weight vectors in the term space, where weights capture the importance of components within
Adaptive multi-class relevance feedback
In order to exploit local data densities effectively, we must first define a multi-class metric upon which to develop a theory of multi-class relevance feedback. We begin our discussion by introducing some classification concepts essential to our theoretical derivation [30], [43]. We state at the outset that the image retrieval problem is opposite to typical classification problems based on nearest neighbor kernel methods [6]. While the goal in classification is to predict the class label of an
Estimation
Since both and in (10) are unknown, we must estimate them using retrievals with relevance feedback: . Here yn∈{1,⋯,J}. The quantity is estimated by considering a neighborhood centered at :where 1(·) is an indicator function such that it returns 1 when its argument is true, and 0 otherwise.
To compute , we introduce a dummy variable gj such thatwhere j
Improving estimation using Stein’s estimator
It is shown [17], [39] that it is possible to improve upon the maximum likelihood estimator in terms of total squared error risk using statistical shrinking. The basic idea is as follows. Suppose we want to estimate several parameters from independent normal observations. Specifically, let
Note that the example typically occurs as a reduction to this canonical form from more complex situations. Also let represent the estimate of θi, and the estimate of overall average
Adaptive multi-class relevance feedback algorithm
The adaptive multi-class relevance feedback algorithm (MURF) has five adjustable tuning (“meta”) parameters:
- •
L: the number of points within the Δ intervals Eqs. (18) and (19);
- •
K1: the number of neighbors in for estimation (17);
- •
T: the positive factor for the exponential weighting scheme (15);
- •
c1: the shrinking factor in (20); and
- •
c2: the shrinking factor in (21).
At the beginning, the estimation of the ri values in (11) is accomplished by using a weighted distance metric (16) with being
Empirical evaluation
In the following we compare two competing methods using both real and simulated data sets.
- •
PFRL—the probabilistic feature relevance learning (PFRL) method described in [30]. PFRL is a two-class relevance feedback technique. It has two procedural (“meta”) parameters that are identical to L and T, two procedural parameters input to MURF (Section 6).
- •
MURF—adaptive multi-class relevance feedback, using the weighted metric (7).
Discussion
The MURF algorithm has five adjustable procedural parameters (Section 6). While L and T are common to both PFRL and MURF, the MURF algorithm has introduced three new parameters, one of which, K1, is shared by all nearest neighbor rules. A discussion on the choice of L and T can be found in [30]. The choice of K1 has been extensively researched within the machine learning community [24]. In all our experiments, K1 was set to either 1 or 3, small enough to avoid bias. K1 should have larger values
Summary
This paper presents a novel multi-class relevance feedback method for content-based retrieval. For a given query, the method estimates the relevance of each feature dimension by approximating a local χ2 distance with multi-class relevance feedback. This information is then used to customize the retrieval metric to rank images. By exploiting multi-class information, the method is able to more accurately predict the local relevance of each feature, thereby creating flexible metrics that better
Acknowledgements
This work was supported in part by NSF Grant IIS-0136348.
References (48)
- et al.
Feature normalization and likelihood-based similarity measures for image retrieval
Pattern Recognition Lett.
(2001) - et al.
Interactive learning with a society of models
Pattern Recognition
(1997) - et al.
The Multi-Class Metric Problem in nearest neighbor discrimination rules
Pattern Recognition
(1990) - et al.
Independent feature analysis for image retrieval
Pattern Recognition Lett.
(2001) - et al.
Probabilistic feature relevance learning for Content-based image retrieval
Comput. Vision Image Understanding
(1999) - et al.
Edge-based structural features for content-based image retrieval
Pattern Recognition Lett.
(2001) - L. Breiman, J.H. Friedman, R.A. Olshen, C.J. Stone, Classification and Regression Trees, Wadsworth,...
- Y. Chen, X. Zhou, T. Huang, One-class SVM for learning in image retrieval, in: Proc. IEEE Internat. Conf. on Image...
- et al.
Trading mips and memory for knowledge engineering
CACM
(1992) - et al.
An Introduction to Support Vector Machines and Other Kernel-based Learning Methods
(2000)
Pattern Classification and Scene Analysis
Data analysis using Stein’s estimator and its generalizations
J. Am. Statist. Assoc.
Query by image and video content: the QBIC system
IEEE Comput.
Discriminant adaptive nearest neighbor classification
IEEE Trans. Pattern Anal. Machine Intell.
Region-based image retrieval using probabilistic feature relevance learning
Pattern Anal. Appl.
Cited by (21)
Incorporating multiple distance spaces in optimum-path forest classification to improve feedback-based learning
2012, Computer Vision and Image UnderstandingCitation Excerpt :Other works try to reduce the number of vectors by feature selection techniques [54,57,8,7]. Some works also allow the user to assign a finer gradation of relevancy, such as somewhat relevant, not sure, and somewhat irrelevant [58,59]. The OPF classifier is a multi-class classifier and can handle multiple gradation levels.
Content-based image retrieval trained by adaboost for mobile application
2006, International Journal of Pattern Recognition and Artificial IntelligenceContent-Based Image Retrieval: Ideas, Influences, and Current Trends
2018, Content-Based Image Retrieval: Ideas, Influences, and Current TrendsFast interactive information retrieval with sampling-based MDS on GPU architectures
2014, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Adaptive learning for relevance feedback: Application to digital mammography
2010, Medical PhysicsA framework of CBIR system based on relevance feedback
2009, 3rd International Symposium on Intelligent Information Technology Application, IITA 2009