Multi-class relevance feedback content-based image retrieval

https://doi.org/10.1016/S1077-3142(03)00013-4Get rights and content

Abstract

Relevance feedback methods for content-based image retrieval have shown promise in a variety of image database applications. These techniques assume two-class relevance feedback: relevant and irrelevant classes. While simple computationally, two-class relevance feedback often becomes inadequate in providing sufficient information to help rapidly improve retrieval performance. In this paper we propose a multi-class form of relevance feedback retrieval to try to exploit multi-class information. For a given query, we use a χ2 analysis to determine the local relevance of each feature dimension with multi-class relevance feedback. This information is then used to customize the retrieval metric to rank images. By exploiting multi-class information, our method is able to create flexible metrics that better capture user perceived similarity. In a number of image data sets, the method achieves a higher level of precision with fewer iterations, demonstrating the potential for substantial improvements over two-class relevance feedback retrieval.

Introduction

We consider the problem of retrieving images or video from a database. Entries in the database consist of feature measurements x=(x1,x2,…,xq)t on q features representing image or video content, where t represents the transpose operator. Our goal is to retrieve images that are similar to a given query with feature vector x0.

Relevance feedback, originally developed in information retrieval [35], is a simple and appealing approach to this problem. The retrieval system finds a set of K nearest images to x0 and then the user interacts with the system by marking the retrieved images as relevant or irrelevant. These marked images constitute training data. From the query and the training data, the system dynamically adjusts its retrieval mechanism, from which a new round of retrieval begins. The above process repeats until the user is satisfied with the results or the system cannot improve its performance from one iteration to the next. Relevance feedback retrieval has shown promise in a variety of image database applications [1], [3], [8], [11], [13], [14], [15], [16], [18], [19], [20], [21], [23], [25], [28], [29], [30], [31], [32], [33], [34], [36], [40], [42], [44], [46], [47], [48].

Relevance feedback retrieval provides an attractive framework upon which to develop a variety of adaptive content-based retrieval techniques. These techniques are capable of generating highly customized metrics for computing image similarity based on a discriminant as well as feature relevance analysis, as in [11], [19], [28], [30], [31], [44], [48], or shifting the input query (in addition to creating the customized metrics) toward relevant retrievals and, at the same time, away from irrelevant ones, as in [13], [15], [32], [34]. These techniques assume two (relevant and irrelevant) class relevance feedback that can be used to iteratively refine retrieval performance. While simple computationally, the assumption often becomes inadequate in providing sufficient information to help rapidly improve retrieval performance. All the information from classes other than relevant and irrelevant ones is lost. As a result, local data information cannot be fully exploited.

Fig. 1(a) illustrates the situation for a simple example. There are three classes in two dimensions that are normally distributed. The means of the classes are randomly generated within (−0.2, 0.2). Two (class one and two) of the classes have identical standard deviations 0.75 along the two dimensions. The third class has a standard deviation 0.25 along the horizontal axis and 1.0 along the vertical axis. There are 200 samples within each class. In this example, the first two classes are similar in that they follow similar distributions. Knowledge about one class reveals a great deal of information about the other.

Fig. 1(b) shows average retrieval precisions as a function of iteration obtained by the PFRL algorithm, a two-class relevance feedback technique described in [30], and a three-class relevance feedback technique to be described in this paper, respectively. Here retrieval precision for query class Q is defined asPrecision=|R∩RQ||R|,where R denotes the set of retrieved data and RQ the set of class Q data in the database. Note that the cardinality of R is set to 20 at each iteration. It can be seen that exploiting local data information helps improve retrieval performance. While the idea of multi-class relevance feedback may seem obvious, we could find few proposals along these lines in the literature.

In order to overcome the limitations associated with two-class relevance feedback, we propose in this paper a novel multi-class form of relevance feedback content-based retrieval to try to exploit local multi-class information. For a given query image, we estimate local feature relevance by approximating a χ2 distance with information provided by multi-class relevance feedback. Local feature relevance is then used to customize the retrieval metric to rank images. By exploiting multi-class information, our method is able to more accurately predict the local relevance of each feature dimension, thereby creating flexible metrics that better capture user perceived similarity. As a result, rapid performance improvement can be achieved in terms of higher precision with fewer iterations. We demonstrate our technique and compare it against other competing methods using both real and simulated data.

There is a subtle but critical difference between existing continuous 0–1 relevance feedback approach [15], [32] and the technique we propose here. Whereas the former uses continuous scores to compute a weighted distance matrix, the latter exploits multi-class (category) information. Providing categorical information places far less burden on the user than continuous one.

The rest of the paper is organized as follows. Section 2 describes related work addressing issues of relevance feedback in the context of image retrieval. Section 3 presents our adaptive multi-class relevance feedback approach to content-based image retrieval that better exploits local data densities over two-class relevance feedback techniques. Section 4 describes an efficient procedure for estimating local feature relevance, hence a flexible distance metric. Section 5 introduces Stein’s estimator that can be used to improve our feature relevance estimates. After that, Section 6 presents our adaptive multi-class relevance feedback algorithm. Section 7 presents experimental results demonstrating the efficacy of our technique using both real and simulated data. Section 8 provides a discussion on our proposal. Finally, Section 9 concludes this paper by pointing out possible extensions to the current work and future research directions.

Section snippets

Related work

There are numerous methods for learning flexible metrics from relevance feedback for image retrieval as well as classification [12], [10], [11], [13], [14], [15], [18], [20], [28], [30], [32], [33], [34], [44], [48]. MARS [34] is a content-based retrieval algorithm that makes use of two-class relevance feedback techniques developed in the field of information retrieval. In MARS, images are represented by weight vectors in the term space, where weights capture the importance of components within

Adaptive multi-class relevance feedback

In order to exploit local data densities effectively, we must first define a multi-class metric upon which to develop a theory of multi-class relevance feedback. We begin our discussion by introducing some classification concepts essential to our theoretical derivation [30], [43]. We state at the outset that the image retrieval problem is opposite to typical classification problems based on nearest neighbor kernel methods [6]. While the goal in classification is to predict the class label of an

Estimation

Since both Pr(j|z) and Pr(j|xi=zi) in (10) are unknown, we must estimate them using retrievals with relevance feedback: {xn,yn}n=1N. Here yn∈{1,⋯,J}. The quantity Pr(j|z) is estimated by considering a neighborhood N1(z) centered at z:Pr̂(j|z)=n=1N1(xn∈N1(z))1(yn=j)n=1N1(xn∈N1(z)),where 1(·) is an indicator function such that it returns 1 when its argument is true, and 0 otherwise.

To compute Pr(j|xi=z)=E[Pr(j|x)|xi=z], we introduce a dummy variable gj such thatgj|x=1ify=j,0otherwise,where j

Improving estimation using Stein’s estimator

It is shown [17], [39] that it is possible to improve upon the maximum likelihood estimator in terms of total squared error risk using statistical shrinking. The basic idea is as follows. Suppose we want to estimate several parameters from independent normal observations. Specifically, letxi∼N(θi,1),i=1,…,k.

Note that the example typically occurs as a reduction to this canonical form from more complex situations. Also let x̄i represent the estimate of θi, and x̄ the estimate of overall average

Adaptive multi-class relevance feedback algorithm

The adaptive multi-class relevance feedback algorithm (MURF) has five adjustable tuning (“meta”) parameters:

  • L: the number of points within the Δ intervals Eqs. (18) and (19);

  • K1: the number of neighbors in N1(z) for estimation (17);

  • T: the positive factor for the exponential weighting scheme (15);

  • c1: the shrinking factor in (20); and

  • c2: the shrinking factor in (21).

At the beginning, the estimation of the ri values in (11) is accomplished by using a weighted distance metric (16) with w being

Empirical evaluation

In the following we compare two competing methods using both real and simulated data sets.

  • PFRL—the probabilistic feature relevance learning (PFRL) method described in [30]. PFRL is a two-class relevance feedback technique. It has two procedural (“meta”) parameters that are identical to L and T, two procedural parameters input to MURF (Section 6).

  • MURF—adaptive multi-class relevance feedback, using the weighted metric (7).

Note that there is a third method, the no relevance feedback method, that

Discussion

The MURF algorithm has five adjustable procedural parameters (Section 6). While L and T are common to both PFRL and MURF, the MURF algorithm has introduced three new parameters, one of which, K1, is shared by all nearest neighbor rules. A discussion on the choice of L and T can be found in [30]. The choice of K1 has been extensively researched within the machine learning community [24]. In all our experiments, K1 was set to either 1 or 3, small enough to avoid bias. K1 should have larger values

Summary

This paper presents a novel multi-class relevance feedback method for content-based retrieval. For a given query, the method estimates the relevance of each feature dimension by approximating a local χ2 distance with multi-class relevance feedback. This information is then used to customize the retrieval metric to rank images. By exploiting multi-class information, the method is able to more accurately predict the local relevance of each feature, thereby creating flexible metrics that better

Acknowledgements

This work was supported in part by NSF Grant IIS-0136348.

References (48)

  • R.O. Duda et al.

    Pattern Classification and Scene Analysis

    (1973)
  • B. Efron et al.

    Data analysis using Stein’s estimator and its generalizations

    J. Am. Statist. Assoc.

    (1975)
  • M. Flickner

    Query by image and video content: the QBIC system

    IEEE Comput.

    (1995)
  • Y. Freund, T. Schapire, Experiments with a new boosting algorithm, in: Proc. 13th Internat. Conf. on Machine Learning,...
  • J.H. Friedman, Flexible metric nearest neighbor classification, Technical Report, Department of Statistics, Stanford...
  • G.D. Guo, A. Jain, W.Y. Ma, H.J. Zhang, Learning similarity measure for natural image retrieval with relevance...
  • T. Hastie et al.

    Discriminant adaptive nearest neighbor classification

    IEEE Trans. Pattern Anal. Machine Intell.

    (1996)
  • D. Heisterkamp, J. Peng, H.K. Dai, Feature relevance learning with query shifting for content-based image retrieval,...
  • D. Heisterkamp, J. Peng, H.K. Dai, An adaptive quasiconformal kernel metric for image retrieval, in: Proc. IEEE Comput....
  • Y. Ishikawa, R. Subramanya, C. Faloutsos, MindReader: query databases through multiple examples, in: Proc. 24th VLDB...
  • R. Jain, S.N.J. Murthy, L. Tran, Similarity measures for image databases, IEEE Conf. on Fuzzy Logic,...
  • W. James, C. Stein, Estimation with quadratic loss, in: Proc. 4th Berkeley Symposium on Mathematical Statistics and...
  • B. Ko et al.

    Region-based image retrieval using probabilistic feature relevance learning

    Pattern Anal. Appl.

    (2001)
  • B. Ko, J. Peng, H. Byun, A new content-based image retrieval system using hand gesture and relevance feedback, in:...
  • Cited by (21)

    • Incorporating multiple distance spaces in optimum-path forest classification to improve feedback-based learning

      2012, Computer Vision and Image Understanding
      Citation Excerpt :

      Other works try to reduce the number of vectors by feature selection techniques [54,57,8,7]. Some works also allow the user to assign a finer gradation of relevancy, such as somewhat relevant, not sure, and somewhat irrelevant [58,59]. The OPF classifier is a multi-class classifier and can handle multiple gradation levels.

    • Content-based image retrieval trained by adaboost for mobile application

      2006, International Journal of Pattern Recognition and Artificial Intelligence
    • Content-Based Image Retrieval: Ideas, Influences, and Current Trends

      2018, Content-Based Image Retrieval: Ideas, Influences, and Current Trends
    • Fast interactive information retrieval with sampling-based MDS on GPU architectures

      2014, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    • A framework of CBIR system based on relevance feedback

      2009, 3rd International Symposium on Intelligent Information Technology Application, IITA 2009
    View all citing articles on Scopus
    View full text