A novel dynamic multi-model relevance feedback procedure for content-based image retrieval

Abstract

This paper deals with the problem of image retrieval in large databases with a big semantic gap by a relevance feedback procedure. We present a novel algorithm for modelling the users׳s preferences in the content-based image retrieval system.

The proposed algorithm considers the probability of an image belonging to the set of those sought by the user, and estimates the parameters of several local logistic regression models whose inputs are the low-level image features. A Principal Component Analysis method is applied to the original vector to reduce its high dimensionality. The relevance probabilities predicted by these local models are combined by means of a weighted average. These weights are obtained according to the variance explained by the group of principal components used for each local model. These models are dynamically estimated in each iteration of the relevance feedback algorithm until the user is satisfied.

This novel procedure has been tested in a collection with a large semantic gap, the Wikipedia collection. Two types of experiments have been performed, one with an automatic user and another with a typical user. The method is compared to some recent similar approaches in literature, obtaining very good performance in terms of the MAP evaluation measure.

Introduction

In the last few years, the increasing number of image databases which need effective and efficient techniques for retrieving multimedia information has motivated the interest in Content-Based Image Retrieval (CBIR) systems [29], [25], [12]. Compared to traditional image retrieval systems based on textual information, CBIR systems represent an improvement taking advantage of the digital information stored in the image itself when image collections are not semantically annotated with textual labels. Thus, visual features are extracted from images in order to describe its content [9], and later be compared with the image query. These visual features used in CBIR systems can be classified into low level features (color, texture and shape) and high level features, which are usually obtained by combining low level features with a predefined model. High level features are not usually suitable for general purpose systems as they have a strong dependency on the application domain, so the extraction of good low level image descriptors in an important research activity in this field.

Nevertheless, although the low level features can easily describe the content of simple images, complex images and high level concepts cannot be properly described. This gap between high level concepts closer to human perception and low level features used to describe images is called semantic gap, and different methods have been proposed to deal with it [18], [20], [35], [8], [28]. In many cases, the strategies proposed are based on the integration of the information provided by the user into the decision process. The procedure in these systems is as follows. First, the user selects from a set of images (resulting from a previous search) those considered to be relevant, according to his/her particular search criterion, and rejects those that are not. Based on this selection, the system learns and offers the user a new set of images closer to his/her search in the next step. This process is repeated until the user considers the offered result as satisfactory. This way, the user has guided the search by indicating his/her preferences. Those systems that take the information provided by the user to improve the results of a new query are said to have a relevance feedback mechanism.

Regarding this relevance feedback mechanism, several researchers have proposed different techniques and algorithms to achieve it [37], [6]. The classical approaches were inspired by other techniques typically used in the context of classical information retrieval, such as the systems proposed by Ishikawa [16] Rui [27] or Ciocca an Schettini [4], based on moving the query point so that it appears closer to the relevant results and farther from non-relevant results, and updating the similarity measure according to user׳s criteria. Other authors propose probabilistic techniques mainly based on Bayesian frameworks that incorporate user preferences ([34], [30], [5], [7]). Another group of approaches are those based on classification and clustering which use support vector machines, a set of supervised learning method for classification that require positive and negative selections ([36], [32]), and others which learn from positive selections only [3], [13], [31].

Being part of this mainstream, in this paper, we present a new algorithm for relevance feedback in large image databases that is within the probabilistic techniques and is based on local regression models improving previous related works [19], [8]. In [19] an iterative relevance feedback scheme was proposed, based on logistic regression analysis for ranking a set of images in decreasing order of their evaluated relevance probabilities. The low level image features are grouped into n subsets with semantically related characteristics obtaining n smaller regression models to be adjusted. These models produce a different relevance probability that are combined by making use of the ordered weighted averaging (OWA) operators to rank the database according to the preferences of users. In [8] a very drastic dimensionality reduction is performed up to 3 dimensions. Each image is described in terms of the distances to the different levels of user preferences (relevant, neutral and non-relevant), changing this feature vector in each iteration. After that, only a global logistic regression model is adjusted to this reduced vector and the associated relevance probabilities are computed.

To improve these algorithms, in the present work we deal with the high dimension of the low-level feature vector by applying a Principal Component Analysis (PCA) to the original feature vector ([17]), obtaining a transformed vector which explains at least the 80% of the sample variability. The problem of the small sample size with respect to the number of features is solved by adjusting several partial generalized local regression models (rather than the global models used in [19], [8]), and combining their relevance probabilities by means of a weighted average where the weights are related to the variability of the vector components used in each model. In this way, we have developed a relevance feedback procedure whose nature is local, and it is able to adapt to the sample size in a dynamic way.

The remainder of this paper is organized as follows. Section 2 gives a detailed explanation of the proposed method, highlighting the chosen low level features, the dimensionality reduction technique and the local model applied. In Section 3 the experimental design as well as the two types of experiments performed are explained. This section also compares the method with some recent similar algorithms in literature. Finally, in Section 4 some conclusions are drawn and further extensions of the work are proposed.