A new SVM-based relevance feedback image retrieval using probabilistic feature and weighted kernel function

Highlights

• By using PCA and AGMM, probabilistic features are extracted and used for fast image retrieval.

• By using improved Relief algorithm, all training sample’ weight values are computed and utilized for feedback.

• SVM kernel function is optimized dynamically according to the feedback samples’ weight values.

Abstract

Relevance feedback (RF) is an effective approach to bridge the gap between low-level visual features and high-level semantic meanings in content-based image retrieval (CBIR). The support vector machine (SVM) based RF mechanisms have been used in different fields of image retrieval, but they often treat all positive and negative feedback samples equally, which will inevitably degrade the effectiveness of SVM-based RF approaches for CBIR. In fact, positive and negative feedback samples, different positive feedback samples, and different negative feedback samples all always have distinct properties. Moreover, each feedback interaction process is usually tedious and time-consuming because of complex visual features, so if too many times of iteration of feedback are asked, users may be impatient to interact with the CBIR system. To overcome the above limitations, we propose a new SVM-based RF approach using probabilistic feature and weighted kernel function in this paper. Firstly, the probabilistic features of each image are extracted by using principal components analysis (PCA) and the adapted Gaussian mixture models (AGMM) based dimension reduction, and the similarity is computed by employing Kullback–Leibler divergence. Secondly, the positive feedback samples and negative feedback samples are marked, and all feedback samples’ weight values are computed by utilizing the samples-based Relief feature weighting. Finally, the SVM kernel function is modified dynamically according to the feedback samples’ weight values. Extensive simulations on large databases show that the proposed algorithm is significantly more effective than the state-of-the-art approaches.

Introduction

Over the past years, content-based image retrieval (CBIR) applications have become increasingly popular. Two potential reasons for that are the vastly growing amount of digital image data and the free access to public image repositories. Both reasons are obviously connected to the dramatic price decline in the digital camera marked. Systems that allow searching, sorting, or retrieval of visual content have turned out to be essential in order to handle the vast amount of data [1]. Many CBIR systems have been developed, including QBIC, Photobook, MARS, NeTra, PicHunter, Blobworld, VisualSEEK, SIMPLIcity, and others [2]. In a typical CBIR system, low-level visual image features (e.g., color, texture, and shape) are automatically extracted for image descriptions and indexing purpose [3]. To search for desirable images, a user presents an image as an example of similarity, and the system returns a set of similar images based on the extracted features. Despite the power of the search strategies, it is very difficult to optimize the retrieval quality of CBIR within only one query process. The hidden problem is that the extracted visual features are too diverse to capture the concept of the user’s query. To solve such problems, the users can pick up some preferred images to refine the image explorations iteratively. The feedback procedure, called relevance feedback (RF), repeats until the user is satisfied with the retrieval results [4], [5].

RF, originally developed for text retrieval systems, refers to a set of approaches learning from an assortment of users’ browsing behaviors on image retrieval [6]. The main idea of RF is to let the user guide the system. The conventional process of RF is as follows: (1) from the retrieved images, the user labels a number of relevant samples as positive feedbacks, and a number of irrelevant samples as negative feedbacks and (2) the CBIR system then refines its retrieval procedure based on these labeled feedback samples to improve retrieval performance. These two steps can be carried out iteratively. As a result, the performance of the system can be enhanced by gradually learning the user’s preferences.

In recent years, there is an unprecedented development in the RF CBIR field, and many RF methods have been introduced. These RF schemes can be roughly divided into Query reweighting, Query point movement, Query expansion, and support vector machine (SVM) based methods [6], [7], [8], as shown in Fig. 1.

In the query reweighting approach, each image is modeled as a set of low-level visual features, each of which is associated with a set of representations. Relevance feedback is achieved by dynamically updating the weights of visual features, as well as the weights of feature representations, in order to accommodate the information needs of the user. Query Reweighting (QR) approaches were first proposed by Rui et al. [9], which convert image feature vectors to weighted-term vectors in early version of MARS. Chang and Chen [10] used genetic algorithms to reweight a user’s query vector, based on the user’s relevance feedback, to improve the performance of information retrieval systems. Based on region representations, Kim and Yu [11] proposed a new region filtering and region weighting method, which filters out unnecessary regions from images and learns region importance from the region size and the spatial location of regions in an image. They weighted the regions optimally and improved the performance of the region-based retrieval system based on relevance feedback. For the query reweighting approaches, no matter how the weighted or generalized distance function is adapted, the diverse visual features extremely limit the effort of image retrieval.

Another solution for enhancing the accuracy of image retrieval is moving the query point toward the contour of the user’s preference in feature space. Query point movement regards multiple positive examples as a new query point at each feedback. After several forceful changes of location and contour, the query point should be close to a convex region of the user’s interest. Liu et al. [12] proposed an efficient query point movement methods, which is able to reach any given target image with fewer iterations in the worst and average cases. They also considered the strategies to minimize the effects of users’ inaccurate relevance feedback. Nguyen et al [13] presented a cluster-based relevance feedback method, which combines two popular techniques of relevance feedback: query point movement and query expansion. From a single point initial query, query expansion provides a multiple point query, which is then enhanced using query point movement. Su et al. [14] proposed an efficient relevance feedback for content-based image retrieval, which made use of three kinds of query refinement strategies, Query Point Movement (QPM), Query Reweighting (QR), and Query Expansion (QEX), to converge the search space toward the user’s intention effectively. Wang et al. [15] proposed a new relevance feedback mechanism, which combines query point movement method and rough set theory. However, the modified query point of each feedback probably moves toward the local optimal centroid. Thus, global optimal results are not easily touched in query point movement like work.

Query expansion usually refers to the technique that uses blind relevance feedback to expand a query with new query terms, and reweigh the query terms, by taking into account a pseudo relevance set. Usually, the pseudo relevance set consists of the top ranked images returned by the first-pass retrieval. These top-ranked images are assumed to be relevant to the topic. Query expansion has proved to be an effective technique for image retrieval [16]. Chum et al. [17] bring query expansion into the visual domain via two novel contributions. Firstly, strong spatial constraints between the query image and each result allow us to accurately verify each return, suppressing the false positives which typically ruin text-based query expansion. Secondly, the verified images can be used to learn a latent feature model to enable the controlled construction of expanded queries. Okabe and Yamada [18] made two refinements to a well-known query expansion method. One uses transductive learning to obtain pseudo relevant images, thereby increasing the total number of source images from which expansion terms can be extracted. The other is a modified parameter estimation method that aggregates the predictions of multiple learning trials to sort candidate terms for expansion by importance. Singh et al. [19] propose a query expansion framework which explores user’s real time implicit feedback provided at the time of search to determine user’s search context and identify relevant query expansion terms. Kaczmarek [20] concerns clustering-by-directions algorithm. The algorithm introduces a novel approach to interactive query expansion. It is designed to support users of search engines in forming Web search queries. When a user executes a query, the algorithm shows potential directions in which the search can be continued. Rahman et al. [21] presented an image retrieval framework based on automatic query expansion in a concept feature space by generalizing the vector space model of information retrieval. Weerkamp et al. [22] proposed a general generative query expansion model that uses external document collections for query expansion: the external expansion model (EEM). The main rationale behind the model is the hypothesis that each query requires its own mixture of external collections for expansion and that an expansion model should account for this. The effectiveness of query expansion is usually better than those of query reweighting and query point movement. Nevertheless, there are still some problems unsolved for query expansion. For example, the relevant query points are too many to be efficient. Adjusting the disjunctive queries causes the expensive search cost and the results cannot escape from the restricted range that the users are able to specify. On the whole, query expansion brings out higher computation cost and more feedbacks in RF.

More recent work on image retrieval treats the relevance feedback problem as a classification problem in which machine learning techniques, for example, support vector machines (SVM), are employed. In such a paradigm, the goal is to discover a decision boundary based on relevant and irrelevant image information collected from the user. Rahman et al. [23] presented a classification-driven biomedical image retrieval framework based on image filtering and similarity fusion by employing supervised learning techniques. In this framework, the probabilistic outputs of a multiclass SVM classifier as category prediction of query and database images are exploited at first to filter out irrelevant images, thereby reducing the search space for similarity matching. Huang et al. [24] proposed a novel paired feature AdaBoost learning system for relevance feedback-based image retrieval. To facilitate density estimation in the feature learning, the author proposed an ID3-like balance tree quantization method to preserve most discriminative information. By using paired feature combination, they mapped all training samples obtained in the relevance feedback process onto paired feature spaces and employed the AdaBoost algorithm to select a few feature pairs with best discrimination capabilities in the corresponding paired feature spaces. Li et al. [25] proposed a method to alleviate the small sample problem in SVM based RF by using semi-supervised active learning algorithm which uses a large amount of unlabeled data together with labeled data to build better models. Wang et al. [26] presented a SVM relevance feedback CBIR algorithm based on feature reconstruction, in which the covariance matrix based kernel empirical orthogonal complement component analysis is utilized. Zagoris et al. [27] proposed an MPEG-like descriptor that contains conventional contour and region shape features with a wide applicability from any arbitrary shape to document retrieval through word spotting. Its size and storage requirements are kept to minimum without limiting its discriminating ability. In addition to that, a RF technique based on SVMs is provided that employs the proposed descriptor with the purpose to measure how well it performs with it. Zhang et al. [28] proposed a novel method of relevance feedback based on support vector machine learning in the content-based image retrieval system. A SVM classifier can be learned from training data of relevance images and irrelevance images marked by users. Using the classifier, the system can retrieve more images relevant to the query in the database efficiently. Liu et al. [29] proposed a SVM-based active feedback in image retrieval using clustering and unlabeled data, in which a new active selection criterion to select images for user’s feedback is designed, and unlabeled images are incorporated within the co-training framework. Lee and Lee [30] proposed an enhanced SVM which not only better selects positive/negative examples considering the reliability of the spoken segments, but emphasizes more on more reliable training examples by modifying the SVM formulation. Besides, Jun et al. [31] proposed a novel ranking model based on the learning to rank framework, in which visual features and click features are simultaneously utilized. Specifically, the proposed approach is based on large margin structured output learning and the visual consistency is integrated with the click features through a hypergraph regularizer term. Jun et al. [32] proposed a multimodal hypergraph learning-based sparse coding method for image click prediction, and applied the obtained click data to the reranking of images. In order to further improve the relevance feedback performance, Tao et al. [33] built an asymmetric bagging and random subspace SVM (ABRS-SVM) by integrating asymmetric bagging-based SVM (AB-SVM) and random subspace SVM (RS-SVM). Yu et al. [34] proposed a novel multi-view hypergraph-based learning (MHL) method that adaptively integrates click data with varied visual features. In particular, MHL considers pairwise discriminative constraints from click data to maximally distinguish images with high click counts from images with no click counts.

Owing to strong generalization capability, SVM-based relevance feedback (RF) image retrieval has become an active area of research, and many SVM-based RF approaches have been proposed in recent years. But, the existing SVM-based RF schemes have two main drawbacks: First, each feedback interaction is usually tedious and time-consuming because original visual features always have poor adaptability, flexibility and robustness. Second, the RF mechanisms treat all positive and negative feedback samples equally, which will inevitably degrade the effectiveness of SVM-based RF approaches for CBIR. In fact, positive and negative training samples, different positive training samples, and different negative training samples all always have distinct properties [35].

In this paper, we present a new SVM-based RF approach using probabilistic feature and weighted kernel function. The novelty of the proposed approach includes: (1) Probabilistic features, which have a lot of merits, such as adaptability to the data, modeling flexibility and robustness, are utilized instead of the original features. (2) All training sample’ weight values are adaptively computed using fast Relief algorithm, and SVM kernel function is optimized dynamically, which can improve significantly the effectiveness of SVM-based RF approaches.

The rest of this paper is organized as follows. Section 2 discusses the probabilistic features extraction and similarity computation. In Section 3, the feedback samples based SVM Kernel function weighting is introduced. Section 4 contains the description of our SVM-based relevance feedback system. Simulation results in Section 5 will show the performance of our scheme. Finally, Section 6 concludes this presentation.