Interactive image retrieval using fuzzy sets

doi:10.1016/S0167-8655(01)00045-9

Pattern Recognition Letters

Volume 22, Issue 9, July 2001, Pages 1021-1031

https://doi.org/10.1016/S0167-8655(01)00045-9 Get rights and content

Abstract

We present an image retrieval system which permits the user to submit a coarse initial query and continuously refine it. The user's relevance feedbacks is modeled by fuzzy sets, and is used to discover and use the more discriminatory features for the given query. The proposed system uses a dissimilarity measure based on the fuzzy integral.

Introduction

The recent increase in the size of multimedia information repositories has made Content-based image retrieval (CBIR) one of the most active research areas in the past few years. Unlike traditional database techniques which retrieve images based on exact matching of keywords, CBIR systems represent the information content of an image by visual features such as color, texture, and shape, and retrieve images based on similarity of features. Therefore, a good similarity (or equivalently a dissimilarity) measure is essential for the effective retrieval in such systems. Unfortunately, the meaning of similarity is rather vague and difficult to define. In general, different similarity measures capture different aspects of perceptual similarity between images. What makes the problem even harder is the fact that different features do not contribute equally, and therefore, cannot be considered equally important for computing the similarity (or dissimilarity) between images. For example, when a user perceives two images as being similar, what he or she really means is that the images are similar either in an individual feature, or in some combination of features. In this paper, we present a novel method that models the user's positive and negative feedbacks by fuzzy sets. These fuzzy sets are then used to learn the feature relevance weights. We show that the learned weights can be used as fuzzy densities in a fuzzy integral based retrieval system. We also show that the learned feature relevance values can be used as weights in a weighted Euclidean distance-based retrieval system.

The rest of this paper is organized as follows: Section 2 summarizes related work on similarity measures and feature relevance. Section 3 gives an overview of fuzzy integrals and fuzzy measures. Section 4 discusses how to use the Choquet integral as a dissimilarity measure. Section 5 describes how to model the user's feedback using fuzzy sets and how to update the feature relevance values. Experimental results over more than 3000 images for testing the learning behavior of the system are given in Section 6. Concluding remarks are given in Section 7.

Section snippets

Similarity measures

The retrieval process of images involves evaluating the degree of similarity between the query image feature vector and the feature vectors of the images stored in the database. Several similarity measures have been used in the past few years. The Euclidean (or weighted Euclidean) distance has been used in several CBIR systems (Flickner et al., 1995, Bach et al., 1998). A hybrid neural network algorithm is used in (Ma and Manjunath, 1996) to learn the visual similarity by clustering patterns in

Fuzzy measures

Let $X ={x_{1},…,x_{n}}$ be an arbitrary set. A set function $g:2^{X} →[0,1]$ is a fuzzy measure if it satisfies the following three axioms:

1.
Boundary conditions: $g(∅)=0,g(X)=1$ ,
2.
Monotonicity: if $A,B⊂ X$ , and A⊂B, then g(A)⩽g(B),
3.
Continuity: if {A_i} is an increasing subsequence of subsets of $X$ , then $lim_{i→∞} g(A_{i})=g ⋃_{i=1}^{∞} A_{i} .$
A fuzzy measure is a Sugeno measure (or a λ-fuzzy measure) if it satisfies the following additional condition for some λ>−1
4.
$∀A,B⊂ X$ with A∩B=∅ $g(A∪B)=g(A)+g(B)+λg(A)g(B).$

For a finite set

X

, the fuzzy

The Choquet integral as a dissimilarity measure

The fuzzy integral can be interpreted as a fuzzy expectation (Sugeno, 1977), the maximal grade of agreement between two opposite tendencies (Wierzchon, 1989), or the maximal grade of agreement between the objective evidence and the expectation (Tahani and Keller, 1990). In this work, we develop a retrieval system that uses the Choquet integral as a dissimilarity measure. The system can be easily modified to use the Sugeno integral.

Let $X ={f_{1},…,f_{n}}$ be a set of the n features used in the CBIR

Feature relevance

Let gⁱ=g({f_i}) represent the relevance of feature i. The relevance values will also serve as the fuzzy densities to be used in the Choquet integral. In this work, we restrict gⁱ to be a λ-measure. This restriction will simplify the system considerably since only the fuzzy densities need to be learned in order to compute the fuzzy integral. Other measures, such as possibility measure, which provide a way to compute the measure of a union set from a pair of component measures, can also be used in

Data and features

The image database for our experiments consists of 3584 images of size 64×64 obtained by dividing 512×512 texture images into 64 subimages. The 56 original images¹ are texture images from the Brodatz album (Brodatz, 1966), and are shown in Fig. 4. We use 24 Gabor filters (4 scales and 6 orientations) for feature extraction. The normalized mean and standard deviation of

Conclusion

In this paper, we have proposed an image retrieval system where human and computer can interact with each other to improve the retrieval performance of a CBIR system. This approach permits the user to submit a coarse initial query and continuously refine it using positive and negative relevance feedbacks. For both the Choquet integral-based dissimilarity and the Euclidean distance, our system requires the user to provide only a vague and more natural description of the retrieved images. These

Acknowledgements

The authors would like to thank the anonymous reviewers for their valuable comments. Partial support of this research was provided by a grant from the University of Memphis New Faculty Research Initiation Awards. This support does not necessarily imply endorsement by the University of research conclusions.

References (25)

M. Grabisch et al.
Fuzzy measure of fuzzy events defined by fuzzy integrals
Fuzzy Sets and Systems
(1992)
T. Murofushi et al.
A theory of fuzzy measures: representations, the Choquet integral, and null sets
J. Math Anal. Appl.
(1991)
Agnew, B., Faloutsos, C., Wang, Z., Welch, D., Xue, X., 1997. Multimedia indexing over the web. In: SPIE Conference,...
Bach, J.R., Fuller, C., Gupta, A., Hampapur, A., Horowitz, B., Jain, R., Shu, C., 1998. The virage image search engine:...
Bhanu, B., Peng, J., Qing, S., 1998. Learning feature relevance and similarity metrics in image databases. In: IEEE...
Bouet, M., Djeraba, C., 1998. Visual content based retrieval in an image database with relevant feedback. In: Internat....
P. Brodatz
Textures: A Photographic Album for Artists & Designers
(1966)
Buckley, C., Salton, G., 1995. Optimization of relevance feedback weights. In:...
A. Celentano et al.
Feature integration and relevance feedback analysis in image similarity evaluation
J. Electronic Imaging
(1998)
M. Flickner et al.
Query by image and video content: the qbic system
IEEE Comput.
(1995)

M. Grabisch

Fuzzy integrals for classification and feature extraction

M. Grabisch et al.

Fundamentals of Uncertainty Claculi with Applications to Fuzzy Inference

(1995)

Cited by (24)

TMR: Towards an efficient semantic-based heterogeneous transportation media big data retrieval
2016, Neurocomputing
Citation Excerpt :
To solve this problem, a feasible solution is to use fuzzy matching. In fact, fuzzy approaches have been applied in many fields of information retrieval such as interactive image retrieval [8], content-based image retrieval [3], decision support system [2] and Web resources retrieval [7]. In current approaches, semantic features are stored in the server׳s database and separated from the media documents.
In media retrieval system for intelligent transportation, media data variety and heterogeneity have been one of the most critical features. Documents with different formats may express similar semantic information, thus, searching documents reflecting users׳ intention has been a crucial and important task. For solving this problem, this paper proposes a novel semantic-based heterogeneous transportation media retrieval (TMR) approach to improve the performance. TMR supports the function of retrieving various media types such as image, video, audio and text by using a single media type. Firstly, semantic fields are extracted from the user annotating and automatic learning to express the users׳ intention. Secondly, ontology is used to represent the semantic fields of a media, and the ontology represented semantic information is saved together with the media document data. Thirdly, the semantic field adjustment process is described. Finally, fuzzy matching is employed to measure the similarity between the users׳ intention and media documents. For the returned results, we carry out the performance evaluation models in comparison with the existing approaches. Experimental result indicates the superiority of TMR in term of precision rate, computing speed, storage cost and user experience.
A new fuzzy logic based ranking function for efficient Information Retrieval system
2015, Expert Systems with Applications
Citation Excerpt :
Chen (2011) presented a new ranking function based on the geometric mean averaging operator to handle the similarity problems of generalized fuzzy numbers. Although, this research work is limited to fuzzy Information Retrieval and fuzzy datasets (Chen & Chen, 2003; Chen, Horng, & Lee, 2001; Chiang, Chow, & Hsien, 1997; Devedzic & Pap, 1999; Frigui, 2001). It is evident that, very few works are reported in literature to develop fuzzy logic based ranking function during last one decade only despite of advantages already mentioned above.
The relevant documents from large data sets are retrieved with the help of ranking function in Information Retrieval system. In this paper, a new fuzzy logic based ranking function is proposed and implemented to enhance the performance of Information Retrieval system. The proposed ranking function is based on the computation of different terms of term-weighting schema such as term frequency, inverse document frequency and normalization. Fuzzy logic is used at two levels to compute relevance score of a document with respect to the query in present work. All the experiments are performed on CACM and CISI benchmark data sets. The experimental results reveal that the performance of our proposed ranking function is much better than the fuzzy based ranking function developed by Rubens along with other widely used ranking function Okapi-BM25 in terms of precision, recall and F-measure.
An evaluation of several fusion algorithms for anti-tank landmine detection and discrimination
2012, Information Fusion
Citation Excerpt :
Exhaustive search can be used to assign weights for small collections of algorithms, but is too computationally burdensome for large collections. The Choquet integral has been investigated for information fusion by many researchers [84–89,45,90–93]. This integral defines a family of generally nonlinear aggregation operators on some function of the algorithm confidence values, which we will refer to as a decision statistic.
Many algorithms have been proposed for detecting anti-tank landmines and discriminating between mines and clutter objects using data generated by a ground penetrating radar (GPR) sensor. Our extensive testing of some of these algorithms has indicated that their performances are strongly dependent upon a variety of factors that are correlated with geographical and environmental conditions. It is typically the case that one algorithm may perform well in one setting and not so well in another. Thus, fusion methods that take advantage of the stronger algorithms for a given setting without suffering from the effects of weaker algorithms in the same setting are needed to improve the robustness of the detection system. In this paper, we discuss, test, and compare seven different fusion methods: Bayesian, distance-based, Dempster–Shafer, Borda count, decision template, Choquet integral, and context-dependent fusion. We present the results of a cross validation experiment that uses a diverse data set together with results of eight detection and discrimination algorithms. These algorithms are the top ranked algorithms after extensive testing. The data set was acquired from multiple collections from four outdoor sites at different locations using the NIITEK GPR system. This collection covers over 41,807 m² of ground and includes 1593 anti-tank mine encounters.
Application of artificial intelligence methods to content-based image retrieval
2011, Advances in Imaging and Electron Physics
Citation Excerpt :
Conversely, there are also those who oppose the interference and proposed auto-relevance feedback methods that did not require human involvement at any time (see Section 3). Frigui (2001), for example, modeled the user's positive and negative feedback by using fuzzy sets to discover the more discriminatory features through relevance weights for a given query. The system used a dissimilarity measure based on the Choquet integral and applied it to the learned feature relevance values of each iteration, which are considered to be fuzzy densities.
Every single day, millions of people worldwide create a vast amount of digital images and the need to index and retrieve these images has become more than obvious. The majority of retrieval systems are based on descriptors created by aggregating certain amounts of a specific feature from the image itself, thus creating a representation of its content. Over the years, many conventional probabilistic methods have been proposed that attempt to solve the problem of image retrieval. In this chapter, a variety of such methods are presented to pave the path that eventually will lead to an answer of whether or not artificial intelligence (AI) methods such as fuzzy logic are the correct way to proceed to enhance the image retrieval process. Moreover, since an abundance of swarm intelligence techniques have been used to find a solution for many non-deterministic polynomial time–complete problems, a discussion is being made concerning the possibility of the successful application of these methods to content-based image retrieval. This chapter provides an introduction to the advances in image retrieval as they occurred over the years, beginning with conventional methods. Readers are familiarized with AI techniques and a fully functional fuzzy ant colony image retrieval system is presented so that readers might grasp the “how,” “where,” and “why” AI techniques are applied to image retrieval.
Using relevance feedback with short-term memory for content-based spine X-ray image retrieval
2009, Neurocomputing
Managing large medical image databases has become a challenging task as more medical images are produced and stored in digital format. Computer-aided decision support for content-based image retrieval (CBIR) is an essential tool for medical image management. This paper presents a novel hybrid relevance feedback (RF) system for shape-based retrieval of spine X-ray images. A new shape similarity measure that considers both whole shape and partial shape matching is presented. The proposed RF architecture includes separate retrieval and feedback modes to solicit user's opinion for refining retrieval results. A unique short-term memory approach is implemented to avoid repeated request for user's feedback on the same, already approved, and retrieved relevant images. An automatic weight updating scheme is developed to present the images on which it is best for the user to provide feedback. Incorporating all these unique features, the proposed RF retrieval system is able to reduce the gap between high-level human visual perception and low-level computerized features. Experimental results show overall retrieval accuracy improvement of 22.0% and 17.5% after the second feedback iteration for retrieving spine X-ray images with similar osteophytes severity and type, respectively.
A relevance feedback CBIR algorithm based on fuzzy sets
2008, Signal Processing: Image Communication
Citation Excerpt :
Regarding the second topic, an interesting approach can be found in [9], where the authors propose a dissimilarity measure based on the fuzzy integral which performs better than the Euclidean distance, and then construct an image retrieval system based on this measure.
CBIR (content-based image retrieval) systems attempt to allow users to perform searches in large picture repositories. In most existing CBIR systems, images are represented by vectors of low level features. Searches in these systems are usually based on distance measurements defined in terms of weighted combinations of the low level features. This paper presents a novel approach to combining features when using multi-image queries consisting of positive and negative selections. A fuzzy set is defined so that the degree of membership of each image in the repository to this fuzzy set is related to the user's interest in that image. Positive and negative selections are then used to determine the degree of membership of each picture to this set. The system attempts to capture the meaning of a selection by modifying a series of parameters at each iteration to imitate user behavior, becoming more selective as the search progresses.
The algorithm has been evaluated against four other representative relevance feedback approaches. Both the performance and usability of the five CBIR systems have been studied. The algorithm presented is easy to use and yields the highest performance in terms of the average number of iterations required to find a specific image. However, it is computationally more expensive and requires more memory than two of the other techniques.

View all citing articles on Scopus

View full text

Interactive image retrieval using fuzzy sets

Abstract

Introduction

Section snippets

Similarity measures

Fuzzy measures

The Choquet integral as a dissimilarity measure

Feature relevance

Data and features

Conclusion

Acknowledgements

Fuzzy Sets and Systems

J. Math Anal. Appl.

Textures: A Photographic Album for Artists & Designers

Feature integration and relevance feedback analysis in image similarity evaluation

J. Electronic Imaging

Query by image and video content: the qbic system

IEEE Comput.

Fuzzy integrals for classification and feature extraction

Fundamentals of Uncertainty Claculi with Applications to Fuzzy Inference