Elsevier

Information Sciences

Volume 289, 24 December 2014, Pages 148-161
Information Sciences

Shape annotation by semi-supervised fuzzy clustering

https://doi.org/10.1016/j.ins.2014.06.025Get rights and content

Abstract

Image annotation is an important and challenging task when managing large image collections. In this paper, a fuzzy shape annotation approach for semi-automatic image annotation is presented. A fuzzy clustering process guided by partial supervision is applied to shapes represented by Fourier descriptors in order to derive a set of shape prototypes representative of a number of semantic categories. Next, prototypes are manually annotated by attaching textual labels related to semantic categories. Based on the labeled proto-types, a new shape is automatically labeled by associating a fuzzy set that provides membership degrees of the shape to all semantic categories. The proposed annotation approach provides an innovative indexing method for shape-based image retrieval. Indeed, shape prototypes represent an inter-mediate indexing level that allows a faster retrieval process since a query is matched against prototypes, instead of the whole shape database, resulting in a speed up of the retrieval. The proposed approach is tested on synthetic and real-word images in order to show its suitability.

Introduction

The exponential diffusion of large digital image collections has given rise to an increasing interest for automated tools able to efficiently store, organize and retrieve images. This has led to the growth of a promising research area referred as Content-Based Image Retrieval (CBIR) that involves the design and the development of systems able to exploit low-level visual features such as texture, color and shape for the indexing and retrieval of images [18], [9].

The extraction of low-level features related to the visual content of images is quite an easy and fairly direct task. Moreover, the use of low-level features enables the definition of suitable similarity measures to develop effective matching methods useful in CBIR processes. However, many users find it difficult to formulate search queries by directly specifying low-level visual attributes. Indeed, humans tend to recognize images and to express their content relying on high-level concepts, i.e. they usually formulate their queries in natural language by employing semantic concepts. Thus, one of the main problems that heavily affects performance of CBIR systems is represented by the “semantic gap”, defined as “the lack of coincidence between the information automatically extracted from the visual data and the semantic meaning, i.e. the interpretation that the same visual data have for a user in a given situation” [31].

In the literature a variety of approaches have been proposed in order to uncover the hidden correlation between low-level visual features and high-level concepts [21], [15]. Among these, a straightforward approach is image annotation also known as image labeling that consists in associating textual descriptions to visual content of images providing the fundamental ability to index images semantically [13], [10]. Manual annotation of images is a subjective, time-consuming and error-prone process. Hence, increasing research efforts have been addressed towards the definition of automatic methods for image annotation based on low-level visual content [35], [11], [22]. Broadly speaking, image annotation can be defined as the process that assigns textual labels to images in order to describe the semantic categories expressed in the image. In image annotation several issues require to be faced such as the choice of low-level features to describe the image visual content, the definition of the semantic categories and the association of a new image to the detected categories.

In particular, the choice of low-level features is a central tenet in image annotation. Several works have proved that visual features such as color, texture, and positioning, though important, are insufficient to convey the information that could be obtained through shape analysis of objects contained into images [3], [19], [30], [32]. Shape plays a critical role for the representation of objects contained into images becoming a key feature exploited in many applications of computer vision and image understanding for indexing and retrieval purposes. Indeed a considerable amount of information is contained in the boundaries of the objects, thus a definite opinion of the scientific community is that shapes should be considered as an essential mean to describe objects in an image. Moreover, it is recognized that shape is an important feature strictly related to human perception. In fact, most users tend to perceive images as composed of individual objects identified by their shapes and they usually retrieve images based on shapes of objects contained in images by giving examples or by sketching [23], [4].

As concerns the definition of semantic categories, in many cases the number of semantic categories is not known in advance. Experts often may have some kind of knowledge about the image domain such as the membership of a number of shapes to specific semantic categories. Of course, the effectiveness of the overall image annotation process can benefit from methods that are able to exploit such kind of domain knowledge in order to automatically discover a number of semantic categories along with the corresponding visual prototypes. Typically, the association of a shape to the semantic categories is a crisp process, i.e. a shape is exactly classified into one semantic category [27], [33]. However, due to the presence of noise and ambiguity in image features, it is very difficult or even impossible to classify shapes precisely into one category. Therefore, the uncertainty underlying the annotation process can be suitably captured by labeling shapes with multiple keywords leading to a fuzzy annotation process that assigns multiple labels to a shape together with values encoding the membership degree of the same shape to each semantic category [8], [7], [1].

To address all these aspects, in this paper we propose a fuzzy shape labeling approach for image annotation. The key idea on the basis of our approach is to equip fuzzy clustering with a partial supervision mechanism that is able to provide a useful guide during the process of automatic discovery of visual shape prototypes representative of a number of semantic categories. Semi-supervised fuzzy clustering allows to identify semantic categories by grouping together similar shapes taking advantage from the domain knowledge expressed in terms of few labeled shapes. Such knowledge may be acquired in different ways. For example, this can be explicitly supplied by a domain expert or automatically derived from the available shape collection as better explained later in this work. After clustering, for each cluster a prototypical shape is determined. Then, each prototype is manually associated to a textual label corresponding to a specific semantic category. Finally, to label a new shape, its visual descriptors are matched with visual descriptors of all prototypes and the similarity values are used to create a fuzzy set expressing the membership degrees of the shape to each identified semantic category. In this way, the proposed approach creates a two-level index structure where the first layer is represented by the database shapes and the second layer is represented by shape prototypes representative of the different clusters formed by similar shapes. This gives an advantage in terms of speed when performing shape-based image retrieval. Actually, the prototype layer acts as a primary filter that reduces the search space quickly in the retrieval process avoiding comparison between the query and all shapes. A shape query is matched only with the derived prototypes (and not with all the database shapes) and shapes belonging to clusters corresponding to the top-n similar prototypes are provided as retrieval result. Therefore the proposed annotation approach provides an innovative and efficient indexing method for shape-based image retrieval.

The remainder of the paper is organized as follows. In Section 2 we provide a brief overview of works adopting shape clustering approaches for annotation and classification. Section 3 describes the proposed approach by detailing the involved steps. Experimental results obtained by testing our approach on synthetic and real-world images are presented in Section 4. Finally, conclusive remarks are drawn in Section 5.

Section snippets

Shape clustering: a brief overview

Annotation of images on the basis of shapes of objects requires an automatic mechanism to associate high-level concepts (i.e. semantic categories) to low-level shape features. The association of semantic concepts to images can be essentially viewed as a classification process. For each shape, the membership to a semantic category has to be derived according to certain similarity measures. Then the shape has to be classified into one of the considered categories. According to the current

The proposed shape annotation approach

In the proposed approach for shape annotation two main steps can be distinguished, namely (i) semi-supervised shape clustering for the automatic derivation of shape prototypes representative of a number of semantic categories and (ii) fuzzy shape labeling for the fuzzy annotation of shapes. In Fig. 1, an overview of our proposed approach is illustrated. The input for the annotation process is represented by a collection of shapes expressed in the form of boundary coordinates. All the shape

Experimental results

To assess the effectiveness of the proposed approach, we performed a series of experiments by employing a benchmark synthetic data set and a collection of real-word images both containing marine animal shapes. Experiments on the benchmark data set were mainly aimed at deriving prototypical shapes representative of a number of semantic categories and evaluating the overall accuracy of the annotation process. Real-world shapes were employed as validation set to test the usefulness of the

Conclusions

In this paper, a fuzzy shape annotation approach has been proposed. The approach exploits a fuzzy clustering technique equipped with a supervision mechanism that is able to provide a helpful guide during the process of shape clustering in order to derive a set of visual prototypes representative of a number of semantic categories. The derived prototypes are manually annotated by attaching labels related to semantic categories. Then, a new shape is automatically labeled by associating a fuzzy

References (34)

  • Y. Rui et al.

    Image retrieval: current techniques, promising directions and open issues

    J. Visual Commun. Image R.

    (1999)
  • T. Athanasiadis et al.

    Semantic image segmentation and object labeling

    IEEE Trans. Circ. Syst. Video Technol.

    (2007)
  • I. Bartolini et al.

    WARP: accurate retrieval of shapes using phase of Fourier descriptors and time warping distance

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2005)
  • S. Belongie et al.

    Shape matching and object recognition using shape contexts

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2002)
  • A. Borras et al.

    Object Image Retrieval by Shape Content in Complex Scenes Using Geometric Constraints

    (2005)
  • G. Castellano et al.

    A fuzzy set approach for shape-based image annotation

  • V. Chander, S. Tapaswi, Shape based automatic annotation and fuzzy indexing of video sequences, in: Proc. of the 2010...
  • Y. Chen et al.

    A region-based fuzzy feature matching approach to content-based image retrieval

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2002)
  • R. Datta et al.
    (2008)
  • E. Akbas, F.Y. Vural, Automatic image annotation by ensemble of visual descriptors, in: Proc. of Conf. on Computer...
  • J. Fan, Y. Gao, H. Luo, Hierarchical classification for automatic image annotation, in: Proc. of the 30th Annual...
  • R.C. Gonzalez et al.

    Digital Image Processing

    (1992)
  • M. Inoue, On the need for annotation-based image retrieval, in: Proc. of the Workshop on Information Retrieval in...
  • D. Yankov, E. Keogh, Manifold clustering of shapes, in: Proc. of the 6th International Conference on Data Mining, 2006,...
  • A. Hauptmann et al.

    Can high-level concepts fill the semantic gap in video retrieval? A case study with broadcast news

    IEEE Trans. Multimedia

    (2007)
  • E. Klassen et al.

    Analysis of planar shapes using geodesic paths on shape spaces

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2004)
  • P. Kontschieder, M. Donoser, H. Bischof, Beyond Pairwise Shape Similarity Analysis. In Proc. of Asian Conference on...
  • Cited by (17)

    • Subspace clustering using a low-rank constrained autoencoder

      2018, Information Sciences
      Citation Excerpt :

      In this paper, we focus on the representation of data using the deep neural network method with a low-rank constraint. Actually, after representing data using LRAE, different clustering algorithms can be used for clustering [2,34,36,38]. To compare LARE with other algorithms, three classic clustering methods, that is, k-means, spectral clustering, and SSC, were used to test the performance of LRAE.

    • FOSIR: Fuzzy-Object-Shape for Image Retrieval applications

      2016, Neurocomputing
      Citation Excerpt :

      However, this approach is found to be more suitable for 3D objects only and for 2D objects the performance is found to be low. Castellano et al. [6] have proposed a fuzzy shape annotation approach for semi-automatic image annotation. The shapes are represented by Fourier descriptors in order to derive a set of shape prototypes and they are representative of a number of semantic categories.

    • Multi-level Thresholding Selection Based on Multi-Verse optimization with Levy Flight for Image Segmentation

      2019, Proceedings of IEEE 14th International Conference on Intelligent Systems and Knowledge Engineering, ISKE 2019
    • A Fuzzy Linearity-based Approach for Contour Salience Points Selection

      2019, IEEE International Conference on Fuzzy Systems
    View all citing articles on Scopus
    View full text