TSS & TSB: Tensor scale descriptors within circular sectors for fast shape retrieval

Highlights

• Two novel shape descriptors are presented, based on the tensor scale (TS) concept.

• The orientation of TS ellipses is extended to 360°.

• Experimental results with MPEG-7 and MNIST datasets validates the methods for CBIR.

• The descriptors achieve a high retrieval rate comparable to state-of-the-art methods.

• Both descriptors uses a simple distance function, with linear complexity in TSB.

Abstract

We propose two novel region-based descriptors for shape-based image retrieval and analysis, which are built upon an extended tensor scale based on the Euclidean Distance Transform (EDT). First the tensor scale algorithm is applied to extract local structure thickness, orientation, and anisotropy as represented by the largest ellipse within a homogeneous region centered at each image pixel. In this work, we extend the local orientation to 360°. Then, for the first proposed descriptor, named Tensor Scale Sector descriptor (TSS), the local distributions of relative orientations within circular sectors are used to compose a fixed-length feature vector for a region-based representation. For the second method, named Tensor Scale Band descriptor (TSB), we consider histograms of relative orientations for each circular concentric band to compose a fixed-length feature vector with linear time matching. Experimental results with MPEG-7 and MNIST datasets are presented to illustrate and validate the methods. TSS can achieve high retrieval values comparable to state-of-the-art methods, which usually rely on time-consuming correspondence optimization algorithms, but uses a simpler and faster distance function, while the even faster linear complexity of TSB leads to a suitable and better solution for very large shape collections.

Introduction

Content-based image retrieval (CBIR) concerns the problem of searching for digital images in large databases, which are similar to a query image. A particular type of CBIR system exploits the shape information as image descriptors and has important applications on several areas (e.g., fast identification of similar species to support biology research on biodiversity, trademark identification, fingerprint identification, visual-search for mobile applications, medical tumor shape retrieval [17], [26], [47], [50]).

In order to accomplish these goals, a shape descriptor should be simple, compact, insensitive to noise, affine-invariant, and at the same time contain all relevant information to distinguish different images [18]. Preferably, the matching algorithm used by a shape descriptor should also be fast in order to be suitable on large datasets, since the feature extraction can often be performed offline. Many proposed shape descriptors yield a high accuracy score but sacrifice performance with a high computational time, mainly relying on Dynamic Programming (DP) for the matching algorithm to establish correspondences. To exemplify, the inner-distance shape context (IDSC-DP) [23] and Height Functions [49] uses a DP scheme for the shape matching.

In this work, we present new shape descriptors designed to achieve a high accuracy with fast distance functions. Besides not using DP, our matching algorithms are also simpler than OCS (Optimal Correspondent Subsequence) used by BAS [5], OSB (Optimal Subsequence Bijection) and the Hungarian algorithm used in [6], and also more efficient than the one used in TSDIZ [2]. In relation to specially proposed methods for dealing with large datasets, such as Hough Transform Statistics (HTS) and HTS-neighborhood (HTSn) [44], our matching algorithms have a lower computational complexity, for non-aligned shapes, and higher retrieval rates.

Our methods are based on the tensor scale concept (TS) [31], [32] — a morphometric parameter yielding a simultaneous representation of local structure orientation, thickness, and anisotropy. Tensor scale has been demonstrated for image filtering [32], [51], segmentation [35], registration [33], shape descriptors [2], [25], detection of contour saliences [3], [4], medical image interpolation [52], [53], removal of partial volume effects in rendering [43] and also in quantifying local morphometry in complex quasi random networks of trabecular bone [37], [38]. The algorithm to compute tensor scale, as originally proposed [31], [32], is computationally expensive. To address this problem, Andaló et al. proposed a simpler and yet effective implementation of the original method [2]. More recently, Xu et al. introduced a precise analytic approach for n-dimensional images [51].

One contribution of this paper is the revision of the algorithmic TS computation, as proposed by Andaló et al., extending the ellipse’s orientation to 360°. Based on this richer TS model, we propose two novel shape descriptors, with greater discrimination power in relation to previous TS-based works, for shape-based image retrieval: Tensor Scale Sector (TSS) and Tensor Scale Band (TSB) descriptors.

The works about shape matching can be coarsely divided into four major categories: pairwise matching, context-based similarity, learning based approaches, and combined classifier approaches. The methods from the pairwise category extract a feature vector from the shapes and use a distance function to compute the perceptually dissimilarity (or similarity) between a pair of shapes. On the other hand, context-based methods exploit the similarity context of all database instances, usually provided as a result of a pairwise matching, to boost the results, and are important to deal with large intra-class variations [8], [28], [54]. Unsupervised learning approaches are usually inspired by the Bag-of-Words (BoW) paradigm [11], [42], where a shape vocabulary is constructed by partitioning the feature space. The quality of this partition is a crucial factor on the discriminative power of the resulting descriptor, which involves the analysis of features extracted from several shapes from the dataset [7]. The methods in the fourth category fuses multiple features obtained with different descriptors, combining their strengths for the purpose of better retrieval accuracy [41].

In this work, we focus on shape description and matching, proposing novel descriptors for pairwise matching that can be used with re-ranking techniques to boost the results, and also with combined classifier approaches.

There are three types of shape descriptors, contour-based, region-based, and skeleton-based [20]. Skeleton matching usually performs better in handling objects with articulated parts, but their complex representations, in the form of a tree or graph, usually require graph edit operations [15], [30] in the matching process leading to a higher computational complexity [39], [40]. Differently from contour-based methods, such as BAS [5], TSDIZ [2] and Height Functions [49], our proposed descriptors are region-based methods, having some advantages, such as not being limited to a specific topology with only one closed contour, and also less sensitive to shape segmentation errors, noise and partial occlusions [27]. Our methods are more versatile, for instance, they can handle non-binary images (multiple labels), objects with holes, and, theoretically, can be extended to 3D.

In relation to previous TS descriptors (TSD [25] and TSDIZ [2]), the proposed descriptors TSS and TSB are more accurate and have faster matching algorithms. TSS incorporates spatial information by the use of circular sectors and TSB by the use of concentric bands around a central point, which are much more discriminative than the simple normalized orientation histogram used by TSD. While TSDIZ is a contour-based method, TSS and TSB are region-based methods, opening new perspectives for novel applications. The features extracted from the ellipses are also more sophisticated, considering 360°, using sector’s relative orientations, and in the case of TSS, applying the sum of the absolute values of the responses, inspired by SURF [9], to detect different orientation patterns.

This paper is organized as follows: Section 2 presents the tensor scale previous relevant work, including its EDT-based implementation and how to extend it to ellipses with 360°, as used in this work. Then, our novel tensor scale descriptors, TSS and TSB, are shown in Sections 3 and 4, respectively, and the analysis of their computational complexity is presented in Section 5. The experimental evaluation is conducted in Section 6. Section 7 states our conclusions and discusses future work.