Landmarks inside the shape: Shape matching using image descriptors

Highlights

• An internal-landmark based planar shape matching scheme is proposed.

• The shape landmarks are obtained using the recently proposed SPEM representation.

• Properties of SPEM that allow the use of image descriptors inside shapes are discussed.

• Statistical evidence indicates that the number of matches can be used as a similarity measure.

Abstract

In the last few decades, significant advances in image matching are provided by rich local descriptors that are defined through physical measurements such as reflectance. As such measurements are not naturally available for silhouettes, existing arsenal of image matching tools cannot be utilized in shape matching. We propose that the recently presented SPEM representation can be used analogous to image intensities to detect local keypoints using invariant image salient point detectors. We devise a shape similarity measure based on the number of matching internal regions. The performance of the similarity measure in planar shape retrieval indicates that the landmarks inside the shape silhouettes provide a strong representation of the regional characteristics of 2D planar shapes.

Introduction

In the past two decades, a large number of shape matching and retrieval methods have been proposed. Perhaps the two most important tasks for shape matching are the representation of shape and the extraction of shape descriptors. A need to measure shape similarity arises in many contexts both generic and application specific as in retrieving the most similar shapes from a database to a given query shape. Shape descriptors are the key to measuring similarity or equivalently distance between two or more shapes, and the choice of shape representation naturally affects the choice of shape descriptors as different representations make different properties explicit. In this paper, we propose internal landmarks based on a set of consistent continuous shape fields, SPEM [1], that facilitate the use of invariant image descriptors for planar shape description (see Fig. 1).

The idea of matching shapes from the same class using correspondences that can be related via geometric transformations, stemming from the work of D׳Arcy Thompson [2], motivates landmark-based methods for shape matching. Typically, the landmark points are acquired by sampling either from shape boundary [3], [4] or medial axes [5], [6], [7], [8], [9], [10]. Such landmark-based shape representations have been successfully used in shape matching, also several robust methods for point-based matching are presented [11], [12], [13], [14].

In a different and very active line of research, local detectors that are covariant to a class of transformations as support regions to compute invariant descriptors have proven to be very successful in image retrieval and object recognition [15], [16], [17], [18], [19], [20]. Matching using such a representation is commonly followed by a spatial verification procedure [21], where a planar homography transformation hypothesis is formed and agreeing matches are kept as inliers. This powerful framework progressively evolved over the years for matching images of objects.

There are, nonetheless, applications where matching silhouette data rather than image data is needed. For example, in computational anatomy, alignment of shapes (via silhouettes or silhouette boundaries) is crucial in constructing anatomical atlases for organs or characterizing change that may be a precursor to certain diseases or defects. Furthermore, availability of depth images via RGB-D cameras made it possible to extract the silhouette data for problems where color and texture may be uninformative. Hence, adopting the highly evolved body of works available for matching images to matching silhouettes is important for not re-inventing equivalent tools for silhouette matching. For instance, Obuchi et al. [22] presented a SIFT based framework for 3D shape retrieval, where the SIFT features are obtained from 2D range images of a 3D model. Yet for 2D silhouettes no information is provided in order to calculate salient local regions inside the shape domain.

On one hand, smooth distance fields given for instance by solutions to the Poisson equation are consistent, but they are highly smooth and not sufficiently expressive; hence, the schemes based on Poisson type shape representations have to employ secondary level representations in the form of skeletons [23] or descriptors such as weighted moments [24]. The reason that the proposed framework can work with planar silhouettes is the underlying shape representation that smoothly encodes pure shape information on the shape domain, is consistent and has rich and revealing characteristics.

In this paper, we show that the recently proposed Screened Poisson Encoding Maps (SPEM) [1] representation is both consistent and expressive in a way that descriptors and geometric assumptions proposed for object images can be used for object silhouettes. As the SPEM representation entails the full regional shape description in the form of a compact set of shape maps, each of which is analogous to an image intensity function, we resort to the robust image feature descriptor SIFT [25]. Our approach allows landmarks to be obtained inside the shape, which is novel. We provide statistical evidence that the landmarks inside shapes and corresponding SIFT descriptors can be used for shape matching. To demonstrate the effectiveness of the approach, we perform shape retrieval experiments on widely experimented common shape datasets and compare retrieval accuracy using precision and Bull׳s eye scores. The promising retrieval performance provides motivation for the use of proposed internal landmarks in a variety of shape analysis applications. In the following sections we will briefly describe the SPEM shape representation and why it is sound to use it with SIFT descriptors, introduce the SIFT-based internal landmark matching framework, which is for the first time utilized in the planar shape-matching problem.