Graph-matching-based correspondence search for nonrigid point cloud registration

Highlights

• We propose a graph-matching model for nonrigid registration and its solver.

• Input 3D data is divided into small rigid pieces to find correspondences.

• We use graph-matching for finding coarse correspondences under large motions.

• We propose a many-to-one matching problem to find semantically identical parts robustly.

• Experiments show that the proposed method outperforms existing algorithms.

Abstract

Nonrigid registration finds transformations to fit a source point cloud/mesh to a target point cloud/mesh. Most nonrigid registration algorithms consist of two steps; finding correspondence and optimization. Among these, finding correspondence plays an important role in registration performance. However, when two point clouds have large displacement, it is hard to know correct correspondences and an algorithm often fails to find correct transformations. In this paper, we propose a novel graph-matching-based correspondence search for nonrigid registration and a corresponding optimization method for finding transformation to complete nonrigid registration. Considering global connectivity as well as local similarity for the correspondence search, the proposed method finds good correspondences according to semantics and consequently finds correct transformations even when the motion is large. Our algorithm is experimentally validated on human body and animal datasets, which verifies that it is capable of finding correct transformations to fit a source to a target.

Introduction

Point cloud registration is a key process for capturing full shapes of 3D objects. Capturing the point cloud of a 3D object is a fundamental process for computer vision and robotics problems such as 3D localization, 3D scene/object recognition, 3D reconstruction in augmented/virtual reality and generating free-viewpoint videos. However, capturing a high-quality 3D object in a single shot is challenging due to missing points occluded by other parts or sparse points in parts captured from distant surfaces. These are inevitable shortcomings we face in acquiring 3D data. Registration of simultaneously captured multi-view point clouds can complement missing and low-resolution regions, however, this is not enough for capturing the hidden sides of an object, such as an armpit or a thigh of a human body. On the other hand, when an object is deformable, nonrigid registration can supplement these by combining point clouds of different frames representing different poses of the object. For example, a hidden or missing human body part of one frame can be complemented by another frame in which the corresponding part is exposed to the camera.

Nonrigid registration can transform another point cloud of the same deformable object into the reference point cloud. Nonrigid registration algorithms generally consist of two procedures, one is finding correspondences and the other is updating parameters in the way that reduces some metric between the correspondence pairs. The difference of nonrigid registration from rigid registration is that nonrigid registration finds transformation parameters for each point to describe the motion to fit a source frame to a target frame whereas rigid registration finds one set of transformation parameters. Hence, for nonrigid registration, it is important to find a correct match for every point to determine where to transform it. In order to acquire a high-quality 3D shape from a deformable object, there are two requirements: A good registration algorithm that can find correct matches between two point clouds of an object that has large motions, and an ability to process two massive point clouds containing hundreds of thousands of points. However, satsifying both conditions is difficult for nonrigid registration.

The nonrigid iterative closest point (nonrigid ICP) method, the most popular nonrigid registration technique, finds correspondences as the closest points in every iteration hoping to find correct matches in the end. However, nearest search based on point position are sensitive to noises. Various feature extraction methods like Shape Contexts (SC) (Belongie et al., 2002) and Point Feature Histogram (PFH) (Rusu et al., 2008a, Rusu et al., 2008b) are developed to improve correspondence search by characterizing local surface and shape of each points. Further, Chui and Rangarajan (2003) relax binary correspondence condition of ICP to soft-assignment. On the other hand, methods to find robust transformations from erroneous matching are developed. Ma et al. (2014) remove outliers from a putative correspondence set containing erroneous matching and Ma et al., 2015, Ma et al., 2013 estimate transformation parameters from erroneous matching using robust $L_{2}E$ estimator. However, the performance degrades when an underlying motion of two point sets is large and often computing features for massive point sets are inexecutable due to memory limit. Another main stream of nonrigid registration is to formulate registration problem as a probability density estimation problem modeled as Gaussian Mixture Models (Jian and Vemuri, 2010, Wang et al., 2015, Wang et al., 2014, Myronenko and Song, 2010). These approaches are proposed to handle noises, missing, and outliers, however, they are prone to errors when two point clouds under registration are related to large motion.

In this work, we propose a novel correspondence matching and initial alignment methods for nonrigid registration based on graph matching. Graph matching finds correspondences of vertices of two graphs by maximizing similarities between vertices and edges of the two graphs. This makes the matching process consider neighborhood connectivity, which contains both the local and global information. In order to incorporate graph matching in nonrigid registration, we first divide point clouds into a number groups and apply graph matching to find groupwise correspondences. In addition, we relax one-to-one matching problem to many-to-one matching problem to improve registration performance. As a consequence, our method can effectively match semantically identical human body parts of two frames even when the motions are large. Furthermore, the search based on a small number of point groups reduces memory usage compared to other nonrigid registration methods based on dense features, which makes our method favorable in handling large-scale point clouds. Using the matching results obtained from graph matching, we compute groupwise transformation parameters to find initial nonrigid alignment. These initial alignments are coarse but they provide great clues for the following refinement step. We evaluate our methods on the Lincoln human motion dataset (Collet et al., 2015) and the TOSCA animal dataset (Bronstein et al., 2008). In short, the contribution of the paper can be summarized as follows:

• We propose a novel graph-matching-based correspondence search method for nonrigid registration, which provide coarsely aligned point cloud even when the underlying motion is large.

• We propose many-to-one matching problem for nonrigid registration and its solver to enhance robustness of matching and empirically show that many-to-one matching preserves more local structures.

• We propose an initial alignment method which refines the rough registration results produced by graph matching procedure.

• We propose a nonrigid registration framework which can handle large-scale point clouds for high-quality 3D reconstruction.

The rest of the paper is organized as follows. Short review of a literature are introduced in 2. The proposed graph-matching-based nonrigid registration method is explained module by module in Section 3. Qualitative and quantitative experiments on human body and animal body datasets are conducted in Section 4, followed by conclusion and future work in Section 5.