Elsevier

Neurocomputing

Volume 259, 11 October 2017, Pages 154-158
Neurocomputing

High resolution non-rigid dense matching based on optimized sampling

https://doi.org/10.1016/j.neucom.2016.07.076Get rights and content

Abstract

A high resolution dense matching algorithm is presented for non-rigid image feature matching in the paper. For high resolution non-rigid images, telephoto lens is helpful in capturing fine scale features like cloth fold, pigmentation and skin pores. It brings us serious image noises which are less texture and bokeh, respectively. In order to avoid mismatch and non-uniform matching, we propose an optimized sampling method based on Gibbs dense sampling considering both texture feature similarity and spatial consistency. In the processing, first we extract connected image patches by triangulation among confidence matched point sets. Then our sampling method is executed in each connected image patch. We propose a judgment for matching points on the image patch boundary. Markov Random Field (MRF) model formulates the problem of dense matching as a Bayes decision task. Experiments are design to demonstrate the effective and efficiency of our method with active skin image data.

Introduction

When we recover object shape through binocular stereo vision, stereo matching is a primary step. The distribution of matched points is beneficial to the recovery of object shape, and the number of matched points determines the degree of recovery for details on object surface. There are two categories of stereo matching phases: sparse matching and dense matching. In sparse matching, candidate feature points are extracted by feature descriptor. In a non-rigid image, we find that most of the extracted feature points were distributed within the texture grid in the form of bright spots. The slight differences between stereo images caused by light influence are the bright spots. In our research, we consider a dense matching algorithm by considering texture feature similarity [1], [2]and spatial consistency.

Compared with rigid dense matching research [3], [4], [5], non-rigid distortion and feature noise are the main important problems that affect the efficiency of dense matching. A large amount of previous work has been done for non-rigid surface reconstruction [6], [7], [8], [9], [10]. Most of the previous approaches that bring the spatial arrangement of points into account are computationally expensive, and are therefore not feasible for dense matching. HaCohen [11] proposed a reliable local sets of dense matching between a pair of images either sharing some content but exhibiting dramatic photometric and geometric variation. By this algorithm matching two general images that exhibit high variability in appearance is far more complicated. Torki [12] proposed an efficient method for dense matching in a lower-dimensional subspace that simultaneously encodes spatial consistency and feature similarity. However, this method still requires exact spectral decomposition for subspace learning. Hamid [13] improved Torkis work by using the subset of high confidence matches as spatial priors, and learning a subspace that reduces the confusion among the remaining set of points. The method of subspace matching could not complete dense matching within less texture images. Except stereo matching, non-rigid surface filtering was proposed by Zollhfer [14] with the help of RGB-D camera. His method obtained the non-rigid shape without dense matching. Tepole [15] combined multi-view stereo and isogeometric analysis together to characterize skin kinematics. They used simple finite element meshes to parameterize the deformation with poor spatial resolution. From the coordinates of a few material points, B-spline tensor product patches with a prescribed parameterization can smoothly interpolate deformations. In our research, we classify texture features [16] without material points on object surface.

Telephoto lenses are installed in SLR camera for capturing high resolution non-rigid images with fine scale features, such as cloth fold, pigmentation and skin pores, etc. In the mean time, it brings us serious image noises with less texture and bokeh, respectively. Fig. 1 shows us the difficulties for high resolution non-rigid dense matching including feature noise by light reflection, less texture and dokeh. The colorful labeled points in Fig. 1 are feature points.

In order to avoid mismatch caused by feature noise and non-uniform matching caused by less texture and dokeh, we propose an optimized sampling method based on Gibbs dense sampling considering both texture feature similarity and spatial consistency. The overview of our method will be illustrated in Fig. 2. In the processing, first we extract connected image patches by triangulation among confidence matched point sets. Then our sampling method is executed in each connected image patch. There will be left matching blanks along the image patches boundary. We design a judgment to deal with the matching points on the boundary. Markov Random Field (MRF) model formulates the problem of dense matching as a Bayes decision task. Our proposed framework is effective with two key elements:

It solves the uniform dense matching within less texture region. Sparse matching can lead to a lot of texture and part of shape missing. It is also the research focus in dense matching. Gibbs dense sampling is useful to balance the matched points in image pairs.

Instead of global optimization in high resolution image, a triangulation is processed relying on confidence matched points, which separated image into many image patches. Fast convergence with high performance is effective for dense matching based on MRF model. Meantime a judgment of point position is proposed to avoid matching blanks on the patch boundary.

The remainder of this paper is organized as follows. Local feature sampling algorithm is detailed in Section 2. After confidence matched points are computed, we presents an optimized sampling method based on Gibbs dense sampling in Section 3, also the judgment for boundary points is illustrated in the section. The process of dense matching among image patches is described in Section 4. Experiment results with non-rigid images and the evaluation are shown in Section 5. We conclude our research in Section 6.

Section snippets

Preliminaries

Rectification is a process used to facilitate the analysis of a stereo pair of images by making it simple to enforce the two view geometric constraint. For a pair of related views the epipolar geometry provides a complete description of relative camera geometry. Once the epipolar geometry has been determined it is possible to constrain the match for a point in one image to lie on the epipolar line in the other image. The process of rectification makes it very simple to impose this constraint by

Optimized sampling method

Gibbs sampling [20] is an algorithm to generate a sequence of samples from the joint probability distribution of two or more random variables. The purpose of such a sequence is to approximate the joint distribution, or to compute an integral. Suppose we want to obtain k samples of X=(x1,,xn) in less texture region from a joint distribution(x1, x2). Denote the ith sample by X(i)=(x1(i),,xn(i)). We proceed as Algorithm 2 in the following:

After local feature sampling, confidence matched points

Dense matching in image patches

Suppose we have a continuous non self-occluding surface. To preserves the topology, each image must have the same topology as the surface does. The disparity gradient is defined as |rLrR|1/2|rLrR|.VectorrL is in left image, and its corresponding vector rrR is in right image. In the paper, there is a poof that a disparity gradient limitation of less than 2 implies that the matches between the two images preserve the topology of the surface.

Suppose pi, pj are matched points, and qi, qj are

Evaluation test

The experimental environment is followed configured in the paper: CPU is the Intel (R) Core (TM) 2 DUO 2.10 GHz; the memory is 2GB; the operating system is windows XP. The experiment is carried on Visual Studio 2010 platform with OpenCV2.0 library. For the evaluation of our proposed method for non-rigid dense matching, we take test images on human skin with resolution of 35042336. The real skin area is about 4mm × 3mm. After sparse matching, triangulation is done. The result is shown in Fig. 1,

Conclusion

In high resolution images, serious image noises which are less texture and bokeh happened. If we use dense matching algorithm directly, not too many corresponding triangles are found because of some vague parts in original images. The disparity map is not satisfied with some edge traces. Also, dense matching is done in related huge regions. For high resolution images, sampling method is the key to complete dense matching. An optimized sampling method is proposed for non-rigid dense matching

Acknowledgment

This research was supported by National Natural Science Foundation of China (No. 61402320, No. 61332015); by grants from National 973 Program (2015CB352501); by the Shandong Provincial Natural Science Foundation, China (ZR2013FQ029).

Qian Zhang She received her Ph.D degree from Dept. of Computer Science in Kyungwon University of Korea, in 2010. She currently does her research in Shandong University as a post doctor. And also she is an associate professor of Taishan University, China. Her research interests include Image Processing, computer vision.

References (26)

  • Y. Hacohen et al.

    Non-rigid dense correspondence with applications for image enhancement

    Proceedings of the ACM Transactions on Graphics

    (2011)
  • M. Torki et al.

    One-shot multi-set non-rigid feature spatial matching

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2010)
  • R. Hamid et al.

    Dense non-rigid point-matching using random projections

    IEEE Conference on Computer Vision and Pattern Recognition

    (2013)
  • Cited by (1)

    Qian Zhang She received her Ph.D degree from Dept. of Computer Science in Kyungwon University of Korea, in 2010. She currently does her research in Shandong University as a post doctor. And also she is an associate professor of Taishan University, China. Her research interests include Image Processing, computer vision.

    Changhe Tu He got his Bachelor, Master, and Ph.D degrees in 1990, 1993 and 2003, respectively, all from Shandong University, China. From 2001.1 to 2002.10, now he is an Associate Dean at School of Computer Science and Technology, Shandong University. His research interests include computer graphics, digital geometry processing and computer aided geometric design.

    View full text