Patch-based reconstruction of surfaces undergoing different types of deformations

Abstract

We deal with the reconstruction of surfaces that deform under a variety of conditions. The deformation can range from no extension to a certain degree of extensibility. The deformed surface is reconstructed from a single image, given a 3D reference shape. This shape corresponds to the undeformed state of the surface and can be computed using any appropriate technique. In particular, we use homographies defined from two views of the surface. To proceed with the 3D reconstruction of the deformed surface, we assume that the deformations are locally homogeneous and that the overall surface deformation can be obtained by combining the local homogeneous deformations. For this purpose, the surface is split into small patches. For each patch, a mapping between the undeformed and the deformed shapes is computed. The mapping is specified by using the quadratic deformation model Fayad et al. (Proceedings of British Machine Vision Conference (BMVC), 2004). As a result, given the undeformed shape, we define an optimization procedure whose goal is to estimate the 3D positions of deformed points in each image. The optimization is performed on each patch, independently of the others. The experimental results show that this approach allows precise reconstruction of a wide class of real deformations.

Appendix: Rigid reconstruction

Assuming that the surface is not in motion and remains in the undeformed state for a while, the camera starts capturing the scene and surface while moving. A selection of frames that cover different fields of view is then made. One of them is adopted as base image. We compose image pairs, each consisting of the base image plus another from the frame selection, called support image. In addition, the points $\mathbf {p}_{i}^{\circ }$ are randomly grouped into distinct sets, each of which includes 3 points and can share no points with other sets. An estimate of the undeformed shape could be found per image pair through the following 2-view rigid reconstruction algorithm:

Fig. 9

Homography between the base and support views

Step 1: The essential matrix between the base image and the support image is determined by using the image points in one image and the corresponding points in the other. Step 2: By decomposing the essential matrix, we can compute the rotation $\mathbf {R}$ and a scaled translation $\bar{\mathbf {t}}$ between the 2 views. As indicated in [23], the decomposition yields four combinations of $\mathbf {R}$ and $\bar{\mathbf {t}}$, one of which is feasible. Positive depth constraint is then used to disambiguate the physically impossible solutions. Let us assume that $\mathbf {R}$ and $\bar{\mathbf {t}}$ contain the correct values. Step 3: Each 3-point set constitutes a plane, for which an scaled homography can be determined according to Fig. 9:

$$\begin{aligned} \mathbf {H}=\mathbf {R}-\frac{\bar{\mathbf {t}}.\mathbf {n}^{T}}{d} \end{aligned}$$

(8)

where $\mathbf {n}$ and d denote the normal to the plane and the distance from the origin, respectively. As a consequence, in order to define the equation of the plane and eventually estimate the 3D positions of the 3 points, $\mathbf {n}$ and d should be known. d can be, however, eliminated by setting it to 1. This way leads to an extra rescaling of the positions without loss of generality. We now build a linear system of 3 equations (one for each point), by which we may achieve a correct analytical solution using the normal equation, thus estimating the normal vector $\mathbf {n}$, which consists in $\mathbf {n}^{T}=\left[ \begin{array}{ccc}n_{x}&n_{y}&n_{z}\end{array}\right]$. Consider that for one of the 3 points, $\mathbf {q}_{b}^{\circ }$ and $\mathbf {q}_{s}^{\circ }$ are 3-vectors corresponding to its homogeneous image points on the base and the support image, respectively. We will therefore have

$$\begin{aligned} \lambda .\mathbf {q}_{s}^{\circ }=\mathbf {H}.\mathbf {q}_{b}^{\circ } \end{aligned}$$

(9)

where $\lambda$ is the ratio of the point depth from support view to that from base image. The three rows of Eq. 9 are linearly dependent. However, we just need one equation per point that is only a linear function of $n_{x}$, $n_{y}$ and $n_{z}$. Such an equation could be derived by simply replacing $\lambda$ in either the first or second row by the last row. The same procedure can apply to the other 2 points, thereby resulting in a 3-equation linear system, where the only unknowns are $n_{x}$, $n_{y}$, and $n_{z}$. This system has a single exact solution that can be obtained via the normal equation. Having estimated the normal vector, it is trivial to calculate the 3D positions of the points in the coordinate system of base view by joining the projection model with the plane equation. Step 4: Step 4 is performed for all the 3-point sets of the image pair to determine all the 3D positions. Step 5: The steps 1–5 are repeated with the other image pairs to make additional 3D estimates. Step 6: Finally, we compute the average of all the estimates from all image pairs to attain a reliable 3D structure for the undeformed shape with respect to the local referential of base view. Note that, this is just a scaling of the true structure.