Equiareal Shape-from-Template

Abstract

This paper studies the 3D reconstruction of a deformable surface from a single image and a reference surface, known as the template. This problem is known as Shape-from-Template and has been recently shown to be well-posed for isometric deformations, for which the surface bends without altering geodesics. This paper studies the case of equiareal deformations. They are elastic deformations where the local area is preserved and thus include isometry as a special case. Elastic deformations have been studied before in Shape-from-Template, yet no theoretical results were given on the existence or uniqueness of solutions. The equiareal model is much more widely applicable than isometry. This paper brings Monge’s theory, widely used for studying the solutions of nonlinear first-order PDEs, to the field of 3D reconstruction. It uses this theory to establish a theoretical framework for equiareal Shape-from-Template and answers the important question of whether it is possible to reconstruct a surface exactly with a much weaker prior than isometry. We prove that equiareal Shape-from-Template has a maximum of two local solutions sufficiently near an initial curve that lies on the surface. In addition, we propose an analytical reconstruction algorithm that can recover the multiple solutions. Our algorithm uses standard numerical tools for ODEs. We use the perspective camera model and give reconstruction results with both synthetic and real examples.

Appendices

A Appendix: Classification of Critical Points

1.1 A.1 Isolated Critical Points

We define isolated critical points as follows.

Definition 3

A point p is an isolated critical point if it is a critical point (${\tilde{\rho }}_u={\tilde{\rho }}_v=0$) and the Hessian of the depth function ${\tilde{\rho }}$ is non-degenerate.

Consequently, there must exist a neighborhood around the critical point p where all points are non-critical except for the point p. By imposing integrability conditions, if the surface has a finite number of these points, there exists only one possible solution to the reconstruction PDE in the neighborhood of p.

1.2 A.2 Critical Curves

Perspective critical curves are spherical curves. We prove that perspective critical curves are contained in a spherical surface whose center is located at the camera center. First of all, we define critical curves as follows.

Definition 4

A curve ${\mathcal {C}}$ in the surface ${\mathcal {S}}$ is a critical curve of the surface ${\mathcal {S}}$ with respect to the perspective parametrization $({\mathcal {I}},X_p)$ if all its points are critical $({\tilde{\rho }}_u={\tilde{\rho }}_v=0)$.

Theorem 7

If ${\mathcal {C}}$ is a critical curve of a surface ${\mathcal {S}}$ with respect to a perspective parametrization $(U,X_p)$, then, the curve ${\mathcal {C}}$ belongs to a spherical surface whose center is located at the camera center.

Fig. 14

Proof that critical curves are spherical curves that lie on a sphere whose center is located at the camera center. If the curve ${\mathcal {C}}$ is critical, then, if there is a point p out of the sphere ${\mathcal {S}}$, either p is a critical point or there is a critical point q in its neighborhood

Proof

Figure 14 illustrates the proof. Suppose that there exist a point p of the critical curve ${\mathcal {C}}$ that does not belong to the sphere ${\mathcal {S}}$ whose center is placed at the camera center and has a radius of $\rho _0$. There are two possibilities. If we assume that p is not a critical point, then, the curve is non-critical, contradicting the hypothesis. If we assume that p is a critical point, there is a neighborhood $\varOmega \subset {\mathcal {C}}$ of p where there exist a point $q\in \varOmega$ that is a non-critical point, contradicting the hypothesis. So, critical curves are spherical curves that belong to a sphere whose radius is located on the camera coordinate center.$\square$

Remark 1

The only possible critical surfaces are the family of conformal spheres centered at the camera center.

Remark 2

Given any surface of revolution whose axis of symmetry is in the xy-plane, that contains the camera center. If its generatrix has a maximum, minimum or a saddle point at $t=0$, then, the surface has as a critical curve formed by rotating this point around the generatrix. An example is a cylinder whose generatrix passes through the camera center, see Fig. 8.

B Refinement Method

We propose a refinement method for equiareal SfT based on minimizing a MAP compound cost functional. We proceed by defining the parametrization of ${\mathcal {S}}$ from the template domain $({\mathcal {U}}, \varphi )$, where $\varphi \in {\mathcal {C}}^2({\mathcal {U}},{\mathbb {R}}^3)$. From the commutative diagram shown in Fig. 2, we have that $X_i = \varphi \circ \eta$, where we recall that $\eta$ is a known function. Working with $\varphi$ instead of $X_i$ allows us to propose a MAP data cost. We define the set of correspondences between the template and the image as pairs of 2D points $(p^i\in {\mathcal {U}}, q^i\in {\mathcal {I}})$ for $i=1,\ldots ,N$. Our cost functional is defined as:

$$\begin{aligned} \varepsilon [\varphi ] = \lambda \varepsilon _\mathrm{data}[\varphi ] + \mu \varepsilon _\mathrm{equiA}[\varphi ] +\kappa \varepsilon _\mathrm{smth}[\varphi ] + \nu \varepsilon _\mathrm{init}[\varphi ], \nonumber \\ \end{aligned}$$

(33)

where $\varepsilon _\mathrm{data}$ is the data term:

$$\begin{aligned} \varepsilon _\mathrm{data}[\varphi ] = \sum ^N_{i=1}||\varPi _{p}(\varphi (p^i)) - q^i||^2. \end{aligned}$$

(34)

$\varepsilon _{equiA}$ is the equiareal constraint defined as:

$$\begin{aligned} \varepsilon _\mathrm{equiA}[\varphi ] = \sum _{p\in P}||{\mathbb {I}}[\varDelta (p)] - {\mathbb {I}}[\varphi (p)]||^2, \end{aligned}$$

(35)

where $P\subset {\mathcal {U}}$ is a regular grid of points defined in the template domain. ${\mathbb {I}}[X_t(p)]$ and ${\mathbb {I}}[\varphi (p)]$ are the determinants of the first fundamental form of surfaces $X_t$ and $\varphi$, respectively, evaluated at point p. $\varepsilon _\mathrm{smth}$ is a functional that minimizes the bending energy of the surface which encourages the solution to be smooth. Finally, the error of the initial conditions is defined by:

$$\begin{aligned} \varepsilon _\mathrm{init}[\varphi ] = \sum _{i\in I}||\varphi (r^i) - s^i||^2, \end{aligned}$$

(36)

where I are the index set and $r^i$ are points in ${\mathcal {U}}$ that correspond to the set of 3D points $s^i\in S$ taken from the initial conditions. The set of hyperparameters $\lambda$, $\mu$, $\kappa$ and $\nu$ are scalars that control the importance of each functional and are fixed.

We use a B-spline to approximate $\varphi$ which converts the variational problem of Eq. (33) into a non-convex polynomial cost function. We use the Levenberg-Marquardt algorithm to iteratively minimize the cost given an initial estimate of $\varphi$.