Auto-calibrating photometric stereo using ring light constraints

Abstract

We propose an auto-calibration method for photometric stereo. Our method exploits constraints placed on light sources to recover their positions and the surface normals of an object up to a scaling and planar rotation ambiguity. The ambiguity is resolved with multi-view consistency constraints leading to a Euclidean reconstruction of the geometric shape of the object. Auto-calibration methods are helpful to overcome the limitations imposed by calibrating objects. We evaluate our algorithm with experiments on real world scenes.

Appendices

Appendix A : Proof of Proposition 1

Proposition 1

The true normal $\mathbf{n}_\mathbf{g}$ of a Lambertian surface illuminated by a ring-light at a distance $d_g$ can be recovered up to a classical bas-relief ambiguity compounded with a planar rotation ambiguity, i.e.

$$\begin{aligned}&\left( {\begin{array}{l} {{n_1}}\\ {{n_2}}\\ {{n_3}} \end{array}} \right) = \left( {\begin{array}{c@{\quad }c@{\quad }c} cos \vartriangle &{} sin \vartriangle &{}0\\ - sin \vartriangle &{} cos \vartriangle &{}0\\ 0&{}0&{}1 \end{array}} \right) \left( {\begin{array}{c@{\quad }c@{\quad }c} 1&{}0&{}0\\ 0&{}1&{}0\\ 0&{}0&{}{\frac{{{d_g}}}{d}} \end{array}} \right) \left( {\begin{array}{l} {{n_{g1}}}\\ {{n_{g2}}}\\ {{n_{g3}}} \end{array}} \right) \nonumber \\ \end{aligned}$$

(18)

$$\begin{aligned}&\text {or} \quad \mathbf{n} = R_{\vartriangle } S\mathbf{n}_g \end{aligned}$$

(19)

Here, $\vartriangle = \theta _* -\theta _0,\,d$ and $\theta _*$ are the assumed distance and initial rotation angles respectively. $n$ is the recovered normal.

Proof

The pseudo inverse of a matrix $\mathbf{L}$ is computed as $\mathbf{L}^{\dag } = (\mathbf{L}^{T}\mathbf{L})^{-1}\mathbf{L}^{T}$. For the matrix $\mathbf{L}$ as defined in Eq. (2) in Sect. 3, we can write the matrix $\mathbf{L}^{T}\mathbf{L}$ as,

$$\begin{aligned} \mathbf{L}^{T}\mathbf{L}= \left[ {\begin{array}{l@{\quad }c@{\quad }c} {{r^2}\sum \limits _{i = 0}^N {{{\cos }^2} \alpha } }&{}{{r^2}\sum \limits _{i = 0}^N {\sin \alpha } \cos \alpha }&{}{dr\sum \limits _{i = 0}^N {\cos \alpha } }\\ {{r^2}\sum \limits _{i = 0}^N {\sin \alpha } \cos \alpha }&{}{{r^2}\sum \limits _{i = 0}^N {{{\sin }^2} \alpha } }&{}{dr\sum \limits _{i = 0}^N {\sin \alpha } }\\ {dr\sum \limits _{i = 0}^N {\cos \alpha } }&{}{dr\sum \limits _{i = 0}^N {\sin \alpha } }&{}{ {N + 1}{d^2}} \end{array}} \right] \nonumber \\ \end{aligned}$$

(20)

where $\alpha =\left( \theta _0+it\right)$ and $i$ is the index in the summation.

This can be rewritten as,

$$\begin{aligned} \mathbf{L}^{T}\mathbf{L}= {r^2}\left[ {\begin{array}{l@{\quad }l@{\quad }c} {\frac{1}{2}\sum \limits _{i = 0}^N {\left( {1 + \cos {2\alpha } } \right) } }&{}{\frac{1}{2}\sum \limits _{i = 0}^N {\sin {2\alpha } } }&{}{\frac{d}{r}\sum \limits _{i = 0}^N {\cos \alpha } } \\ {\frac{1}{2}\sum \limits _{i = 0}^N {\sin {2\alpha } } }&{}{\frac{1}{2}\sum \limits _{i = 0}^N {\left( {1 - \cos {2\alpha } } \right) } }&{}{\frac{d}{r}\sum \limits _{i = 0}^N {\sin \alpha } } \\ {\frac{d}{r}\sum \limits _{i = 0}^N {\cos \alpha } }&{}{\frac{d}{r}\sum \limits _{i = 0}^N {\sin \alpha } }&{}{(N + 1)\frac{{{d^2}}}{{{r^2}}}} \end{array}} \right] \nonumber \\ \end{aligned}$$

(21)

From [12], we have the following identities,

$$\begin{aligned} \sum \limits _{i = 0}^N {\cos \left( {{\theta _0} + it} \right) } = \frac{{\sin \left( {\frac{{\left( {N + 1} \right) t}}{2}} \right) \cos \left( {{\theta _0} + \frac{{Nt}}{2}} \right) }}{{\sin \left( {\frac{t}{2}} \right) }} \end{aligned}$$

and

$$\begin{aligned} \sum \limits _{i = 0}^N {\sin \left( {{\theta _0} + it} \right) } = \frac{{\sin \left( {\frac{{\left( {N + 1} \right) t}}{2}} \right) \sin \left( {{\theta _0} + \frac{{Nt}}{2}} \right) }}{{\sin \left( {\frac{t}{2}} \right) }} \end{aligned}$$

(22)

Substituting the above identities in Eq. (21), the inverse of the matrix $\mathbf{L}^{T}\mathbf{L}$ is calculated as,

$$\begin{aligned} (\mathbf{L}^\mathbf{T}\mathbf{L})^{ - 1} = \frac{1}{{{d^2}K}}\left[ {\begin{array}{c@{\quad }c@{\quad }c} d^2 A&{}d^2 B&{}dC\\ d^2 B&{}d^2 D&{}dC\\ dC&{}dE&{}F \end{array}} \right] \end{aligned}$$

(23)

where $A,B,C,D,E,F$ and $K$ are terms which are functions of the parameters ${\theta _0},r,N$ and $t$ and independent of $d$.

The pseudo inverse of $\mathbf{L}$ is computed as,

$$\begin{aligned} \mathbf{L}^{\dag }=(\mathbf{L}^{\mathbf{T}}\mathbf{L})^{-1}\mathbf{L}^\mathbf{T}= \left[ {\begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} a_{11}&{}\ldots &{}a_{1k}&{} \ldots &{}a_{1N}\\ a_{21}&{}\ldots &{}a_{2k}&{} \ldots &{}a_{2N}\\ {\frac{1}{d}{a_{31}}}&{}\ldots &{}{\frac{1}{d}{a_{3k}}}&{} \ldots &{}{\frac{1}{d}{a_{3N}}} \end{array}} \right] \nonumber \\ \end{aligned}$$

(24)

where $a_{ij}$ are a function of ${\theta _0},r,N$ and $t$.

The normal is obtained by multiplying this with the intensity matrix $\mathbf{I}$,

$$\begin{aligned} \rho \mathbf{n} = \mathbf{L}^{\dag } \mathbf{I}&= \left[ {\begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} a_{11}&{} \ldots &{}a_{1k}&{} \ldots &{}a_{1N}\\ a_{21}&{}\ldots &{}a_{2k}&{} \ldots &{}a_{2N}\\ {\frac{1}{d}{a_{31}}}&{}\ldots &{}{\frac{1}{d}{a_{3k}}}&{} \ldots &{}{\frac{1}{d}{a_{3N}}} \end{array}}\right] \left[ {\begin{array}{l} {{I_1}}\\ \vdots \\ {{I_k}}\\ \vdots \\ I_N \end{array}}\right] \nonumber \\&= \left[ {\begin{array}{l} b_1\\ b_2\\ {\frac{1}{d}{b_3}} \end{array}} \right] \end{aligned}$$

(25)

where, $b_{ij}$ are also functions of ${\theta _0},r,N$ and $t$.

The error term at each pixel is now computed as,

$$\begin{aligned}&e\left( d \right) = \left\| \mathbf{I} - \mathbf{L}\left( d \right) \cdot \rho \mathbf{n} \right\| \nonumber \\&\quad = \left\| \mathbf{I} - \left[ {\begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} {r\cos {\theta _0}}&{}{r\sin {\theta _0}}&{}d\\ \vdots &{} \vdots &{} \vdots \\ {r\cos \left( {{\theta _0} + kt} \right) }&{}{r\sin \left( {{\theta _0} + kt} \right) }&{}d\\ \vdots &{} \vdots &{} \vdots \\ {r\cos \left( {{\theta _0} + Nt} \right) }&{}{r\sin \left( {{\theta _0} + Nt} \right) }&{}d \end{array}}\right] \left[ {\begin{array}{l} b_1\\ b_2\\ {\frac{1}{d}{b_3}} \end{array}} \right] \right\| \nonumber \\ \end{aligned}$$

(26)

or

$$\begin{aligned} e\left( d \right) = \left\| \mathbf{I} - \left[ {\begin{array}{l} c_1\\ \vdots \\ c_k\\ \vdots \\ c_N \end{array}}\right] \right\| \end{aligned}$$

(27)

where, $c_i$ are also functions of ${\theta _0},r,N$ and $t$.

Hence, we observe that although the normal obtained from pseudo inverse of $\mathbf{L}$ is a function of the distance $d$, yet the product of $\mathbf{L}$ and $\mathbf{n}$ is independent of $d$. As the distance is varied, since all other parameters remain constant, the error remains constant while the computed normal changes as a function of $d$. In other words, we can say that for two different values of the distance term, say $d$ and $d_g$, and the corresponding light source directions $\mathbf{L}\left( d\right)$ and $\mathbf{L}\left( d_g\right)$,

$$\begin{aligned} \mathbf{L}\left( d \right) \cdot \rho \mathbf{n} = \mathbf{L}\left( d_g \right) \cdot \rho \mathbf{n}_\mathbf{g} \end{aligned}$$

(28)

where $\mathbf{n}$ and $\mathbf{n}_\mathbf{g}$ are the respective normals at the distance $d$ and $d_g$.

From Eq. (25), the normals obtained at a distance $d$ and $d_g$ are

$$\begin{aligned} \mathbf{n} = \left[ \begin{array}{c@{\quad }c@{\quad }c} b_1&b_2&\frac{1}{d}b_3 \end{array}\right] ^{T} \text {and}\; \mathbf{n}_\mathbf{g} = \left[ \begin{array}{c@{\quad }c@{\quad }c} b_1&b_2&\frac{1}{d_g}b_3 \end{array}\right] ^{T} \end{aligned}$$

(29)

Here the $b_i$’s are independent of the term $d$.

We can now rewrite $\mathbf{n}$ as

$$\begin{aligned} \mathbf{n}=\left[ \begin{array}{c@{\quad }c@{\quad }c} b_1&b_2&\frac{1}{d}b_3 \end{array}\right] ^{T} = \left[ \begin{array}{c@{\quad }c@{\quad }c} b_1&b_2&\left( \frac{d_g}{d} \frac{1}{d_g}b_3\right) \end{array}\right] ^{T} \end{aligned}$$

Thus, we have,

$$\begin{aligned} \left[ {\begin{array}{l} n_1\\ n_2\\ n_3 \end{array}} \right] = \left[ {\begin{array}{c@{\quad }c@{\quad }c} 1&{}0&{}0\\ 0&{}1&{}0\\ 0&{}0&{}{\frac{{{d_g}}}{d}} \end{array}} \right] \left[ {\begin{array}{*{20}{c}} n_{g1}\\ n_{g2}\\ n_{g3} \end{array}}\right] \text {or} \; \mathbf{n} = S\mathbf{n}_\mathbf{g} \end{aligned}$$

(30)

where $\mathbf{n}$ and $\mathbf{n}_\mathbf{g}$ are as defined in the statement of Proposition 1.

Next, we show that the recovered normals are up to a rotational ambiguity. Note that these two are independent of each other. Let $\theta _{g}$ be the initial angle and $\theta _*$ be an arbitrary estimate of the initial angle. We know the position of each light source $\mathbf{l}_{g}$ is given by

$$\begin{aligned} \mathbf{l}_\mathbf{g}=\left[ \begin{array}{c@{\quad }c@{\quad }c} r\cos (\theta _{g}+(i-1)t)&r\sin (\theta _{g}+(i-1)t)&d \end{array}\right] \end{aligned}$$

(31)

where, $t = \frac{2\pi }{N}$ and $i$ is the index of the light source $(i = 1,\ldots ,N)$

Now, for an arbitrary $\theta _*$, the light source position $\mathbf{l}_*$at the same index is computed as

$$\begin{aligned} \mathbf{l}_{*}&= \left[ \begin{array}{c@{\quad }c@{\quad }c} r\cos (\theta _{*}+(i-1)t)&r\sin (\theta _{*}+(i-1)t)&d \end{array}\right] \\&= \left[ \begin{array}{c@{\quad }c@{\quad }c} r\cos (\theta _{g}\!+\!(i-1)t\!+\!\theta _* -\theta _{g})&r\sin (\theta _{g}+(i-1)t+\theta _* -\theta _{g})&d \end{array}\right] \\&= \left[ \begin{array}{c@{\quad }c@{\quad }c} r\cos (\theta _{g}+(i-1)t)&r\sin (\theta _{g}+(i-1)t)&d \end{array}\right] \mathbf{R}_{\vartriangle }\\&= \mathbf{l}_\mathbf{g}\mathbf{R}_{\vartriangle } \end{aligned}$$

where $\vartriangle = \theta _* -\theta _0$ and

$$\begin{aligned} \mathbf{R}_{\vartriangle } = \left[ \begin{array}{c@{\quad }c@{\quad }c} \cos \vartriangle &{} \sin \vartriangle &{} 0\\ -\sin \vartriangle &{} \cos \vartriangle &{} 0\\ 0 &{} 0 &{} 1 \end{array}\right] \end{aligned}$$

(32)

From (32), we observe a planar rotation ambiguity (in the $xy$ plane) between the estimated and the true light source positions. We can derive the following relationship between the true normal and estimated normal as,

$$\begin{aligned} \mathbf{I}&= \rho \mathbf{l}_{*}\mathbf{n}\\&= \rho \mathbf{l}_\mathbf{g}\mathbf{R}_\vartriangle \mathbf{n}_\mathbf{g}\\&= \rho \mathbf{l}_\mathbf{g} \mathbf{n}_\mathbf{g} \end{aligned}$$

We can now say that

$$\begin{aligned} \mathbf{n}=\mathbf{R}_\vartriangle \mathbf{n}_\mathbf{g} \end{aligned}$$

(33)

In other words, for any arbitrary $\theta _*$, the recovered normals are rotated by an angle $\vartriangle =(\theta _*-\theta _{g})$.

Appendix B : Parameters

If $\mathbf{n}_i=\left[ \begin{array}{ccc} n_{i1}&n_{i2}&n_{i3}\end{array}\right]$ and $T = \left\{ t_{ij}\right\} _{3 \times 3}$, then the constants $A_{ij}$ are,

$$\begin{aligned} A_{11}&= n_{12}n_{21}t_{31}+n_{12}n_{22}t_{32}\nonumber \\ A_{12}&= n_{12}t_{22}n_{23}\nonumber \\ A_{13}&= -n_{13}n_{23}t_{23}\nonumber \\ A_{14}&= -n_{13}t_{21}n_{21}-n_{13}n_{21}t_{21}\nonumber \\ A_{21}&= -n_{21}t_{31}n_{11}-n_{22}t_{32}n_{11}\nonumber \\ A_{22}&= n_{13}n_{23}t_{13}\nonumber \\ A_{23}&= -n_{11}n_{23}t_{22}\nonumber \\ A_{24}&= n_{13}t_{11}n_{21}+n_{13}n_{22}t_{12}\nonumber \\ A_{31}&= n_{11}t_{23}n_{23}-n_{12}n_{23}t_{13}\nonumber \\ A_{32}&= n_{11}t_{21}n_{21}+n_{11}t_{22}n_{22}-n_{12}t_{11}n_{21}-n_{12}t_{12}n_{22}\nonumber \\ \end{aligned}$$

(34)