A lighting robust fitting approach of 3D morphable model for face reconstruction

Abstract

Three-dimensional morphable model (3DMM) is a powerful tool for recovering 3D shape and texture from a single facial image. The success of 3DMM relies on two things: an effective optimization strategy and a realistic approach to synthesizing face images. However, most previous methods have focused on developing an optimization strategy under Phong’s synthesis approach. In this paper, we adopt a more realistic synthesis technique that fully considers illumination and reflectance in the 3DMM fitting process. Using the sphere harmonic illumination model (SHIM), our new synthesis approach can account for more lighting factors than Phong’s model. Spatially varying specular reflectance is also introduced into the synthesis process. Under SHIM, the cost function is nearly linear for all parameters, which simplifies the optimization. We apply our new optimization algorithm to determine the shape and texture parameters simultaneously. The accuracy of the recovered shape and texture can be improved significantly by considering the spatially varying specular reflectance. Hence, our algorithm produces an enhanced shape and texture compared with previous SHIM-based methods that recover shape from feature points. Although we use just a single input image in a profile pose, our approach gives plausible results. Experiments on a well-known image database show that, compared to state-of-the-art methods based on Phong’s model, the proposed approach enhances the robustness of the 3DMM fitting results under extreme lighting and profile pose.

Appendix

1.1 The gradient of texture cost function

The key in the whole optimization is get the gradient of cost function. We offer the formula for how to compute the gradient ${\nabla e_v^c}$. As $e_v^c = E_d + E_s - I_\mathrm{input}^c({\pi _\gamma }(v))$,

$$\begin{aligned}&\frac{{\partial e_v^c}}{{\partial \xi }} = \frac{{\partial {E_d}}}{{\partial \xi }} + \frac{{\partial {E_s}}}{{\partial \xi }} - \frac{{\partial I_\mathrm{input}^c({\pi _\gamma }(v))}}{{\partial \xi }} \end{aligned}$$

(19)

$$\begin{aligned}&\frac{{\partial {E_d}}}{{\partial \xi }} = \frac{{\partial {t^c}(v)N(v){M^c}{N^T}(v)}}{{\partial \xi }} \end{aligned}$$

(20)

$$\begin{aligned}&\frac{{\partial {E_s}}}{{\partial \xi }} = \frac{{\partial \sum \nolimits _{l = 0}^{{l_{\max }}} {\sum \nolimits _{m = - l}^l {\rho _l^s{\varLambda _l}{L_{lm}}{Y_{lm}}(v)} } }}{{\partial \xi }} \end{aligned}$$

(21)

$$\begin{aligned} \frac{{\partial I_\mathrm{input}^c({\pi _\gamma }(v))}}{{\partial \xi }}= & {} \frac{{\partial I_\mathrm{input}^c(\pi _\gamma ^x,\pi _\gamma ^y)}}{{\partial \xi }} \nonumber \\= & {} \frac{{\partial I_\mathrm{input}^c(p_x,p_y)}}{{\partial x}}\frac{{p_x}}{{\partial \xi }}+ \frac{{\partial I_\mathrm{input}^c(p_{x},p_{y})}}{{\partial y}}\frac{{p_y}}{{\partial \xi }}\nonumber \\ \end{aligned}$$

(22)

Here, $\xi = \left( {\alpha ,\beta ,\gamma ,\iota _{d},\iota _{s}}, \mu \right)$.

1.2 Prior cost function

According to the theory of PCA, the parameters $\alpha$ and $\beta$ obey K-dimension normal distribution:

$$\begin{aligned} \begin{array}{ll} \alpha \sim \frac{1}{{{{\left( {2\pi } \right) }^{K/2}}}}{e^{ - \frac{{{{\left\| \alpha \right\| }^2}}}{2}}}\\ \beta \sim \frac{1}{{{{\left( {2\pi } \right) }^{K/2}}}}{e^{ - \frac{{{{\left\| \beta \right\| }^2}}}{2}}} \end{array} \end{aligned}$$

(23)

Another parameter with several variates is $\iota _{s}$. Nevertheless, we have no prior information about the same. According to the SHT, the height order of $\iota _{s}$ is little, so we add regularization constrain as prior information when we optimize it. Thus, the prior cost function is

$$\begin{aligned} {C_p} = {\tau _{p1}}{\left\| \alpha \right\| ^2} + {\tau _{p2}}{\left\| \beta \right\| ^2} + {\tau _{p3}}{\left\| {{\iota _s}} \right\| ^2} \end{aligned}$$

(24)