Gradient Shape Model

Abstract

For years, the so-called Constrained Local Model (CLM) and its variants have been the gold standard in face alignment tasks. The CLM combines an ensemble of local feature detectors whose locations are regularized by a shape model. Fitting such a model typically consists of an exhaustive local search using the detectors and a global optimization that finds the CLM’s parameters that jointly maximize all the responses. However, one major drawback of CLMs is the inefficiency of the local search, which relies on a large amount of expensive convolutions. This paper introduces the Gradient Shape Model (GSM), a novel approach that addresses this limitation. We are able to align a similar CLM model without the need for any convolutions at all. We also use true analytical gradient and Hessian matrices, which are easy to compute, instead of their approximations. Our formulation is very general, allowing an optional 3D shape term to be seamlessly included. Additionally, we expand the GSM formulation through a cascade regression framework. This revised technique allows a substantially reduction in the complexity/dimensionality of the data term, making it possible to compute a denser, more accurate, regression step per cascade level. Experiments in several standard datasets show that our proposed models perform faster than state-of-the-art CLMs and better than recent cascade regression approaches.

Gradient Definitions

1.1 Hessian of the 2D Regularization Term ($\mathbf{H}_{\text {R}}$)

The Hessian of the 2D regularization term is a $(2v+4)$ square matrix of the form:

$$\begin{aligned} \mathbf{H}_{\text {R}} = \left[ \begin{array}{ccccccccc} \ddots &{} &{} \ldots &{} &{} &{} \vdots &{} \vdots &{} \vdots &{} \vdots \\ &{} \frac{\partial ^2 R}{\partial x_i^2} &{} &{} &{} &{} \frac{\partial ^2 R}{\partial x_i \partial a} &{} \frac{\partial ^2 R}{\partial x_i \partial b} &{} \frac{\partial ^2 R}{\partial x_i \partial t_x} &{} \frac{\partial ^2 R}{\partial x_i \partial t_y}\\ \vdots &{} &{} \ddots &{} &{} \vdots &{} \vdots &{} \vdots &{} \vdots &{} \vdots \\ &{} &{} &{} \frac{\partial ^2 R}{\partial y_i^2} &{} &{} \frac{\partial ^2 R}{\partial y_i \partial a} &{} \frac{\partial ^2 R}{\partial y_i \partial b} &{} \frac{\partial ^2 R}{\partial y_i \partial t_x} &{} \frac{\partial ^2 R}{\partial y_i \partial t_y}\\ &{} &{} \ldots &{} &{} \ddots &{} \vdots &{} \vdots &{} \vdots &{} \vdots \\ \ldots &{} \frac{\partial ^2 R}{\partial a \partial x_i} &{} \ldots &{} \frac{\partial ^2 R}{\partial a \partial y_i} &{} \ldots &{} \frac{\partial ^2 R}{\partial a^2} &{} \frac{\partial ^2 R}{\partial a \partial b} &{} \frac{\partial ^2 R}{\partial a \partial t_x} &{} \frac{\partial ^2 R}{\partial a \partial t_y} \\ \ldots &{} \frac{\partial ^2 R}{\partial b \partial x_i} &{} \ldots &{} \frac{\partial ^2 R}{\partial b \partial y_i} &{} \ldots &{} \frac{\partial ^2 R}{\partial b \partial a} &{} \frac{\partial ^2 R}{\partial b^2} &{} \frac{\partial ^2 R}{\partial b \partial t_x} &{} \frac{\partial ^2 R}{\partial b \partial t_y} \\ \ldots &{} \frac{\partial ^2 R}{\partial t_x \partial x_i} &{} \ldots &{} \frac{\partial ^2 R}{\partial t_x \partial y_i} &{} \ldots &{} \frac{\partial ^2 R}{\partial t_x \partial a} &{} \frac{\partial ^2 R}{\partial t_x \partial b} &{} \frac{\partial ^2 R}{\partial t_x^2} &{} \frac{\partial ^2 R}{\partial t_x \partial t_y} \\ \ldots &{} \frac{\partial ^2 R}{\partial t_y \partial x_i} &{} \ldots &{} \frac{\partial ^2 R}{\partial t_y \partial y_i} &{} \ldots &{} \frac{\partial ^2 R}{\partial t_y \partial a} &{} \frac{\partial ^2 R}{\partial t_y \partial b} &{} \frac{\partial ^2 R}{\partial t_y \partial t_x} &{} \frac{\partial ^2 R}{\partial t_y^2} \end{array} \right] \end{aligned}$$

(52)

where the main $(2v \times 2v)$ sub-matrix (constant, therefore can be precomputed) is

$$\begin{aligned} \frac{\partial ^2 R}{\partial \mathbf{s}^2} = 2 \Sigma _{\mathbf{s}}^{-1}. \end{aligned}$$

(53)

The 2D pose diagonal terms are given by

$$\begin{aligned} \frac{\partial ^2 R}{\partial a^2}= & {} \left( \frac{\partial \mathbf{s}_\text {BM}}{\partial a} \right) ^T \frac{\partial ^2 R}{\partial \mathbf{s}_\text {BM}^2} \frac{\partial \mathbf{s}_\text {BM}}{\partial a} \nonumber \\= & {} 2 \left( \mathbf{s} - \mathbf{s}_m \right) ^T \Sigma _{\mathbf{s}}^{-1} \left( \mathbf{s} - \mathbf{s}_m \right) \end{aligned}$$

(54)

$$\begin{aligned} \frac{\partial ^2 R}{\partial b^2}= & {} \left( \frac{\partial \mathbf{s}_\text {BM}}{\partial b} \right) ^T \frac{\partial ^2 R}{\partial \mathbf{s}_\text {BM}^2} \frac{\partial \mathbf{s}_\text {BM}}{\partial b} \nonumber \\= & {} 2 \left( \frac{\mathbf{s}_m^y - \mathbf{s}^y}{\mathbf{s}^x - \mathbf{s}_m^x} \right) ^T \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{s}_m^y - \mathbf{s}^y }{\mathbf{s}^x - \mathbf{s}_m^x} \right) \end{aligned}$$

(55)

$$\begin{aligned} \frac{\partial ^2 R}{\partial t_x^2}= & {} \left( \frac{\partial \mathbf{s}_\text {BM}}{\partial t_x} \right) ^T \frac{\partial ^2 R}{\partial \mathbf{s}_\text {BM}^2} \frac{\partial \mathbf{s}_\text {BM}}{\partial t_x} = 2 \left( \frac{\mathbf{1 }_v}{\mathbf{0 }_v} \right) ^T \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{1 }_v}{\mathbf{0 }_v}\right) \nonumber \\ \end{aligned}$$

(56)

$$\begin{aligned} \frac{\partial ^2 R}{\partial t_y^2}= & {} \left( \frac{\partial \mathbf{s}_\text {BM}}{\partial t_y} \right) ^T \frac{\partial ^2 R}{\partial \mathbf{s}_\text {BM}^2} \frac{\partial \mathbf{s}_\text {BM}}{\partial t_y} = 2 \left( \frac{\mathbf{0 }_v }{\mathbf{1 }_v} \right) ^T \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{0 }_v}{\mathbf{1 }_v}\right) \nonumber \\ \end{aligned}$$

(57)

where $\mathbf{0 }_v$ and $\mathbf{1 }_v$ are v sized vectors filled with zeros and ones, respectively. In the previous, $\mathbf{s}^x$ and $\mathbf{s}^y$ represent the x and y components (v sized vectors) of the shape $\mathbf{s}$. Additionally, note that $\mathbf{s}_m$ (2v expanded vector that defines the base mesh centre of mass) is constant.

The 2D pose mixed terms are given by

$$\begin{aligned} \frac{\partial ^2 R}{\partial a \partial b}= & {} \frac{\partial ^2 R}{\partial b \partial a} = 2 \left( \mathbf{s} - \mathbf{s}_m \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{s}_m^y - \mathbf{s}^y}{\mathbf{s}^x - \mathbf{s}_m^x} \right) \end{aligned}$$

(58)

$$\begin{aligned} \frac{\partial ^2 R}{\partial a \partial t_x}= & {} \frac{\partial ^2 R}{\partial t_x \partial a} = 2 \left( \mathbf{s} - \mathbf{s}_m \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{1 }_v}{\mathbf{0 }_v} \right) \end{aligned}$$

(59)

$$\begin{aligned} \frac{\partial ^2 R}{\partial a \partial t_y}= & {} \frac{\partial ^2 R}{\partial t_y \partial a} = 2 \left( \mathbf{s} - \mathbf{s}_m \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{0 }_v}{\mathbf{1 }_v} \right) \end{aligned}$$

(60)

$$\begin{aligned} \frac{\partial ^2 R}{\partial b \partial t_x}= & {} \frac{\partial ^2 R}{\partial t_x \partial b} = 2 \left( \frac{\mathbf{s}_m^y - \mathbf{s}^y}{\mathbf{s}^x - \mathbf{s}_m^x} \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{1 }_v}{\mathbf{0 }_v} \right) \end{aligned}$$

(61)

$$\begin{aligned} \frac{\partial ^2 R}{\partial b \partial t_y}= & {} \frac{\partial ^2 R}{\partial t_y \partial b} = 2 \left( \frac{\mathbf{s}_m^y - \mathbf{s}^y}{\mathbf{s}^x - \mathbf{s}_m^x } \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{0 }_v}{\mathbf{1 }_v}\right) \end{aligned}$$

(62)

$$\begin{aligned} \frac{\partial ^2 R}{\partial t_x \partial t_y}= & {} \frac{\partial ^2 R}{\partial t_y \partial t_x} = 2 \left( \frac{\mathbf{1 }_v}{\mathbf{0 }_v}\right) ^T \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{0 }_v}{\mathbf{1 }_v}\right) . \end{aligned}$$

(63)

Finally, the remaining mixed terms, are

$$\begin{aligned} \frac{\partial ^2 R}{\partial x_i \partial a}= & {} \frac{\partial ^2 R}{\partial a \partial x_i} = 2 \left( \frac{a \varvec{\delta }_i }{b \varvec{\delta }_{i}} \right) \Sigma _{\mathbf{s}}^{-1} \left( \mathbf{s} - \mathbf{s}_m \right) \nonumber \\&+ 2 \left( \frac{\varvec{\delta }_i }{\mathbf{0 }_v} \right) \Sigma _{\mathbf{s}}^{-1} ( \mathbf{s}_\text {BM} - \mathbf{s}_0 )\end{aligned}$$

(64)

$$\begin{aligned} \frac{\partial ^2 R}{\partial y_i \partial a}= & {} \frac{\partial ^2 R}{\partial a \partial y_i} = 2 \left( \frac{-b \varvec{\delta }_i }{a \varvec{\delta }_{i}} \right) \Sigma _{\mathbf{s}}^{-1} \left( \mathbf{s} - \mathbf{s}_m \right) \nonumber \\&+ 2 \left( \frac{\mathbf{0 }_v}{\varvec{\delta }_i } \right) \Sigma _{\mathbf{s}}^{-1} ( \mathbf{s}_\text {BM} - \mathbf{s}_0 ) \end{aligned}$$

(65)

$$\begin{aligned} \frac{\partial ^2 R}{\partial x_i \partial b}= & {} \frac{\partial ^2 R}{\partial b \partial x_i} = 2 \left( \frac{a \varvec{\delta }_i}{b \varvec{\delta }_{i}} \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{\mathbf{s}_m^y - \mathbf{s}^y}{\mathbf{s}^x - \mathbf{s}_m^x} \right) \nonumber \\&+ \left( \frac{ \mathbf{0 }_v}{\varvec{\delta }_i } \right) \Sigma _{\mathbf{s}}^{-1} ( \mathbf{s}_\text {BM} - \mathbf{s}_0 )\end{aligned}$$

(66)

$$\begin{aligned} \frac{\partial ^2 R}{\partial y_i \partial b}= & {} \frac{\partial ^2 R}{\partial b \partial y_i} = 2 \left( \frac{ -b \varvec{\delta }_i}{ a \varvec{\delta }_{i} } \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{ \mathbf{s}_m^y - \mathbf{s}^y}{ \mathbf{s}^x - \mathbf{s}_m^x } \right) \nonumber \\+ & {} \left( \frac{ -\varvec{\delta }_i}{ \mathbf{0 }_v } \right) \Sigma _{\mathbf{s}}^{-1} ( \mathbf{s}_\text {BM} - \mathbf{s}_0 )\end{aligned}$$

(67)

$$\begin{aligned} \frac{\partial ^2 R}{\partial x_i \partial t_x}= & {} \frac{\partial ^2 R}{\partial t_x \partial x_i} = 2 \left( \frac{ a \varvec{\delta }_i}{ b \varvec{\delta }_{i} } \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{ \mathbf{1 }_v}{ \mathbf{0 }_v } \right) \end{aligned}$$

(68)

$$\begin{aligned} \frac{\partial ^2 R}{\partial y_i \partial t_x}= & {} \frac{\partial ^2 R}{\partial t_x \partial y_i} = 2 \left( \frac{ -b \varvec{\delta }_i}{ a \varvec{\delta }_{i} } \right) \Sigma _{\mathbf{s}}^{-1} \left( \frac{ \mathbf{1 }_v}{ \mathbf{0 }_v } \right) . \end{aligned}$$

(69)

where $\varvec{\delta }_i$ is a v-dimensional vector filled with zeros, except on a scalar of 1 at the ith element location.

1.2 Gradients of the 2D $+$ 3D Model

The gradient of the 3D regularization term, in Eq. 42, is

$$\begin{aligned} \nabla _{\text {R3D}}(\overline{\mathbf{s}}) = \left[ \begin{array}{ccc} \mathbf{0 }_{2v+4}&2 \Sigma _{\overline{\mathbf{s}}}^{-1} (\overline{\mathbf{s}} - \overline{\mathbf{s}}_0)&\mathbf{0 }_6 \end{array} \right] . \end{aligned}$$

(70)

Recalling gradient of the 3D to 2D projection error, defined by a $(2v) \times (2v+4+3v+6)$ matrix as

$$\begin{aligned} \nabla \mathbf{r } = \left[ \begin{array}{ccccccccc} \frac{\partial \mathbf{r} }{\partial \mathbf{s}}&\frac{\partial \mathbf{r} }{\partial \varvec{ \theta }}&\frac{\partial \mathbf{r} }{\partial \overline{\mathbf{s}}}&\frac{\partial \mathbf{r} }{\partial \sigma }&\frac{\partial \mathbf{r} }{\partial \Delta \theta _x}&\frac{\partial \mathbf{r} }{\partial \Delta \theta _y}&\frac{\partial \mathbf{r} }{\partial \Delta \theta _z}&\frac{\partial \mathbf{r} }{\partial o_x}&\frac{\partial \mathbf{r} }{\partial o_y} \end{array} \right] \end{aligned}$$

with

$$\begin{aligned}&\frac{\partial \mathbf{r} }{\partial \mathbf{s}} = \mathbf{I}_{2v}, \quad \frac{\partial \mathbf{r} }{\partial \varvec{ \theta }} = \mathbf{0 }_{2v}, \quad \frac{\partial \mathbf{r} }{\partial \overline{\mathbf{s}}} = -\mathbf{P} \otimes \mathbf{I}_v, \quad \frac{\partial \mathbf{r} }{\partial \sigma } = - \mathbf{I}_v \otimes \mathbf{R} _o ~ \overline{\mathbf{s}}, \\&\frac{\partial \mathbf{r} }{\partial \Delta \theta _x} = \mathbf{I}_v \otimes \left( \mathbf{P} \left[ \begin{array}{ccc} 0 &{} 0 &{} 0\\ 0 &{} 0 &{} -1 \\ 0 &{} 1 &{} 0 \end{array} \right] \right) \overline{\mathbf{s}}, \frac{\partial \mathbf{r} }{\partial \Delta \theta _y} = \mathbf{I}_v \otimes \left( \mathbf{P} \left[ \begin{array}{ccc} 0 &{} 0 &{} 1\\ 0 &{} 0 &{} 0 \\ -1 &{} 0 &{} 0 \end{array}\right] \right) \overline{\mathbf{s}}, \\&\frac{\partial \mathbf{r} }{\partial \Delta \theta _z} = \mathbf{I}_v \otimes \left( \mathbf{P} \left[ \begin{array}{ccc} 0 &{} -1 &{} 0\\ 1 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 \end{array} \right] \right) \overline{\mathbf{s}}, \frac{\partial \mathbf{r} }{\partial o_x} = \left( \frac{ - \mathbf{1 }_v}{ \mathbf{0 }_v } \right) , \frac{\partial \mathbf{r} }{\partial o_y} = \left( \frac{ \mathbf{0 }_v}{ - \mathbf{1 }_v } \right) \end{aligned}$$

where $\mathbf{I}_{n}$ represents a n dimensional identity matrix and the $\otimes$ symbol is the Kronecker product.

Finally, the Hessian of the 3D shape regularization term (in Eq. 43) is a $(2v+4+3v+6)$ square matrix given by

$$\begin{aligned} \mathbf{H}_{\text {R3D}}(\overline{\mathbf{s}}) = \left[ \begin{array}{cccc} \mathbf{0 }_{2v} &{}\quad \mathbf{0 }_{4} &{}\quad \mathbf{0 }_{3v} &{}\quad \mathbf{0 }_6 \\ \mathbf{0 }_{2v} &{}\quad \mathbf{0 }_{4} &{}\quad \mathbf{0 }_{3v} &{}\quad \mathbf{0 }_6\\ \mathbf{0 }_{2v} &{}\quad \mathbf{0 }_{4} &{}\quad 2 \Sigma _{\overline{\mathbf{s}}}^{-1} &{}\quad \mathbf{0 }_6\\ \mathbf{0 }_{2v} &{}\quad \mathbf{0 }_{4} &{}\quad \mathbf{0 }_{3v} &{}\quad \mathbf{0 }_6\\ \end{array} \right] \end{aligned}$$

(71)

note that this matrix is constant, therefore, it can be precomputed.