Unsupervised joint face alignment with gradient correlation coefficient

Abstract

This work proposes an unsupervised joint alignment framework, referred to as “Gradient Correlation Congealing,” which aligns an image ensemble by maximizing a sum of gradient correlation coefficient function defined over all images. We, respectively, develop two different formulations to optimize the objective function regarding the role of “template.” While most existing face alignment methods suffer from outliers, e.g., occlusions, the proposed algorithms are able to align faces undergoing partial occlusions. Moreover, our algorithms can cope with nonuniform illumination changes (even extremely difficult ones), and also, they do not require any predefined templates. We test the novel approaches against four typical joint alignment methods including Least-Squares Congealing, Learned-Miller Congealing, Lucas–Kanade entropy Congealing, and RASL using three challenging face databases: AR, Yale B, and LFW. Experimental results prove the efficiency of our approaches under different conditions, especially when faces are partially occluded, and the proposed algorithms perform much better than all considered methods.

Appendix

Using the Lucas–Kanade algorithm [4], we assume that the current $\varvec{p}$ is known, and then, $\varvec{p}$ can be iteratively updated using $\Delta \varvec{p}$:

$$\varvec{p}\leftarrow \varvec{p}+\Delta \varvec{p}$$

(21)

However, a first-order Taylor expansion of $\widetilde{\varvec{g}}_i$ with respect to $\Delta \varvec{p}$ yields a linear function of $\Delta \varvec{p}$ which is maximized when $\Delta \varvec{p}\rightarrow \infty$. To solve this problem, since $\left\| \widetilde{\varvec{g}}_{i}(k)\right\| _{2} = 1$, Eq. (6) is exactly equal to:

$$\begin{aligned} \psi (\varvec{I}_i,\varvec{I}_j)=\frac{\sum _{k=1}^N\left[ \widetilde{\varvec{g}}_{i,x}(k)\widetilde{\varvec{g}}_{j,x}(k)+\widetilde{\varvec{g}}_{i,y}(k)\widetilde{\varvec{g}}_{j,y}(k)\right] }{\sqrt{\sum _{k=1}^N\left[ \widetilde{\varvec{g}}_{i,x}^2(k)+\widetilde{\varvec{g}}_{i,y}^2(k)\right] }} \end{aligned}$$

(22)

Equation (22) can also be written in a vector expression:

$$\begin{aligned} \psi (\varvec{I}_i,\varvec{I}_j)=\frac{\widetilde{\varvec{g}}_{i,x}^{\rm T}\widetilde{\varvec{g}}_{j,x}+\widetilde{\varvec{g}}_{i,y}^{\rm T}\widetilde{\varvec{g}}_{j,y}}{\sqrt{\widetilde{\varvec{g}}_{i,x}^{\rm T}\widetilde{\varvec{g}}_{i,x}+\widetilde{\varvec{g}}_{i,y}^{\rm T}\widetilde{\varvec{g}}_{i,y}}} \end{aligned}$$

(23)

In Lucas–Kanade framework, the objective function Eq. (7) becomes:

$$\begin{aligned} \psi (\varvec{I}_i(\varvec{W}(\varvec{x};\varvec{p+\Delta p})),\varvec{I}_j) = \frac{\widetilde{\varvec{g}}_{i,x}^{\rm T}\left[ \varvec{p+\Delta p}\right] \widetilde{\varvec{g}}_{j,x}+\widetilde{\varvec{g}}_{i,y}^{\rm T}\left[ \varvec{p+\Delta p}\right] \widetilde{\varvec{g}}_{j,y}}{\sqrt{\widetilde{\varvec{g}}_{i,x}^{\rm T}\left[ \varvec{p+\Delta p}\right] \widetilde{\varvec{g}}_{i,x}\left[ \varvec{p+\Delta p}\right] +\widetilde{\varvec{g}}_{i,y}^{\rm T}\left[ \varvec{p+\Delta p}\right] \widetilde{\varvec{g}}_{i,y}\left[ \varvec{p+\Delta p}\right] }} \end{aligned}$$

(24)

where the symbol $\varvec{g}_{i,x}\left[ \varvec{p}\right]$ represents a vector obtained by writing $\varvec{G}_{i,x}(\varvec{W}(\varvec{x};\varvec{p}))$ in lexicographic ordering.

As presented in [46], by maximizing Eq. (24), we can achieve Eq. (9).