Graph Maximum Margin Criterion for Face Recognition

Abstract

In this paper, we propose a graph maximum margin criterion (GMMC) which provides a unified method for overcoming the small sample size problem encountered by the algorithms interpreted in a general graph embedding framework. The proposed GMMC-based feature extraction algorithms compute the discriminant vectors by maximizing the difference between the graph between-class scatter matrix and the graph within-class scatter matrix and then the singularity problem is avoided. An efficient and stable algorithm for implementing GMMC is also proposed. We also reveal the eigenvalue distribution of GMMC. Experiments on the ORL, PIE and AR face databases show the effectiveness of the proposed GMMC-based feature extraction methods.

Appendix

Theorem 1 Suppose $u\in {\mathbb {R}}^{n\times 1}$ to be the eigenvector of the matrix $(L^{b}-\alpha L^{w})X^{T}X$ corresponding to the eigenvalue $\lambda$. Then, Xu is the eigenvector of the matrix $S_{b}^{L} -\alpha S_{w}^{L}$ corresponding to the eigenvalue $\lambda$.

Proof

Since $u\in {\mathbb {R}}^{n\times 1}$ is the eigenvector of the matrix $(L^{b}-\alpha L^{w})X^{T}X$ corresponding to the eigenvalue $\lambda$, we have

$$\begin{aligned} \left( L^{b}-\alpha L^{w}\right) X^{T}Xu&=\lambda u \\ \Rightarrow X\left( L^{b}-\alpha L^{w}\right) X^{T}Xu&=\lambda Xu \\ \Rightarrow \left( S_{b}^{L} -\alpha S_{w}^{L}\right) Xu&=\lambda Xu \\ \end{aligned}$$

$\square$

Lemma 1 Let $R_{w}$ and $\tilde{R}_{w}$, respectively, be the orthonormal eigenvectors of $\hat{{S}}_{w}^{L}$ corresponding to non-zero and zero eigenvalues, then $\tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w}$ is positive definite.

Proof

It is obvious the space spanned by $\tilde{R}_{w}$ is the null space of $\hat{{S}}_{w}^{L}$. Suppose $\mathbf{a}$ is an arbitrary vector and $\mathbf{a}\ne 0$, we have

$$\begin{aligned} \mathbf{a}^{T}\tilde{R}_{w}^{T} \hat{{S}}_{w}^{L} \tilde{R}_{w} \mathbf{a}=0 \end{aligned}$$

(9)

It is obvious that $\hat{{S}}_{t}^{L}$ is positive definite, then we can obtain $\tilde{R}_{w}^{T} \hat{{S}}_{t}^{L} \tilde{R}_{w}$ is also positive definite and

$$\begin{aligned} \mathbf{a}^{T}\tilde{R}_{w}^{T} \hat{{S}}_{t}^{L} \tilde{R}_{w} \mathbf{a}>0 \end{aligned}$$

(10)

By combining Eqs. (9), (10) and $\tilde{R}_{w}^{T} \hat{{S}}_{t}^{L} \tilde{R}_{w} =\tilde{R}_{w}^{T} \hat{{S}}_{w}^{L} \tilde{R}_{w} +\tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w}$, we get

$$\begin{aligned} \mathbf{a}^{T}\tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w} \mathbf{a}>0 \end{aligned}$$

(11)

Then we can obtain $\tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w}$ is positive definite $\square$

Theorem 2 $\delta (S_{b}^{L} -\alpha S_{w}^{L} )\ge d-r_{t}$, where $r_{t} =rank(S_{t}^{L})$

Proof

Since column vectors in $\tilde{Q}_{t}$ are the eigenvectors of $S_{t}^{L}$ corresponding to zero eigenvalues, we have

$$\begin{aligned} S_{t}^{L} \tilde{Q}_{t} =0, \tilde{Q}_{t}^{T} S_{t}^{L} \tilde{Q}_{t} =0 \end{aligned}$$

(12)

Since $S_{w}^{L}$ and $S_{b}^{L}$ are both positive semi-definite and $S_{t}^{L} =S_{w}^{L} +S_{b}^{L}$, we can obtain

$$\begin{aligned} \tilde{Q}_{t}^{T} S_{t}^{L} \tilde{Q}_{t}= & {} 0\Leftrightarrow \tilde{Q}_{t}^{T} (S_{w}^{L} +S_{b}^{L} )\tilde{Q}_{t} =0\Leftrightarrow \tilde{Q}_{t}^{T} S_{w}^{L} \tilde{Q}_{t} +\tilde{Q}_{t}^{T} S_{b}^{L} \tilde{Q}_{t} =0\Leftrightarrow \tilde{Q}_{t}^{T} S_{w}^{L} \tilde{Q}_{t}\nonumber \\= & {} 0, \tilde{Q}_{t}^{T} S_{b}^{L} \tilde{Q}_{t} =0 \end{aligned}$$

(13)

Then we have

$$\begin{aligned} S_{w}^{L} \tilde{Q}_{t} =0, S_{b}^{L} \tilde{Q}_{t} =0, \left( S_{b}^{L} -\alpha S_{w}^{L} \right) \tilde{Q}_{t} =0 \end{aligned}$$

(14)

Note that $\tilde{Q}_{t}$ is a $d\times (d-r_{t})$ matrix. Then we can obtain $\delta (S_{b}^{L} -\alpha S_{w}^{L})\ge d-r_{t}$. $\square$

Lemma 3 Suppose $\lambda$ to be an eigenvalue of $\hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L}$. Then $\lambda$ is also an eigenvalue of $S_{b}^{L} -\alpha S_{w}^{L}$.

Proof

Suppose $\varphi$ to be eigenvector of the matrix $\hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L}$ corresponding to the eigenvalue $\lambda$, we have

$$\begin{aligned}&\left( \hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L} \right) \varphi =\lambda \varphi \Rightarrow \nonumber \\&\quad Q_{t} Q_{t}^{T} \left( S_{b}^{L} -\alpha S_{w}^{L} \right) Q_{t} \varphi =\lambda Q_{t} \varphi \end{aligned}$$

(15)

From Eq. (14) we can obtain

$$\begin{aligned} \tilde{Q}_{t} \tilde{Q}_{t}^{T} S_{b}^{L} =0, \tilde{Q}_{t} \tilde{Q}_{t}^{T} S_{w}^{L} =0 \end{aligned}$$

(16)

By combining Eqs. (15) and (16) we have

$$\begin{aligned} \left( Q_{t} Q_{t}^{T} + \tilde{Q}_{t} \tilde{Q}_{t}^{T} \right) \left( S_{b}^{L} -\alpha S_{w}^{L} \right) Q_{t} \varphi =\lambda Q_{t} \varphi \end{aligned}$$

(17)

Let $P=[{\begin{array}{ll} {Q_{t}}&{} {\tilde{Q}_{t}} \\ \end{array}}]$, we have

$$\begin{aligned} PP^{T}=Q_{t} Q_{t}^{T} +\tilde{Q}_{t} \tilde{Q}_{t}^{T} \end{aligned}$$

(18)

Note that P is a unitary matrix, then we have

$$\begin{aligned} \left( S_{b}^{L} -\alpha S_{w}^{L} \right) Q_{t} \varphi =\lambda Q_{t} \varphi \end{aligned}$$

(19)

From Eq. (19) we know $\lambda$ is also an eigenvalue of $S_{b}^{L} -\alpha S_{w}^{L}$. $\square$

Theorem 3 $\pi (S_{b}^{L} -\alpha S_{w}^{L} )\ge r_{t} -r_{w} ,\nu (S_{b}^{L} -\alpha S_{w}^{L} )\ge r_{t} -r_{b}$, where $r_{b} =rank\left( {S_{b}^{L}} \right)$ and $r_{w} =rank(S_{w}^{L})$, respectively.

Proof

Since $r_{w} =rank(S_{w}^{L}), R_{w}$ is a $r_{t} \times r_{w}$ matrix and $\tilde{R}_{w}$ is a $r_{t} \times (r_{t} -r_{w})$ matrix. Let $P=[\tilde{R}_{w} R_{w}]$, we have

$$\begin{aligned} P^{T}\left( \hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L} \right) P=\left[ {\begin{array}{cc} {\tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w}}&{} C \\ {C^{T}}&{} D \\ \end{array} }\right] \end{aligned}$$

(20)

where $C=\tilde{R}_{w}^{T} (\hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L} )R_{w}$ is a $(r_{t} -r_{w} )\times r_{w}$ matrix and $D=R_{w}^{T} (\hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L} )R_{w}$ is a $r_{w} \times r_{w}$ matrix. From Lemma 1, we know $\tilde{R}_w^T \hat{{S}}_b^L \tilde{R}_w$is positive definite. Then there exists $(\tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w} )^{-1}$. Let $\tilde{P}$ be a matrix defined as

$$\begin{aligned} \tilde{P}=\left[ {\begin{array}{cc} {I_{1}}&{}\quad {-\left( \tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w}\right) ^{-1}C} \\ O&{}\quad {I_{2}} \\ \end{array} }\right] \end{aligned}$$

(21)

where $I_{1}$ is a $(r_{t} -r_{w} )\times (r_{t} -r_{w})$ identity matrix, $I_{2}$ is a $r_{w} \times r_{w}$ identity matrix, and O is a $r_{w} \times (r_{t} -r_{w})$ zero matrix. Then we have

$$\begin{aligned} \tilde{P}^{T}P^{T}\left( \hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L} \right) P\tilde{P}=\left[ {\begin{array}{cc} {\tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w} }&{} {O^{T}} \\ O&{} {D-C^{T}\left( \tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w} \right) ^{-1}C} \\ \end{array} }\right] \end{aligned}$$

(22)

Note that $\tilde{R}_{w}^{T} \hat{{S}}_{b}^{L} \tilde{R}_{w}$ is a positive definite $(r_{t} -r_{w} )\times (r_{t} -r_{w})$ matrix, from Eq. (22) we can obtain $\pi (\tilde{P}^{T}P^{T}(\hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L} )P\tilde{P})\ge r_{t} -r_{w}$. Since P and $\tilde{P}$ are both non-degenerate matrix, from Lemma 2, we can obtain $\pi (\hat{{S}}_{b}^{L} -\alpha \hat{{S}}_{w}^{L} )\ge r_{t} -r_{w}$. By using Lemma 3 we can obtain $\pi (S_{b}^{L} -\alpha S_{w}^{L} )\ge r_{t} -r_{w}$

Following the similar deduction as $\pi (S_{b}^{L} -\alpha S_{w}^{L})\ge r_{t} -r_{w}$, we can get $\nu (S_{b}^{L} -\alpha S_{w}^{L} ) \ge r_{t}-r_{b}$ if we use $R_{b}$ and $\tilde{R}_{b}$ to substitute $R_{w}$ and $\tilde{R}_{w}$, respectively. $\square$

Lemma 4 If $S_{w}^{L}$ is nonsingular, then the function $f(\alpha )$ is strictly monotonic decreasing.

Proof

Let $0<\alpha _{1} <\alpha _{2} ,\mathbf{w}_{1}$ and $\mathbf{w}_{2}$ are the eigenvectors corresponding to the maximal eigenvalues of $(S_{b}^{L} -\alpha _{1} S_{w}^{L})$ and $(S_{b}^{L} -\alpha _{2} S_{w}^{L})$, respectively. Then we have

$$\begin{aligned} f(\alpha _{1} )&=\mathbf{w}_{1}^{T} \left( S_{b}^{L} -\alpha _{1} S_{w}^{L} \right) \mathbf{w}_{1} \ge \mathbf{w}_{2}^{T}\left( S_{b}^{L} -\alpha _{1} S_{w}^{L}\right) \mathbf{w}_{2} \nonumber \\&=\mathbf{w}_{2}^{T}\left( S_{b}^{L} -\alpha _{2} S_{w}^{L} \right) \mathbf{w}_{2} +(\alpha _{2} -\alpha _{1} )\mathbf{w}_{2}^{T}S_{w}^{L} \mathbf{w}_{2} \nonumber \\&=f(\alpha _{2} )+ \left( \alpha _{2} -\alpha _{1} \right) \mathbf{w}_{2}^{T}S_{w}^{L} \mathbf{w}_{2} \end{aligned}$$

(23)

Since $S_{w}^{L}$ is nonsingular, we have $\mathbf{w}_{2}^{T}S_{w}^{L} \mathbf{w}_{2} >0, (\alpha _{2} -\alpha _{1} )\mathbf{w}_{2}^{T}S_{w}^{L} \mathbf{w}_{2} >0$ and $f(\alpha _{1} )>f(\alpha _{2})$. This proves the strictly monotonic decreasing property of $f(\alpha )$. $\square$

Theorem 4 The eigenvector corresponding to the maximal eigenvalue of $S_{b}^{L} -\alpha _{0} S_{w}^{L}$ is equivalent to the eigenvector corresponding to the maximal eigenvalue of $(S_{w}^{L} )^{-1}S_{b}^{L}$.

Proof

Suppose $\mathbf{w}^{{*}}$ is the eigenvector corresponding to the maximal eigenvalue of $(S_{w}^{L} )^{-1}S_{b}^{L}$ and $c_{1} =\frac{(\mathbf{w}^{{*}})^{T}S_{b}^{L} \mathbf{w}^{{*}}}{(\mathbf{w}^{{*}})^{T}S_{w}^{L} \mathbf{w}^{{*}}}$, where $c_{1}$ is also the maximal eigenvalue of $(S_{w}^{L} )^{-1}S_{b}^{L}$. Then we have

$$\begin{aligned} (\mathbf{w}^{{*}})^{T}S_{b}^{L} \mathbf{w}^{{*}}-c_{1} (\mathbf{w}^{{*}})^{T}S_{w}^{L} \mathbf{w}^{{*}}=0 \end{aligned}$$

(24)

and

$$\begin{aligned} \frac{\mathbf{w}^{T}S_{b}^{L} \mathbf{w}}{\mathbf{w}^{T}S_{w}^{L} \mathbf{w}}\le c_{1} , \mathbf{w}^{T}S_{b}^{L} \mathbf{w}-c_{1} \mathbf{w}^{T}S_{w}^{L} \mathbf{w}\le 0 \end{aligned}$$

(25)

where $\mathbf{w}$ is an arbitrary projection vector. By combining Eqs. (24) and (25) we have

$$\begin{aligned} (\mathbf{w}^{{*}})^{T}S_{b}^{L} \mathbf{w}^{{*}}-c_{1} (\mathbf{w}^{{*}})^{T}S_{w}^{L} \mathbf{w}^{{*}}=0\ge \mathbf{w}^{T}S_{b}^{L} \mathbf{w}-c_{1} \mathbf{w}^{T}S_{w}^{L} \mathbf{w} \end{aligned}$$

(26)

From Eq. (26) we know $\mathbf{w}^{{*}}$ is also the eigenvector corresponding to the maximal eigenvalue of $S_{b}^{L} -c_{1} S_{w}^{L}$. That is

$$\begin{aligned} f(c_{1} )=(\mathbf{w}^{{*}})^{T}S_{b}^{L} \mathbf{w}^{{*}}-c_{1} (\mathbf{w}^{{*}})^{T}S_{w}^{L} \mathbf{w}^{{*}}=0 \end{aligned}$$

(27)

Note that the zero point of $f(\alpha )$ is unique, then we have $c_1 =\alpha _{0}$ $\square$