Robust $l_{2,1}$ Norm-Based Sparse Dictionary Coding Regularization of Homogenous and Heterogenous Graph Embeddings for Image Classifications

Abstract

In the field of manifold learning, Marginal Fisher Analysis (MFA), Discriminant Neighborhood Embedding (DNE) and Double Adjacency Graph-based DNE (DAG-DNE) construct the graph embedding for homogeneous and heterogeneous k-nearest neighbors (i.e. double adjacency) before feature extraction. All of them have two shortcomings: (1) vulnerable to noise; (2) the number of feature dimensions is fixed and likely very large. Taking advantage of the sparsity effect and de-noising property of sparse dictionary, we add the $l_{2,1}$ norm-based sparse dictionary coding regularization term to the graph embedding of double adjacency, to form an objective function, which seeks a small amount of significant dictionary atoms for feature extraction. Since our initial objective function cannot generate the closed-form solution, we construct an auxiliary function instead. Theoretically, the auxiliary function has closed-form solution w.r.t. dictionary atoms and sparse coding coefficients in each iterative step and its monotonously decreased value can pull down the initial objective function value. Extensive experiments on the synthetic dataset, the Yale face dataset, the UMIST face dataset and the terrain cover dataset demonstrate that our proposed algorithm has the ability of pushing the separability among heterogenous classes onto much fewer dimensions, and robust to noise.

Appendix: Theoretical Proofs of the SDCR-DAGE Algorithm

Theorem 1

$l(D,U,A)\ge f(D,U,A)$ always holds.

Proof

Based on $M=I+\alpha L_{SD}$ and $Y=XM^{-1}$, from Eq. (12) it gets that:

$$\begin{aligned} f(D,U,A)=tr\{M(A^{T}D^{T}DA-2Y^{T}DA+Y^{T}Y)+X^{T}X(I-M^{-1})+\beta A^{T}UA+\gamma D^{T}D\} \end{aligned}$$

In light of triangle inequality,

$$\begin{aligned} tr\{M(A^{T}D^{T}DA-2Y^{T}DA+Y^{T}Y)\}\le tr(M)\cdot tr(Y-DA)^{T}(Y-DA) \end{aligned}$$

Therefore, $l(D,U,A)\ge f(D,U,A)$.$\square$

Theorem 2

The term $I-M^{-1}$ in Eq. (14) is identical to $\alpha L_{SD}^{\frac{1}{2}}M^{-1}L_{SD}^{\frac{1}{2}}$.

Proof

Let $L_{SD}^{\frac{1}{2}}=U\Sigma V^{T}$, since $L_{SD}$ is symmetric and positive definite, then $U=V$, and $VV^{T}=V^{T}V=I$, so $V^{T}=V^{-1}$.

Besides, since $(I+BC)^{-1}=I-B(I+CB)^{-1}C$ and $(AB)^{-1}=B^{-1}A^{-1}$.

$$\begin{aligned}&M^{-1}=(I+\sqrt{\alpha }V\Sigma ^{2}V^{T}\sqrt{\alpha })^{-1} \\&=I-\sqrt{\alpha }V\Sigma (I+\Sigma V^{T}\alpha V\Sigma )^{-1}\Sigma V^{T}\sqrt{\alpha } \\&=I-\alpha V\Sigma (I+\alpha \Sigma ^{2})^{-1}\Sigma V^{T} \end{aligned}$$

Therefore,

$$\begin{aligned}&I-M^{-1}=\alpha V\Sigma V^{T}V(I+\alpha \Sigma ^{2})^{-1}V^{T}V\Sigma V^{T} \\&=\alpha L_{SD}^{\frac{1}{2}}(V^{T})^{-1}(I+\alpha \Sigma ^{2})^{-1}V^{-1}L_{SD}^{\frac{1}{2}} \\&=\alpha L_{SD}^{\frac{1}{2}}(I+\alpha V\Sigma ^{2}V^{T})^{-1}L_{SD}^{\frac{1}{2}} \\&=\alpha L_{SD}^{\frac{1}{2}}M^{-1}L_{SD}^{\frac{1}{2}} \end{aligned}$$

$\square$

Theorem 3

The monotonous decrease of l(D, U, A) can pull down the value of f(D, A) in Eq. (11).

Proof

In light of Eq. (15), let

$$\begin{aligned} l(D,A)= & {} \underset{D,A}{argmin}\quad tr(M)\cdot \parallel Y-DA\parallel ^{2}_{F}+\beta \parallel A\parallel _{2,1}+ \gamma \parallel D\parallel ^{2}_{F}\nonumber \\&+\,\alpha \cdot tr(X^{T}XL_{SD}^{\frac{1}{2}}M^{-1}L_{SD}^{\frac{1}{2}}) \end{aligned}$$

Due to the monotonous decrease of l(D, U, A), if we fix U as $U^{t}$ after the t-th iteration and update $D^{t+1}$ and $A^{t+1}$, since $\parallel A\parallel _{2,1}=\sum _{k=1}^{s}\parallel a^{k}\parallel _{2}$, then: $l(D^{t+1},U^{t},A^{t+1})\le l(D^{t},U^{t},A^{t} )$, i.e.

$$\begin{aligned}&tr(M)\cdot \parallel Y-D^{t+1}A^{t+1}\parallel _{F}^{2}+\gamma tr(D^{t+1^{T}}D^{t+1})+\beta \parallel A^{t+1}\parallel _{2,1}\\&\qquad +\,\beta \sum _{k=1}^{d}(\frac{\parallel a^{k^{t+1}}\parallel _{2}^{2}}{2\parallel a^{k^{t}}\parallel _{2}}-\parallel a^{k^{t+1}}\parallel _{2})\\&\quad \le tr(M)\cdot \parallel Y-D^{t}A^{t}\parallel _{F}^{2}+\gamma tr(D^{t^{T}}D^{t})+\,\beta \parallel A^{t}\parallel _{2,1} +\beta \sum _{k=1}^{s}(\frac{\parallel a^{k^{t}}\parallel _{2}^{2}}{2\parallel a^{k^{t}}\parallel _{2}}-\parallel a^{k^{t}}\parallel _{2}) \end{aligned}$$

Based on

$$\begin{aligned} \frac{\parallel a^{k^{t+1}}\parallel _{2}^{2}}{2\parallel a^{k^{t}}\parallel _{2}}-\parallel a^{k^{t+1}}\parallel _{2} \ge \frac{\parallel a^{k^{t}}\parallel _{2}^{2}}{2\parallel a^{k^{t}}\parallel _{2}}-\parallel a^{k^{t}}\parallel _{2} \end{aligned}$$

it gets:

$$\begin{aligned}&tr(M)\cdot \parallel Y-D^{t+1}A^{t+1}\parallel _{F}^{2}+\gamma tr(D^{t+1^{T}}D^{t+1})+\beta \parallel A^{t+1}\parallel _{2,1}\\&\quad \le tr(M)\cdot \parallel Y-D^{t}A^{t}\parallel _{F}^{2}+\gamma tr(D^{t^{T}}D^{t})+\beta \parallel A^{t}\parallel _{2,1} \end{aligned}$$

Therefore for any iteration t, $l(D^{t+1},A^{t+1})\le l(D^{t},A^{t})$. Using triangle inequality again, similar to Theorem 1, $l(D,A)\ge f(D,A)$. Therefore, the monotonous decrease of l(D, U, A) can pull down the value of f(D, A). $\square$

Theorem 4

If the SVD decompositions $X=U\Sigma V^{T}$ and $MV=B\theta Q^{T}$, then $YB=U\Sigma Q\theta ^{-1}$.

Proof

Since M is of full rank, therefore $M^{-1}$ exists. In light of $V^{T}M^{-1}MV=I$, we get $V^{T}M^{-1}B\theta Q^{T}=I$, then $XM^{-1}B=U\Sigma V^{T}M^{-1}B\theta Q^{T}Q\theta ^{-1}=U\Sigma Q\theta ^{-1}$. $\square$