Simultaneous dimensionality reduction and dictionary learning for sparse representation based classification

Abstract

Learning dictionaries from the training data has led to promising results for pattern classification tasks. Dimensionality reduction is also an important issue for pattern classification. However, most existing methods perform dimensionality reduction (DR) and dictionary learning (DL) independently, which may result in not fully exploiting the discriminative information of the training data. In this paper, we propose a simultaneous dimensionality reduction and dictionary learning (SDRDL) model to learn a DR projection matrix and a class-specific dictionary (i.e., the dictionary atoms correspond to the class labels) simultaneously. Since simultaneously learning makes the learned projection and dictionary fit better with each other, more effective pattern classification can be achieved using the representation residual. In SDRDL model, not only the representation residual is discriminative, but the representation coefficients are also discriminative. Therefore, a classification scheme associated with SDRDL is presented by exploiting such discriminative information. Experimental results on a series of benchmark image databases show that our proposed method outperforms many state-of-the-art discriminative dictionary learning methods.

Appendix

φ _i (Z _i ) is convex and continuously differentiable with Lipschitz continuous gradient L(φ _i ):

$$ \left\Vert \nabla {\varphi}_i(x)-\nabla {\varphi}_i(y)\right\Vert \le L\left({\varphi}_i\right)\left\Vert x-y\right\Vert, \forall x,y\in {R}^n. $$

(21)

where ‖ ⋅ ‖ denotes the standard Euclidean norm and L(φ _i ) > 0 is the Lipschitz constant of ∇φ _i .

In Eq. (10)

$$ {\varphi}_i\left({Z}_i\right)={\left\Vert {B}_i-D{Z}_i\right\Vert}_F^2+{\left\Vert {B}_i-{D}_i{Z}_i^i\right\Vert}_F^2+{\displaystyle {\sum}_{j\ne i}{\left\Vert {D}_j{Z}_i^j\right\Vert}_F^2}+{\lambda}_2{\displaystyle {\sum}_{j\ne i}{\left\Vert {\tilde{Z}}_j^T{Z}_i\right\Vert}_F^2} $$

(22)

Let Z ⁱ _i  = P ⁱ Z _i and Z ^j _i  = P ^j Z _i where P ⁱ(P ^j) are projection matrixes which keeps components of Z _i (Z _j ) associated with D _i (D _j ) unchanged but sets other components to be zero. Hence, we can rewrite Eq. (22) as:

$$ {\varphi}_i\left({Z}_i\right)={\left\Vert {B}_i-D{Z}_i\right\Vert}_F^2+{\left\Vert {B}_i-D{P}^i{Z}_i\right\Vert}_F^2+{\displaystyle {\sum}_{j\ne i}{\left\Vert D{P}^j{Z}_i\right\Vert}_F^2}+{\lambda}_2{\displaystyle {\sum}_{j\ne i}{\left\Vert {\tilde{Z}}_j^T{Z}_i\right\Vert}_F^2} $$

(23)

Let DP ⁱ = D ⁱ and DP ^j = D ^j. Equation (23) equals to:

$$ {\varphi}_i\left({Z}_i\right)={\left\Vert {B}_i-D{Z}_i\right\Vert}_F^2+{\left\Vert {B}_i-{D}^i{Z}_i\right\Vert}_F^2+{\displaystyle {\sum}_{j\ne i}{\left\Vert {D}^j{Z}_i\right\Vert}_F^2}+{\lambda}_2{\displaystyle {\sum}_{j\ne i}{\left\Vert {\tilde{Z}}_j^T{Z}_i\right\Vert}_F^2} $$

(24)

The stacking operator introduced in [30] can be used to write B _i and Z _i as a column vector. We form ${\varPsi}_i={\left[{b}_{i,1},{b}_{i,2},\cdots, {b}_{i,{n}_i}\right]}^T$, ${\chi}_i={\left[{z}_{i,1},{z}_{i,2},\cdots, {z}_{i,{n}_i}\right]}^T$ where a _i,i , z _i,i  ∈ R ^m × 1 and thus ${\varPsi}_i,{\chi}_i\in {R}^{\left(m\cdot {n}_i\right)\times 1}$. Hence, Eq. (24) can be rewrite as:

$$ \begin{array}{l}{\varphi}_i\left({\chi}_i\right)={\left\Vert {\varPsi}_i-\mathrm{diag}(D){\chi}_i\right\Vert}_2^2+{\left\Vert {\varPsi}_i-\mathrm{diag}\left({D}^i\right){\chi}_i\right\Vert}_2^2+{\displaystyle {\sum}_{j\ne i}{\left\Vert \mathrm{diag}\left({D}^j\right){\chi}_i\right\Vert}_2^2}+\hfill \\ {}\kern4em {\lambda}_2{\displaystyle {\sum}_{j\ne i}{\left\Vert \mathrm{diag}\left({\tilde{\chi}}_j^T\right){\chi}_i\right\Vert}_2^2}\hfill \end{array} $$

(25)

where diag(T) is a block diagonal matrix with each block on the diagonal being matrix T. And also φ _i (χ _i ) equals to:

$$ \begin{array}{c}\hfill 2{\varPsi}_i^T{\varPsi}_i-2{\varPsi}_i^T\left(\mathrm{diag}(D)+\mathrm{diag}\left({D}^i\right)\right){\chi}_i+{\chi}_i^T\left(\mathrm{diag}\left({D}^TD\right)+\mathrm{diag}\left({D^i}^T{D}^i\right)\right.+\hfill \\ {}\hfill\ \left.\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{D^j}^T{D}^j}\right)+{\lambda}_2\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{\tilde{\chi}}_j{\tilde{\chi}}_j^T}\right)\right){\chi}_i\hfill \end{array} $$

(26)

The convexity of φ _i (χ _i ) depends on its Hessian matrix ∇² φ _i (χ _i ) is whether positive semi-definite or not [5]. We could write the Hessian matrix of φ _i (χ _i ) as:

$$ \begin{array}{c}\hfill {\nabla}^2{\varphi}_i\left({\chi}_i\right)=2\mathrm{diag}\left({D}^TD\right)+2\mathrm{diag}\left({D^i}^T{D}^i\right)+2\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{D^j}^T{D}^j}\right)+\hfill \\ {}\hfill 2{\lambda}_2\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{\tilde{\chi}}_j{\tilde{\chi}}_j^T}\right)\hfill \end{array} $$

(27)

Since diag(D ^T D), diag(D ^iT D ⁱ), diag(∑ _j ≠ i D ^jT D ^j) and $\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{\tilde{\chi}}_j{\tilde{\chi}}_j^T}\right)$ are all Hermite matrix, they are all positive semi-definite. Therefore, Hessian matrix ∇² φ _i (χ _i ) is positive semi-definite. Based on this, we claim that φ _i (χ _i ) is a convex function.

Via Eq. (26), we have:

$$ \begin{array}{l}\nabla {\varphi}_i\left({\chi}_i\right)=-2{\varPsi}_i^T\left(\mathrm{diag}(D)+\mathrm{diag}\left({D}^i\right)\right)+2\left(\mathrm{diag}\left({D}^TD\right)+\mathrm{diag}\left({D^i}^T{D}^i\right) + \right.\hfill \\ {}\left.\kern5em \mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{D^j}^T{D}^j}\right)+{\lambda}_2\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{\tilde{\chi}}_j{\tilde{\chi}}_j^T}\right)\right){\chi}_i\hfill \end{array} $$

(28)

From Eq. (28), we can easy see that ∇φ _i (χ _i ) is continuously differentiable to χ _i . And via Eq. (28), we have:

$$ \begin{array}{c}\hfill \nabla {\varphi}_i(x)-\nabla {\varphi}_i(y)=2\left(\mathrm{diag}\left({D}^TD\right)+\mathrm{diag}\left({D^i}^T{D}^i\right)+\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{D^j}^T{D}^j}\right)+\right.\hfill \\ {}\hfill \left.{\lambda}_2\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{\tilde{\chi}}_j{\tilde{\chi}}_j^T}\right)\right)\left(x-y\right)\hfill \end{array} $$

(29)

Hence, we obtain:

$$ \begin{array}{c}\hfill \left\Vert \nabla {\varphi}_i(x)-\nabla {\varphi}_i(y)\right\Vert =\left\Vert 2\left(\mathrm{diag}\left({D}^TD\right)+\mathrm{diag}\left({D^i}^{{}^T}{D}^i\right)+\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{D^j}^{{}^T}{D}^j}\right)+\right.\right.\hfill \\ {}\hfill \left.\left.\ {\lambda}_2\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{\tilde{\chi}}_j{\tilde{\chi}}_j^T}\right)\right)\left(x-y\right)\right\Vert \hfill \\ {}\hfill \le 2\left(\mathrm{diag}\left({D}^TD\right)+\mathrm{diag}\left({D^i}^{{}^T}{D}^i\right)+ diag\left({\displaystyle {\sum}_{j\ne i}{D^j}^{{}^T}{D}^j}\right)+\right.\hfill \\ {}\hfill \left.{\lambda}_2\mathrm{diag}\left({{\displaystyle {\sum}_{j\ne i}\tilde{\chi}}}_j{\tilde{\chi}}_j^T\right)\right)\left\Vert \left\Vert \left(x-y\right)\right\Vert \right.\hfill \\ {}\hfill \le 2\ \left\Vert {\lambda}_{max}^1+{\lambda}_{max}^2+{\lambda}_{max}^3+{\lambda}_2{\lambda}_{max}^4\right\Vert \left\Vert \left(x-y\right)\right\Vert \hfill \end{array} $$

(30)

where λ ¹ _max  = λ _max(diag(D ^T D)), λ ² _max  = λ _max(diag(D ^iT D ⁱ)), λ ³ _max  = λ _max(diag(∑ _j ≠ i D ^jT D ^j)) and ${\lambda}_{\max}^4={\lambda}_{\max}\left(\mathrm{diag}\left({\displaystyle {\sum}_{j\ne i}{\tilde{\chi}}_j{\tilde{\chi}}_j^T}\right)\right)$. So the (smallest) Lipschitz constant of the gradient ∇φ _i (χ _i ) is L(φ _i ) = 2(λ ¹ _max  + λ ² _max  + λ ³ _max  + λ ₂ λ ⁴ _max ).

Therefore, we claim that φ _i (Z _i ) is continuously differentiable with Lipschitz continuous gradient L(φ _i ).