Generalized robust linear discriminant analysis for jointly sparse learning

Abstract

Linear discriminant analysis (LDA) is a well-known supervised method that can perform dimensionality reduction and feature extraction effectively. However, traditional LDA-based methods need to be turned into the trace ratio form to compute the closed-form solution, in which the within-class scatter matrix should be nonsingular. In this article, we design a new model named generalized robust linear discriminant analysis (GRLDA) method to tackle this disadvantage and improve the robustness. GRLDA uses ${L}_{\mathrm{2,1}}$-norm on both loss functions to reduce the influence of outliers and on regularization term to obtain joint sparsity simultaneously. The intrinsic graph and the penalty graph are constructed to characterize the intraclass similarity and interclass separability, respectively. A novel optimization method is proposed to solve the proposed model, in which a quadratic problem on the Stiefel manifold is involved to avoid the inverse computation on a singular matrix. We also analyze the computational complexity rigorously. Finally, the experimental results on face, object, and medical images exhibit the superiority of GRLDA.

Appendices

Appendix

First, we introduce a lemma as follows:

Lemma 3

Assuming that f(x) is a convex function of x where x can be a scalar, vector or matrix variable, then we obtain:

$$\mathrm{f}({\mathrm{x}}_{1})-\mathrm{f}({\mathrm{x}}_{2})\ge \mathrm{Tr}({({\mathrm{f}}{{^{\prime}}}({\mathrm{x}}_{2}))}^{\mathrm{T}}({\mathrm{x}}_{1}-{\mathrm{x}}_{2})$$

(28)

where f'(x₂) is the super-gradient of f(x) at x₂.

Proof of the theorem 2

It is easy to know that ${\mathrm{f}}_{\mathrm{i}}({\mathrm{h}}_{\mathrm{i}}(\mathrm{m}))$ is an arbitrary convex function w.r.t. ${\mathrm{h}}_{\mathrm{i}}(\mathrm{m})$  under the arbitrary constraint of $\mathrm{m}\in\Omega$. We assume that ${\mathrm{f}}_{\mathrm{i}}({\mathrm{h}}_{\mathrm{i}}(\mathrm{m}))\le0$. In the t-th iteration, we denote ${\mathrm{G}}_{\mathrm{i}}^{\mathrm{t}}={{\mathrm{f}}{\prime}}_{\mathrm{i}}({\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}-1}))$. For each i, according to the lemma 3, we have:

$$\begin{array}{l}{\mathrm{f}}_{\mathrm{i}}({\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}}))-{\mathrm{f}}_{\mathrm{i}}({\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}-1}))\\ \ge Tr({(}^{{\mathrm{G}}_{\mathrm{i}}^{\mathrm{t}}}{\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}}))-Tr({(}^{{\mathrm{G}}_{\mathrm{i}}^{\mathrm{t}}}{\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}-1}))\end{array}$$

(29)

According to (21), the following can be derived:

$$\mathrm{Tr}({({\mathrm{G}}_{\mathrm{i}}^{\mathrm{t}})}^{\mathrm{T}}{\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}}))\ge \mathrm{Tr}({({\mathrm{G}}_{\mathrm{i}}^{\mathrm{t}})}^{\mathrm{T}}{\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}-1}))$$

(30)

Summing (29) and (30), we have:

$$\sum_{\mathrm{i}}{\mathrm{f}}_{\mathrm{i}}({\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}}))\ge \sum_{\mathrm{i}}{\mathrm{f}}_{\mathrm{i}}({\mathrm{h}}_{\mathrm{i}}({\mathrm{m}}^{\mathrm{t}-1}))$$

(31)

Summing (31) and ${\mathrm{f}}_{\mathrm{i}}({\mathrm{h}}_{\mathrm{i}}(\mathrm{m}))\le 0$, the value of the objective function (16) will monotonically increase until convergence.