Discriminative least squares regression for multiclass classification based on within-class scatter minimization

Abstract

Least square regression has been widely used in pattern classification, due to the compact form and efficient solution. However, two main issues limit its performance for solving the multiclass classification problems. The first one is that employing the hard discrete labels as the regression targets is inappropriate for multiclass classification. The second one is that it focus only on exactly fitting the instances to the target matrix while ignoring the within-class similarity of the instances, resulting in overfitting. To address this issues, we propose a discriminative least squares regression for multiclass classification based on within-class scatter minimization (WCSDLSR). Specifically, a ε-dragging technique is first introduced to relax the hard discrete labels into the slack soft labels, which enlarges the between-class margin for the soft labels as much as possible. The within-class scatter for the soft labels is then constructed as a regularization term to make the transformed instances of the same class closer to each other. These factors ensure WCSDLSR can learn a more compact and discriminative transformation for classification, thus avoiding the overfitting problems. Furthermore, the proposed WCSDLSR can obtain a closed-form solution in each iteration with the lower computational costs. Experimental results on the benchmark datasets demonstrate that the proposed WCSDLSR achieves the better classification performance with the lower computational costs.

Appendix A:: Proof of Theorem 1

Proof

For simplicity, Let L denotes the optimization problem (8). The KKT conditions for (8) are derived as follows (because the process of solving W and E does not involve the Lagrange multipliers, the KKT conditions for them are omitted):

$$ \begin{array}{@{}rcl@{}} \textbf{U}=\textbf{T}. \end{array} $$

(20)

$$ \begin{array}{@{}rcl@{}} \frac{\partial{L}}{\partial{\textbf{T}}}&=&(1+\alpha+\mu)\textbf{T}\\&&-\left( \textbf{X}\textbf{W}+\alpha(\textbf{Y}+\textbf{Y}\odot\textbf{E}) +\mu\textbf{U}-\textbf{H}\right)=0. \end{array} $$

(21)

$$ \begin{array}{@{}rcl@{}} \frac{\partial{L}}{\partial{\textbf{U}_{j}}}&=&\left( (\beta+\mu)\textbf{I}_{n_{j}}-\frac{\beta}{n_{j}}\textbf{1}_{n_{j}}\right)\textbf{U}_{j} \\&&-(\mu\textbf{T}_{j}+\textbf{H}_{j})=0. \end{array} $$

(22)

First, the Lagrangian multiplier H can be obtained from Algorithm 1, as follows

$$ \begin{array}{@{}rcl@{}} \textbf{H}^{k}=\textbf{H}+\mu(\textbf{T}-\textbf{U}). \end{array} $$

(23)

If sequence $\{\textbf {H}^{k}\}^{\infty }_{k=1}$ converges to a stationary point, i.e., $(\textbf {H}^k-\textbf {H})\rightarrow 0$, then $(\textbf {T}-\textbf {U})\rightarrow 0$. Thus, the first KKT condition (20) is proved.

For the second KKT condition, the following equation can be obtained from (10)

$$ \begin{array}{@{}rcl@{}} \textbf{T}^{k}-\textbf{T} &=&(1+\alpha+\mu)^{-1}\left( \textbf{X}\textbf{W}+\alpha(\textbf{Y}+\textbf{Y}\odot\textbf{E})\right.\\&&\left.+\mu\textbf{U}-\textbf{H}\right)-\textbf{T} \end{array} $$

(24)

From the (24), the following equation can be obtained

$$ \begin{array}{@{}rcl@{}} &&(1+\alpha+\mu)(\textbf{T}^{k}-\textbf{T})\\ &=&\left( \textbf{X}\textbf{W}+\alpha(\textbf{Y}+\textbf{Y}\odot\textbf{E})+\mu\textbf{U}-\textbf{H}\right)-(1+\alpha+\mu)\textbf{T} \end{array} $$

(25)

(25) indicates that when the sequence $\{\textbf {T}^{k}\}^{\infty }_{k=1}$ converges to a stationary point, i.e., $(\textbf {T}^k-\textbf {T})\rightarrow 0$, the second KKT condition (21) holds.

For the third KKT condition (22), the following equation is obtained by using the results of (12)

$$ \begin{array}{@{}rcl@{}} \textbf{U}_{j}^{k} - \textbf{U}_{j} = \left( (\beta+\mu)\textbf{I}_{n_{j}} - \frac{\beta}{n_{j}}\textbf{1}_{n_{j}}\right)^{-1} (\mu\textbf{T}_{j} + \textbf{H}_{j}) - \textbf{U}_{j} \end{array} $$

(26)

which is equivalent to

$$ \begin{array}{@{}rcl@{}} &\left( (\beta+\mu)\textbf{I}_{n_{j}}-\frac{\beta}{n_{j}}\textbf{1}_{n_{j}}\right)(\textbf{U}_{j}^{k}-\textbf{U}_{j})\\ &=(\mu\textbf{T}_{j}+\textbf{H}_{j})-\left( (\beta+\mu)\textbf{I}_{n_{j}}-\frac{\beta}{n_{j}}\textbf{1}_{n_{j}}\right)\textbf{U}_{j} \end{array} $$

(27)

(27) indicates that when the sequence $\{\textbf {U}^{k}\}^{\infty }_{k=1}$ converges to a stationary point, i.e., $(\textbf {U}^k-\textbf {U})\rightarrow 0$, the third KKT condition (22) holds.

In summary, the sequence solution $\{{{\varTheta }}^k\}^{\infty }_{k=1}$ is bounded and $\lim _{k\rightarrow \infty }({{\varTheta }}^{k+1}-{{\varTheta }}^{k})=0$ can deduce that the limit points of $\{{{\varTheta }}^k\}^{\infty }_{k=1}$ is the Karush-Kuhn-Tucker (KKT) point of problem (8). Thus, the proof is complete. □