Subspace quadratic regularization method for group sparse multinomial logistic regression

Abstract

Sparse multinomial logistic regression has recently received widespread attention. It provides a useful tool for solving multi-classification problems in various fields, such as signal and image processing, machine learning and disease diagnosis. In this paper, we first study the group sparse multinomial logistic regression model and establish its optimality conditions. Based on the theoretical results of this model, we hence propose an efficient algorithm called the subspace quadratic regularization algorithm to compute a stationary point of a given problem. This algorithm enjoys excellent convergence properties, including the global convergence and locally quadratic convergence. Finally, our numerical results on standard benchmark data clearly demonstrate the superior performance of our proposed algorithm in terms of logistic loss value, sparsity recovery and computational time.

Appendix A: Proofs of Lemmas 1 and 2

Proof of Lemma 1:

Since the results of Lemma 1 (i)–(iii) can be directly calculated, we omit their proofs and give the following detailed derivation of 1 (iv).

Denote

$$\begin{aligned} C^{(k)}(W)={\mathrm {Diag}}\Big (\frac{\exp ({w^{(k)}}^{\top }x_i)}{({\mathbf {1}}^{\top }\exp (W^{\top }x_i)+1)^{2}}\Big |\ i=1,2,\ldots ,n\Big ). \end{aligned}$$

(27)

If no confusion arises, we use the simple notation $A=A(W),\ B=B(W),\ C=C(W)$. Since $A^{(k)}=C^{(k)}-\sum \limits _{\begin{array}{c} j\ne k\\ j=1 \end{array}}^{m-1}B^{(k,j)},\ k=1,2,\ldots ,(m-1)$, we have

$$\begin{aligned} X^{\top }A^{(k)}X=X^{\top }(C^{(k)}-\sum \limits _{\begin{array}{c} j\ne k\\ j=1 \end{array}}^{m-1}B^{(k,j)})X. \end{aligned}$$

(28)

Therefore, we can rewrite the Hessian matrix as

$$\begin{aligned} \nabla ^{2}\ell (W)=\left( \begin{array}{ccc} X^{\top }(C^{(1)}-\sum \limits _{j=2}^{m-1}B^{(1,j)})X & \cdots & X^{\top }B^{(1,m-1)}X \\ \vdots & \ddots & \vdots \\ X^{\top }B^{(m-1,1)}X & \cdots & X^{\top }(C^{(m-1)}-\sum \limits _{j=1}^{m-2}B^{(m-1,j)})X \\ \end{array} \right) . \end{aligned}$$

For any $z=(z_1; z_2;\ldots ;z_{m-1}) \in {\mathbb {R}}^{p(m-1)}$ with $z_i \in {\mathbb {R}}^{p},\ i=1,2,\ldots ,(m-1)$, we have

$$\begin{aligned} z^{\top }\nabla ^{2}\ell (W) & z= \sum \limits _{k=1}^{m-1}z_k^\top X^{\top }C^{(k)}Xz_k -\sum \limits _{k=1}^{m-2}\sum \limits _{j=k+1}^{m-1}(z_k-z_j)^\top X^{\top }B^{(k,j)}X(z_k-z_j)\\ & \ge 0, \quad (\text {because of}\ C^{(k)}\succ {\mathbf {0}},\ B^{(k,j)}\preceq {\mathbf {0}}) \nonumber \end{aligned}$$

(29)

which means that $\nabla ^{2}\ell (W)\succeq {\mathbf {0}}$.

To verify the second inequality of (3), D. Böhning [4] has shown that the Hessian matrix is upper bounded by a positive definite matrix that does not depend on W, i.e.,

$$\begin{aligned} \nabla ^{2} \ell (W) \preceq \frac{1}{2}[I_{m-1}-{\mathbf {1}}{\mathbf {1}}^{\top }/m]\otimes X^{\top }X. \end{aligned}$$

(30)

We next prove the Lipschitz continuity of the Hessian matrix. Denote

$$\begin{aligned} h(\mathbf{z })=\displaystyle \frac{\exp (\mathbf{z }^k)}{({\mathbf {1}}^{\top }\exp (\mathbf{z })+1)^{2}},\,\,\, g(\mathbf{z })=\displaystyle \frac{\exp (\mathbf{z }^k)\exp (\mathbf{z }^j)}{({\mathbf {1}}^{\top }\exp (\mathbf{z })+1)^{2}}, \end{aligned}$$

where $\mathbf{z }=(\mathbf{z }^{1},\mathbf{z }^{2},\ldots ,\mathbf{z }^{m-1})^{\top }=W^{\top }x_i \in {\mathbb {R}}^{m-1}$ with $\mathbf{z }^k={w^{(k)}}^{\top }x_i$ for $i \in \{1,2,\ldots ,n\}$. We first consider the following two cases.

Case 1. If $j\ne k$,

$$\begin{aligned} \Big |\frac{\partial h(\mathbf{z })}{\partial \mathbf{z }^j}\Big |=\Big |\frac{-2\exp (\mathbf{z }^k)\exp (\mathbf{z }^j)}{({\mathbf {1}}^{\top }\exp (\mathbf{z })+1)^{3}}\Big | \le 2. \end{aligned}$$

Case 2. If $j= k$,

$$\begin{aligned} \Big |\frac{\partial h(\mathbf{z })}{\partial \mathbf{z }^j}\Big |=\Big |\frac{\exp (\mathbf{z }^k)}{({\mathbf {1}}^{\top }\exp (\mathbf{z })+1)^{2}}-\frac{2\exp ^2(\mathbf{z }^k)}{({\mathbf {1}}^{\top }\exp (\mathbf{z })+1)^{3}}\Big | \le 2. \end{aligned}$$

Hence, $\Vert \nabla h(\mathbf{z } )\Vert \le 2\sqrt{m-1}.$ Then by the mean value theorem, there exists $\theta \in (0,1)$ such that for any $i \in \{1,2,\ldots ,n\}$,

$$\begin{aligned} h(W^{\top }x_i)-h(\widehat{W}^{\top }x_i)& = h(\mathbf{z })-h(\hat{\mathbf{z }})=\nabla h(\mathbf{z }+\theta (\hat{\mathbf{z }}-\mathbf{z }))^{\top }(\mathbf{z }-\hat{\mathbf{z }})\\& \le \Vert \nabla h(\mathbf{z }+\theta (\hat{\mathbf{z }}-\mathbf{z }) )\Vert \Vert \mathbf{z }-\hat{\mathbf{z }}\Vert \le 2\sqrt{m-1}\Vert \mathbf{z }-\hat{\mathbf{z }}\Vert \\ & \le 2\sqrt{m-1}\Vert W-\widehat{W}\Vert \Vert x_i\Vert \le \varpi \Vert W-\widehat{W}\Vert , \end{aligned}$$

where $\varpi := 2\sqrt{m-1}\Vert X\Vert _\infty$. Similarly, we have

$$\begin{aligned} g(W^{\top }x_i)-g(\widehat{W}^{\top }x_i)\le \varpi \Vert W-\widehat{W}\Vert . \end{aligned}$$

Notice that for any $k,j \in \{1,2,\ldots ,m-1\}$,

$$\begin{aligned} \begin{array}{cl} &\Vert C^{(k)}(W)-C^k(\widehat{W})\Vert _2 =\max \limits _{i \in \{1,2,\ldots ,n\}}\{h(W^{\top }x_i)-h(\widehat{W}^{\top }x_i)\} \le \varpi \Vert W-\widehat{W}\Vert \\ &\\ &\Vert B^{(k,j)}(W)-B^{(k,j)}(\widehat{W})\Vert _2=\max \limits _{i \in \{1,2,\ldots ,n\}}\{g(W^{\top }x_i)-g(\widehat{W}^{\top }x_i)\} \le \varpi \Vert W-\widehat{W}\Vert \end{array} \end{aligned}$$

(31)

Therefore,

$$\begin{aligned}&\Vert \nabla ^{2}\ell (W)-\nabla ^{2}\ell (\widehat{W})\Vert \\&\le \Vert \sum \limits _{k=1}^{m-1} X^{\top }(A^{(k)}(W)-A^k(\widehat{W})X\Vert +2\Vert \sum \limits _{k=1}^{m-2}\sum \limits _{j=k+1}^{m-1} X^{\top }(B^{(k,j)}(W)-B^{(k,j)}(\widehat{W}))X \Vert \\&\overset{(28)}{\le } \Vert \sum \limits _{k=1}^{m-1} X^{\top }(C^{(k)}(W)-C^k(\widehat{W})X\Vert +4\Vert \sum \limits _{k=1}^{m-2}\sum \limits _{j=k+1}^{m-1} X^{\top }(B^{(k,j)}(W)-B^{(k,j)}(\widehat{W}))X \Vert \\&\le (m-1)\max \limits _{k,j} \{\Vert X^{\top }(C^{(k)}(W)-C^k(\widehat{W}))X\Vert +2(m-2)\Vert X^{\top }(B^{(k,j)}(W)-B^{(k,j)}(\widehat{W}))X \Vert \}\\&\le (m-1)\max \limits _{k,j} \{\Vert C^{(k)}(W)-C^k(\widehat{W})\Vert _2+2(m-2)\Vert B^{(k,j)}(W)-B^{(k,j)}(\widehat{W})\Vert _2\}\Vert X^{\top }X \Vert \\&\overset{(31)}{\le }2\varpi (m-1)^2 \Vert X^{\top }X \Vert \Vert W-\widehat{W}\Vert , \end{aligned}$$

which completes the proof of the Lemma. $\square$

Proof of Lemma 2:

For any $W,D\in {\mathbb {R}}^{p\times {(m-1)}}$ and $\bar{t} \in [0,1]$, there exists $\varXi = W+\bar{t}(D-W)$ such that

$$\begin{aligned} \Vert \nabla \ell (W+\bar{t}D)-\nabla \ell (W)\Vert& = \Vert \bar{t}\nabla ^2 \ell (\varXi )\mathrm{vec}(D)\Vert \quad (\text {by mean-value theorem}) \nonumber \\&\overset{(30)}{\le }\bar{t}\Vert \frac{1}{2}[I_{m-1}-{\mathbf {1}}{\mathbf {1}}^{\top }/m]\otimes X^{\top }X\Vert _2\Vert D\Vert \nonumber \\&\le \frac{\bar{t}}{2}\Vert [I_{m-1}-{\mathbf {1}}{\mathbf {1}}^{\top }/m]\Vert _2\Vert X^{\top }X\Vert _2\Vert D\Vert \nonumber \\&\le \frac{\bar{t}}{2}\lambda _{\max }( X^{\top }X)\Vert D\Vert :=\bar{t}\lambda _{{\mathrm {x}}}\Vert D\Vert . \end{aligned}$$

(32)

Let t be a scalar parameter and $g(t)=\ell (W+tD)$. The chain rule yields

$$\begin{aligned} \frac{\partial g}{\partial t}(t)=\langle \nabla \ell (W+tD),D \rangle . \end{aligned}$$

(33)

Then, we have

$$\begin{aligned} \ell (W+tD)-\ell (W)&=g(1)-g(0)=\int _{0}^{1}\frac{\partial g}{\partial t}(t)dt \overset{(33)}{=} \int _{0}^{1}\langle \nabla \ell (W+tD),D \rangle dt\\&\le \int _{0}^{1}\langle \nabla \ell (W),D \rangle dt+\Big |\int _{0}^{1}\langle \nabla \ell (W+tD)-\nabla \ell (W),D \rangle dt\Big |\\&\le \langle \nabla \ell (W),D \rangle +\int _{0}^{1}\Vert \nabla \ell (W+tD)-\nabla \ell (W)\Vert \Vert D\Vert dt\\&\overset{(32)}{\le } \langle \nabla \ell (W),D \rangle + \Vert D\Vert \int _{0}^{1} \lambda _{{\mathrm {x}}}t\Vert D\Vert dt\\&=\langle \nabla \ell (W),D \rangle + \frac{\lambda _{{\mathrm {x}}}}{2}\Vert D\Vert ^{2}. \end{aligned}$$

Hence, we have completed the proof. $\square$