A sparsity preserving stochastic gradient methods for sparse regression

Abstract

We propose a new stochastic first-order algorithm for solving sparse regression problems. In each iteration, our algorithm utilizes a stochastic oracle of the subgradient of the objective function. Our algorithm is based on a stochastic version of the estimate sequence technique introduced by Nesterov (Introductory lectures on convex optimization: a basic course, Kluwer, Amsterdam, 2003). The convergence rate of our algorithm depends continuously on the noise level of the gradient. In particular, in the limiting case of noiseless gradient, the convergence rate of our algorithm is the same as that of optimal deterministic gradient algorithms. We also establish some large deviation properties of our algorithm. Unlike existing stochastic gradient methods with optimal convergence rates, our algorithm has the advantage of readily enforcing sparsity at all iterations, which is a critical property for applications of sparse regressions.

Appendix:  Some technical proofs

This section presents the proofs of Lemma 2, Lemma 4, and Proposition 1.

1.1 A.1 Proof of Lemma 2

According to the definition of $\widehat{x}$, there is a $\eta\in \partial h(\widehat{x})$ such that

$$ \bigl\langle G(\bar{x})+\bar{L}(\widehat{x}-\bar {x})+\eta,x- \widehat{x} \bigr\rangle \geq0\quad \text{for all}\ x\in\mathbb{R}^n. $$

(56)

By the convexity of h(x), we will have

$$ h(x)\geq h(\widehat{x})+ \langle\eta,x-\widehat{x} \rangle. $$

(57)

It then follow from (2), (56) and (57) that

$$\begin{aligned} &\phi(x)-\frac{\mu}{2}\|x-\bar{x}\|^2 \\ &\quad =f(x)-\frac{\mu}{2}\|x-\bar{x}\|^2+h(x) \\ &\quad \geq f(\bar{x})+ \bigl\langle \nabla f(\bar{x}),x-\bar{x} \bigr\rangle +h(\widehat{x})+ \langle\eta,x-\widehat{x} \rangle\quad \bigl(\text{by (2) and (57)}\bigr) \\ &\quad \geq f(\bar{x})+ \bigl\langle \nabla f(\bar{x}),x-\bar{x} \bigr\rangle +h(\widehat{x})- \bigl\langle G(\bar{x})+\bar{L}(\widehat {x}-\bar{x}),x- \widehat{x} \bigr\rangle \quad\bigl(\text{by (56)}\bigr) \\ &\quad \geq f(\widehat{x})-\frac{L}{2}\|\widehat{x}-\bar{x} \|^2-M\| \widehat{x}-\bar{x}\|- \bigl\langle \nabla f(\bar {x}),\widehat {x}-\bar{x} \bigr\rangle + \bigl\langle \nabla f(\bar{x}),x-\bar {x} \bigr\rangle +h(\widehat{x}) \\ &\qquad - \bigl\langle G(\bar{x},\xi)+\bar{L}(\widehat{x}-\bar{x}),x- \widehat{x} \bigr\rangle \quad\bigl(\text{by (2)}\bigr) \\ &\quad =\phi(\widehat{x})-\frac{L}{2}\|\widehat{x}-\bar{x} \|^2-M\| \widehat{x}-\bar{x}\|+ \langle \varDelta ,\widehat{x}-x \rangle- \bigl\langle \bar{L}(\widehat{x}-\bar{x}),x-\widehat{x} \bigr\rangle \\ &\qquad(\text{by the definition of $\varDelta $}) \\ &\quad =\phi(\widehat{x})-\frac{L}{2\bar{L}^2}\|g\|^2-M\| \widehat{x}-\bar{x}\|+ \langle \varDelta ,\widehat{x}-x \rangle+ \langle g,\bar{x}- \widehat{x} \rangle+ \langle g,x-\bar{x} \rangle\\ &\qquad(\text{by the definition of $g$}) \\ &\quad =\phi(\widehat{x})+\frac{2\bar{L}-L}{2\bar{L}^2}\|g\|^2+ \langle g,x-\bar{x} \rangle+ \langle \varDelta ,\widehat{x}-x \rangle-M\| \widehat{x}-\bar{x} \|. \end{aligned}$$

Notice that the inequalities above hold for any realization of stochastic gradient G(x,ξ). Hence, (16) holds almost surely.  □

1.2 A.2 Proof of Lemma 4

By the choice of V ₀(x), (20) holds for k=0 with $V_{0}^{*}=\phi(x_{0})$, ϵ ₀=0 and δ ₀=0. Suppose we have $V_{k}(x)=V_{k}^{*}+\frac{\gamma_{k}}{2}\|x-z_{k}\|^{2}+ \langle\epsilon _{k},x \rangle$. According to the updating equation (19), V _k+1(x) is also a quadratic function whose coefficient on ∥x∥² is $\frac{(1-\alpha_{k})\gamma_{k}+\alpha_{k}\mu}{2}$. Therefore, V _k+1(x) can definitely be represented as $V_{k+1}(x)=V_{k+1}^{*}+\frac{\gamma_{k+1}}{2}\|x-z_{k+1}\|^{2}+ \langle\epsilon_{k+1},x \rangle$ with γ _k+1=(1−α _k )γ _k +α _k μ and some $V_{k+1}^{*}$, z _k+1 and ϵ _k+1.

It follows from (19) that

$$\begin{aligned} V_{k+1}(x) &=(1-\alpha_k)V_k^*+ \frac{(1-\alpha_k)\gamma_k}{2}\|x-z_k\| ^2+(1-\alpha_k) \langle\epsilon_k,x \rangle \\ &\quad +\alpha_k \biggl[\phi(x_{k+1})+ \langle g_k,x-y_k \rangle+\frac{2L_k-L}{2L_k^2}\|g_k \|^2+\frac{\mu}{2}\|x-y_k\| ^2 \\ &\quad + \langle \varDelta _k,x_{k+1}-x \rangle-M\|x_{k+1}-y_k \| \biggr] \\ &=\underbrace{\frac{(1-\alpha_k)\gamma_k}{2}\|x-z_k\|^2+ \frac {\alpha_k\mu}{2}\|x-y_k\|^2+\alpha_k \langle g_k,x-y_k \rangle}_{T_1} \\ &\quad +(1- \alpha_k) \langle\epsilon_k,x \rangle- \alpha_k \langle \varDelta _k,x \rangle+(1-\alpha_k)V_k^* \\ &\quad +\alpha_k \biggl[\phi(x_{k+1})+\frac {2L_k-L}{2L_k^2}\|g_k \|^2+ \langle \varDelta _k,x_{k+1} \rangle-M \|x_{k+1}-y_k\| \biggr] \end{aligned}$$

(58)

We take the term $T_{1}=\frac{(1-\alpha_{k})\gamma_{k}}{2}\|x-z_{k}\| ^{2}+\frac{\alpha_{k}\mu}{2}\|x-y_{k}\|^{2}+\alpha_{k} \langle g_{k},x-y_{k} \rangle$ out from V _k+1(x) and consider it separately. We define γ _k+1 and z _k+1 according to (21) and (23), we get

$$\begin{aligned} T_1 &=\frac{(1-\alpha_k)\gamma_k}{2}\|x-z_k \|^2+\frac{\alpha_k\mu }{2}\|x-y_k\|^2+ \alpha_k \langle g_k,x-y_k \rangle \\ &=\frac{\gamma_{k+1}}{2}\|x\|^2+ \bigl\langle x,(1-\alpha_k) \gamma _kz_k+\alpha_k\mu y_k- \alpha_kg_k \bigr\rangle \\ &\quad +\frac{(1-\alpha_k)\gamma_k}{2} \|z_k\|^2+\frac{\alpha_k\mu}{2}\| y_k \|^2-\alpha_k \langle g_k,y_k \rangle \\ &=\frac{\gamma_{k+1}}{2}\|x-z_{k+1}\|^2-\frac{\gamma _{k+1}}{2}\biggl\| \frac{(1-\alpha_k)\gamma_kz_k+\alpha_k\mu y_k-\alpha_kg_k}{\gamma_{k+1}}\biggr\| ^2 \\ &\quad +\frac{(1-\alpha_k)\gamma_k}{2}\|z_k \|^2+\frac{\alpha_k\mu}{2}\| y_k\|^2- \alpha_k \langle g_k,y_k \rangle \\ &=\frac{\gamma_{k+1}}{2}\|x-z_{k+1}\|^2+\frac{\alpha_k(1-\alpha _k)\gamma_k\mu}{2\gamma_{k+1}} \|y_k-z_k\|^2-\frac{\alpha _k^2}{2\gamma_{k+1}} \|g_k\|^2 \\ &\quad +\frac{\alpha_k(1-\alpha_k)\gamma_k}{\gamma_{k+1}} \langle g_k, z_k-y_k \rangle. \end{aligned}$$

(59)

(The last step follows from a straightforward, though somewhat tedious algebraic expansion.) Therefore, replacing T ₁ in (58) with (59), we can rewrite V _k+1(x) as

$$\begin{aligned} V_{k+1}(x) =&\frac{\gamma_{k+1}}{2}\|x-z_{k+1}\|^2+ \underbrace{(1-\alpha_k) \langle\epsilon_k,x \rangle- \alpha_k \langle \varDelta _k,x \rangle}_{ \langle\epsilon_{k+1},x \rangle} \\ &{}+(1-\alpha_k)V_k^*+\alpha_k \phi(x_{k+1})+\biggl(\alpha_k\frac {2L_k-L}{2L_k^2}- \frac{\alpha_k^2}{2\gamma_{k+1}}\biggr)\|g_k\|^2 \\ &{}+\frac{\alpha_k(1-\alpha_k)\gamma_k}{\gamma_{k+1}}\biggl(\frac {\mu }{2}\|y_k-z_k \|^2+ \langle g_k,z_k-y_k \rangle \biggr) \\ &{}+\alpha_k \langle \varDelta _k,x_{k+1} \rangle-\alpha_kM\| x_{k+1}-y_k\| \end{aligned}$$

In order to represent V _k+1(x) by the formulation (20), it suffices to define ϵ _k+1 and $V_{k+1}^{*}$ as (22) and (24). □

1.3 A.3 Proof of Proposition 1

We prove (13) is true by induction. According to Lemma 4, V _k (x) is given by (20). Hence, we just need to prove $V_{k}^{*}\geq\phi(x_{k})-B_{k}$. Since $V_{0}^{*}=\phi(x_{0})$ and B ₀=0, (13) is true when k=0. Suppose (13) is true for k. We are going to show that it is also true for k+1.

Dropping the non-negative term $\frac{\mu}{2}\|y_{k}-z_{k}\|^{2}$ in (24) and applying the assumption $V_{k}^{*}\geq\phi(x_{k})-B_{k}$, we can bound $V_{k+1}^{*}$ from below as

$$\begin{aligned} V_{k+1}^* \geq&(1-\alpha_k)\phi(x_k)-(1- \alpha_k)B_k+\alpha_k\phi(x_{k+1})+ \biggl(\alpha_k\frac{2L_k-L}{2L_k^2}-\frac{\alpha_k^2}{2\gamma _{k+1}}\biggr) \|g_k\|^2 \\ & {}+\frac{\alpha_k(1-\alpha_k)\gamma_k}{\gamma_{k+1}} \langle g_k,z_k-y_k \rangle+\alpha_k \langle \varDelta _k,x_{k+1} \rangle-\alpha_kM\|x_{k+1}-y_k\|. \end{aligned}$$

(60)

According to Lemma 2 (with $\widehat{x} = x_{k+1}, \; \bar{x} = y_{k}$) and (17), we have

$$\begin{aligned} \phi(x_k) \geq& \phi(x_{k+1})+ \langle g_k,x_k-y_k \rangle+\frac {2L_k-L}{2L_k^2} \|g_k\|^2+\frac{\mu}{2}\|x_k-y_k \|^2 \\ &{}+ \langle \varDelta _k,x_{k+1}-x_k \rangle-M\|x_{k+1}-y_k\| \\ \geq& \phi(x_{k+1})+ \langle g_k,x_k-y_k \rangle+\frac {2L_k-L}{2L_k^2}\|g_k\|^2\\ &{}+ \langle \varDelta _k,x_{k+1}-x_k \rangle-M \|x_{k+1}-y_k\|. \end{aligned}$$

Applying this to ϕ(x _k ) in (60), it follows that

$$\begin{aligned} V_{k+1}^* \geq&\phi(x_{k+1})+ \biggl( \frac{2L_k-L}{2L_k^2}-\frac{\alpha _k^2}{2\gamma_{k+1}} \biggr)\|g_k \|^2-(1-\alpha_k)B_k\\ &{} +(1-\alpha_k) \biggl\langle g_k,\underbrace{\frac{\alpha_k\gamma _k}{\gamma_{k+1}}(z_k-y_k)+x_k-y_k}_{T_2} \biggr\rangle \\ &{}+ \bigl\langle \varDelta _k,x_{k+1}-(1- \alpha_k)x_k \bigr\rangle -M\| x_{k+1}-y_k \|. \end{aligned}$$

Due to (26), the term T ₂ above is zero. Hence, $V_{k+1}^{*}$ can be bounded from below as

$$\begin{aligned} V_{k+1}^* \geq&\phi(x_{k+1})+ \biggl(\frac{2L_k-L}{2L_k^2}- \frac{\alpha _k^2}{2\gamma_{k+1}} \biggr)\|g_k\|^2-(1- \alpha_k)B_k \\ &{} + \langle \varDelta _k,x_{k+1}-y_k \rangle-M \|x_{k+1}-y_k\| + \bigl\langle \varDelta _k,y_k-(1- \alpha_k)x_k \bigr\rangle \\ \geq&\phi(x_{k+1})+ \biggl(\frac{2L_k-L}{2L_k^2}-\frac{\alpha _k^2}{2\gamma_{k+1}} \biggr)\|g_k\|^2-(1-\alpha_k)B_k \\ &{} -(\|\varDelta _k\|+M)\|x_{k+1}-y_k\|+ \bigl\langle \varDelta _k,y_k-(1-\alpha_k)x_k \bigr\rangle \\ &{}\quad\text{(Cauchy-Schwarz and (26))} \\ \geq&\phi(x_{k+1})-\frac{(\|\varDelta _k\|+M)^2}{4L_k^2A_k}-(1-\alpha _k)B_k+ \bigl\langle \varDelta _k,y_k-(1- \alpha_k)x_k \bigr\rangle \\ =&\phi(x_{k+1})-B_{k+1}. \end{aligned}$$

Here, the last inequality is from the definition of A _k given by (28) and the following inequality

$$ A_k\|g_k\|^2-\|g_k\| \frac{\|\varDelta _k\|+M}{L_k}\geq-\frac{(\|\varDelta _k\|+M)^2}{4L_k^2A_k}. $$

Hence, (13) holds for k+1. □