Stochastic variance reduced gradient with hyper-gradient for non-convex large-scale learning

Abstract

Non-convex optimization, which can better capture the problem structure, has received considerable attention in the applications of machine learning, image/signal processing, statistics, etc. With faster convergence rate, there have been tremendous studies on developing stochastic variance reduced algorithms to solve these non-convex optimization problems. However, as a crucial hyper-parameter for stochastic variance reduced algorithms, that how to select an appropriate step size is less researched in solving non-convex optimization problems. To address this gap, we propose a new class of stochastic variance reduced algorithms based on hyper-gradient, which has the ability to automatically obtain the online step size. Specifically, we focus on the variance-reduced stochastic optimization algorithms, the stochastic variance reduced gradient (SVRG) algorithm, which computes a full gradient periodically. We analyze theoretically the convergence of the proposed algorithm for non-convex optimization problems. Moreover, we show that the proposed algorithm enjoys the same complexities as state-of-the-art algorithms for solving non-convex problems in terms of finding an approximate stationary point. Thorough numerical results on empirical risk minimization with non-convex loss functions validate the efficacy of our method.

Appendix A: Proofs for MSVRG-HD

1.1 A.1 Proof of Lemma 1

Proof

According to Algorithm 1 and F has a $\sigma$-bounded gradient, we have

$$\begin{aligned}&\mathbb {E}[\eta _k] \le |\eta _0|+\mathbb {E}\biggl [\beta \sum _{i=1}^{m-1} |\nabla F_{\hat{S}}(w_{i+1}^{s+1})\cdot \nabla F_{\hat{S}}(w_i^{s+1})|\biggr ] \\&\le |\eta _0|+\beta \sum _{i=1}^{m-1}\Vert \nabla F(w_{i+1}^{s+1})\Vert \Vert \nabla F(w_i^{s+1})\Vert \le \eta _0+m \beta \sigma ^2. \end{aligned}$$

Notice that the above inequality holds because we choose the sample i independently by adopting a uniformly randomly (with replacement) sample from [n]. In other word, the resulting algorithm (MSVRG-HD) utilizes an unbiased estimator of gradient per iterative step. Additionally, the evaluation of the step size $\eta _k$ in Algorithm 1 can utilize different batch samples uniformly randomly selecting from [n] for $\nabla F_{\hat{S}}(w_{i+1}^{s+1})$ and $\nabla F_{\hat{S}}(w_{i}^{s+1})$. Due to the tactic of uniformly randomly (with replacement), the inequality in the above can be held. $\square$

1.2 A.2 Proof of Lemma 2

Proof

From (3) and $w_{k+1}^{s+1}=w_{k}^{s+1}-\eta _{k+1} \phi _{k}^{s+1}$ in Algorithm 1, we have

$$\begin{aligned} \mathbb {E}[ F(w_{k+1}^{s+1})]\le & {} \mathbb {E}\biggl [ F(w_{k}^{s+1}) - \langle \nabla F(w_{k}^{s+1}), w_{k+1}^{s+1}-w_{k}^{s+1}\rangle \nonumber \\{} & {} + \frac{L}{2} \Vert w_{k+1}^{s+1}-w_{k}^{s+1}\Vert ^2 \biggr ] \nonumber \\= & {} \mathbb {E}\biggl [F(w_k^{s+1})-\eta _{k+1}\langle \nabla F(w_k)^{s+1}, \phi _{k}^{s+1}\rangle \rangle \nonumber \\ {}{} & {} +\frac{L}{2}\Vert \eta _{k+1}\phi _{k}^{s+1}\Vert ^2\biggr ] \nonumber \\= & {} \mathbb {E}\biggl [F(w_k^{s+1})+\frac{L\eta _{k+1}^2}{2}\Vert \phi _{k}^{s+1}\Vert ^2\biggr ]\nonumber \\{} & {} -\mathbb {E}\left[ \eta _{k+1}\right] \Vert \nabla F(w_k^{s+1})\Vert ^2, \end{aligned}$$

(14)

where the last equality holds due to $\mathbb {E}[\phi _{k}^{s+1}]=\nabla F(w_k^{s+1})$.

We emphasize here that the computation of the step size $\eta _k$ in (5) can use different samples uniformly randomly drawing from [n] for $\nabla f_i(w_{k-1})$ and $\nabla f_i(w_{k-1})$. As a consequence, because of uniformly randomly (with replacement) sample from [n], (14) can be satisfied, which is similar to the proof in Lemma 1.

Now we consider the Lyapunov function, i.e.,

$$\begin{aligned} R_k^{s+1}= \mathbb {E}[F(w_k^{s+1})+c_k\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]. \end{aligned}$$

In order to bound it, we provide the following

$$\begin{aligned}{} & {} \mathbb {E}[\Vert w_{k+1}^{s+1}-\widetilde{w}^s\Vert ^2] =\mathbb {E}[\Vert w_{k+1}^{s+1}-w_k^{s+1}+w_k^{s+1}-\widetilde{w}^s\Vert ^2] \nonumber \\= & {} \mathbb {E}[\Vert w_{k+1}^{s+1}-w_k^{s+1}\Vert ^2+\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2+2\langle w_{k+1}^{s+1}\nonumber \\{} & {} -w_k^{s+1}, w_k^{s+1}-\widetilde{w}^s\rangle ]\nonumber \\= & {} \mathbb {E}[\Vert w_{k+1}^{s+1}-w_k^{s+1}\Vert ^2+\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2\nonumber \\{} & {} +2\langle -\eta _{k+1}\nabla \phi _k^{s+1}, w_k^{s+1}-\widetilde{w}^s\rangle ]\nonumber \\= & {} \mathbb {E}[\eta _{k+1}^2\Vert \phi _k^{s+1}\Vert ^2+\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]\nonumber \\{} & {} -2\mathbb {E}[\eta _{k+1}]\langle \nabla F(w_k^{s+1}), w_{k}^{s+1}-\widetilde{w}^s\rangle \nonumber \\\le & {} \mathbb {E}[\eta _{k+1}^2\Vert \phi _k^{s+1}\Vert ^2+\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]+2\\{} & {} \cdot \mathbb {E}[\eta _{k+1}]\biggl (\frac{1}{2\alpha _k}\Vert \nabla F(w_k^{s+1})\Vert ^2\!+\frac{\alpha _k}{2}\mathbb {E}\left[ \Vert w_k^{s+1}\!-\widetilde{w}^s\Vert ^2\right] \biggr ), \nonumber \end{aligned}$$

(15)

where in the third equality we used $w_{k+1}^{s+1}=w_{k}^{s+1}-\eta _{k+1} \phi _{k}^{s+1}$, in the fourth equality we used $\mathbb {E}[\phi _{k}^{s+1}]=\nabla F(w_k^{s+1})$ and in the last inequality we used Cauchy-Schwarz and Young’s inequality. Here, we also used the hypothesis that the samples were chosen from $\{1, \ldots ,n\}$ with replacement independently.

Taking (14) and (15) into $R_{k+1}^{s+1}$, we have the following boundary:

$$\begin{aligned} R_{k+1}^{s+1}\le & {} \mathbb {E}\biggl [F(w_k^{s+1})-\eta _{k+1}\Vert \nabla F(w_k^{s+1})\Vert ^2+\frac{L\eta _{k+1}^2}{2}\Vert \phi _{k}^{s+1}\Vert ^2\biggr ] \\{} & {} +\mathbb {E}[c_{k+1}\eta _{k+1}^2\Vert \phi _k^{s+1}\Vert ^2+c_{k+1}\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]+2c_{k+1}\\{} & {} \mathbb {E}[\eta _{k+1}]\biggl (\frac{1}{2\alpha _k}\Vert \nabla F(w_k^{s+1})\Vert ^2\!+\frac{\alpha _k}{2}\mathbb {E}\left[ \Vert w_k^{s+1}\!-\widetilde{w}^s\Vert ^2\right] \biggr )\\\le & {} \mathbb {E}\left[ F(w_k^{s+1})\right] -\mathbb {E}\biggl [\eta _{k+1}-\frac{c_{k+1}\eta _{k+1}}{\alpha _k}\biggr ]\Vert \nabla F(w_k^{s+1})\Vert ^2\\{} & {} +\mathbb {E}\biggl [\biggl (\frac{L\eta _{k+1}^2}{2}+c_{k+1}\eta _{k+1}^2\biggr )\Vert \phi _{k}^{s+1}\Vert ^2\biggr ]\\{} & {} +\mathbb {E}[c_{k+1}+c_{k+1}\eta _{k+1}\alpha _k]\mathbb {E}[\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]. \end{aligned}$$

According to the upper boundary of $\phi _{k}^{s+1}$, i.e., Lemma 3, we ascertain that

$$\begin{aligned} R_{k+1}^{s+1}\le & {} \mathbb {E}[F(w_k^{s+1})]-\mathbb {E}\biggl [\eta _{k+1}-\frac{c_{k+1}\eta _{k+1}}{\alpha _k}\biggr ]\Vert \nabla F(w_k^{s+1})\Vert ^2\\{} & {} +\mathbb {E}\biggl [\frac{L\eta _{k+1}^2}{2}+c_{k+1}\eta _{k+1}^2\biggr ]\biggl ( 2\mathbb {E}[\Vert \nabla F(w_k^{s+1})\Vert ^2]+\frac{2L^2}{b}\\{} & {} \mathbb {E}[\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]\biggr )+\mathbb {E}[c_{k+1}+c_{k+1}\eta _{k+1}\alpha _k]\mathbb {E}[\Vert w_k^{s+1}\nonumber \\{} & {} -\widetilde{w}^s\Vert ^2]\\= & {} \mathbb {E}[F(w_k^{s+1})]-\mathbb {E}\biggl [\eta _{k+1}-\frac{c_{k+1}\eta _{k+1}}{\alpha _k}-L\eta _{k+1}^2\\{} & {} -2c_{k+1}\eta _{k+1}^2\biggr ]\Vert \nabla F(w_k^{s+1})\Vert ^2+\mathbb {E}\biggl [c_{k+1}+c_{k+1}\eta _{k+1}\alpha _k\\{} & {} +\frac{L^3\eta _{k+1}^2}{b}+\frac{2L^2c_{k+1}\eta _{k+1}^2}{b}\biggr ]\mathbb {E}[\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]. \end{aligned}$$

Further, utilizing Lemma 1, we have

$$\begin{aligned} R_{k+1}^{s+1}\le & {} \mathbb {E}[F(w_k^{s+1})]-\biggl (Q_m-\frac{c_{k+1}Q_m}{\alpha _k}-LQ_m^2\\{} & {} -2c_{k+1}Q_m^2\biggr )\cdot \Vert \nabla F(w_k^{s+1})\Vert ^2\\{} & {} +\biggl (c_{k+1}+c_{k+1}Q_m\alpha _k+\frac{L^3Q_m^2}{b}+\frac{2L^2c_{k+1}Q_m^2}{b}\biggr )\\{} & {} \cdot \mathbb {E}[\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]\\= & {} \mathbb {E}[F(w_k^{s+1})]+c_k\mathbb {E}[\Vert w_k^{s+1}-\widetilde{w}^s\Vert ^2]\\{} & {} -\biggl (Q_m-\frac{c_{k+1}Q_m}{\alpha _k}-LQ_m^2-2c_{k+1}Q_m^2\biggr )\\{} & {} \times \Vert \nabla F(w_k^{s+1})\Vert ^2, \end{aligned}$$

where the first equality holds due to Lemma 2.

Thereby, we complete the proof of Lemma 2.

Notice that to make ${\varGamma }_{k, m}>0$, we only require $2c_{k+1}Q_m+LQ_m+\frac{c_{k+1}}{\alpha _k}<1$. When choosing $c_k$, $Q_m$ and $\alpha _k$ from (0, 1), such condition is easy to be satisfied. From the definition of $Q_m$, we can choose enough small parameters, $\eta _0$ and $\beta$, making $Q_m$ small enough. Also, from the definition of $c_k$, we can make $c_k$ small enough. Specifically, as an example, when setting $c_{k+1}\ll \alpha _k$ at the same time, we have the conclusion, $2c_{k+1}Q_m+LQ_m+\frac{c_{k+1}}{\alpha _k}\ll 1$. Therefore, we ascertain that the condition ${\varGamma }_{k, m}>0$ is satisfied when choosing the appropriate parameters, $c_k$, $Q_m$ and $\alpha _k$. $\square$

1.3 A.3 Proof of Theorem 1

Proof

Combining Lemma 2 and $\gamma _m=\min _k {\varGamma }_{k, m}$, by summing over $k=0, \ldots , m-1$, we have

$$\begin{aligned} \sum _{k=0}^{m-1} \mathbb {E}\left[ \Vert \nabla F(w_k^{s+1})\Vert ^2\right] \le \frac{R_0^{s+1}-R_m^{s+1}}{\gamma _m} \end{aligned}$$

Above mentioned inequality indicates that

$$\begin{aligned} \sum _{k=0}^{m-1} \mathbb {E}\left[ \Vert \nabla F(w_k^{s+1})\Vert ^2\right] \le \frac{\mathbb {E}[F(\widetilde{w}^s)-F(\widetilde{w}^{s+1})]}{\gamma _m}, \end{aligned}$$

(16)

where we took the fact that $R_m^{s+1}=\mathbb {E}[F(w_m^{s+1})]=\mathbb {E}[F(\widetilde{w}^{s+1})]$ (since $c_m=0$) and that $R_0^{s+1}=\mathbb {E}[f(\widetilde{w}^s)]$ (since $w_0^{s+1}=\widetilde{w}^s$).

By summing over all epochs, we have

$$\begin{aligned} \frac{1}{T} \sum _{s=0}^{S-1}\sum _{k=0}^{m-1} \mathbb {E}\left[ \Vert \nabla F(w_k^{s+1})\Vert ^2\right] \le \frac{F(w^0)-F(w^s)}{T\gamma _m}. \end{aligned}$$

(17)

The above inequality used the fact that $\widetilde{w}^0=w^0$. Thus, we complete the proof of Theorem 1. Note that, although the output of our algorithm is the last iteration, $w_{m}^{s+1}$, of the inner loop, rather than sampling randomly from the set $\{w_k^s\}$ for $s=0, \ldots ,S-1$ and $k=0, \ldots ,m-1$, the quantitative relationship in (17) still can be used in our proof. This is indeed the case. Many studies had pointed out that these two ways in determining the last iterate achieved almost similar numerical performance in many problems.

Further, considering Assumption 1, i.e., $\Vert \nabla f_i(w)-\nabla f_i(v)\Vert \le \Vert w-v\Vert$, we have

$$\begin{aligned} \Vert \nabla F(\tilde{w}^S)-\nabla F(w^*)\Vert ^2\le L^2\Vert \tilde{w}^S-w^*\Vert ^2. \end{aligned}$$

To satisfy the result in Theorem 1, $\mathbb {E}[\Vert \nabla F(\widetilde{w}^{S})\Vert ^2]\le \frac{F(w^{0})-F(w^*)}{T\gamma _m}$, it is enough to hold the following condition

$$\begin{aligned} \mathbb {E}[L^2\Vert \tilde{w}^S-w^*\Vert ^2]\le \frac{F(w^{0})-F(w^*)}{T\gamma _m}. \end{aligned}$$

Therefore, the conclusion in Theorem 1 can be rewritten as

$$\begin{aligned} \mathbb {E}[\Vert \tilde{w}^S-w^*\Vert ^2]\le \frac{F(w^{0})-F(w^*)}{L^2T\gamma _m}. \end{aligned}$$

$\square$