Adaptive step size rules for stochastic optimization in large-scale learning

Abstract

The importance of the step size in stochastic optimization has been confirmed both theoretically and empirically during the past few decades and reconsidered in recent years, especially for large-scale learning. Different rules of selecting the step size have been discussed since the arising of stochastic approximation methods. The first part of this work reviews the studies on several representative techniques of setting the step size, covering heuristic rules, meta-learning procedure, adaptive step size technique and line search technique. The second component of this work proposes a novel class of accelerating stochastic optimization methods by resorting to the Barzilai–Borwein (BB) technique with a diagonal selection rule for the metric, particularly, termed as DBB. We first explore the theoretical and empirical properties of variance reduced stochastic optimization algorithms with DBB. Especially, we study the theoretical and numerical properties of the resulting method under strongly convex and non-convex cases respectively. To great show the efficacy of the step size schedule of DBB, we extend it into more general stochastic optimization methods. The theoretical and empirical properties of such the case also developed under different cases. Extensive numerical results in machine learning are offered, suggesting that the proposed algorithms show much promise.

Appendices

Appendix A Proofs for mS2GD-DBB

1.1 A.1 Proof of Lemma 2

For clarity, we provide a brief proof for the Lemma 2.

Proof

From (17), i.e., $\widetilde{\eta }_k^{SBB2}=\frac{\theta }{m}\cdot \frac{\langle s_{k}, y_{k}\rangle }{\Vert y_k\Vert ^2}$, we have

$$\begin{aligned} \widetilde{\eta }_k^{SBB2}\ge & {} \frac{\theta }{m}\cdot \frac{1}{L} \frac{\Vert y_k\Vert ^2}{\Vert y_k\Vert ^2}=\frac{\theta }{mL}, \end{aligned}$$

where in the first inequality we took Lemma 1 (a.k.a. (24)).

In addition, from (16), i.e., $\widetilde{\eta }_k^{SBB1}=\frac{\theta }{m}\cdot \frac{\Vert s_{k}\Vert ^{2}}{\langle s_{k}, y_{k}\rangle }$, we have

$$\begin{aligned}{} & {} \widetilde{\eta }_k^{SBB1} \le \frac{\theta }{m}\cdot \frac{\Vert s_k\Vert ^2}{\mu \Vert s_k\Vert ^2}=\frac{\theta }{\mu m}, \end{aligned}$$

where in the first inequality we took Assumption 2.

Further, using the Cauchy–Schwartz inequality, we have the result that $\widetilde{\eta }_k^{SBB1} > \widetilde{\eta }_k^{SBB2}$.

Here, we finished the proof of Lemma 2. $\square$

1.2 A.2 Proof of Theorem 1

Proof

According to the definition of $w_{k+1}$, i.e., $w_{k+1}=w_k-(U_s)^{-1} v_k$, in Algorithm 1, we obtain

$$\begin{aligned}{} & {} \mathbb {E}[\Vert w_{k+1}-w_*\Vert ^2] = \mathbb {E}[\Vert w_k-(U_s)^{-1}v_k-w_*\Vert ^2] \\{} & {} \quad = \Vert w_k-w_*\Vert ^2-2\langle w_k-w_*, (U_s)^{-1}\nabla F(w_k)\rangle \\{} & {} \qquad +\,\Vert (U_s)^{-1}v_k\Vert ^2\\{} & {} \quad =\Vert w_k-w_*\Vert ^2-2\langle W, Y\rangle \bar{U}+\Vert (U_s)^{-1}v_k\Vert ^2 \end{aligned}$$

where the second equality holds due to $\mathbb {E}[v_k]=\nabla F(w_k)$. In the last equality, we set $W=[(w_k-w_*)^1, \ldots , (w_k-w_*)^d]^T$, and $\bar{U}=[(U_s)^{-1}_{1, 1}, \ldots , (U_s)^{-1}_{d, d}]^T$.

Utilizing the Cauchy–Schwartz inequality, we have

$$\begin{aligned} \mathbb {E}[\Vert w_{k+1}-w_*\Vert ^2]\le & {} \Vert w_k-w_*\Vert ^2-2\langle W, Y\rangle \bar{U}\\{} & {} +\,\Vert (U_s)^{-1}\Vert ^2\Vert v_k\Vert ^2 \\\le & {} \Vert w_k-w_*\Vert ^2-2\langle W, Y\rangle \bar{U}\\{} & {} +\,\Vert (U_s)^{-1}\Vert ^2\biggl [\frac{4L}{b} \bigl [ F(w_{k})\\{} & {} -F(w_{*})+F(\widetilde{w})-F(w_*)]\\ {}{} & {} \quad +\frac{2}{b}\Vert \nabla F(w_{k-1})\Vert ^2\biggr ], \end{aligned}$$

where the last inequality holds due to Lemma 3.

Further, combining the convexity of the objective function, F(w), that is $F(v)\ge F(w)+\langle \nabla F(w), v-w\rangle$, and the result (23), shown in Lemma 1, we have

$$\begin{aligned}{} & {} \mathbb {E}[\Vert w_{k+1}-w_*\Vert ^2]\nonumber \\ {}{} & {} \quad \le \Vert w_k-w_*\Vert ^2-2\langle Z, \bar{U}\rangle \nonumber \\{} & {} \qquad +\,\Vert (U_s)^{-1}\Vert ^2\biggl [\frac{4L}{b} \bigl [ F(w_{k})-F(w_{*})+F(\widetilde{w})\nonumber \\{} & {} \qquad -\,F(w_*)\biggr ]+\frac{4L\Vert (U_s)^{-1}\Vert ^2}{b}[F(w_k)-F(w_*)]\nonumber \\{} & {} \quad \le \Vert w_k-w_*\Vert ^2-2\langle Z, \bar{U}\rangle \nonumber \\{} & {} \qquad +\,\frac{4\theta ^2Ld}{m^2\mu ^2b}[ F(w_{k})-F(w_{*})\nonumber \\{} & {} \qquad +\,F(\widetilde{w})-F(w_*)]\nonumber \\{} & {} \qquad +\,\frac{4\theta ^2Ld}{m^2\mu ^2b}[F(w_k)-F(w_*)], \end{aligned}$$

(A.1)

where we set $Z=[F(w_k)-F(w_*), \ldots , F(w_k)-F(w_*)]^T$ $\in \mathbb {R}^d$, and in the last inequality, we use the fact $\frac{\theta }{mL}\textrm{I}\le U^{-1} \le \frac{\theta }{m\mu }\textrm{I}$.

To hold the inequality (A.1), we only required that the following condition was satisfied:

$$\begin{aligned} \mathbb {E}[\Vert w_{k+1}-w_*\Vert ^2]\le & {} \Vert w_k-w_*\Vert ^2-\frac{2d\theta }{m\mu }[F(w_k)\\{} & {} -\,F(w_*)]+\frac{4\theta ^2Ld}{m^2\mu ^2b}[ F(w_{k})\\{} & {} -F(w_{*})+F(\widetilde{w})-F(w_*)]\\{} & {} +\,\frac{4\theta ^2Ld}{m^2\mu ^2b}[F(w_k)-F(w_*)]\\= & {} \Vert w_k-w_*\Vert ^2-\biggl (\frac{2d\theta }{m\mu }\\{} & {} -\,\frac{8\theta ^2Ld}{m^2\mu ^2b}\biggr )[F(w_k)-F(w_*)]\\{} & {} +\,\frac{4\theta ^2Ld}{m^2\mu ^2b}[F(\widetilde{w})-F(w_*)] \end{aligned}$$

In addition, according to the definition of $\widetilde{w}_s$ in Algorithm 1, we have

$$\begin{aligned}{} & {} \mathbb {E}[F(\widetilde{w}_s)]=\frac{1}{m} \sum _{k=0}^{m-1}\mathbb {E}[F(w_{k+1})]. \end{aligned}$$

Through summing the previous inequality over $k=0, \ldots , m-1$, using expectation with all the history, and taking $\widetilde{w}_{s}=w_{m}$ in Algorithm 1, we obtain

$$\begin{aligned}{} & {} \mathbb {E}[\Vert w_m-w_*\Vert ^2] + \biggl (\frac{2d\theta }{\mu }-\frac{8\theta ^2Ld}{\mu ^2mb}\biggr )\mathbb {E}[F(\widetilde{w}_s)-F(w_*)]\\{} & {} \quad \le \mathbb {E}[\Vert w_0-w_*\Vert ^2]+\frac{4\theta ^2Ld}{\mu ^2mb}[F(\widetilde{w})\\{} & {} \qquad -\,F(w_*)]\\{} & {} \quad = \mathbb {E}[\Vert \widetilde{w}-w_*\Vert ^2]+\frac{4\theta ^2Ld}{\mu ^2mb}\\{} & {} \qquad [F(\widetilde{w})-F(w_*)]\\{} & {} \quad \le \frac{2}{\mu }[F(\widetilde{w})-F(w_*)]\\{} & {} \qquad +\,\frac{4\theta ^2Ld}{\mu ^2mb}[F(\widetilde{w})-F(w_*)]\\{} & {} \quad = \biggl (\frac{2}{\mu }+\frac{4\theta ^2Ld}{\mu ^2mb}\biggr )\\{} & {} \qquad [F(\widetilde{w})-F(w_*)] \end{aligned}$$

where in the last inequality we used the inequality (22), appearing in Assumption 2.

Further, we have

$$\begin{aligned}{} & {} \mathbb {E}[F(\widetilde{w}_s)-F(w_*)] \\{} & {} \quad \le \biggl (\frac{\mu mb+2\theta ^2Ld}{d\mu \theta mb-4\theta ^2Ld}\biggr )\mathbb {E}[F(\widetilde{w}_{s-1})-F(w_*)] \end{aligned}$$

When setting $\rho =\frac{\mu mb+2\theta ^2Ld}{d\mu \theta mb-4\theta ^2Ld}$, we had the desired bound. $\square$

1.3 A.3 Proof of Theorem 2

Proof

Combining $w_{k+1}=w_k-(U_s)^{-1} v_k$ and Lemma 4, we reached to the following conclusion

$$\begin{aligned} F(w_{k+1})\le & {} F(w_k)+\langle \nabla F(w_k), w_{k+1}-w_{k}\rangle \\{} & {} +\,\frac{L}{2}\Vert w_{k+1}-w_{k}\Vert ^2\\= & {} F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}v_k\rangle \\{} & {} +\,\frac{L}{2}\Vert (U_s)^{-1}v_k\Vert ^2\\\le & {} F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}v_k\rangle \\{} & {} +\,\frac{L}{2}\Vert (U_s)^{-1}\Vert ^2\Vert v_k\Vert ^2\\= & {} F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}v_k\rangle \\{} & {} +\,\frac{L}{2}\Vert (U_s)^{-1}\Vert ^2\Vert \nabla F_S(w_k)\\{} & {} -\,\nabla F(w_k)+\nabla F(w_k)\\{} & {} -\,\nabla F_S(\tilde{w})+\nabla F(\tilde{w})\Vert ^2 \end{aligned}$$

where the second inequality used the Cauchy–Schwartz inequality and the second equality used the definition of $v_k=\nabla F_S(w_k)-\nabla F_S(\tilde{w})+\nabla F(\tilde{w})$.

Additionally, according to the fact $(a_1+a_2+ \cdots +a_n)^2\le n(a_1^2+a_2^2+ \cdots + a_n^2)$, we have

$$\begin{aligned} \mathbb {E}[F(w_{k+1})]\le & {} \mathbb {E}\biggl [F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}v_k\rangle \nonumber \\{} & {} +\,\frac{3L}{2}\Vert (U_s)^{-1}\Vert ^2[\Vert \nabla F_S(w_k)\nonumber \\{} & {} -\,\nabla F(w_k)\Vert ^2+\Vert \nabla F(w_k)\Vert ^2+\Vert \nabla F_S(\tilde{w})\nonumber \\{} & {} -\,\nabla F(\tilde{w})\Vert ^2]\biggr ]\nonumber \\\le & {} F(w_k)-\frac{\theta d}{m\zeta }\Vert \nabla F(w_k)\Vert ^2\nonumber \\{} & {} +\,\frac{3L}{2}\cdot \frac{d\theta ^2}{m^2\zeta ^2}\cdot \frac{\sigma ^2}{b}\nonumber \\{} & {} +\,\frac{3L}{2}\cdot \frac{d\theta ^2}{m^2\zeta ^2}\Vert \nabla F(w_k)\Vert ^2\nonumber \\{} & {} +\,\frac{3L}{2}\cdot \frac{d\theta ^2}{m^2\zeta ^2}\cdot \frac{\sigma ^2}{b}\nonumber \\= & {} F(w_k)-\biggl (\frac{\theta d}{m\zeta }\frac{3Ld\theta ^2}{2m^2\zeta ^2}\biggr )\Vert \nabla F(w_k)\Vert ^2 \nonumber \\{} & {} +\frac{3Ld\theta ^2\sigma ^2}{bm^2\zeta ^2}, \end{aligned}$$

(A.2)

where the first inequality can be hold due to , and Assumption 3.

Via summing the inequality (A.2), we easily arrive at

$$\begin{aligned} \mathbb {E}[F(w_m)]\le & {} F(w_0)-\biggl (\frac{\theta d}{m\zeta }-\frac{3Ld\theta ^2}{2m^2\zeta ^2}\biggr )\\{} & {} \sum _{k=0}^{n}\Vert \nabla F(w_k)\Vert ^2+\frac{3Ld\theta ^2\sigma ^2}{bm\zeta ^2}. \end{aligned}$$

Further, we obtain

$$\begin{aligned}{} & {} \sum _{k=0}^{n}\Vert \nabla F(w_k)\Vert ^2\le \frac{2m^2\zeta ^2}{2m\zeta \theta d-3Ld\theta ^2}\\{} & {} \quad \mathbb {E}[F(w_0)-F(w_{m})]+\frac{6mL\theta \sigma ^2}{b(2m\zeta -3L\theta )}. \end{aligned}$$

Since $\mathbb {E}[\Vert \nabla F(w_m)\Vert ^2]=\frac{1}{m}\sum _{k=0}^{m-1}\Vert \nabla F(w_k)\Vert$, we induced

$$\begin{aligned}{} & {} \mathbb {E}[\Vert \nabla F(w_m)\Vert ^2]\nonumber \\{} & {} \quad \le \frac{2m\zeta ^2}{2m\zeta \theta d-3Ld\theta ^2}\mathbb {E}[F(w_0)-F(w_*)]\nonumber \\{} & {} \quad +\frac{6mL\theta \sigma ^2}{b(2m\zeta -3L\theta )}, \end{aligned}$$

(A.3)

where this inequality used the condition $w_*=\mathop {\mathrm {arg\,min}} F(w)$.

Note that, according to the inequality (4)

$$\begin{aligned} F(w)\le F(v)+ \langle \nabla F(v), w-v \rangle +\frac{L}{2}\parallel w-v\parallel ^2, \end{aligned}$$

we observed that the right hand of the above inequality defined a quadratic function, where it reached to a minimal value at the point $\bar{w}=v-\frac{\nabla f(v)}{L}$.

Practically, to satisfy the above inequality, it was an adequate to satisfy the following condition:

$$\begin{aligned} F(w)\le & {} F(v)+\langle \nabla F(v), \bar{w}-v \rangle \\{} & {} +\,\frac{L}{2}\Vert \bar{w}-v\Vert ^2\\= & {} F(v)-\frac{1}{2L}\Vert \nabla F(v)\Vert ^2. \end{aligned}$$

When taking $w=w_*$ and $v=w_0$, according to above inequality, we obtained the following result

$$\begin{aligned} \frac{1}{2L}\Vert \nabla F(w_0)\Vert ^2\le F(w_0)-F(w_*). \end{aligned}$$

(A.4)

Therefore, to hold the inequality (A.3), we only required that the following conclusion was satisfied:

$$\begin{aligned} \mathbb {E}[\Vert \nabla F(w_m)\Vert ^2]\le & {} \frac{m\zeta ^2}{2Lm\zeta \theta d-3L^2d\theta ^2}\\{} & {} \mathbb {E}[\Vert \nabla F(w_0)\Vert ^2]+\frac{6mL\theta \sigma ^2}{b(2m\zeta -3L\theta )}. \end{aligned}$$

According to the definition $\tilde{w}_{s+1}=w_m$ and $w_0=\tilde{w}_s$, denoted in Algorithm 1, we obtained

$$\begin{aligned}{} & {} \mathbb {E}[\Vert \nabla F(\tilde{w}_{s+1})\Vert ^2]\\{} & {} \quad \le \frac{m\zeta ^2}{2Lm\zeta \theta d-3L^2d\theta ^2}\mathbb {E}[\Vert \nabla F(\tilde{w}_s)\Vert ^2]+\frac{6mL\theta \sigma ^2}{b(2m\zeta -3L\theta )}. \end{aligned}$$

Resorting summing the above inequality over $s=0, 1, \ldots , \hat{S}-1$, we obtained the results.

$\square$

Appendix B Proofs for SGD-DBB

1.1 B.1 Proof of Theorem 3

Proof

According to $w_{k+1}=w_k-(U_s)^{-1}\nabla F_S(w_k)$, defined in Algorithm 2, we deduced

$$\begin{aligned}{} & {} \mathbb {E}[\Vert w_{k+1}-w_*\Vert ^2] \mathbb {E}[\Vert w_{k}-(U_s)^{-1}\nonumber \\{} & {} \quad \nabla F_S(w_k)-w_{*}\Vert ^2] \nonumber \\{} & {} \quad =\mathbb {E}[\Vert w_k-w_*\Vert ^2-2\langle w_k-w_*, (U_s)^{-1}\nabla F_S(w_k)\rangle \nonumber \\{} & {} \qquad +\,\Vert (U_s)^{-1}\nabla F_S(w_k)\Vert ^2]\nonumber \\{} & {} \quad \le \mathbb {E}[\Vert w_k-w_*\Vert ^2]-2\langle w_k-w_*, (U_s)^{-1}\nonumber \\{} & {} \quad \nabla F(w_k)\rangle \nonumber \\{} & {} \qquad +\,\mathbb {E}[\Vert (U_s)^{-1}\Vert ^2\Vert \nabla F_S(w_k)\Vert ^2]\nonumber \\{} & {} \quad \le \mathbb {E}[\Vert w_k-w_*\Vert ^2]-2\langle w_k-w_*, (U_s)^{-1}\nonumber \\{} & {} \quad \nabla F(w_k)\rangle \nonumber \\{} & {} \qquad +\,2L\Vert (U_s)^{-1}\Vert ^2[F(w_k)\nonumber \\{} & {} \qquad -\,F(w_*)], \end{aligned}$$

(B.1)

where the first inequality holds due to $\mathbb {E}[\nabla F_S(w_k)]=\nabla F(w_k)$ and Cauchy–Schwartz inequality. The second inequality used the conclusion (23), shown in Lemma 1.

To satisfy the inequality (B.1), we only required

$$\begin{aligned} \mathbb {E}[\Vert w_{k+1}-w_*\Vert ^2]\le & {} \mathbb {E}[\Vert w_k-w_*\Vert ^2]\\{} & {} -\,\frac{2\theta d}{m\mu }[F(w_k)-F(w_*)]+\frac{2Ld\theta ^2}{m^2\mu ^2}\\{} & {} [F(w_k)-F(w_*)]\\= & {} \mathbb {E}[\Vert w_k-w_*\Vert ^2]-\biggl (\frac{2\theta d}{mu}\\ {}{} & {} \quad -\frac{2Ld \theta ^2}{m^2\mu ^2} \biggr )[F(w_k)-F(w_*)], \end{aligned}$$

where in the first inequality, we used the convexity property of the objective function, i.e., $F(v)\ge F(w)+\nabla F(w)(v-w)$, $w_*=\mathop {\mathrm {arg\,min}}\nolimits _{w\in \mathbb {R}^d} F(w)$ and $U^{-1}\le \frac{\theta }{mu}\textrm{I}$.

By summing the above inequality over $k=0, 1, \ldots , m-1$, we obtain

$$\begin{aligned}{} & {} \mathbb {E}[\Vert \tilde{w}_{s+1}-w_*\Vert ^2] \le \mathbb {E}[\Vert w_{0}-w_*\Vert ^2]\\{} & {} \quad -\,\biggl (\frac{2\theta d}{mu}-\frac{2Ld \theta ^2}{m^2\mu ^2} \biggr )\sum _{k=0}^{m-1}[F(w_k)-F(w_*)], \end{aligned}$$

where this inequality used $\tilde{w}_{s+1}=w_m$, denoted in Algorithm 2.

Since $\mathbb {E}[F(\tilde{w}_s)]=\frac{1}{m}\sum _{k=0}^{m-1}\mathbb {E}[F(w_k)]$ and take expectation on both sides, we ascertained

$$\begin{aligned}{} & {} \mathbb {E}[\Vert \tilde{w}_{s+1}-w_*\Vert ^2]+ \biggl (\frac{2\theta d}{u}-\frac{2Ld \theta ^2}{m\mu ^2} \biggr )\\{} & {} \quad \mathbb {E}[F(\tilde{w}_s)-F(w_*)] \le \mathbb {E}[\Vert w_{0}-w_*\Vert ^2]. \end{aligned}$$

Further, we have

$$\begin{aligned} \mathbb {E}[F(\tilde{w}_{s+1})-F(w_*)]\le & {} \frac{m\mu ^2}{2m\mu d\theta -2Ld\theta ^2}\\{} & {} \mathbb {E}[\Vert w_{0}-w_*\Vert ^2]\\= & {} \frac{m\mu ^2}{2m\mu d\theta -2Ld\theta ^2}\\{} & {} \mathbb {E}[\Vert \tilde{w}_{s}-w_*\Vert ^2]\\\le & {} \frac{m\mu }{md\mu \theta -Ld\theta ^2}\mathbb {E}[F(\tilde{w}_s)-F(w_*)], \end{aligned}$$

where the first inequality holds due to the definition $w_0=\tilde{w}_s$ and the second inequality holds because of Assumption 2 (a.k.a. (22)).

Finally, summing the above inequality, we obtain

$$\begin{aligned} \mathbb {E}[F(\tilde{w}_{s})-F(w_*)]\le \tau ^{s} \mathbb {E}[F(\tilde{w}_0)-F(w_*)], \end{aligned}$$

where we set $\tau =\frac{m\mu }{md\mu \theta -Ld\theta ^2}$. $\square$

1.2 B.2 Proof of Theorem 4

Proof

Combining the fact $w_{k+1}=w_k-(U_s)^{-1}\nabla F_S(w_k)$ and Lemma 4, we have

$$\begin{aligned} F(w_{k+1})\le & {} F(w_k)+\langle \nabla F(w_k), w_{k+1}-w_{k}\rangle \nonumber \\{} & {} +\,\frac{L}{2}\Vert w_{k+1}-w_{k}\Vert ^2 \nonumber \\= & {} F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}\nabla F_S(w_k)\rangle \nonumber \\{} & {} +\,\frac{L}{2}\Vert (U_s)^{-1}\nabla F_S(w_k)\Vert ^2\nonumber \\\le & {} F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}\nabla F_S(w_k)\rangle \nonumber \\{} & {} +\,\frac{L}{2}\Vert (U_s)^{-1}\Vert ^2\Vert \nabla F_S(w_k)\Vert ^2\nonumber \\= & {} F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}\nabla F_S(w_k)\rangle \nonumber \\{} & {} +\,\frac{L}{2}\Vert (U_s)^{-1}\Vert ^2\Vert \nabla F_S(w_k)\nonumber \\{} & {} -\,\nabla F(w_k)+\nabla F(w_k)\Vert ^2\nonumber \\\le & {} F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}\nabla F_S(w_k)\rangle \nonumber \\{} & {} +\,L\Vert (U_s)^{-1}\Vert ^2[\Vert \nabla F_S(w_k)\nonumber \\{} & {} -\,\nabla F(w_k)\Vert ^2+\Vert \nabla F(w_k)\Vert ^2], \end{aligned}$$

(B.2)

where the second inequality held due to the Cauchy–Schwartz inequality and the last inequality held because of $\Vert a+b\Vert ^2\le 2(\Vert a\Vert ^2+\Vert b\Vert ^2)$.

In virtue of Assumption 3 and taking expectation on both sides of (B.2), we obtained

$$\begin{aligned} \mathbb {E}[F(w_{k+1})]\le & {} \mathbb {E}[ F(w_k)-\langle \nabla F(w_k), (U_s)^{-1}\nonumber \\{} & {} \nabla F_S(w_k)\rangle +L\Vert (U_s)^{-1}\Vert ^2[\Vert \nabla F_S(w_k)\nonumber \\{} & {} -\,\nabla F(w_k)\Vert ^2+\Vert \nabla F(w_k)\Vert ^2] \nonumber \\\le & {} F(w_k)-\frac{\theta d}{m\zeta }\Vert \nabla F(w_k)\Vert ^2\nonumber \\{} & {} +\,\frac{L\theta ^2d}{m^2\zeta ^2}\cdot \frac{\sigma ^2}{b}\nonumber \\{} & {} +\,\frac{Ld\theta ^2}{m^2\zeta ^2}\Vert \nabla F(w_k)\Vert ^2\nonumber \\= & {} F(w_k)-\biggl (\frac{\theta d}{m\zeta }-\frac{L\theta ^2d}{m^2\zeta ^2}\biggr )\Vert \nabla F(w_k)\Vert ^2\nonumber \\{} & {} +\,\frac{Ld\theta ^2\sigma ^2}{bm^2\zeta ^2} \end{aligned}$$

(B.3)

where the second inequality used the conditions $\mathbb {E}[\nabla F_S(w_k)]=\nabla F(w_k)$ and .

By recursively adding the inequality (B.3) over $k=0, 1, \ldots , m-1$, we easily had the following result:

$$\begin{aligned} \mathbb {E}[F(w_{m})]\le & {} \mathbb {E}[F(w_0)]-\biggl (\frac{\theta d}{m\zeta }-\,\frac{L\theta ^2d}{m^2\zeta ^2}\biggr )\nonumber \\{} & {} \sum _{k=0}^{m-1}\Vert \nabla F(w_k)\Vert ^2\,+\frac{Ld\theta ^2\sigma ^2}{bm\zeta ^2}. \end{aligned}$$

(B.4)

Further, according to (B.4), we arrive at

$$\begin{aligned}{} & {} \sum _{k=0}^{m-1}\Vert \nabla F(w_k)\Vert ^2\le \frac{m^2\zeta ^2}{m\zeta \theta d-Ld\theta ^2}\mathbb {E}[F(w_0)-F(w_m)]\\{} & {} \quad +\frac{Lm\theta \sigma ^2}{b(m\zeta -L\theta )}. \end{aligned}$$

Since $\mathbb {E}[\Vert \nabla F(w_m)\Vert ^2]=\frac{1}{m}\sum _{k=0}^{m-1}\Vert \nabla F(w_k)\Vert$, we further derived

$$\begin{aligned} \mathbb {E}[\Vert \nabla F(w_m)\Vert ^2]\le & {} \frac{m\zeta ^2}{m\zeta \theta d-Ld\theta ^2}\mathbb {E}[F(w_0)-F(w_m)]\\{} & {} +\,\frac{L\theta \sigma ^2}{b(m\zeta -L\theta )}\\\le & {} \frac{m\zeta ^2}{m\zeta \theta d-Ld\theta ^2}\mathbb {E}[F(w_0)-F(w_*)]\\{} & {} +\,\frac{L\theta \sigma ^2}{b(m\zeta -L\theta )} \end{aligned}$$

where the second inequality adopted the condition $w_*=\mathop {\mathrm {arg\,min}} F(w)$.

In addition, according to $w_0=\tilde{w}_s$ and $\tilde{w}_{s+1}=w_m$, we induced

$$\begin{aligned}{} & {} \mathbb {E}[\Vert \nabla F(\tilde{w}_{s+1})\Vert ^2] \le \frac{m\zeta ^2}{m\zeta \theta d-Ld\theta ^2}\mathbb {E}[F(\tilde{w}_s)-F(w_*)]\nonumber \\{} & {} \quad +\,\frac{L\theta \sigma ^2}{b(m\zeta -L\theta )}. \end{aligned}$$

(B.5)

In order to satisfy (B.5), we just required that the following condition be held.

$$\begin{aligned}{} & {} \mathbb {E}[\Vert \nabla F(\tilde{w}_{s+1})\Vert ^2] \le \frac{m\zeta ^2}{2Lm\zeta \theta d-2L^2d\theta ^2}\nonumber \\{} & {} \qquad \mathbb {E}[\Vert \nabla F(\tilde{w}_s)\Vert ^2]+\frac{L\theta \sigma ^2}{b(m\zeta -L\theta )}, \end{aligned}$$

(B.6)

where this inequality used the conclusion in (A.4).

Finally, summing the inequality (B.6) over $s=0, 1, \ldots , \hat{S}-1$, we obtained the desired results. $\square$