Block Mirror Stochastic Gradient Method For Stochastic Optimization

Abstract

In this paper, a block mirror stochastic gradient method is developed to solve stochastic optimization problems involving convex and nonconvex cases, where the feasible set and the variables are treated as multiple blocks. The proposed method combines the features of the classic mirror descent stochastic method and the block coordinate gradient descent method. Acquiring the stochastic gradient information by stochastic oracles, our method updates all the blocks of variables in the Gauss–Seidel type. We establish the convergence for both convex and nonconvex cases. The analysis of our method is challenging because the typical unbiasedness assumption of stochastic gradient fails to hold in the Gauss–Seidel renewal type and requires more specific assumptions. The proposed algorithm is tested on the conditional value-at-risk problem and the stochastic LASSO problem to demonstrate the efficiency of our algorithm.

Appendix A: Proofs of some lemmas in this paper

Proof of Lemma 1

From the strong convexity of $\omega _{i}$, it holds

$$\begin{aligned} \begin{aligned} \omega _{i}(\textbf{x}_{i})-\omega _{i}(\textbf{y}_{i}) \ge&\langle \nabla \omega _{i}(\textbf{y}_{i}),\textbf{x}_{i}-\textbf{y}_{i}\rangle +\frac{\alpha _{i}}{2}\Vert \textbf{y}_{i}-\textbf{x}_{i}\Vert _{\mathcal {E}_i}^{2} \\ \ge&-\frac{1}{2\alpha _{i}}\left\| \nabla \omega _{i}(\textbf{y}_{i}) \right\| _{\mathcal {E}_i,*}^{2}, \end{aligned} \end{aligned}$$

where the second inequality follows from (6). It implies

$$\begin{aligned} \omega _{i}(\textbf{y}_{i})-\omega _{i}(\textbf{x}_{i}) \le \frac{1}{2\alpha _{i}}\left\| \nabla \omega _{i}(\textbf{y}_{i}) \right\| _{\mathcal {E}_i,*}^{2}. \end{aligned}$$

Let us consider the function $\varphi _{\textbf{x}_{i}}(\textbf{z})=\omega _{i}(\textbf{z}) -\langle \nabla \omega _{i}(\textbf{x}_{i}),\textbf{z}\rangle$. It is easy to see that $\varphi _{\textbf{x}_{i}}(\textbf{z})$ is the strongly convex function with the same parameter $\alpha _{i}$ since

$$\begin{aligned} \begin{aligned} \langle \nabla \varphi _{\textbf{x}_{i}}(\textbf{z}_{1}) -\nabla \varphi _{\textbf{x}_{i}}(\textbf{z}_{2}),\textbf{z}_{1}-\textbf{z}_{2}\rangle =&\langle \nabla \omega _{i}(\textbf{z}_{1})-\nabla \omega _{i}(\textbf{z}_{2}),\textbf{z}_{1}-\textbf{z}_{2}\rangle \\ \ge&\alpha _{i}\Vert \textbf{z}_{1}-\textbf{z}_{2}\Vert _{\mathcal {E}_i,*}^{2}. \end{aligned} \end{aligned}$$

Then we have

$$\begin{aligned} \begin{aligned} \omega _{i}(\textbf{y}_{i})-\omega _{i}(\textbf{x}_{i})-\langle \nabla \omega _{i}(\textbf{x}_{i}),\textbf{y}_{i}-\textbf{x}_{i}\rangle =&\varphi _{\textbf{x}_{i}}(\textbf{y}_{i})-\varphi _{\textbf{x}_{i}}(\textbf{x}_{i}) \\ \le&\frac{1}{2\alpha _{i}}\left\| \nabla \varphi _{\textbf{x}_{i}}(\textbf{y}_{i}) \right\| _{\mathcal {E}_i,*}^{2}\\ =&\frac{1}{2\alpha _{i}} \left\| \nabla \omega _{i}(\textbf{y}_{i})-\nabla \omega _{i}(\textbf{x}_{i}) \right\| _{\mathcal {E}_i,*}^{2}, \end{aligned} \end{aligned}$$

which gives the result. $\square$

Proof of Lemma 2

Note that $\sum _{t=1}^{T-1}\textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) - \textbf{g}_{i}(\textbf{x}^{k})$ is independent of $\textbf{G}_{i}(\textbf{x}^{k},\xi _{T}^{k}) - \textbf{g}_{i}(\textbf{x}^{k})$ on $\{\xi _{t}^{k}\}_{t=1}^{T-1}$ for any $T \ge 2$. This together with (2) yields that

$$\begin{aligned} {\mathbb {E}}\left[ \langle \sum _{t=1}^{T-1} \textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) -\textbf{g}_{i}(\textbf{x}^{k}), \textbf{G}_{i}(\textbf{x}^{k},\xi _{T}^{k}) - \textbf{g}_{i}(\textbf{x}^{k}) \rangle | \{\xi _{t}^{k}\}_{t=1}^{T-1} \right] = 0,\quad \forall ~T\ge 2. \end{aligned}$$

Then for Euclidean norm $\Vert \cdot \Vert$, we have

$$\begin{aligned} \begin{aligned} {\mathbb {E}}\left[ \left\| \sum _{t=1}^{T_{k}}\textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) - \textbf{g}_{i}(\textbf{x}^{k}) \right\| ^{2}\right] =&{\mathbb {E}}\left[ \left\| \sum _{t=1}^{T_{k}-1}\textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) - \textbf{g}_{i}(\textbf{x}^{k}) \right\| ^{2} \right] \\&+{\mathbb {E}}\left[ \left\| \textbf{G}_{i}(\textbf{x}^{k},\xi _{T_{k}}^{k}) - \textbf{g}_{i}(\textbf{x}^{k}) \right\| ^{2} \right] \\ =&\sum _{t=1}^{T_{k}}{\mathbb {E}}\left[ \left\| \textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) - \textbf{g}_{i}(\textbf{x}^{k}) \right\| ^{2} \right] . \end{aligned} \end{aligned}$$

Under Assumption 1, it implies that

$$\begin{aligned} {\mathbb {E}}\left[ \left\| \bar{\textbf{G}}_{i}^{k} - \textbf{g}_{i}(\textbf{x}^{k}) \right\| ^{2} \right] =\frac{1}{T_{k}^2} {\mathbb {E}}\left[ \left\| \sum _{t=1}^{T_{k}}\textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) - \textbf{g}_{i}(\textbf{x}^{k}) \right\| ^{2} \right] \le \frac{\sigma _{i}^{2}}{T_{k}}. \end{aligned}$$

Then the equivalence relation between the norm $\Vert \cdot \Vert _{\mathcal {E}_i,*}$ and the norm $\Vert \cdot \Vert$ on ${\mathbb {R}}^{n_i}$ completes the proof. $\square$

Proof of Lemma 3

By Remark 2, it can be seen from (10), (11), and (14) that for any i and k

$$\begin{aligned} \begin{aligned} {\mathbb {E}}\left[ \left\| \textbf{G}_{i}(\textbf{x}^{k},\xi ) \right\| _{\mathcal {E}_i,*}^{2} \right] \le&2\left\| \textbf{g}_{i}(\textbf{x}^{k}) \right\| _{\mathcal {E}_i,*}^{2} +2{\mathbb {E}}\left[ \left\| \textbf{G}_{i}(\textbf{x}^{k},\xi ) -\textbf{g}_{i}(\textbf{x}^{k}) \right\| _{\mathcal {E}_i,*}^{2} \right] \\ \le&2\left( 2\left\| \textbf{g}_{i}(\textbf{x}^{k})-\textbf{g}_{i}(\textbf{x}^{1}) \right\| _{\mathcal {E}_i,*}^{2} +2\Vert \textbf{g}_{i}(\textbf{x}^{1})\Vert _{\mathcal {E}_i,*}^{2} \right) +2\sigma _{i}^{2},\\ \le&4L_{i}^{2}\Vert \textbf{x}^{k}-\textbf{x}^{1}\Vert _{\mathcal {E}_i}^{2} +4\left\| \textbf{g}_{i}(\textbf{x}^{1}) \right\| _{\mathcal {E}_i,*}^{2}+2\sigma _{i}^{2}\\ \le&16L_{i}^{2}\rho ^{2}+4\left\| \textbf{g}_{i}(\textbf{x}^{1}) \right\| _{\mathcal {E}_i,*}^{2} +2\sigma _{i}^{2}\\ \le&M_{i}^{2} + 2\sigma _{i}^{2}.\\ \end{aligned} \end{aligned}$$

Also it holds by Lemma 2 that

$$\begin{aligned} {\mathbb {E}}\left[ \Vert \bar{\textbf{G}}_{i}^{k}\Vert _{\mathcal {E}_i,*}^{2} \right] \le 2\left\| \textbf{g}_{i}(\textbf{x}^{k}) \right\| _{\mathcal {E}_i,*}^{2} +\frac{2c_{i}^{2}\sigma _{i}^{2}}{T_{k}} \le M_{i}^{2} + \frac{2c_{i}^{2}\sigma _{i}^{2}}{T_{k}}. \end{aligned}$$

We complete the proof of Lemma 3. $\square$

Proof of Lemma 4

By (8) and the optimality of $\textbf{x}_{i}^{k+1}$, we have

$$\begin{aligned} \langle \tilde{\textbf{G}}_{i}^{k}+\frac{1}{\gamma _{i}^{k}}(\nabla \omega _{i}(\textbf{x}_{i}^{k+1}) -\nabla \omega _{i}(\textbf{x}_{i}^{k})),\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}\rangle +r_{i}(\textbf{x}_{i}^{k})-r_{i}(\textbf{x}_{i}^{k+1})\ge 0. \end{aligned}$$

Because of the strong convexity of $\omega _{i}$, we obtain

$$\begin{aligned} \begin{aligned} \langle \tilde{\textbf{G}}_{i}^{k},\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}\rangle +r_{i}(\textbf{x}_{i}^{k})-r_{i}(\textbf{x}_{i}^{k+1}) \ge&\frac{1}{\gamma _{i}^{k}}\langle \nabla \omega _{i}(\textbf{x}_{i}^{k}) -\nabla \omega _{i}(\textbf{x}_{i}^{k+1}),\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}\rangle \\ \ge&\frac{\alpha _{i}}{\gamma _{i}^{k}}\Vert \textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}\Vert _{\mathcal {E}_i}^{2}. \end{aligned} \end{aligned}$$

On the other side, it holds that by Assumption 3

$$\begin{aligned} \langle \tilde{\textbf{G}}_{i}^{k},\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}\rangle +r_{i}(\textbf{x}_{i}^{k})-r_{i}(\textbf{x}_{i}^{k+1}) \le \left( \Vert \tilde{\textbf{G}}_{i}^{k}\Vert _{\mathcal {E}_i,*}+L_{r_i} \right) \Vert \textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}\Vert _{\mathcal {E}_i}. \end{aligned}$$

Then, we obtaind that

$$\begin{aligned} \Vert \textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}\Vert _{\mathcal {E}_i} \le \frac{\gamma _{i}^{k}}{\alpha _{i}} \left( \Vert \tilde{\textbf{G}}_{i}^{k}\Vert _{\mathcal {E}_i,*}+L_{r_i} \right) . \end{aligned}$$

In addition, under Assumption 2, for any $\xi _{t}^{k} \in \Xi ^{k} \subseteq \Xi$, it holds that

$$\begin{aligned} \begin{aligned} \left\| \textbf{G}_{i}(\textbf{x}_{<i}^{k+1},\textbf{x}_{\ge i}^{k},\xi _{t}^{k}) - \textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) \right\| _{\mathcal {E}_i,*}^{2}&\le L_{i}(\xi _{t}^{k})^{2}\left( \sum _{j<i}\Vert \textbf{x}_{j}^{k+1}-\textbf{x}_{j}^{k}\Vert _{\mathcal {E}_{j}}^{2}\right) \\&\le 2L_{i}(\xi _{t}^{k})^{2}\left( \sum _{j<i}\left( \frac{\gamma _{j}^{k}}{\alpha _{j}}\right) ^{2} \left( \Vert \tilde{\textbf{G}}_{j}^{k}\Vert _{\mathcal {E}_j,*}^{2}+L_{r_j}^{2}\right) \right) \\&= 2\sum _{j<i}\left( \frac{\gamma _{j}^{k}}{\alpha _{j}}\right) ^{2}\left( L_{i}(\xi _{t}^{k})^{2}\Vert \tilde{\textbf{G}}_{j}^{k}\Vert _{\mathcal {E}_j,*}^{2} +L_{i}(\xi _{t}^{k})^{2}L_{r_j}^{2}\right) . \\ \end{aligned} \end{aligned}$$

Note that

$$\begin{aligned} L_{i}(\xi _{t}^{k})^{2}\Vert \tilde{\textbf{G}}_{j}^{k}\Vert _{\mathcal {E}_j,*}^{2}\le \frac{1}{T_k}\sum _{s=1}^{T_k} L_{i}(\xi _{t}^{k})^{2}\left\| \textbf{G}_{j}(\textbf{x}_{<j}^{k+1},\textbf{x}_{\ge j}^{k},\xi _{s}^{k}) \right\| _{\mathcal {E}_j,*}^{2}, \end{aligned}$$

(68)

then by the uncorrelated condition, we have

$$\begin{aligned} {\mathbb {E}}\left[ L_{i}(\xi _{t}^{k})^{2}\Vert \tilde{\textbf{G}}_{j}^{k}\Vert _{\mathcal {E}_j,*}^{2} \right] \le \frac{1}{T_k}\sum _{s=1}^{T_k}L_{i}^{2}{\mathbb {E}}\left[ \left\| \textbf{G}_{j}(\textbf{x}_{<j}^{k+1},\textbf{x}_{\ge j}^{k},\xi _{s}^{k}) \right\| _{\mathcal {E}_j,*}^{2} \right] . \end{aligned}$$

(69)

By using the above observation and an auxiliary notation $\Lambda$ denoted by

$$\begin{aligned} \Lambda = \max _{j} {\mathbb {E}}\left[ \left\| \textbf{G}_{j}(\textbf{x}_{<j}^{k+1},\textbf{x}_{\ge j}^{k},\xi ) \right\| _{\mathcal {E}_j,*}^{2}\right] , \end{aligned}$$

we get that

$$\begin{aligned} {\mathbb {E}}\left[ \left\| \textbf{G}_{i}(\textbf{x}_{<i}^{k+1},\textbf{x}_{\ge i}^{k},\xi _{t}^{k}) - \textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) \right\| _{\mathcal {E}_i,*}^{2} \right] \le 2mL^{2}\left( \Lambda +L_{r}^{2}\right) \left( \frac{\gamma _{max}^{k}}{\alpha }\right) ^{2}. \end{aligned}$$

(70)

Also, observe that

$$\begin{aligned} \left\| \textbf{G}_{i}(\textbf{x}_{<i}^{k+1},\textbf{x}_{\ge i}^{k},\xi _{t}^{k}) \right\| _{\mathcal {E}_i,*}^{2} \le 2\left\| \textbf{G}_{i}(\textbf{x}_{<i}^{k+1},\textbf{x}_{\ge i}^{k},\xi _{t}^{k}) - \textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) \right\| _{\mathcal {E}_i,*}^{2} + 2\left\| \textbf{G}_{i}(\textbf{x}^{k},\xi _{t}^{k}) \right\| _{\mathcal {E}_i,*}^{2}, \end{aligned}$$

and by the same argument in the proof of Lemma 3 and the definition of $\Lambda$, it implies that

$$\begin{aligned} \Lambda \le \frac{4mL^{2}L_{r}^{2}(\gamma _{max}^{k})^{2} + 2\alpha ^{2}(M^{2}+2\sigma ^{2})}{\alpha ^{2}-4mL^{2}(\gamma _{max}^{k})^{2}} = \delta (\gamma _{max}^{k}). \end{aligned}$$

(71)

Moreover, it is clear that by Assumption 2

$$\begin{aligned} \begin{aligned}&{\mathbb {E}}\left[ \Vert \tilde{\textbf{G}}_{i}^{k}-\bar{\textbf{G}}_{i}^{k}\Vert _{\mathcal {E}_i,*}^{2}\right] \le \frac{1}{T_k}{\mathbb {E}}\left[ \sum _{t=1}^{T_k} L_{i}(\xi _{t}^{k})^{2}\left( \sum _{j<i}\Vert \textbf{x}_{j}^{k+1}-\textbf{x}_{j}^{k}\Vert _{\mathcal {E}_{j}}^{2}\right) \right] \\&\quad \le \frac{1}{T_k}{\mathbb {E}}\left[ 2\sum _{t=1}^{T_k} L_{i}(\xi _{t}^{k})^{2}\left( \sum _{j<i}\left( \frac{\gamma _{j}^{k}}{\alpha _{j}}\right) ^{2} \left( \Vert \tilde{\textbf{G}}_{j}^{k}\Vert _{\mathcal {E}_j,*}^{2}+L_{r_j}^{2}\right) \right) \right] \\&\quad \le \frac{1}{T_k}{\mathbb {E}}\left[ 2\sum _{t=1}^{T_k} \left( \sum _{j<i}\left( \frac{\gamma _{j}^{k}}{\alpha _{j}}\right) ^{2} \left( \frac{1}{T_k}\sum _{s=1}^{T_k} L_{i}(\xi _{t}^{k})^{2}\Vert \textbf{G}_{j}(\textbf{x}_{<j}^{k+1},\textbf{x}_{\ge j}^{k},\xi _{s}^{k})\Vert _{\mathcal {E}_j,*}^{2}\right) \right) \right] \\&\qquad +\frac{1}{T_k}{\mathbb {E}}\left[ 2\sum _{t=1}^{T_k} \left( \sum _{j<i}L_{i}(\xi _{t}^{k})^{2}L_{r_j}^{2} \left( \frac{\gamma _{j}^{k}}{\alpha _{j}}\right) ^{2}\right) \right] \\&\quad \le 2mL_{i}^{2}\left( \delta (\gamma _{max}^{k}) + L_{r}^{2}\right) \left( \frac{\gamma _{max}^{k}}{\alpha }\right) ^{2}, \end{aligned} \end{aligned}$$

(72)

which, together with Lemma 3, gives that

$$\begin{aligned} \begin{aligned} {\mathbb {E}}\left[ \Vert \tilde{\textbf{G}}_{i}^{k}\Vert _{\mathcal {E}_i,*}^{2} \right]&\le 2{\mathbb {E}}\left[ \Vert \tilde{\textbf{G}}_{i}^{k}-\bar{\textbf{G}}_{i}^{k}\Vert _{\mathcal {E}_i,*}^{2} \right] +2{\mathbb {E}}\left[ \Vert \bar{\textbf{G}}_{i}^{k}\Vert _{\mathcal {E}_i,*}^{2} \right] \\&\le 4mL_{i}^{2}\left( \delta (\gamma _{max}^{k})^{2} + L_{r}^{2}\right) \left( \frac{\gamma _{max}^{k}}{\alpha }\right) ^{2} + 2\left( M_{i}^{2} + \frac{2c_{i}^{2}\sigma _{i}^{2}}{T_{k}}\right) \\&= \varepsilon _{i}(\gamma _{max}^{k},T_{k}). \end{aligned} \end{aligned}$$

Furthermore, by the above observation, we conclude that

$$\begin{aligned} {\mathbb {E}}\left[ \Vert \textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}\Vert _{\mathcal {E}_i}^{2} \right] \le 2\left( \varepsilon _{i}(\gamma _{max}^{k},T_{k}) + L_{r_i}^{2}\right) \left( \frac{\gamma _{i}^{k}}{\alpha _{i}}\right) ^{2}, \end{aligned}$$

(73)

completing the proof. $\square$

Proof of Lemma 7

Let $\textbf{x}_{\textbf{g}_{i}}^{k+1} =\textbf{x}_{i}^{k}-\gamma _{i}^{k}\mathcal {G}_{i}\left( \textbf{x}_{i}^{k},\textbf{g}(\textbf{x}^{k}),\gamma _{i}^{k} \right) =P_{i}\left( \textbf{x}_{i}^{k},\textbf{g}(\textbf{x}^{k}),\gamma _{i}^{k} \right)$. By the optimality condition of (48), and the definition of $\textbf{x}_{i}^{k+1}$ and $\textbf{x}_{\textbf{g}_{i}}^{k+1}$, we have

$$\begin{aligned} \left\langle \tilde{\textbf{G}}_{i}^{k}+\frac{1}{\gamma _{i}^{k}}\left( \nabla \omega _{i}(\textbf{x}_{i}^{k+1}) -\nabla \omega _{i}(\textbf{x}_{i}^{k}) \right) ,\textbf{x}_{\textbf{g}_{i}}^{k+1}-\textbf{x}_{i}^{k+1}\right\rangle +r_{i}(\textbf{x}_{\textbf{g}_{i}}^{k+1})-r_{i}(\textbf{x}_{i}^{k+1})\ge 0, \end{aligned}$$

and

$$\begin{aligned} \langle \textbf{g}_{i}(\textbf{x}^{k})+\frac{1}{\gamma _{i}^{k}}\left( \nabla \omega _{i}(\textbf{x}_{\textbf{g}_{i}}^{k+1}) -\nabla \omega _{i}(\textbf{x}_{i}^{k}) \right) ,\textbf{x}_{i}^{k+1}-\textbf{x}_{\textbf{g}_{i}}^{k+1}\rangle +r_{i}(\textbf{x}_{i}^{k+1})-r_{i}(\textbf{x}_{\textbf{g}_{i}}^{k+1})\ge 0, \end{aligned}$$

respectively. Summing up the above two inequalities, one has

$$\begin{aligned} \begin{aligned} \langle \tilde{\textbf{G}}_{i}^{k}-\textbf{g}_{i}(\textbf{x}^{k}),\textbf{x}_{\textbf{g}_{i}}^{k+1}-\textbf{x}_{i}^{k+1}\rangle \ge&\frac{1}{\gamma _{i}^{k}}\langle \nabla \omega _{i}(\textbf{x}_{\textbf{g}_{i}}^{k+1}) -\nabla \omega _{i}(\textbf{x}_{i}^{k+1}),\textbf{x}_{\textbf{g}_{i}}^{k+1}-\textbf{x}_{i}^{k+1}\rangle \\ \ge&\frac{\alpha _{i}}{\gamma _{i}^{k}}\Vert \textbf{x}_{\textbf{g}_{i}}^{k+1}-\textbf{x}_{i}^{k+1}\Vert _{\mathcal {E}_i}^{2}. \end{aligned} \end{aligned}$$

where the second inequality follows from the strong convexity of $\omega _{i}$. Then, it holds that

$$\begin{aligned} \frac{1}{\gamma _{i}^{k}}\Vert \textbf{x}_{\textbf{g}_{i}}^{k+1}-\textbf{x}_{i}^{k+1}\Vert _{\mathcal {E}_i}\le \frac{1}{\alpha _{i}}\Vert \tilde{\textbf{G}}_{i}^{k}-\textbf{g}_{i}(\textbf{x}^{k})\Vert _{\mathcal {E}_i,*}. \end{aligned}$$

Using the above relation, we obtain

$$\begin{aligned} \begin{aligned} \left\| \mathcal {G}_{i}(\textbf{x}_{i}^{k},\textbf{g}(\textbf{x}^{k}),\gamma _{i}^{k}) -\frac{1}{\gamma _{i}^{k}}(\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1})\right\| _{\mathcal {E}_i} =&\left\| \frac{1}{\gamma _{i}^{k}}(\textbf{x}_{i}^{k}-\textbf{x}_{\textbf{g}_{i}}^{k+1}) -\frac{1}{\gamma _{i}^{k}}(\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}) \right\| _{\mathcal {E}_i} \\ \le&\frac{1}{\alpha _{i}}\Vert \tilde{\textbf{G}}_{i}^{k}-\textbf{g}_{i}(\textbf{x}^{k})\Vert _{\mathcal {E}_i,*}. \end{aligned} \end{aligned}$$

Hence, we have

$$\begin{aligned} \begin{aligned} \left\| \mathcal {G}_{i}(\textbf{x}_{i}^{k},\textbf{g}(\textbf{x}^{k}),\gamma _{i}^{k})\right\| _{\mathcal {E}_i}^{2} =&\left\| \mathcal {G}_{i}(\textbf{x}_{i}^{k},\textbf{g}(\textbf{x}^{k}),\gamma _{i}^{k}) -\frac{1}{\gamma _{i}^{k}}(\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1}) +\frac{1}{\gamma _{i}^{k}}(\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1})\right\| _{\mathcal {E}_i}^{2} \\ \le&2\left( \frac{1}{\alpha _{i}^{2}}\Vert \tilde{\textbf{G}}_{i}^{k}-\textbf{g}_{i}(\textbf{x}^{k})\Vert _{\mathcal {E}_i,*}^{2} +\left\| \frac{1}{\gamma _{i}^{k}}(\textbf{x}_{i}^{k}-\textbf{x}_{i}^{k+1})\right\| _{\mathcal {E}_i}^{2}\right) , \end{aligned} \end{aligned}$$

and obtain the result as required in (51).