An inexact proximal generalized alternating direction method of multipliers

Abstract

This paper proposes and analyzes an inexact variant of the proximal generalized alternating direction method of multipliers (ADMM) for solving separable linearly constrained convex optimization problems. In this variant, the first subproblem is approximately solved using a relative error condition whereas the second one is assumed to be easy to solve. In many ADMM applications, one of the subproblems has a closed-form solution; for instance, $\ell _1$ regularized convex composite optimization problems. The proposed method possesses iteration-complexity bounds similar to its exact version. More specifically, it is shown that, for a given tolerance $\rho >0$, an approximate solution of the Lagrangian system associated to the problem under consideration is obtained in at most $\mathcal {O}(1/\rho ^2)$ (resp. $\mathcal {O}(1/\rho )$ in the ergodic case) iterations. Numerical experiments are presented to illustrate the performance of the proposed scheme.

Appendix A: Proof of Proposition 1

The proof of Proposition 1 is divided into the following steps. First we prove the inclusion in Proposition 1(a), and then we establish two technical lemmas which will be used to prove the inequality in Propositions1(a) and 1(b).

First, note that the definitions of $\tilde{\gamma }_{k}$ and $\gamma _{k}$ given in (12) and (14), respectively, imply that

$$\begin{aligned} \tilde{\gamma }_k-\gamma _{k-1}=\frac{\beta }{\alpha } B(y_{k}-y_{k-1})+\frac{1}{\alpha }\left( \gamma _{k}-\gamma _{k-1}\right) , \quad \forall k\ge 1. \end{aligned}$$

(44)

Proof of the inclusion in Proposition 1(a): From the inclusion in (11) and the first relation in (14), we have

$$\begin{aligned} \frac{1}{\beta }(x_{k-1}-x_k)=v_k\in \partial f({{\tilde{x}}}_k)-A^{*}\tilde{\gamma }_{k}. \end{aligned}$$

(45)

Now, the first-order optimality condition for (13) and the definition of $\gamma _k$ in (14) imply that

$$\begin{aligned} 0 \in \partial g(y_k)-B^{*}\gamma _{k}+H(y_k- y_{k-1}). \end{aligned}$$

(46)

On the other hand, it follows from (44) that

$$\begin{aligned} \gamma _k= \tilde{\gamma }_k-\frac{1-\alpha }{\alpha }(\gamma _{k}-\gamma _{k-1})-\frac{\beta }{\alpha }B(y_{k}-y_{k-1}), \end{aligned}$$

which, combined with (46), yields

$$\begin{aligned} \left(H+\frac{\beta }{\alpha } B^{*}B\right)\left( y_{k-1}-y_{k}\right) +\frac{1-\alpha }{\alpha }B^{*}\left( \gamma _{k-1}-\gamma _{k}\right) \in \partial g(y_k)-B^{*}\tilde{\gamma }_k. \end{aligned}$$

(47)

From the second equality in (14), we obtain

$$\begin{aligned} \frac{1-\alpha }{\alpha }B \left( y_{k-1}-y_{k}\right) +\frac{1}{\alpha \beta }\left( \gamma _{k-1}-\gamma _{k}\right) = A{\tilde{x}}_k+By_k-b. \end{aligned}$$

(48)

Therefore, the inclusion in Proposition 1(a) now follows by combining (45), (47), (48) and the definitions of M and T in (16).

In order to prove the remaining statements of Proposition 1, we need to establish two technical results. Note first that the relation in (44) implies that

$$\begin{aligned} \left\| \tilde{\gamma }_{k}-\gamma _{k-1}\right\| ^{2}=\frac{\beta }{\alpha ^{2}}\left\| (y_{k}-y_{k-1},\gamma _{k}-\gamma _{k-1})\right\| ^{2}_{S}, \quad \text{ where }\quad S = \left[ \begin{array}{cc} \beta B^{*}B &{} B^{*}\\ B&{}\frac{1}{\beta }I \end{array} \right] . \end{aligned}$$

(49)

For simplicity, we also consider the following symmetric matrices

$$\begin{aligned} N=\left[ \begin{array}{cc} \left[ 1+\alpha (2-\alpha )\right] \beta B^{*}B &{}(1+\alpha -\alpha ^{2})B^{*}\\ (1+\alpha -\alpha ^{2})B&{}\frac{1}{\beta }I \end{array} \right] ,\quad P=\left[ \begin{array}{cc} \beta B^{*}B&{}(1-\alpha )B^{*}\\ (1-\alpha )B &{}\frac{(1-\alpha )^{2}}{\beta }I \end{array} \right] . \end{aligned}$$

(50)

It is easy to verify that S, N and P are positive semidefinite for every $\beta >0$ and $\alpha \in (0,2)$.

Lemma 1

Let $\{z_k\}$ and $\{{{\tilde{z}}}_k\}$ be as in (18). Then, for every $k\ge 1$, the following hold:

$$\begin{aligned} \left\| {\tilde{z}}_{k}-z_{k-1}\right\| _{M}^{2}\ge \frac{1}{\beta }\left\| {\tilde{x}}_{k}-x_{k-1}\right\| ^{2} + \frac{1}{\alpha ^{3}}\left\| \left( y_{k}-y_{k-1},\gamma _{k}-\gamma _{k-1}\right) \right\| _{N}^{2} \end{aligned}$$

(51)

and

$$\begin{aligned} \left\| {\tilde{z}}_{k}-z_{k}\right\| _{M}^{2}=\frac{1}{\beta }\left\| {\tilde{x}}_{k}-x_{k}\right\| ^{2}+\frac{1}{\alpha ^{3}}\left\| \left( y_{k}-y_{k-1},\gamma _{k}-\gamma _{k-1}\right) \right\| _{P}^{2}, \end{aligned}$$

(52)

where the matrices M, N and P are as in (16) and (50).

Proof

Using the fact that ${\tilde{z}}_{k}-z_{k-1}=({\tilde{x}}_{k}-x_{k-1},y_{k}-y_{k-1},\tilde{\gamma }_{k}-\gamma _{k-1})$ and the definition of M in (16), we obtain

$$\begin{aligned} \Vert {\tilde{z}}_{k}- z_{k-1}\Vert _{M}^2=&\frac{1}{\beta }\Vert {\tilde{x}}_k-x_{k-1}\Vert ^2+\Vert y_{k}-y_{k-1}\Vert _{H}^2+ \frac{\beta }{\alpha }\Vert B(y_{k}-y_{k-1})\Vert ^2\\&+\frac{2(1-\alpha )}{\alpha }\left\langle {B(y_{k}-y_{k-1})},{\tilde{\gamma }_{k}-\gamma _{k-1}}\right\rangle + \frac{1}{\alpha \beta }\Vert {\tilde{\gamma }}_k-\gamma _{k-1}\Vert ^2. \end{aligned}$$

On the other hand, equality (44) implies that

$$\begin{aligned} \left\langle {B(y_{k}-y_{k-1})},{\tilde{\gamma }_{k}-\gamma _{k-1}}\right\rangle =\frac{\beta }{\alpha }\Vert B (y_{k}-y_{k-1})\Vert ^2+\frac{1}{\alpha }\langle {B(y_{k}-y_{k-1})},{\gamma _{k}-\gamma _{k-1}}\rangle , \end{aligned}$$

and

$$\begin{aligned}&\left\| \tilde{\gamma }_{k} -\gamma _{k-1}\right\| ^2=\frac{\beta ^{2}}{\alpha ^{2}}\Vert B(y_{k} -y_{k-1})\Vert ^2+\frac{2\beta }{\alpha ^{2}}\langle {B(y_{k}-y_{k-1})},{\gamma _{k}-\gamma _{k-1}}\rangle \\&\quad +\frac{1}{\alpha ^{2}}\Vert \gamma _{k} -\gamma _{k-1}\Vert ^2. \end{aligned}$$

Combining the last three equalities, we find

$$\begin{aligned} \Vert {\tilde{z}}_{k}- z_{k-1}\Vert _{M}^2\ge & {} \frac{1}{\beta }\Vert {\tilde{x}}_k-x_{k-1}\Vert ^2+ \left( \frac{1}{\alpha }+\frac{2(1-\alpha )}{\alpha ^2}+\frac{1}{\alpha ^{3}}\right) \beta \Vert B(y_{k}-y_{k-1})\Vert ^2\\&+\left( \frac{2(1-\alpha )}{\alpha ^{2}}+\frac{2}{\alpha ^{3}}\right) \left\langle {B(y_{k}-y_{k-1})},{\gamma _{k}-\gamma _{k-1}}\right\rangle + \frac{1}{\alpha ^{3}\beta }\Vert \gamma _k-\gamma _{k-1}\Vert ^2. \end{aligned}$$

Thus, (51) follows from the last equality and the definition of N in (50).

Let us now prove (52). Using ${\tilde{z}}_{k}-z_{k}=({\tilde{x}}_{k}-x_{k},0,\tilde{\gamma }_{k}-\gamma _{k})$ [see (18)] and the definition of M in (16), we have

$$\begin{aligned} \Vert {{\tilde{z}}}_k-z_k\Vert ^2_{M}= \frac{1}{\beta }\Vert {\tilde{x}}_k-x_{k}\Vert ^2+\frac{1}{\alpha \beta }\Vert \tilde{\gamma }_k-\gamma _{k}\Vert ^2. \end{aligned}$$

It follows from (44) and some algebraic manipulations that

$$\begin{aligned} \left\| \tilde{\gamma }_k-\gamma _k\right\| ^{2}= & {} \frac{\beta ^{2}}{\alpha ^{2}}\Vert B (y_{k}-y_{k-1})\Vert ^2+ \frac{2(1-\alpha )\beta }{\alpha ^{2}}\langle { B(y_{k}-y_{k-1})},{\gamma _k-\gamma _{k-1}}\rangle \\&+\frac{(1-\alpha )^{2}}{\alpha ^{2}}\left\| \gamma _{k}-\gamma _{k-1}\right\| ^{2}. \end{aligned}$$

Therefore, the desired equality now follows by combining the last two equalities and the definition of P in (50). $\square$

Lemma 2

Let $\{(x_k,y_k,\gamma _k)\}$ be generated by Algorithm 1. Then, the following hold:

(a):

$2\langle {B(y_1-y_{0})},{\gamma _1-\gamma _{0}}\rangle \ge \Vert y_1-y_{0}\Vert _{H}^2 - 4d_0^2$, where $d_0$ is as in (19);

(b):

$2\langle B(y_k-y_{k-1}),\gamma _k-\gamma _{k-1} \rangle \ge \Vert y_k-y_{k-1}\Vert _{H}^2-\Vert y_{k-1}-y_{k-2}\Vert _{H}^2$, for every$k\ge 2$.

Proof

(a) Consider $z_0,z_1$ and ${{\tilde{z}}}_1$ as in (18), and let an arbitrary $z^{*}:=(x^*,y^*,\gamma ^*)\in \varOmega ^{*}$ (see Assumpiton 1). Note that, in view of the definition of $d_0$ in (19), in order to establish (a), it is sufficient to prove that

$$\begin{aligned} \varTheta :=\Vert y_1-y_{0}\Vert _{H}^2-2\langle {B(y_1-y_{0})},{\gamma _1-\gamma _{0}}\rangle \le 4 \Vert z^{*}-z_{0}\Vert ^{2}_{M}, \end{aligned}$$

(53)

where M is as in (16). Let us then show (53). From the definitions of M and $\{z_k\}$, we have

$$\begin{aligned} \left\| z_1-z_0\right\| ^{2}_{M}&=\frac{1}{\beta }\Vert x_1-x_0\Vert ^2+\Vert y_1-y_{0}\Vert _{H+\frac{\beta }{\alpha } B^*B}^2\\&\quad +\frac{2(1-\alpha )}{\alpha }\langle {B(y_1-y_{0})},{\gamma _1-\gamma _{0}}\rangle \\&\quad +\frac{1}{\alpha \beta }\Vert \gamma _1-\gamma _{0}\Vert ^2\\&= \frac{1}{\beta }\Vert x_1-x_0\Vert ^2+\varTheta +\left\| \frac{ \sqrt{\beta }}{\sqrt{\alpha }} B(y_{1}-y_{0})+\frac{1}{\sqrt{\alpha \beta }}(\gamma _1-\gamma _{0})\right\| ^2. \end{aligned}$$

Hence, we obtain

$$\begin{aligned} \varTheta \le \left\| z_1-z_0\right\| ^{2}_{M}\le 2\left( \left\| z^{*}-z_1\right\| ^{2}_{M}+\left\| z^{*}-z_0\right\| ^{2}_{M}\right) , \end{aligned}$$

(54)

where the last inequality is due to $\Vert z+z^{\prime }\Vert _{M}^2\le 2\left( \Vert z\Vert _{M}^2+\Vert z^{\prime }\Vert _{M}^2\right)$ for all $z, z^{\prime }$. We will now prove that

$$\begin{aligned} \left\| z^{*}-z_1\right\| ^{2}_{M}\le \left\| z^{*}-z_0\right\| ^{2}_{M}. \end{aligned}$$

(55)

Since we have already proved that the inclusion in Proposition 1(a) holds, we have $M(z_0-z_1) \in T(\tilde{z}_1)$ where M and T are as in (16). Thus, using that $0 \in T(z^*)$ and T is monotone, we obtain $\langle M(z_0-z_1),z^*- {\tilde{z}}_1 \rangle \le 0$. Hence,

$$\begin{aligned} \Vert z^{*}-z_{1}\Vert ^{2}_{M}-\Vert z^{*}-z_{0}\Vert ^{2}_{M}&=\Vert (z^{*}-{\tilde{z}}_1)+({\tilde{z}}_1-z_1)\Vert ^{2}_{M}-\Vert (z^{*}-{\tilde{z}}_1)+({\tilde{z}}_1-z_{0})\Vert ^{2}_{M}\\&= \Vert {\tilde{z}}_1-z_{1}\Vert ^{2}_{M}+2\langle M(z_{0}-z_1),z^{*}-{\tilde{z}}_1 \rangle -\Vert {\tilde{z}}_1-z_{0}\Vert ^{2}_{M}\\&\le \Vert {\tilde{z}}_1-z_{1}\Vert ^{2}_{M}-\Vert {\tilde{z}}_1-z_{0}\Vert ^{2}_{M}. \end{aligned}$$

Using (52), the inequality in (11), and the first equality in (14) (all with $k=1$), we have

$$\begin{aligned} \left\| {\tilde{z}}_1-z_{1}\right\| ^{2}_{M}&\le \frac{\tau _{1}}{\beta }\left\| \tilde{\gamma }_1-\gamma _{0}\right\| ^{2}+\frac{\tau _{2}}{\beta }\left\| {\tilde{x}}_1-x_{0}\right\| ^{2}+\frac{1}{\alpha ^{3}}\left\| (y_{1}-y_{0},\gamma _{1}-\gamma _{0})\right\| ^{2}_{P}, \end{aligned}$$

where P is as in (50). Now, (51) with $k=1$ becomes

$$\begin{aligned} \Vert {\tilde{z}}_1-z_{0}\Vert _{M}^{2}\ge \frac{1}{\beta }\left\| {\tilde{x}}_{1}-x_{0}\right\| ^{2}+ \frac{1}{\alpha ^{3}}\left\| (y_{1}-y_{0},\gamma _{1}-\gamma _{0})\right\| _{N}^{2} \end{aligned}$$

where N is as in (50). Combining the last three inequalities and the fact that $\tau _{2}<1$ (see Algorithm 1), we find

$$\begin{aligned} \nonumber \left\| z^{*}-z_{1}\right\| ^{2}_{M}-\left\| z^{*}-z_{0}\right\| ^{2}_{M}&\le \frac{\tau _{1}}{\beta }\left\| \tilde{\gamma }_{1}-\gamma _{0}\right\| ^{2}\\\nonumber&\quad +\frac{1}{\alpha ^{3}}\left( \left\| (y_{1}-y_{0},\gamma _{1}-\gamma _{0})\right\| _{P}^{2}-\left\| (y_{1}-y_{0},\gamma _{1}-\gamma _{0})\right\| _{N}^{2}\right) \\&= \frac{\tau _{1}}{\beta }\left\| \tilde{\gamma }_{1}-\gamma _{0}\right\| ^{2} - \frac{2-\alpha }{\alpha ^{2}}\left\| (y_{1}-y_{0},\gamma _{1}-\gamma _{0})\right\| _{S}^{2}, \end{aligned}$$

(56)

where the last equality is due to the fact that $P-N=-\alpha (2-\alpha )S$, with S given in (49). The last inequality, (49) with $k=1$ and the fact that $\alpha \in (0,2-\tau _{1})$ yield

$$\begin{aligned} \left\| z^{*}-z_{1}\right\| ^{2}_{M}-\left\| z^{*}-z_{0}\right\| ^{2}_{M}\le \frac{\alpha +\tau _{1}-2}{\alpha ^{2}}\left\| (y_{1}-y_{0},\gamma _{1}-\gamma _{0})\right\| _{S}^{2}\le 0, \end{aligned}$$

which implies that (55) holds. Therefore, (a) now follows by combining (54) and (55).

(b) From the first-order optimality condition for (13) and the second relation in (14), we obtain

$$\begin{aligned} B^{*}\gamma _{j}-H(y_{j}-y_{j-1})\in \partial g(y_j) \qquad \forall \,j\ge 1. \end{aligned}$$

Hence, for every $k\ge 2$, using the above inclusion with $j \leftarrow k$ and $j \leftarrow k-1$ and the monotonicity of $\partial g$ , we have

$$\begin{aligned} \left\langle B^*(\gamma _k-\gamma _{k-1}),y_{k}-y_{k-1}\right\rangle&\ge \left\| y_k-y_{k-1}\right\| ^{2}_{H}-\left\langle H(y_{k-1}-y_{k-2}),y_k-y_{k-1}\right\rangle \\&\ge \frac{1}{2}\left\| y_k-y_{k-1}\right\| ^{2}_{H}-\frac{1}{2}\left\| y_{k-1}-y_{k-2}\right\| ^{2}_{H}, \end{aligned}$$

where the last inequality is due to the fact that $2\left\langle Hy,y^{\prime }\right\rangle \le \Vert y\Vert _{H}^{2}+\Vert y^{\prime }\Vert _{H}^{2}$ for all $y, y^{\prime }$. Therefore, (b) follows trivially from the last inequality. $\square$

We are now ready to prove the remaining statements of Proposition 1.

Proof of the inequality in Proposition 1(a) Using (52) and the first relation in (14), we have

$$\begin{aligned} \left\| {\tilde{z}}_{k}-z_{k}\right\| ^{2}_{M}&=\frac{1}{\beta }\left\| {\tilde{x}}_{k}-x_{k-1}+\beta v_{k}\right\| ^{2}+\frac{1}{\alpha ^{3}}\left\| (y_{k}-y_{k-1},\gamma _{k}-\gamma _{k-1})\right\| ^{2}_{P}\\&\le \frac{\tau _{1}}{\beta }\left\| \tilde{\gamma }_{k}-\gamma _{k-1}\right\| ^{2}+\frac{\tau _{2}}{\beta }\left\| {\tilde{x}}_{k}-x_{k-1}\right\| ^{2}+\frac{1}{\alpha ^{3}}\left\| (y_{k}-y_{k-1},\gamma _{k}-\gamma _{k-1})\right\| ^{2}_{P}, \end{aligned}$$

where the inequality is due to the second condition in (11). It follows from the last inequality, (51) and the fact that ${\sigma }\ge \tau _{2}$ [see (20)] that

$$\begin{aligned} \sigma \Vert \tilde{z}_{k}-z_{k-1}\Vert _{M}^2- \Vert {\tilde{z}}_k-z_{k}\Vert _{M}^2\ge a_{k} \end{aligned}$$

(57)

where

$$\begin{aligned} a_{k}:= & {} -\frac{\tau _{1}}{\beta }\left\| \tilde{\gamma }_{k}-\gamma _{k-1}\right\| ^{2}+\frac{1}{\alpha ^{3}}\left( \sigma \left\| (y_{k}-y_{k-1},\gamma _{k}-\gamma _{k-1})\right\| ^{2}_{N}-\left\| (y_{k}-y_{k-1},\gamma _{k}\right. \right. \\&\left. \left. -\gamma _{k-1})\right\| ^{2}_{P}\right) . \end{aligned}$$

We will show that $a_{k}\ge \eta _{k}-\eta _{k-1}$, where the sequence $\{\eta _{k}\}$ is defined in (21). From (49), we find

$$\begin{aligned} \frac{\tau _{1}}{\beta }\left\| \tilde{\gamma }_{k}-\gamma _{k-1}\right\| ^{2}= \frac{1}{\alpha ^{3}}\left\| (y_{k}-y_{k-1},\gamma _{k}-\gamma _{k-1})\right\| ^{2}_{\alpha \tau _{1} S}, \end{aligned}$$

which, combined with definition of $a_k$, yields

$$\begin{aligned} a_{k}= \frac{1}{\alpha ^{3}}\left\| (y_{k}-y_{k-1},\gamma _{k}-\gamma _{k-1})\right\| ^{2}_{\sigma N-\alpha \tau _1 S-P}. \end{aligned}$$

Hence, using the definitions of N, S and P in (49) and (50), we obtain

$$\begin{aligned} a_{k}=\frac{1}{\alpha ^{3}}\left( {\hat{\xi }}\beta \Vert B(y_{k}-y_{k-1})\Vert ^2+ 2\bar{\xi }\left\langle {B(y_{k}-y_{k-1})},{{\gamma }_{k}-\gamma _{k-1}}\right\rangle + \frac{\tilde{\xi }}{\beta }\Vert \gamma _{k}-\gamma _{k-1}\Vert ^{2}\right) , \end{aligned}$$

(58)

where

$$\begin{aligned} {\hat{\xi }}= & {} \sigma (1+\alpha (2-\alpha ))-\alpha \tau _{1}-1, \nonumber \\ \bar{\xi }= & {} \sigma (1+\alpha -\alpha ^{2})+(1-\tau _{1})\alpha -1, \nonumber \\ \tilde{\xi }= & {} \sigma -\alpha \tau _{1}-(1-\alpha )^{2}. \end{aligned}$$

(59)

Now, from the definition of $\sigma$ given in (20), we obtain ${\sigma }\ge (1+\alpha \tau _{1})/(1+\alpha (2-\alpha ))$. Hence, ${\hat{\xi }}\ge 0$ and

$$\begin{aligned} \tilde{\xi }\ge \frac{1+\alpha \tau _{1}}{1+\alpha (2-\alpha )}-\alpha \tau _1 -(1-\alpha )^{2}=\frac{\alpha ^{2}(2-\tau _{1}-\alpha )(2-\alpha )}{1+\alpha (2-\alpha )}>0, \end{aligned}$$

where the last inequality is due to the fact that $\alpha \in (0,2-\tau _{1})$. Moreover, since $\sigma \in (0,1)$ (see (20)), we find

$$\begin{aligned} \bar{\xi }=\sigma (1+\alpha -\alpha ^{2})+\alpha -\tau _{1}\alpha -1>\sigma (1+\alpha (2-\alpha ))-\alpha \tau _{1}-1={\hat{\xi }}. \end{aligned}$$

Thus, $\bar{\xi } >{\hat{\xi }}\ge 0$, and $\tilde{\xi }\ge 0$. Hence, from (58), Lemma 2 and the fact that $\bar{\xi }=\alpha ^{3}\xi$ [see (20) and (59)], it follows that

$$\begin{aligned} a_{k}\ge & {} \frac{2\bar{\xi }}{\alpha ^{3}}\left\langle {B(y_{k}-y_{k-1})},{{\gamma }_{k}-\gamma _{k-1}}\right\rangle = 2\xi \left\langle {B(y_{k}-y_{k-1})},{{\gamma }_{k}-\gamma _{k-1}}\right\rangle \\\ge & {} {\left\{ \begin{array}{ll} \xi \left\| y_{1}-y_{0}\right\| _{H}^{2}- 4\xi d_{0}^{2}, &{} k=1, \\ \xi \left\| y_{k}-y_{k-1}\right\| _{H}^{2}-\xi \left\| y_{k-1}-y_{k-2}\right\| _{H}^{2}, &{} k\ge 2, \end{array}\right. } \end{aligned}$$

which, combined with the definitions of $\{\eta _{k}\}$ in (21) yields $a_{k}\ge \eta _{k}-\eta _{k-1}$ for every $k\ge 1$. Hence, the desired inequality now follows from (57).

Proof of Proposition 1(b) First, for every $z^*=(x^*,y^*,\gamma ^*)\in \varOmega ^*$, we have

$$\begin{aligned}&\Vert z^*-z_{k}\Vert ^{2}_{M}-\Vert z^*-z_{k-1}\Vert ^{2}_{M} \\&\quad = \Vert (z^*-{\tilde{z}}_k)+({\tilde{z}}_k-z_k)\Vert ^{2}_{M}-\Vert (z^*-{\tilde{z}}_k)+({\tilde{z}}_k-z_{k-1})\Vert ^{2}_{M}\\&\quad = \Vert {\tilde{z}}_k-z_{k}\Vert ^{2}_{M}-\Vert {\tilde{z}}_k-z_{k-1}\Vert ^{2}_{M}+2\langle M(z_{k-1}-z_k),z^*-{\tilde{z}}_k \rangle . \end{aligned}$$

Now, since $M(z_{k-1}-z_k)\in T(\tilde{z}_k)$ (see (22)), $0 \in T(z^*)$, and T is monotone, we trivially obtain $\langle M(z_{k-1}-z_k),{\tilde{z}}_k-z^* \rangle \ge 0$. Therefore, combining the last two inequalities and (22), we obtain

$$\begin{aligned} \Vert z^{*}-z_{k}\Vert ^{2}_{M}-\Vert z^{*}-z_{k-1}\Vert ^{2}_{M}\le \eta _{k-1}-\eta _{k}-(1-\sigma )\Vert {\tilde{z}}_k-z_{k-1}\Vert ^{2}_{M}, \end{aligned}$$

which is equivalent to the desired inequality.