The modified second APG method for DC optimization problems

Abstract

In this paper, we construct a variant of the second accelerated proximal gradient method introduced by Nesterov (Introductory lectures on convex optimization, Kluwer Academic Publisher, Dordrecht, 2004) and Auslender and Teboulle (SIAM J Optim 16:697–725, 2006) [and named by Tseng (Math Program 125:263–295, 2010)] for solving the minimization of DC functions (difference of two convex functions). Under some suitable assumptions such as level boundedness, Kurdyka–Łojasiewicz property, and locally Lipschitz differentiability, we prove that the sequence generated by our algorithm locally linearly converges to some stationary point of the given DC function. Numerical results show that our method performs well and fast, comparing to some other often used algorithms.

Appendix: Proof of Lemma 2

Proof

(Lemma 2) By the definition of $z^{k+1}$ in Algorithm 1 and together with 3-Point Property (see [37, Lemma 2.2] and [3, Section 5]), we have

$$\begin{aligned}&\langle \nabla f(y^{k})-\xi ^k,z^{k+1}-y^{k} \rangle +P_1(z^{k+1})+\displaystyle \frac{\theta _k L}{2}\Vert z^{k+1}-z^{k}\Vert ^2\nonumber \\&\quad \le \,\langle \nabla f(y^{k})-\xi ^k,x^{k}-y^{k} \rangle -\displaystyle \frac{\theta _k L}{2}\Vert z^{k+1}-x^{k}\Vert ^2\nonumber \\&\quad \quad +\,\displaystyle \frac{\theta _k L}{2}\Vert x^{k}-z^{k}\Vert ^2+P_1(x^{k}). \end{aligned}$$

(24)

On the other hand, we have

$$\begin{aligned} F(x^{k+1})= & {} f(x^{k+1})+P(x^{k+1})\le f(y^{k})+\langle \nabla f(y^{k}),x^{k+1}-y^{k} \rangle \nonumber \\&+\,\frac{L}{2}\Vert x^{k+1}-y^{k}\Vert ^2+P(x^{k+1})\nonumber \\= & {} f(y^{k})+\langle \nabla f(y^{k}),(1-\theta _{k})x^{k}+\theta _{k}z^{k+1}-y^k \rangle +P_1(x^{k+1})-P_2(x^{k+1})\nonumber \\&+\,\frac{L}{2}\Vert x^{k+1}-y^{k}\Vert ^2\nonumber \\\le & {} \, (1-\theta _{k})\left[ f(y^{k})+\langle \nabla f(y^{k}),x^{k}-y^{k} \rangle \right] \nonumber \\&+\,\theta _{k}\left[ f(y^{k}) +\langle \nabla f(y^{k}),z^{k+1}-y^{k}\rangle \right] \nonumber \\&+\,(1-\theta _{k})P_1(x^{k})+\theta _{k}P_1(z^{k+1})\nonumber \\&-\,P_2(x^{k}) -\langle \xi ^k,x^{k+1}-x^{k} \rangle +\displaystyle \frac{L}{2}{{\theta ^2_{k}}}\Vert z^{k+1}-z^{k}\Vert ^2\nonumber \\= & {} (1-\theta _{k})\left[ f(y^{k})+\langle \nabla f(y^{k}),x^{k}-y^{k} \rangle +P_1(x^{k})\right] \nonumber \\&-\,P_2(x^{k}) -\langle \xi ^k,x^{k+1}-x^{k} \rangle \nonumber \\&+\,\theta _{k}[f(y^{k})+\langle \nabla f(y^{k}),z^{k+1}-y^{k} \rangle +P_1(z^{k+1})+\frac{L}{2}\theta _{k}\Vert z^{k+1}-z^{k}\Vert ^2]\nonumber \\= & {} (1-\theta _{k})\left[ f(y^{k})+\langle \nabla f(y^{k}),x^{k}-y^{k} \rangle +P_1(x^{k})\right] \nonumber \\&-\,P_2(x^{k}) -\theta _{k}\langle \xi ^k,z^{k+1}-x^{k} \rangle \nonumber \\&+\,\theta _{k}[f(y^{k})+\langle \nabla f(y^{k}),z^{k+1}-y^{k} \rangle +P_1(z^{k+1})+\frac{L}{2}\theta _{k}\Vert z^{k+1}-z^{k}\Vert ^2]\nonumber \\= & {} (1-\theta _{k})\left[ f(y^{k})+\langle \nabla f(y^{k}),x^{k}-y^{k} \rangle +P_1(x^{k})\right] -P_2(x^{k})\nonumber \\&+\,\theta _{k}[f(y^{k})+\langle \nabla f(y^{k}),z^{k+1}-y^{k} \rangle -\langle \xi ^k,z^{k+1}-x^{k} \rangle \nonumber \\&+\,P_1(z^{k+1})+\frac{L}{2}\theta _{k}\Vert z^{k+1}-z^{k}\Vert ^2]\nonumber \\= & {} (1-\theta _{k})\left[ f(y^{k})+\langle \nabla f(y^{k}),x^{k}-y^{k} \rangle +P_1(x^{k})\right] -P_2(x^{k}) +\theta _{k}[f(y^{k})\nonumber \\&+\,\langle \nabla f(y^{k}),z^{k+1}-y^{k} \rangle -\langle \xi ^k,z^{k+1}-y^{k}+y^{k}-x^{k} \rangle \nonumber \\&+\,P_1(z^{k+1})+\frac{L}{2}\theta _{k}\Vert z^{k+1}-z^{k}\Vert ^2] \end{aligned}$$

(25)

where the first inequality is obtained since $\nabla f$ is Lipschitz continuous with the modulus $L> 0$, the second inequality follows from the convexity of $P_1$ and $P_2$ and the facts that $\xi ^k\in \partial P_2(x^k)$ and together with the relation $x^{k+1}-y^k=\theta _{k}(z^{k+1}-z^k)$, and $x^{k+1}=(1-\theta _{k})x^{k}+\theta _{k}z^{k+1}$. Now, we obtain further from (25) that

$$\begin{aligned} F(x^{k+1})&\le (1-\theta _{k})\left[ f(y^{k})+\langle \nabla f(y^{k}),x^{k}-y^{k} \rangle +P_1(x^{k})\right] -P_2(x^{k})\\&+\,\theta _{k}[f(y^{k})+\langle \nabla f(y^{k})-\xi ^k,z^{k+1}-y^{k} \rangle +P_1(z^{k+1})\\&+\,\frac{L}{2}\theta _{k}\Vert z^{k+1}-z^{k}\Vert ^2]-\theta _{k}\langle \xi ^k,y^{k}-x^{k} \rangle . \end{aligned}$$

This together with (24) imply

$$\begin{aligned} \begin{aligned} F(x^{k+1})&\le (1-\theta _{k})\left[ f(y^{k})+\langle \nabla f(y^{k}),x^{k}-y^{k} \rangle +P_1(x^{k})\right] -P_2(x^{k})\\&\quad +\,\theta _{k}[f(y^{k})+\langle \nabla f(y^{k})-\xi ^k,x^{k}-y^{k} \rangle +P_1(x^{k})\\&\quad +\,\frac{L}{2}\theta _{k}\Vert x^{k}-z^{k}\Vert ^2-\frac{L}{2}\theta _{k}\Vert z^{k+1}-x^{k}\Vert ^2] -\theta _{k}\langle \xi ^k,y^{k}-x^{k} \rangle \\&=f(y^{k})+\langle \nabla f(y^{k}),x^{k}-y^{k}\rangle +P_1(x^{k})-P_2(x^{k})\\&\quad +\,\frac{L}{2}{\theta ^2_{k}}\Vert x^{k}-z^{k}\Vert ^2 -\frac{L}{2}{\theta ^2_{k}}\Vert x^{k}-z^{k+1}\Vert ^2. \end{aligned} \end{aligned}$$

(26)

Using the convexity of f,

$$\begin{aligned} f(y^k)+\langle \nabla f(y^{k}),x^{k}-y^{k} \rangle \le f(x^k). \end{aligned}$$

(27)

On the other hand, we have

$$\begin{aligned} x^k-z^k=(1-\theta _{k-1})x^{k-1}+\theta _{k-1}z^k-z^k=(1-\theta _{k-1}) (x^{k-1}-z^k). \end{aligned}$$

This relation and (26), (27) together imply

$$\begin{aligned} \begin{aligned}&[F(x^{k+1})+\frac{\alpha _{k+1}}{2}\Vert x^{k}-z^{k+1}\Vert ^2] - [F(x^{k})+\frac{\alpha _{k}}{2}\Vert x^{k-1}-z^{k}\Vert ^2]\\&\le \, \frac{1}{2}[{\theta ^2_{k}}(1-{\theta _{k-1}})^2L-\alpha _{k}]\Vert x^{k-1}-z^{k}\Vert ^2 +\frac{1}{2}[\alpha _{k+1}-{\theta ^2_{k}}L]\Vert x^{k}-z^{k+1}\Vert ^2. \end{aligned} \end{aligned}$$

(28)

Since ${\theta ^2_{k}}(1-{\theta _{k-1}})^2L -\alpha _{k}\le 0,\alpha _{k+1}-{\theta ^2_{k}}L\le -\delta$, we obtain (6) by (5). $\square$