A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

Abstract

In this paper, we consider the problem that minimizing the sum of a nonsmooth function with a smooth one in the nonconvex setting, which arising in many contemporary applications such as machine learning, statistics, and signal/image processing. To solve this problem, we propose a new nonmonotone accelerated proximal gradient method with variable stepsize strategy. Note that incorporating inertial term into proximal gradient method is a simple and efficient acceleration technique, while the descent property of the proximal gradient algorithm will lost. In our algorithm, the iterates generated by inertial proximal gradient scheme are accepted when the objective function values decrease or increase appropriately; otherwise, the iteration point is generated by proximal gradient scheme, which makes the function values on a subset of iterates are decreasing. We also introduce a variable stepsize strategy, which does not need a line search or does not need to know the Lipschitz constant and makes the algorithm easy to implement. We show that the sequence of iterates generated by the algorithm converges to a critical point of the objective function. Further, under the assumption that the objective function satisfies the Kurdyka–Łojasiewicz inequality, we prove the convergence rates of the objective function values and the iterates. Moreover, numerical results on both convex and nonconvex problems are reported to demonstrate the effectiveness and superiority of the proposed method and stepsize strategy.

Appendices

Appendix A: Proof of Lemma 3.1

Proof

By the adaptive non-monotone stepsize strategy, if ${\lambda _{k + 1}}$ is generated by (7), then,

$$\begin{aligned}{\lambda _{k + 1}} - {\lambda _k} < 0 \le E\left( k \right) ;\end{aligned}$$

otherwise,

$$\begin{aligned}{\lambda _{k + 1}} - {\lambda _k} \le E\left( k \right) .\end{aligned}$$

Hence, for any $i \ge 1$ we have

$$\begin{aligned} {\lambda _{i + 1}} - {\lambda _i} \le \textrm{E}\left( i \right) . \end{aligned}$$

(88)

Denote that

$$\begin{aligned} {\lambda _{i + 1}} - {\lambda _i} = {\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) ^ + } - {\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) ^ - },\mathrm{{where}}{\left( \cdot \right) ^ + } = \max \{ 0, \cdot \} ,{\left( \cdot \right) ^ - } = - \min \{ 0, \cdot \} , \end{aligned}$$

(89)

we have

$$\begin{aligned} {\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) ^ + } \le \textrm{E}\left( i \right) ,\forall i = 1,2, \ldots , \end{aligned}$$

(90)

which implies that $\sum \nolimits _{i = 1}^\infty {{{\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) }^ + }}$ is convergent from the fact that $\sum \nolimits _{i = 1}^\infty {\textrm{E}\left( i \right) }$ is a convergent positive series.

The convergence of $\sum \nolimits _{i = 1}^\infty {{{\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) }^ - }}$ also can be proved as follows.

Assume by contradiction that $\sum \nolimits _{i = 1}^\infty {{{\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) }^ - }} \mathrm{{ = + }}\infty .$ Based on the convergence of $\sum \nolimits _{i = 1}^\infty {{{\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) }^ + }}$ and the equality

$$\begin{aligned} {\lambda _{k + 1}} - {\lambda _1}\mathrm{{ = }}\sum \limits _{i = 1}^k {\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) = \sum \limits _{i = 1}^k {{{\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) }^ + }} - \sum \limits _{i = 1}^k {{{\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) }^ - }} }. \end{aligned}$$

(91)

We can easily deduce $\mathop {\lim }\nolimits _{k \rightarrow \infty } {\lambda _k} = - \infty ,$ which is a contradiction with ${\lambda _{k}} > 0,$ $\forall k \ge 1.$ Therefore, $\sum \nolimits _{i = 1}^\infty {{{\left( {{\lambda _{i + 1}} - {\lambda _i}} \right) }^ - }}$ is a convergent series. Then, in view of (91), we obtain the sequence $\left\{ {{\lambda _k}} \right\}$ is convergent.

We can easily prove that $\forall k \ge 1,$ ${\lambda _k} \ge \min \left\{ {{\lambda _1},\frac{{{\mu _1}}}{{{L_f}}}} \right\}$ holds by induction. $\square$

Appendix B: Proof of Lemma 3.2

Proof

Suppose that the conclusion is not true, there exists a $\left\{ {{k_j}} \right\}$ with ${k_j} \rightarrow \infty$ such that

$$\begin{aligned} 2\left( {f\left( {\varvec{x_{{k_j}}}} \right) - f\left( {\varvec{y_{{k_j}}}} \right) - \left\langle {\nabla f\left( {\varvec{y_{{k_j}}}} \right) ,\varvec{x_{{k_j}}} - \varvec{y_{{k_j}}}} \right\rangle } \right) > \frac{{{\mu _0}}}{{{\lambda _{{k_j}}}}}{\left\| {\varvec{x_{{k_j}}} - \varvec{y_{{k_j}}}} \right\| ^2} \end{aligned}$$

(92)

holds. Then, based on the scheme of adaptive nonmonotone stepsize, we have

$$\begin{aligned} {\lambda _{{k_j} + 1}} = \frac{{{\mu _1} \cdot {{\left\| {\varvec{x_{{k_j}}} - \varvec{y_{{k_j}}}} \right\| }^2}}}{{2\left( {f\left( {\varvec{x_{{k_j}}}} \right) - f\left( {\varvec{y_{{k_j}}}} \right) - \left\langle {\nabla f\left( {\varvec{y_{{k_j}}}} \right) ,\varvec{x_{{k_j}}} - \varvec{y_{{k_j}}}} \right\rangle } \right) }}. \end{aligned}$$

(93)

From the above two formulas, it is easy to obtain

$$\begin{aligned}&{\left\| {\varvec{x_{{k_j}}} - \varvec{y_{{k_j}}}} \right\| ^2} < \frac{{2{\lambda _{{k_j}}}}}{{{\mu _0}}}\left( {f\left( {\varvec{x_{{k_j}}}} \right) - f\left( {\varvec{y_{{k_j}}}} \right) - \left\langle {\nabla f\left( {\varvec{y_{{k_j}}}} \right) ,\varvec{x_{{k_j}}} - \varvec{y_{{k_j}}}} \right\rangle } \right) \nonumber \\&\quad = \frac{{{\mu _1}{\lambda _{{k_j}}}}}{{{\mu _0}{\lambda _{{k_j} + 1}}}}{\left\| {\varvec{x_{{k_j}}} - \varvec{y_{{k_j}}}} \right\| ^2}. \end{aligned}$$

(94)

Based on the facts that ${\mu _1} < {\mu _0}$ and the sequence $\left\{ {{\lambda _k}} \right\}$ is convergent, we can get

$$\begin{aligned} \frac{{{\mu _1}{\lambda _{{k_j}}}}}{{{\mu _0}{\lambda _{{k_j} + 1}}}} \rightarrow \frac{{{\mu _1}}}{{{\mu _0}}} < 1, \end{aligned}$$

(95)

which contradicts with (94). Therefore, (15) will holds constantly after a finite step ${\hat{k}}.$ $\square$