Skip to main content
SpringerLink
Log in

On the convergence of the iterates of proximal gradient algorithm with extrapolation for convex nonsmooth minimization problems

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

In this paper, we consider the proximal gradient algorithm with extrapolation for solving a class of convex nonsmooth minimization problems. We show that for a large class of extrapolation parameters including the extrapolation parameters chosen in FISTA (Beck and Teboulle in SIAM J Imaging Sci 2:183–202, 2009), the successive changes of iterates go to 0. Moreover, based on the Łojasiewicz inequality, we establish the global convergence of iterates generated by the proximal gradient algorithm with extrapolation with an additional assumption on the extrapolation coefficients. The assumption is general enough to allow the threshold of the extrapolation coefficients to be 1. In particular, we prove the length of the iterates is finite. Finally, we perform numerical experiments on the least squares problems with \(\ell _1\) regularization to illustrate our theoretical results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions invoving analytic features. Math. Program. Ser. B 116, 5–16 (2009)

    Article  Google Scholar 

  2. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka–Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    Article  MathSciNet  Google Scholar 

  3. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward backward splitting, and regularized Gauss Seidel methods. Math. Progam. Ser. A 137, 91–129 (2013)

    Article  MathSciNet  Google Scholar 

  4. Attouch, H., Chbani, Z.: Fast inertial dynamics and FISTA algorithms in convex optimization. Perturbation aspects. arXiv preprint arXiv:1507.01367v1

  5. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)

    Article  MathSciNet  Google Scholar 

  6. Becker, S., Candès, E.J., Grant, M.C.: Templates for convex cone problems with applications to sparse signal recovery. Math. Program. Comput. 3, 165–218 (2011)

    Article  MathSciNet  Google Scholar 

  7. Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17, 1205–1223 (2007)

    Article  Google Scholar 

  8. Borwein, J.M., Lewis, A.: Convex Analysis and Nonlinear Optimization, 2nd edn. Springer, New York (2006)

    Book  Google Scholar 

  9. Boţ, R.I., Csetnek, E.R., László, S.C.: An inertial forward-backward algorithm for the minimization of the sum of two nonconvex functions. EURO J. Comput. Optim. 4, 3–25 (2016)

    Article  MathSciNet  Google Scholar 

  10. Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9, 717–772 (2009)

    Article  MathSciNet  Google Scholar 

  11. Candès, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51, 4203–4215 (2005)

    Article  MathSciNet  Google Scholar 

  12. Chambolle, A., Dossal, Ch.: On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm”. J. Optim. Theory Appl. 166, 968–982 (2015)

    Article  MathSciNet  Google Scholar 

  13. Chen, X., Pan, W., Kwok, J.K., Carbonell, J.G. Accelerated gradient method for multi-task sparse learning problem. In: 2009 Ninth IEEE International Conference on Data Mining

  14. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52, 1289–1306 (2006)

    Article  MathSciNet  Google Scholar 

  15. O’Donoghue, B., Candès, E.J.: Adaptive restart for accelerated gradient schemes. Found. Comput. Math. 15, 715–732 (2015)

    Article  MathSciNet  Google Scholar 

  16. Ji, S., Ye, J.: An accelerated gradient method for trace norm minimization. In: Proceedings of the 26th International Conference on Machine Learning, Montreal, Canada (2009)

  17. Johnstone, P.R., Moulin, P.: A Lyapunov analysis of FISTA with local linear convergence for sparse optimization. arXiv preprint arXiv:1502.02281v4

  18. Kiwiel, K.C.: A bundle bregman proximal method for convex nondifferentiable minimization. Math. Program. 85, 241–258 (1999)

    Article  MathSciNet  Google Scholar 

  19. Li, G., Pong, T.K.: Calculus of the exponent of Kurdyka-Łojasiewicz inequality and its applications to linear convergence of first-order methods. Found. Comput. Math. 18, 1199–1232 (2018)

    Article  MathSciNet  Google Scholar 

  20. Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)

    Article  MathSciNet  Google Scholar 

  21. Liu, H., So, A.M.-C., Wu, W.: Quadratic optimization with orthogonality constraint: explicit Łojasiewicz exponent and linear convergence of retraction-based line-search and stochastic variance-reduced gradient methods. Math. Program. https://doi.org/10.1007/s10107-018-1285-1

  22. Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels. Les Équations aux Dérivées Partielles, Éditions du centre National de la Recherche Scientifique, Paris, pp. 87–89 (1963)

  23. Łojasiewicz, S.: Sur la géométrie semi-et sous-analytique. Ann. L’Institut Fourier 43, 1575–1595 (1993)

    Article  MathSciNet  Google Scholar 

  24. Mäkelä, M.M.: Survey of bundle methods for nonsmooth optimization. Optim. Softw. 17, 1–29 (2002)

    Article  MathSciNet  Google Scholar 

  25. Nesterov, Y.: A method of solving a convex programming problem with convergence rate \({O}(\frac{1}{k^{2}})\). Sov. Math. Dokl. 27, 372–376 (1983)

    Google Scholar 

  26. Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Boston (2004)

    Book  Google Scholar 

  27. Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. Ser. A 103, 127–152 (2005)

    Article  MathSciNet  Google Scholar 

  28. Nesterov, Y.: Gradient methods for minimizing composite objective function. CORE Discussion Paper (2007)

  29. Nesterov, Y.: Dual extrapolation and its applications to solving variational inequalities and related problems. Math. Program. Ser. B 109, 319–344 (2007)

    Article  MathSciNet  Google Scholar 

  30. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    Book  Google Scholar 

  31. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, New York (1998)

    Book  Google Scholar 

  32. Schramm, H., Zowe, J.: A version of the bundle idea for minimizing a nonsmooth functions: conceptual idea, convergence analysis, numercial results. SIAM J. Optim. 2, 121–152 (1992)

    Article  MathSciNet  Google Scholar 

  33. Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. J. Mach. Learn. Res. 17, 1–43 (2016)

    MathSciNet  MATH  Google Scholar 

  34. Tseng, P.: Approximation accuracy, gradient methods, and error bound for structured convex optimization. Math. Program. Ser. B 125, 263–295 (2010)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the editor and two anonymous referees for their helpful comments.

Author information

Authors and Affiliations

  1. Institute of Mathematics, Hebei University of Technology, Tianjin, People’s Republic of China

    Bo Wen

  2. Department of Mathematics, Harbin Institute of Technology, Harbin, People’s Republic of China

    Bo Wen & Xiaoping Xue

  3. Institute for Advanced Study in Mathematics, Harbin Institute of Technology, Harbin, People’s Republic of China

    Xiaoping Xue

Authors
  1. Bo Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoping Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoping Xue.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by the NSFC 11731010, NSFC 11671109, NSFC 11801131 and a scientific grant of Hebei Educational Committee (QN2018101).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, B., Xue, X. On the convergence of the iterates of proximal gradient algorithm with extrapolation for convex nonsmooth minimization problems. J Glob Optim 75, 767–787 (2019). https://doi.org/10.1007/s10898-019-00789-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-019-00789-8

Keywords

Search

Navigation