Skip to main content
SpringerLink
Log in

A family of optimal weighted conjugate-gradient-type methods for strictly convex quadratic minimization

  • Original Paper
  • Published:
Numerical Algorithms Aims and scope Submit manuscript

Abstract

We introduce a family of weighted conjugate-gradient-type methods, for strictly convex quadratic functions, whose parameters are determined by a minimization model based on a convex combination of the objective function and its gradient norm. This family includes the classical linear conjugate gradient method and the recently published delayed weighted gradient method as the extreme cases of the convex combination. The inner cases produce a merit function that offers a compromise between function-value reduction and stationarity which is convenient for real applications. We show that each one of the infinitely many members of the family exhibits q-linear convergence to the unique solution. Moreover, each one of them enjoys finite termination and an optimality property related to the combined merit function. In particular, we prove that if the n × n Hessian of the quadratic function has p < n different eigenvalues, then each member of the family obtains the unique global minimizer in exactly p iterations. Numerical results are presented that demonstrate that the proposed family is promising and exhibits a fast convergence behavior which motivates the use of preconditioning strategies, as well as its extension to the numerical solution of general unconstrained optimization problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. The SuiteSparse Matrix Collection tool–box is available in https://sparse.tamu.edu/.

References

  1. Andreani, R., Raydan, M.: Properties of the delayed weighted gradient method. Comput. Optim. Appl. 78, 167–180 (2021)

    Article  MathSciNet  Google Scholar 

  2. Barzilai, J., Borwein, J: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    Article  MathSciNet  Google Scholar 

  3. Brezinski, C: Hybrid methods for solving systems of equations. NATO ASI Ser. C Math. Phys. Sci.-Adv. Study Inst. 508, 271–290 (1998)

    MathSciNet  MATH  Google Scholar 

  4. Brezinski, C., Redivo-Zaglia, M: Hybrid procedures for solving linear systems. Numer. Math. 67, 1–19 (1994)

    Article  MathSciNet  Google Scholar 

  5. Burdakov, O., Dai, Y.-H., Huang, N: Stabilized Barzilai-Borwein method. J. Comput. Math. 37, 916–936 (2019)

    Article  MathSciNet  Google Scholar 

  6. Cauchy, A: Méthode générale pour la résolution des systemes d’équations simultanées. Comptes Rendus Sci. Paris 25, 536–538 (1847)

    Google Scholar 

  7. Dai, Y.-H., Fletcher, R.: On the asymptotic behaviour of some new gradient methods. Math. Program. Ser. A 13, 541–559 (2005)

    Article  MathSciNet  Google Scholar 

  8. Dai, Y.-H., Huang, Y., Liu, X.: A family of spectral gradient methods for optimization. Comput. Optim. Appl. 74, 43–65 (2019)

    Article  MathSciNet  Google Scholar 

  9. Davis, T.-A., Hu, Y.: The University of Florida sparse matrix collection. ACM Trans. Math. Softw. 38, 1–25 (2011)

    MathSciNet  MATH  Google Scholar 

  10. De Asmundis, R., Di Serafino, D., Riccio, F., Toraldo, G: On spectral properties of steepest descent methods. IMA J. Numer. Anal. 33(4), 1416–1435 (2013)

    Article  MathSciNet  Google Scholar 

  11. De Asmundis, R., Di Serafino, D., Hager, W., Toraldo, G., Zhang, H: An efficient gradient method using the Yuan steplength. Comput. Optim. Appl. 59, 541–563 (2014)

    Article  MathSciNet  Google Scholar 

  12. Di Serafino, D., Ruggiero, V., Toraldo, G., Zanni, L: On the steplength selection in gradient methods for unconstrained optimization. Appl. Math. Comput. 318, 176–195 (2018)

    MathSciNet  MATH  Google Scholar 

  13. Fletcher, R.: On the Barzilai-Borwein method. In: Qi, L., Teo, K., Yang, X. (eds.) Optimization and Control with Applications, vol. 96, pp 235–256. Series in Applied Optimization, Kluwer (2005)

  14. Fletcher, R: A limited memory steepest descent method. Math. Program. Ser. A 135, 513–436 (2012)

    Article  MathSciNet  Google Scholar 

  15. Frassoldati, G., Zanni, L., Zanghirati, G.: New adaptive stepsize selections in gradient methods. J. Ind. Manag. Optim. 4(2), 299–312 (2008)

    Article  MathSciNet  Google Scholar 

  16. Friedlander, A., Martínez, J.M., Molina, B., Raydan, M.: Gradient method with retards and generalizations. SIAM J. Numer. Anal. 36(1), 275–289 (1999)

    Article  MathSciNet  Google Scholar 

  17. Gonzaga, C., Schneider, R.M.: On the steepest descent algorithm for quadratic functions. Comput. Optim. Appl. 63(2), 523–542 (2016)

    Article  MathSciNet  Google Scholar 

  18. Hestenes, M.R., Stiefel, E.: Method of conjugate gradient for solving linear system. J. Res. Nat. Bur. Stand. 49, 409–436 (1952)

    Article  MathSciNet  Google Scholar 

  19. Huang, Y., Dai, Y.-H., Liu, X.-W., Zhang, H: Gradient methods exploiting spectral properties. Optim. Meth. Soft. 35(4), 681–705 (2020)

    Article  MathSciNet  Google Scholar 

  20. Lemaréchal, C., Sagastizábal, C.: Practical aspects of the Moreau–Yosida regularization: theoretical preliminaries. SIAM J. Optim. 7(2), 367–385 (1997)

    Article  MathSciNet  Google Scholar 

  21. Liu, Z., Liu, H., Dong, X.: An efficient gradient method with approximate optimal stepsize for the strictly convex quadratic minimization problem. Optimization 67(3), 427–440 (2018)

    Article  MathSciNet  Google Scholar 

  22. Luenberger, D.: Introduction to Linear and Nonlinear Programming, 2nd ed. Addison Wesley, Amsterdam (1984)

    MATH  Google Scholar 

  23. Moreau, J. -J.: Propriétés des applications “prox”. C. R. Acad. Sci. Paris 256, 1069–1071 (1963)

    MathSciNet  MATH  Google Scholar 

  24. Moreau, J.-J.: Proximité et dualité dans un espace Hilbertien. Bull. Soc. Math. France 22, 5–35 (1965)

    MATH  Google Scholar 

  25. Nocedal, J., Sartenaer, A., Zhu, C: On the behavior of the gradient norm in the steepest descent method. Comput. Optim. Appl. 22, 5–35 (2002)

    Article  MathSciNet  Google Scholar 

  26. Nocedal, J., Wright, S.: Numerical Optimization, 2nd ed. Springer, New York (2006)

    MATH  Google Scholar 

  27. Oviedo-Leon, H.F.: A delayed weighted gradient method for strictly convex quadratic minimization. Comput. Optim. Appl. 74, 729–746 (2019)

    Article  MathSciNet  Google Scholar 

  28. Oviedo, H., Dalmau, O., Herrera, R.: A hybrid gradient method for strictly convex quadratic programming, to appear in Numerical Linear Algebra with Applications, 28(4), https://doi.org/10.1002/nla.2360https://doi.org/10.1002/ https://doi.org/10.1002/nla.2360nla.2360 (2020)

  29. Planiden, C., Wang, X.: Proximal Mappings and Moreau Envelopes of Single-Variable Convex Piecewise Cubic Functions and Multivariable Gauge Functions. In: Hosseini, S., Mordukhovich, B., Uschmajew, A. (eds.) Nonsmooth Optimization and Its Applications, International Series of Numerical Mathematics, vol. 170, pp 89–130. Springer Nature, Birkhäuser (2019)

  30. Steihaug, T: The conjugate gradient method and trust regions in large scale optimization. SIAM J. Numer. Anal. 20, 626–637 (1983)

    Article  MathSciNet  Google Scholar 

  31. Stella, L., Themelis, A., Patrinos, P: Forward–backward quasi-Newton methods for nonsmooth optimization problems. Comput. Optim. Appl. 67, 443–487 (2017)

    Article  MathSciNet  Google Scholar 

  32. Trefethen, L.N., Bau, D.: Numerical Linear Algebra. SIAM, Philadelphia (1997)

    Book  Google Scholar 

  33. Yuan, Y.: A new stepsize for the steepest descent method. J. Comput. Math. 24, 149–156 (2006)

    MathSciNet  MATH  Google Scholar 

  34. Zhou, B., Gao, L., Dai, Y.-H.: Gradient methods with adaptive step-sizes. Comput. Optim. Appl. 35, 69–86 (2006)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors are very grateful to two anonymous referees whose constructive remarks have improved the quality of the paper. In particular, one referee suggested considering dense matrices in Section 5, and the connection to the Moreau envelope developed in Remark 2 was inspired by a comment from the other referee.

Funding

The first author was financially supported by FGV (Fundação Getulio Vargas) through the excellence post–doctoral fellowship program. The second author was financially supported by FAPESP (Projects 2013/05475-7 and 2017/18308-2) and CNPq (Project 301888/2017-5). The third author was financially supported by the Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology) through the project UIDB/MAT/00297/2020 (Centro de Matemática e Aplicações).

Author information

Authors and Affiliations

  1. Escola de Matemática Aplicada, Fundação Getulio Vargas (FGV/EMAp), Rio de Janeiro, RJ, Brazil

    Harry Oviedo

  2. Department of Applied Mathematics, IMECC-UNICAMP, University of Campinas, Distrito Barão Geraldo, 13083-859, Campinas, SP, Brazil

    Roberto Andreani

  3. Centro de Matemática e Aplicações (CMA), FCT, UNL, 2829–516, Caparica, Portugal

    Marcos Raydan

Authors
  1. Harry Oviedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roberto Andreani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcos Raydan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marcos Raydan.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Oviedo, H., Andreani, R. & Raydan, M. A family of optimal weighted conjugate-gradient-type methods for strictly convex quadratic minimization. Numer Algor 90, 1225–1252 (2022). https://doi.org/10.1007/s11075-021-01228-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11075-021-01228-0

Keywords

Search

Navigation