Skip to main content
Springer Nature Link
Log in

Cubic regularization in symmetric rank-1 quasi-Newton methods

  • Full Length Paper
  • Published:
Mathematical Programming Computation Aims and scope Submit manuscript

Abstract

Quasi-Newton methods based on the symmetric rank-one (SR1) update have been known to be fast and provide better approximations of the true Hessian than popular rank-two approaches, but these properties are guaranteed under certain conditions which frequently do not hold. Additionally, SR1 is plagued by the lack of guarantee of positive definiteness for the Hessian estimate. In this paper, we propose cubic regularization as a remedy to relax the conditions on the proofs of convergence for both speed and accuracy and to provide a positive definite approximation at each step. We show that the n-step convergence property for strictly convex quadratic programs is retained by the proposed approach. Extensive numerical results on unconstrained problems from the CUTEr test set are provided to demonstrate the computational efficiency and robustness of the approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Anandkumar, A., Ge, R.: Efficient approaches for escaping higher order saddle points in non-convex optimization. (2016). arXiv preprint arXiv:1602.05908

  2. Bellavia, S., Morini, B.: Strong local convergence properties of adaptive regularized methods for nonlinear least squares. IMA J. Numer. Anal. 35, dru021 (2014)

    MathSciNet  Google Scholar 

  3. Benson, H.Y., Shanno, D.F.: Interior-point methods for nonconvex nonlinear programming: cubic regularization. Comput. Optim. Appl. 58, 323 (2014)

    Article  MathSciNet  Google Scholar 

  4. Bianconcini, T., Liuzzi, G., Morini, B., Sciandrone, M.: On the use of iterative methods in cubic regularization for unconstrained optimization. Comput. Optim. Appl. 60(1), 35–57 (2015)

    Article  MathSciNet  Google Scholar 

  5. Bianconcini, T., Sciandrone, M.: A cubic regularization algorithm for unconstrained optimization using line search and nonmonotone techniques. Optim. Methods Softw. 1–28 (2016)

  6. Birgin, E.G., Gardenghi, J.L., Martınez, J.M., Santos, S.A., Toint, Ph. L.: Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models. Report naXys-05-2015, University of Namur, Belgium (2015)

  7. Broyden, C.G.: Quasi-newton methods and their application to function minimisation. Math. Comput. 21(99), 368–381 (1967)

    Article  MathSciNet  Google Scholar 

  8. Broyden, G.C.: The convergence of a class of double-rank minimization algorithms 2. the new algorithm. IMA J. Appl. Math. 6(3), 222–231 (1970)

    Article  MathSciNet  Google Scholar 

  9. Byrd, R.H., Khalfan, H.F., Schnabel, R.B.: Analysis of a symmetric rank-one trust region method. SIAM J. Optim. 6, 1025–1039 (1996)

    Article  MathSciNet  Google Scholar 

  10. Cartis, C., Gould, N.I.M., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Math. Program. Ser. A 127, 245–295 (2011)

    Article  MathSciNet  Google Scholar 

  11. Cartis, C., Gould, N.I.M., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity. Math. Program. Ser. A 130, 295–319 (2011)

    Article  MathSciNet  Google Scholar 

  12. Conn, A.R., Gould, N.I.M., Toint, P.L.: Convergence of quasi-newton matrices generated by the symmetric rank one update. Math. Program. 50(1–3), 177–195 (1991)

    Article  MathSciNet  Google Scholar 

  13. Conn, A.R., Gould, N., Toint, Ph.L.: Constrained and unconstrained testing environment. http://www.cuter.rl.ac.uk/Problems/mastsif.shtml. Accessed 01 Feb 2018

  14. Conn, A.R., Gould, N.I.M., Toint, P.L.: Convergence of quasi-newton matrices generated by the symmetric rank one update. Math. Program. 50, 177–195 (1991)

    Article  MathSciNet  Google Scholar 

  15. Davidon, W.C.: Variance algorithm for minimization. Comput. J. 10(4), 406–410 (1968)

    Article  MathSciNet  Google Scholar 

  16. Davidon, W.C.: Variable metric method for minimization. SIAM J. Optim. 1(1), 1–17 (1991)

    Article  MathSciNet  Google Scholar 

  17. Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Technical report, Argonne National Laboratory (2001)

  18. Dussault, Jean-Pierre: Simple unified convergence proofs for the trust-region and a new arc variant. Technical report, Technical report, University of Sherbrooke, Sherbrooke, Canada (2015)

  19. Feldman, S.I.: A fortran to c converter. In: ACM SIGPLAN Fortran Forum, vol. 9, pp. 21–22. ACM (1990)

  20. Fiacco, A.V., McCormick, G.P.: Nonlinear programming: sequential unconstrained minimization techniques. Research Analysis Corporation, McLean Virginia. Republished in 1990 by SIAM, Philadelphia (1968)

  21. Fletcher, R.: Practical Methods of Optimization. Wiley, Chichester (1987)

    MATH  Google Scholar 

  22. Fletcher, R.: A new approach to variable metric algorithms. Comput. J. 13(3), 317–322 (1970)

    Article  MathSciNet  Google Scholar 

  23. Fletcher, R., Powell, M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 6(2), 163–168 (1963)

    Article  MathSciNet  Google Scholar 

  24. Fourer, R., Gay, D.M., Kernighan, B.W.: AMPL: A Modeling Language for Mathematical Programming. Scientific Press, London (1993)

    MATH  Google Scholar 

  25. Goldfarb, D.: Sufficient conditions for the convergence of a variable metric algorithm. In: Fletcher, R. (ed.) Optimization, pp. 273–281. Academic Press, New York (1969)

    Google Scholar 

  26. Goldfarb, D.: A family of variable-metric methods derived by variational means. Mathematics of computation 24(109), 23–26 (1970)

    Article  MathSciNet  Google Scholar 

  27. Göllner, T., Hess, W., Ulbrich, S.: Geometry optimization of branched sheet metal products. PAMM 12(1), 619–620 (2012)

    Article  Google Scholar 

  28. Gould, N.I.M., Porcelli, M., Toint, P.L.: Updating the regularization parameter in the adaptive cubic regularization algorithm. Comput. Optim. Appl. 53(1), 1–22 (2012)

    Article  MathSciNet  Google Scholar 

  29. Griewank, A.: The modification of Newton’s method for unconstrained optimization by bounding cubic terms. Technical Report NA/12, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1981)

  30. Griewank, A., Fischer, J., Bosse, T.: Cubic overestimation and secant updating for unconstrained optimization of c 2, 1 functions. Optim. Methods Softw. 29(5), 1075–1089 (2014)

    Article  MathSciNet  Google Scholar 

  31. Hsia, Y., Sheu, R.-L., Yuan, Y.-X.: On the p-regularized trust region subproblem (2014). arXiv preprint arXiv:1409.4665

  32. Huang, H.Y.: Unified approach to quadratically convergent algorithms for function minimization. J. Optim. Theory Appl. 5(6), 405–423 (1970)

    Article  MathSciNet  Google Scholar 

  33. Khalfan, H.F., Byrd, R.H., Schnabel, R.B.: A theoretical and experimental study of the symmetric rank-one update. SIAM J. Optim. 3(1), 1–24 (1993)

    Article  MathSciNet  Google Scholar 

  34. Liu, X., Sun, J.: Global convergence analysis of line search interior-point methods for nonlinear programming without regularity assumptions. J. Optim. Theory Appl. 125(3), 609–628 (2005)

    Article  MathSciNet  Google Scholar 

  35. Sha, L., Wei, Z., Li, L.: A trust region algorithm with adaptive cubic regularization methods for nonsmooth convex minimization. Comput. Optim. Appl. 51(2), 551–573 (2012)

    Article  MathSciNet  Google Scholar 

  36. Martınez, J.M., Raydan, M.: Cubic-regularization counterpart of a variable-norm trust-region method for unconstrained minimization. Technical report (2015)

  37. Murtagh, B.A., Sargent, R.W.H.: A constrained minimization method with quadratic convergence. Optimization, pp. 215–246 (1969)

  38. Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Math. Program. Ser. A 108, 177–205 (2006)

    Article  MathSciNet  Google Scholar 

  39. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research. Springer, Berlin (1999)

    Google Scholar 

  40. Oren, S.S., Luenberger, D.G.: Self-scaling variable metric (ssvm) algorithms. Manag. Sci. 20(5), 863–874 (1974)

    Article  Google Scholar 

  41. Oren, S.S., Luenberger, D.G.: Self-scaling variable metric (ssvm) algorithms: Part i: Criteria and sufficient conditions for scaling a class of algorithms. Manag. Sci. 20(5), 845–862 (1974)

    Article  Google Scholar 

  42. Oren, S.S., Spedicato, E.: Optimal conditioning of self-scaling variable metric algorithms. Math. Program. 10(1), 70–90 (1976)

    Article  MathSciNet  Google Scholar 

  43. Powell, M.J.D.: Recent advances in unconstrained optimization. Math. Program. 1, 26–57 (1971)

    Article  MathSciNet  Google Scholar 

  44. Schiela, A.: A flexible framework for cubic regularization algorithms for non-convex optimization in function space. Technical report (2014)

  45. Shanno, D.D., Phua, K.H.: Remark on algorithm 500. minimization of unconstrained multivariate functions. Trans. Math. Softw. 6(4), 618–622 (1980)

    Article  Google Scholar 

  46. Shanno, D.F.: Conditioning of quasi-newton methods for function minimization. Math. Comput. 24(111), 647–656 (1970)

    Article  MathSciNet  Google Scholar 

  47. Shanno, D.F.: Conjugate gradient methods with inexact searches. Math. Oper. Res. 3(3), 244–256 (1978)

    Article  MathSciNet  Google Scholar 

  48. Shanno, D.F., Phua, K.H.: Matrix conditioning and nonlinear optimization. Math. Program. 14(1), 149–160 (1978)

    Article  MathSciNet  Google Scholar 

  49. Sherman, J., Morrison, W.J.: Adjustment of an inverse matrix corresponding to changes in the elements of a given column or a given row of the original matrix. In: Annals of Mathematical Statistics, volume 20(4), pp. 621–621. INST MATHEMATICAL STATISTICS IMS BUSINESS OFFICE-SUITE 7, 3401 INVESTMENT BLVD, HAYWARD, CA 94545 (1949)

  50. Vanderbei, R.J.: AMPL models. http://orfe.princeton.edu/~rvdb/ampl/nlmodels. Accessed 01 August 2016

  51. Vanderbei, R.J., Shanno, D.F.: An interior-point algorithm for nonconvex nonlinear programming. Comput. Optim. Appl. 13, 231–252 (1999)

    Article  MathSciNet  Google Scholar 

  52. Weiser, M., Deuflhard, P., Erdmann, B.: Affine conjugate adaptive Newton methods for nonlinear elastomechanics. Optim. Methods Softw. 22(3), 413–431 (2007)

    Article  MathSciNet  Google Scholar 

  53. Wolfe, P.: Another variable metric method. Technical report, working paper (1967)

  54. Wolfe, P.: Convergence conditions for ascent methods. SIAM Rev. 11(2), 226–235 (1969)

    Article  MathSciNet  Google Scholar 

  55. Wolfe, P.: Convergence conditions for ascent methods. ii: Some corrections. SIAM Rev. 13(2), 185–188 (1971)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

We would like to thank Daniel Bienstock and Andreas Waechter for their handling of the paper as Editor and Associate Editor, respectively, for MPC. We would also like to thank the two anonymous referees whose feedback and suggestions have greatly improved the paper.

Author information

Authors and Affiliations

  1. Drexel University, Philadelphia, PA, USA

    Hande Y. Benson

  2. Emeritus, Rutgers University, New Brunswick, NJ, USA

    David F. Shanno

Authors
  1. Hande Y. Benson
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. David F. Shanno
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Hande Y. Benson.

Appendix

Appendix

See Table 1.

Table 1 Iteration and runtime (in CPU seconds) results for the proposed modified SR1 approach and BFGS on unconstrained NLPs from the CUTEr test set
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Benson, H.Y., Shanno, D.F. Cubic regularization in symmetric rank-1 quasi-Newton methods. Math. Prog. Comp. 10, 457–486 (2018). https://doi.org/10.1007/s12532-018-0136-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12532-018-0136-7

Keywords

Mathematics Subject Classification