Skip to main content
Log in

Local analysis of a spectral correction for the Gauss-Newton model applied to quadratic residual problems

  • Original Paper
  • Published:
Numerical Algorithms Aims and scope Submit manuscript

Abstract

A simple spectral correction for the Gauss-Newton model applied to nonlinear least squares problems is presented. Such a correction consists in adding a sign-free multiple of the identity to the Hessian of the Gauss-Newton model, being the multiple based on spectral approximations for the Hessians of the residual functions. A detailed local convergence analysis is provided for the resulting method applied to the class of quadratic residual problems. Under mild assumptions, the proposed method is proved to be convergent for problems for which the convergence of the Gauss-Newton method might not be ensured. Moreover, the rate of linear convergence is proved to be better than the Gauss-Newton’s one for a class of non-zero residue problems. These theoretical results are illustrated by numerical examples with quadratic and non-quadratic residual problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bartholomew-Biggs, M.C.: The estimation of the Hessian matrix in nonlinear least squares problems with non-zero residuals. Math. Program. 12, 67–80 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  2. Barzilai, J., Borwein, J.: Two-point step size gradient methods. IMA J. Numer. Anal. 8(1), 141–148 (1988). doi:10.1093/imanum/8.1.141

    Article  MathSciNet  MATH  Google Scholar 

  3. Behling, R., Fischer, A.: A unified local convergence analysis of inexact constrained Levenberg-Marquardt methods. Optim. Lett 6(5), 927–940 (2012). doi:10.1007/s11590-011-0321-3

    Article  MathSciNet  MATH  Google Scholar 

  4. Bellavia, S., Cartis, C., Gould, N., Morini, B., Toint, Ph.: L.: Convergence of a regularized Euclidean residual algorithm for nonlinear least-squares. SIAM J. Numer. Anal. 48, 1–29 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bellavia, S., Morini, B.: Strong local convergence properties of adaptive regularized methods for nonlinear least squares. IMA J. Numer. Anal. 35(2), 947–968 (2015). doi:10.1093/imanum/dru021 [http://imajna.oxfordjournals.org/content/early/2014/05/14/imanum.dru021.abstract]

    Article  MathSciNet  MATH  Google Scholar 

  6. Birgin, E.G., Martínez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim., 1196–1211 (2000). doi:10.1.1.145.9706

  7. Brown, K.M., Dennis Jr., J.E.: Derivative free analogues of the Levenberg-Marquardt and Gauss algorithms for nonlinear least squares approximation. Numer. Math. 18, 289–297 (1971/72)

    Article  MathSciNet  MATH  Google Scholar 

  8. Chen, P.: Hessian matrix vs. Gauss-newton Hessian matrix. SIAM J. Numer. Anal 49(4), 1417–1435 (2011). doi:10.1137/100799988

    Article  MathSciNet  MATH  Google Scholar 

  9. Dennis Jr., J.E.: Non-linear least squares and equations, pp 269–312. Academic Press, London (1977)

  10. Dennis Jr., J.E., Schnabel, R.B.: Numerical methods for unconstrained optimization and nonlinear equations, Classics in Applied Mathematics, vol. 16. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA. doi:10.1137/1.9781611971200, Corrected reprint of the 1983 original (1996)

  11. Dennis Jr., J.E., Song-Bai, S., Vu, P.A.: A memoryless augmented Gauss-Newton method for nonlinear least-squares problems. J. Comput. Math 6(4), 355–374 (1988)

    MathSciNet  MATH  Google Scholar 

  12. Ferreira, O.P., Gonçalves, M.L.N., Oliveira, P.R.: Local convergence analysis of the Gauss-Newton method under a majorant condition. J. Complex. 27 (1), 111–125 (2011). doi:10.1016/j.jco.2010.09.001

    Article  MathSciNet  MATH  Google Scholar 

  13. Ferreira, O.P., Gonçalves, M.L.N., Oliveira, P.R.: Convergence of the Gauss-Newton method for convex composite optimization under a majorant condition. SIAM J. Optim. 23(3), 1757–1783 (2013). doi:10.1137/110841606

    Article  MathSciNet  MATH  Google Scholar 

  14. Gill, P.E., Murray, W.: Algorithms for the solution of the nonlinear least-squares problem. SIAM J. Numer. Anal. 15(5), 977–992 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  15. Gould, N. I. M.: Ph.l.toint: FILTRANE, a Fortran 95 filter-trust-region package for solving nonlinear least-squares and nonlinear feasibility problems. ACM Trans. Math. Softw. 33(1), 23 (2007)

    Article  Google Scholar 

  16. Gratton, S., Lawless, A.S., Nichols, N.K.: Approximate Gauss-Newton methods for nonlinear least squares problems. SIAM J. Optim 18(1), 106–132 (2007). doi:10.1137/050624935

    Article  MathSciNet  MATH  Google Scholar 

  17. Kanzow, C., Yamashita, N., Fukushima, M.: Levenberg–marquardt methods with strong local convergence properties for solving nonlinear equations with convex constraints. J. Comput. Appl. Math. 172(2), 375–397 (2004). doi:10.1016/j.cam.2004.02.013. http://www.sciencedirect.com/science/article/pii/ S0377042704001256

    Article  MathSciNet  MATH  Google Scholar 

  18. Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2, 164–168 (1944)

    MathSciNet  MATH  Google Scholar 

  19. Marquardt, D.: An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11(2), 431–441 (1963). doi:10.1137/0111030

    Article  MathSciNet  MATH  Google Scholar 

  20. McKeown, J.: Specialised versus general purpose algorithms for minimising functions that are sums of squared terms. Math. Program. 9, 57–68 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  21. Meyer, C.D.: Matrix analysis and applied linear algebra. SIAM philadelphia (2000)

  22. Morrison, D.D.: Methods for nonlinear least squares problems and convergence proofs. In: Lorell, J., Yagi, F. (eds.) Proceedings of the seminar on tracking programs and orbit determination, pp 1–9. Jet Propulsion Laboratory, Pasadena, CA, USA (1960)

    Google Scholar 

  23. Nesterov, Y.: Modified Gauss-Newton scheme with worst-case guarantees for global performance. Optim. Methods Softw. 22, 469–483 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  24. Nocedal, J., Wright, S.J.: Numerical optimization. Springer Series in Operations Research. Springer, New York (1999). doi:10.1007/b98874

    Google Scholar 

  25. Raydan, M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optim. 7(1), 26–33 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  26. Salzo, S., Villa, S.: Convergence analysis of a proximal Gauss-Newton method. Comput. Optim. Appl 53, 557–589 (2012). doi:10.1007/s10589-012-9476-9

    Article  MathSciNet  MATH  Google Scholar 

  27. Smith, F.B., Shanno, D.F.: An improved Marquardt procedure for nonlinear regressions. Technometrics 13(1), 63–74 (1971). doi:10.1080/00401706.1971.10488754

    Article  MATH  Google Scholar 

  28. Wilkinson, J.H.: The algebraic eigenvalue problem. Oxford university press new york (1988)

  29. Yuan, Y.X.: Recent advances in numerical methods for nonlinear equations and nonlinear least squares. Numer. Algebra Control Optim. 1(1), 15–34 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  30. Zhang, H.C., Conn, A.R.: On the local convergence of a derivative-free algorithm for least-squares minimization. Comput. Optim. Appl. 51, 481–507 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  31. Zhang, H.C., Conn, A.R., Scheinberg, K.: A derivative-free algorithm for the least-squares minimization. SIAM J. Optim. 20, 3555–3576 (2010)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Douglas S. Gonçalves.

Additional information

Partially supported by CNPq (Grant 304032/2010-7), FAPESP (Grants 2013/05475-7 and 2013/07375-0) and PRONEX-Optimization.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gonçalves, D.S., Santos, S.A. Local analysis of a spectral correction for the Gauss-Newton model applied to quadratic residual problems. Numer Algor 73, 407–431 (2016). https://doi.org/10.1007/s11075-016-0101-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11075-016-0101-3

Keywords

Mathematics Subject Classification

Navigation