Skip to main content
SpringerLink
Log in

Greatest Descent Algorithms in Unconstrained Optimization

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

We show that, for an unconstrained optimization problem, the long-term optimal trajectory consists of a sequence of greatest descent directions and a Newton method in the final iteration. The greatest descent direction can be computed approximately by using a Levenberg-Marquardt like formula. This implies the view that the Newton method approximates a Levenberg-Marquardt like formula at a finite distance from the minimum point, instead of the standard view that the Levenberg-Marquadt formula is a way to approximate the Newton method. With the insight gained from this analysis, we develop a two-dimensional version of a Levenberg-Marquardt like formula. We make use of the two numerically largest components of the gradient vector to define here new search directions. In this way, we avoid the need of inverting a high-dimensional matrix. This reduces also the storage requirements for the full Hessian matrix in problems with a large number of variables.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999)

    Book  MATH  Google Scholar 

  2. Goh, B.S.: Algorithms for unconstrained optimization problems via control theory. J. Optim. Theory Appl. 92, 581–604 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  3. Vincent, T.L., Grantham, W.J.: Nonlinear and Optimal Control Systems. Wiley, New York (1997)

    Google Scholar 

  4. Kalman, R.E., Bertram, J.E.: Control system analysis and design via the second method of Liapunov. II. Discrete-time systems. ASME J. Basic Eng. 82, 394–400 (1960)

    MathSciNet  Google Scholar 

  5. LaSalle, J.P.: The Stability of Dynamical Systems. SIAM, Philadelphia (1976)

    MATH  Google Scholar 

  6. Goh, B.S.: Management and Analysis of Biological Populations. Elsevier, Amsterdam (1980)

    Google Scholar 

  7. Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    Article  MATH  MathSciNet  Google Scholar 

  8. Yuan, Y.X.: Step-sizes for the gradient method, Edited by K.S. Liu, Z.P. Xin and S.T. Yau. In: Third International Congress of Chinese Mathematicians (AMS/IP Studies in Advanced Mathematics), pp. 785–796 (2008)

  9. Fletcher, R.: On the Barzilai-Borwein method. Research Report NA207, University of Dundee, UK (2001)

  10. Dai, Y.H.: Alternative step gradient method. Optimization 52, 395–415 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  11. Kelley, C.T.: Iterative Methods in Optimization. SIAM, Philadelphia (1999)

    Google Scholar 

  12. Fan, J.Y., Yuan, Y.X.: On the quadratic convergence of the Levenberg-Marquardt method without nonsingularity assumption. Computing 74, 23–39 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  13. Grantham, W.J.: Trajectory following optimization by gradient transformation differential equations. In: Proceedings of 42nd IEEE Conference on Decision and Control, pp. 5496–5501. Hawaii (2003)

  14. Grantham, W.J.: Gradient transformation trajectory following algorithms for determining stationary min-max saddle points. In: Jørgensen, S.J., Vincent, T.L., Quincampoix, M. (eds.) Advances in Dynamic Game Theory and Applications, pp. 639–657. Birkhaüser, Boston (2006)

    Google Scholar 

  15. Fletcher, R.: Practical Methods of Optimization. Wiley, New York (1987)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mathematics Dept., Nanjing University, Nanjing, Jiangsu, 210093, China

    B. S. Goh

Authors
  1. B. S. Goh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to B. S. Goh.

Additional information

Communicated T.L. Vincent.

The author thanks Mark Wu, Professors Sanyang Liu, Junmin Li, Shuisheng Zhou and Feng Ye for support and help in this research as well as the referees for helpful comments.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Goh, B.S. Greatest Descent Algorithms in Unconstrained Optimization. J Optim Theory Appl 142, 275–289 (2009). https://doi.org/10.1007/s10957-009-9533-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-009-9533-4

Keywords

Search

Navigation