Abstract
This paper presents a unified control Liapunov function (CLF) approach to the design of heavy ball with friction (HBF) and conjugate gradient (CG) neural networks that aim to minimize scalar nonconvex functions that have continuous first- and second-order derivatives and a unique global minimum. This approach leads naturally to the design of second-order differential equations which are the mathematical models of the corresponding implementations as neural networks. Preliminary numerical simulations indicate that, on a small suite of benchmark test problems, a continuous version of the well known conjugate gradient algorithm, designed by the proposed CLF method, has better performance than its HBF competitor.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cichocki, A., Unbehauen, R.: Neural Networks for Optimization and Signal Processing. John Wiley, Chichester (1993)
Chu, M.T.: Linear algebra algorithms as dynamical systems. Acta Numerica, 1–86 (2008)
Bhaya, A., Kaszkurewicz, E.: Control perspectives on numerical algorithms and matrix problems. In: Advances in Control. SIAM, Philadelphia (2006)
Bhaya, A., Kaszkurewicz, E.: Iterative methods as dynamical systems with feedback control. In: Proc. 42nd IEEE Conference on Decision and Control, Maui, Hawaii, USA, December 2003, pp. 2374–2380 (2003)
Bhaya, A., Kaszkurewicz, E.: A control-theoretic approach to the design of zero finding numerical methods. IEEE Transactions on Automatic Control 52(6), 1014–1026 (2007)
Pazos, F., Bhaya, A., Kaszkurewicz, E.: Unified control Liapunov function based design of neural networks that aim at global minimization of nonconvex functions. In: Proc. of the International Joint Conference on Neural Networks, Atlanta, U.S.A. (2009)
Polyak, B.T.: Some methods of speeding up the convergence of iterative methods. USSR Computational Mathematics and Mathematical Physics 4(5), 1–17 (1964); Zh. Vychisl. Mat. Mat. Fiz., pp. 791-803 (Russian edn.)
Polyak, B.T.: Introduction to optimization. Optimization software, New York (1987)
Attouch, H., Goudou, X., Redont, P.: The heavy ball with friction method. The continuous dynamical system. Communications in contemporary math. 2, 1–34 (2000)
Cabot, A.: Inertial gradient-like dynamical system controlled by a stabilizing term. Journal of Optimization Theory and Applications 120(2), 275–303 (2004)
Alvarez, F., Attouch, H., Bolte, J., Redont, P.: A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics. J. Math. Pures Appl. 81, 747–779 (2002)
Bhaya, A., Kaszkurewicz, E.: Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method. Neural Networks 17(1), 65–71 (2004)
Cortés, J.: Discontinuous dynamical systems. A tutorial on notions of solutions, nonsmooth analysis, and stability. IEEE Control systems magazine, 36–73 (June 2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bhaya, A., Pazos, F.A., Kaszkurewicz, E. (2009). Comparative Study of the CG and HBF ODEs Used in the Global Minimization of Nonconvex Functions. In: Alippi, C., Polycarpou, M., Panayiotou, C., Ellinas, G. (eds) Artificial Neural Networks – ICANN 2009. ICANN 2009. Lecture Notes in Computer Science, vol 5768. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04274-4_69
Download citation
DOI: https://doi.org/10.1007/978-3-642-04274-4_69
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04273-7
Online ISBN: 978-3-642-04274-4
eBook Packages: Computer ScienceComputer Science (R0)