Sotiris Konstantinidis
ABSTRACT
The Levenberg-Marquardt (LM) algorithm is a very popular training method in Neural Networks due to its accuracy and robustness. LM outperforms gradient based methods that use direct calculation of the first derivative of the error cost function through back-propagation. In this paper we will examine how the direct computation of the diagonal elements of the Hessian matrix of the error cost function can be used to improve the performance of the original LM algorithm.
- R. Battiti, "First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method," Neural Computation, vol. 4, no. 2, pp. 141--266, 1992. Google Scholar
Digital Library
- K. Levenberg, "A Method for the Solution of Certain Non-Linear Problems in Least Squares," Quarterly of Applied Mathematics, vol. 2, no. 2, pp. 164--168, 1944.Google ScholarCross Ref
- J. Peng, K. Li and G. Irwin, "A new Jacobian matrix for optimal learning of single-layer neural networks," IEEE Trans Neural Netw, vol. 19, no. 1, pp. 119--29, 2008. Google ScholarDigital Library
- Y.-T. Kwak and J. Heesung, "A Layer-by-Layer Levenberg-Marquardt algorithm for Feedforward Multilayer Perceptron," Applied Mathematics & Information Sciences, vol. 6, pp. 505--511, 2011.Google Scholar
- Y. A. LeCun, L. Bottou, G. B. Orr and K.-R. Muller, "Efficient BackProp," in Neural Networks: Tricks of the trade, Springer, 1998, pp. 9--47. Google ScholarDigital Library
- G. Tesauro and B. Janssens, "Scaling Relationships in Back-propagation Learninng," Complex Systems, vol. 2, pp. 39--44, 1988. Google ScholarDigital Library
- K. J. Lang and M. J. Witbrock, "Learning to Tell Two Spirals Apart," in In Proceeding of the Connectioninst Models Summer School, 1988.Google Scholar
Index Terms
- Enhancing the Levenberg-Marquardt Method in Neural Network training using the direct computation of the Error Cost Function Hessian
Recommendations
Levenberg–Marquardt multi-classification using hinge loss function
AbstractIncorporating higher-order optimization functions, such as Levenberg–Marquardt (LM) have revealed better generalizable solutions for deep learning problems. However, these higher-order optimization functions suffer from very large ...
A Study on Hepatitis Disease Diagnosis Using Multilayer Neural Network with Levenberg Marquardt Training Algorithm
In this study, a hepatitis disease diagnosis study was realized using neural network structure. For this purpose, a multilayer neural network structure was used. Levenberg---Marquardt algorithm was used as training algorithm for the weights update of ...
A Radial Basis Function Neural Network (RBFNN) Approach for Structural Classification of Thyroid Diseases
The thyroid is a gland that controls key functions of body. Diseases of the thyroid gland can adversely affect nearly every organ in human body. The correct diagnosis of a patient's thyroid disease clarifies the choice of drug treatment and also allows ...
Comments