research-article

Enhancing the Levenberg-Marquardt Method in Neural Network training using the direct computation of the Error Cost Function Hessian

Authors:

Sotiris Konstantinidis,

Pythagoras Karampiperis,

Miguel-Angel SiciliaAuthors Info & Claims

EANN '15: Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (INNS)

Article No.: 2, Pages 1 - 5

https://doi.org/10.1145/2797143.2797162

Published: 25 September 2015 Publication History

Get Access

Abstract

The Levenberg-Marquardt (LM) algorithm is a very popular training method in Neural Networks due to its accuracy and robustness. LM outperforms gradient based methods that use direct calculation of the first derivative of the error cost function through back-propagation. In this paper we will examine how the direct computation of the diagonal elements of the Hessian matrix of the error cost function can be used to improve the performance of the original LM algorithm.

References

[1]

R. Battiti, "First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method," Neural Computation, vol. 4, no. 2, pp. 141--266, 1992.

Digital Library

Google Scholar

[2]

K. Levenberg, "A Method for the Solution of Certain Non-Linear Problems in Least Squares," Quarterly of Applied Mathematics, vol. 2, no. 2, pp. 164--168, 1944.

Crossref

Google Scholar

[3]

J. Peng, K. Li and G. Irwin, "A new Jacobian matrix for optimal learning of single-layer neural networks," IEEE Trans Neural Netw, vol. 19, no. 1, pp. 119--29, 2008.

Digital Library

Google Scholar

[4]

Y.-T. Kwak and J. Heesung, "A Layer-by-Layer Levenberg-Marquardt algorithm for Feedforward Multilayer Perceptron," Applied Mathematics & Information Sciences, vol. 6, pp. 505--511, 2011.

Google Scholar

[5]

Y. A. LeCun, L. Bottou, G. B. Orr and K.-R. Muller, "Efficient BackProp," in Neural Networks: Tricks of the trade, Springer, 1998, pp. 9--47.

Digital Library

Google Scholar

[6]

G. Tesauro and B. Janssens, "Scaling Relationships in Back-propagation Learninng," Complex Systems, vol. 2, pp. 39--44, 1988.

Digital Library

Google Scholar

[7]

K. J. Lang and M. J. Witbrock, "Learning to Tell Two Spirals Apart," in In Proceeding of the Connectioninst Models Summer School, 1988.

Google Scholar

Cited By

View all

Nasar FBakar NMahmud J(2024)Prediction of In-vitro Dissolution Profile for Oral Dispersible Film (ODF) using Artificial Neural Network2024 IEEE 22nd Student Conference on Research and Development (SCOReD)10.1109/SCOReD64708.2024.10872643(192-196)Online publication date: 19-Dec-2024
https://doi.org/10.1109/SCOReD64708.2024.10872643
Sharma PSengupta JSuri P(2019)WLI Fuzzy Clustering and Adaptive Lion Neural Network (ALNN) for Cloud Intrusion DetectionInternational Journal of Distributed Artificial Intelligence10.4018/IJDAI.201901010111:1(1-17)Online publication date: 1-Jan-2019
https://doi.org/10.4018/IJDAI.2019010101
Iqbal UGhazali R(2016)Chebyshev Multilayer Perceptron Neural Network with Levenberg Marquardt-Back Propagation Learning for Classification TasksRecent Advances on Soft Computing and Data Mining10.1007/978-3-319-51281-5_17(162-170)Online publication date: 29-Dec-2016
https://doi.org/10.1007/978-3-319-51281-5_17

Index Terms

Enhancing the Levenberg-Marquardt Method in Neural Network training using the direct computation of the Error Cost Function Hessian

Recommendations

Levenberg–Marquardt multi-classification using hinge loss function
Abstract
Incorporating higher-order optimization functions, such as Levenberg–Marquardt (LM) have revealed better generalizable solutions for deep learning problems. However, these higher-order optimization functions suffer from very large ...
A Study on Hepatitis Disease Diagnosis Using Multilayer Neural Network with Levenberg Marquardt Training Algorithm

In this study, a hepatitis disease diagnosis study was realized using neural network structure. For this purpose, a multilayer neural network structure was used. Levenberg---Marquardt algorithm was used as training algorithm for the weights update of ...
A Radial Basis Function Neural Network (RBFNN) Approach for Structural Classification of Thyroid Diseases

The thyroid is a gland that controls key functions of body. Diseases of the thyroid gland can adversely affect nearly every organ in human body. The correct diagnosis of a patient's thyroid disease clarifies the choice of drug treatment and also allows ...

Comments

Information & Contributors

Information

Published In

EANN '15: Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (INNS)

September 2015

266 pages

ISBN:9781450335805

DOI:10.1145/2797143

Editors:
Lazaros Iliadis
Democritus University of Thrace, Greece
,
Chrisina Jane
Coventry University, UK

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Aristotle University of Thessaloniki
INNS: International Neural Network Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

16th EANN workshops

16th EANN workshops: 16th Engineering Applications of Neural Networks Conference WORKSHOPS

September 25 - 28, 2015

Island, Rhodes, Greece

Acceptance Rates

EANN '15 Paper Acceptance Rate 36 of 60 submissions, 60%;

Overall Acceptance Rate 36 of 60 submissions, 60%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
112
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Nasar FBakar NMahmud J(2024)Prediction of In-vitro Dissolution Profile for Oral Dispersible Film (ODF) using Artificial Neural Network2024 IEEE 22nd Student Conference on Research and Development (SCOReD)10.1109/SCOReD64708.2024.10872643(192-196)Online publication date: 19-Dec-2024
https://doi.org/10.1109/SCOReD64708.2024.10872643
Sharma PSengupta JSuri P(2019)WLI Fuzzy Clustering and Adaptive Lion Neural Network (ALNN) for Cloud Intrusion DetectionInternational Journal of Distributed Artificial Intelligence10.4018/IJDAI.201901010111:1(1-17)Online publication date: 1-Jan-2019
https://doi.org/10.4018/IJDAI.2019010101
Iqbal UGhazali R(2016)Chebyshev Multilayer Perceptron Neural Network with Levenberg Marquardt-Back Propagation Learning for Classification TasksRecent Advances on Soft Computing and Data Mining10.1007/978-3-319-51281-5_17(162-170)Online publication date: 29-Dec-2016
https://doi.org/10.1007/978-3-319-51281-5_17

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Levenberg–Marquardt multi-classification using hinge loss function

A Study on Hepatitis Disease Diagnosis Using Multilayer Neural Network with Levenberg Marquardt Training Algorithm

A Radial Basis Function Neural Network (RBFNN) Approach for Structural Classification of Thyroid Diseases

Comments

Information

Published In

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations