Abstract
In this work, a novel modification on the standard Levenberg-Marquardt (LM) algorithm is proposed for eliminating the necessity of the validation set for avoiding overfitting, thereby shortening the training time while maintaining the test performance. The idea is that training points with smaller magnitudes of training errors are much liable to cause overfitting and that they should be excluded from the training set at each epoch. The proposed modification has been compared to the standard LM on three different problems. The results shown that even though the modified LM does not use the validation data set, it reduces the training time without compromising the test performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sarle, W.S.: Stopped Training and Other Remedies for Overfitting. In: 27th Symposium on the Interface, pp. 352–360 (1995). https://doi.org/10.1.1.42.3920
Poggio, T., Girosi, F.: Networks for Approximation and Learning. Proc. IEEE 78(9), 1481–1497 (1990). https://doi.org/10.1109/5.58326
Zur, R.M., Jiang, Y., Pesce, L.L., Drukker, K.: Noise Injection for Training ANNs: A Comparison with Weight Decay and Early Stopping. Medical Phys. 36(10), 4810–4818 (2009). https://doi.org/10.1118/1.3213517
Liu, Y., Starzyk, J.A., Zhu, Z.: Optimized Approximation Algorithm in Neural Networks without Overfitting. IEEE Trans. Neural Networks 19(6), 983–995 (2008). https://doi.org/10.1109/TNN.2007.915114
Nocedal, J., Wright, S.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer, New York (2006)
Piotrowski, A.P., Napiorkowski, J.J.: A comparison of methods to avoid overfitting in NNs training in the case of catchment runoff modelling. J. Hydrol. 476, 97–111 (2013). https://doi.org/10.1016/j.jhydrol.2012.10.019
Hagan, M.T., Demuth, H.B., Beale, M.: Neural Network Design. PWS Publishing Co., Boston (1996)
Kwak, Y., Hwang, J., Yoo, C.: A new damping strategy of Levenberg-Marquardt algorithm for Multilayer Perceptrons. Neural Network World 21(4), 327–340 (2011). https://doi.org/10.14311/NNW.2011.21.020
Sunspot Index and Long-term Solar Observations. http://www.sidc.be/silso
Purwar, S., Kar, I.N., Jha, A.N.: On-line system identification of complex systems using chebyshev neural networks. Appl. Soft Comput. 7, 364–372 (2007). https://doi.org/10.1016/j.asoc.2005.08.001
Wu, W., Chou, Y.S.: Adaptive feedforward and feedback control of nonlinear time-varying uncertain systems. Int. J. Control 72(12), 1127–1138 (1999). https://doi.org/10.1080/002071799220489
Acknowledgements
This work is supported by Pamukkale University Scientific Research Projects Council under the grand number 2018KRM002-035.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Iplikci, S., Bilgi, B., Menemen, A., Bahtiyar, B. (2019). A Novel Modification on the Levenberg-Marquardt Algorithm for Avoiding Overfitting in Neural Network Training. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning. ICANN 2019. Lecture Notes in Computer Science(), vol 11728. Springer, Cham. https://doi.org/10.1007/978-3-030-30484-3_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-30484-3_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30483-6
Online ISBN: 978-3-030-30484-3
eBook Packages: Computer ScienceComputer Science (R0)