Abstract
To enhance the generalization of neural network model, we proposed a novel neural network, Minimum Risk Neural Networks (MRNN), whose principle is the combination of minimizing the sum of squares of error and maximizing the classification margin, based on the principle of structural risk minimization. Therefore, the objective function of MRNN is the combination of the sum of squared error and the sum of squares of the slopes of the classification function. Besides, we derived a more sophisticated formula similar to the traditional weight decay technique from the MRNN, establishing a more rigorous theoretical basis for the technique. This study employed several real application examples to test the MRNN. The results led to the following conclusions. (1) As long as the penalty coefficient was in the appropriate range, MRNN performed better than pure MLP. (2) MRNN may perform better in difficult classification problems than MLP using weight decay technique.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Wu, L.Z., Moody, J.: A Smoothing Regularizer for Feedforward and Recurrent Neural Networks. Neural Computation 8(3), 461–489 (1996)
Krogh, A., Hertz, J.A.: A Simple Weight Decay Can Improve Generalization. In: Moody, J.E., Hanson, S.J., Lippmann, R.P. (eds.) Advances in Neural Information Processing Systems, San Mateo, CA, pp. 450–957 (1992)
Krogh, A., Hertz, J.A.: A Simple Weight Decay Can Improve Generalization. In: Advances in Neural Information Processing Systems, vol. 4, pp. 950–957 (1992)
Hinton, G.E., Camp, D.: Keeping the Neural Networks Simple by Minimizing the Description Length of the Weights. In: Proceedings of the Sixth Annual Conference on Computational Learning Theory, pp. 5–13 (1993)
Treadgold, N.K., Gedeon, T.D.: Simulated Annealing and Weight Decay in Adaptive Learning: the SARPROP algorithm. IEEE Transactions on Neural Networks 9(4), 662–668 (1998)
Cortes, F., Vapnik, V.: Support Vector Networks. Machine Learning 20(3), 273–297 (1995)
Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995)
Drucker, H., Wu, D., Vapink, V.: Support Vector Machines for Spam Categorization. IEEE Transactions on Neural Networks 10(5), 1048–1054 (1999)
Burges, C.: A Tutorial on Support Vector Machines for Pattern Recognitionl. Data Mining and Knowledge Discovery 2(2), 121–167 (1998)
Fan, R.E., Chen, P.H., Lin, C.J.: Working Set Selection using Second Order Information for Training Support Vector Machines. The Journal of Machine Learning Research 6, 1889–1918 (2005)
UCI Machine Learning Repository Content Summary (2008), http://archive.ics.uci.edu/ml/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Yeh, IC., Tseng, PY., Huang, KC., Kuo, YH. (2012). Minimum Risk Neural Networks and Weight Decay Technique. In: Huang, DS., Gupta, P., Zhang, X., Premaratne, P. (eds) Emerging Intelligent Computing Technology and Applications. ICIC 2012. Communications in Computer and Information Science, vol 304. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31837-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-31837-5_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31836-8
Online ISBN: 978-3-642-31837-5
eBook Packages: Computer ScienceComputer Science (R0)