Abstract
In the parameter space of MLP(J), multilayer perceptron with J hidden units, there exist flat areas called singular regions created by applying reducibility mappings to the optimal solution of MLP(\(J-1\)). Since such singular regions cause serious stagnation of learning, a learning method to avoid singular regions has been desired. However, such avoiding does not guarantee the quality of the final solutions. This paper proposes a new learning method which does not avoid but makes good use of singular regions to stably and successively find excellent solutions commensurate with MLP(J). The proposed method worked well in our experiments using artificial and real data sets.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Amari S (1998) Natural gradient works efficiently in learning. Neural Comput 10(2):251–276
Amari S, Park H, Fukumizu K (2000) Adaptive method of realizing natural gradient learning for multilayer perceptrons. Neural Comput 12(6):1399–1409
Cousseau F, Oseki T, Amari S (2008) Dynamics of learning in multilayer perceptrons near singularities. IEEE Trans Neural Networks 19(8):1313–1328
Duda RO, Hart PE, Stork DG (2001) Pattern classification, 2nd edn. Wiley, New York
Fukumizu K, Amari S (2000) Local minima and plateaus in hierarchical structure of multilayer perceptrons. Neural Netw 1(3):317–327
Hamey LGC (1998) XOR has no local minima: a case study in neural network error surface. Neural Netw 11(4):669–681
Hecht-Nielsen H (1990) Neurocomputing. Addison–Wesley Publishing Company, Reading
Minnett RCJ, Smith AT, Hecht-Nielsen R (2011) Neural network tomography: network replication from output surface geometry. Neural Netw 24(5):484–492
Luenberger DG (1984) Linear and nonlinear programming. Addison–Wesley Publishing Company, Reading
Nakano R, Saito K (2002) Discovering polynomials to fit multivariate data having numeric and nominal variables. LNAI 2281:482–493
Nakano R, Satoh S, Ohwaki T (2011) Learning method utilizing singular region of multilayer perceptron. In: Proceedings of the 3rd International Conference on Neural Computation Theory and Applications, Paris, pp. 106–111
Saito K, Nakano R (1997) Partial BFGS update and efficient step-length calculation for three-layer neural networks. Neural Comput 9(1):239–257
Sussmann HJ (1992) Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Netw 5(4):589–593
Wan W (2006) Implementing online natural gradient learning: problems and solutions. IEEE Trans Neural Netw 17(2):317–329
Watanabe S (2008) A formula of equations of states in singular learning machines. In: Proceedings of the International Joint Conference on Neural Networks 2008, Hong Kong, pp. 2099–2106
Watanabe S (2009) Algebraic geometry and statistical learning theory. Cambridge University Press, Cambridge
Acknowledgments
This work was supported by Grants-in-Aid for Scientific Research (C) 22500212 and Chubu University Grant 24IS27A.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Satoh, S., Nakano, R. Fast and Stable Learning Utilizing Singular Regions of Multilayer Perceptron. Neural Process Lett 38, 99–115 (2013). https://doi.org/10.1007/s11063-013-9283-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-013-9283-z