Abstract
We attempt to quantify the significance of increasing the number of neurons in the hidden layer of a feedforward neural network architecture using the singular value decomposition (SVD). Through this, we extend some well-known properties of the SVD in evaluating the generalizability of single hidden layer feedforward networks (SLFNs) with respect to the number of hidden neurons. The generalization capability of the SLFN is measured by the degree of linear independency of the patterns in hidden layer space, which can be indirectly quantified from the singular values obtained from the SVD, in a post-learning step.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Huang, S., Huang, Y.: Bounds on Number of Hidden Neurons of Multilayer Perceptrons in Classification and Recognition. IEEE International Symposium on Circuits and Systems 4, 2500–2503 (1990)
Sartori, M., Antsaklis, P.: A Simple Method to Derive Bounds on the Size and to Train Multi-Layer Neural Networks. IEEE Trans. on Neural Networks 2(4), 467–471 (1991)
Tamura, S.: Capabilities of a Three-Layer Feedforward Neural Network. In: Proc. Int. Joint Conf. on Neural Networks (1991)
Tamura, S., Tateishi, M.: Capabilities of a Four-Layered Feedforward Neural Network: Four Layers Versus Three. IEEE Trans. on Neural Networks 8(2), 251–255 (1997)
Huang, G., Babri, H.: Upper Bounds on the Number of Hidden Neurons in Feedforward Networks with Arbitrary Bounded Nonlinear Activation Functions. IEEE Trans. Neural Networks 9(1) (1998)
Huang, G.: Learning Capability and Storage Capacity of Two-Hidden-Layer Feedforward Networks. IEEE Trans. on Neural Networks 14(2), 274–281 (2003)
Cover, T.: Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition. IEEE Trans. Electronic. Comput. 14, 326–334 (1965)
Hayashi, M.: A Fast Algorithm for the Hidden Units in a Multilayer Perceptron. Proc. Int. Joint Conf. on Neural Networks 1, 339–342 (1993)
Tamura, S., Tateishi, M., Matsumoto, M., Akita, S.: Determination of the Number of Redundant Hidden Units in a Three-Layered Feedforward Neural Network. Proc. Int. Joint Conf. on Neural Networks 1, 335–338 (1993)
Psichogios, D., Ungar, L.: SVD-NET: An Algorithm that Automatically Selects Network Structure. IEEE Trans. on Neural Networks 5(3), 513–515 (1994)
Xiang, C., Ding, S., Lee, T.: Geometrical Interpretation and Architecture Selection of MLP. IEEE Trans. Neural Networks 16(1), 84–96 (2005)
Stewart, G.: Determining Rank in the Presence of Error. Technical Report (TR-92-108) Institute for Advanced Computer Studies (TR-2972) Department of Computer Science, University of Maryland, College Park (October 1992)
Golub, G.H., Van Loan, C.: Matrix Computations, 3rd edn. John Hopkins University Press, Baltimore (1999)
Hansen, P.: Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion. SIAM, Philadelphia (1998)
Golub: G., Klema, V., Stewart, G.: Rank Degeneracy and the Least Squares Problem. Technical Report (STAN-CS-76-559), Computer Science Department, School of Humanities and Sciences, Stanford University (1976)
Ljung, L.: System Identification: Theory for the User. Prentice-Hall, Englewood Cliffs (1987)
Konstantinides, K., Yao, K.: Statistical Analysis of Effective Singular Values in Matrix Rank Determination. IEEE Trans. on Acoustics, Speech and Signal Processing 36(5), 757–763 (1988)
Moller, M.: A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning. Neural Networks 6, 525–533 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Teoh, E.J., Xiang, C., Tan, K.C. (2006). Estimating the Number of Hidden Neurons in a Feedforward Network Using the Singular Value Decomposition. In: Wang, J., Yi, Z., Zurada, J.M., Lu, BL., Yin, H. (eds) Advances in Neural Networks - ISNN 2006. ISNN 2006. Lecture Notes in Computer Science, vol 3971. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11759966_126
Download citation
DOI: https://doi.org/10.1007/11759966_126
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34439-1
Online ISBN: 978-3-540-34440-7
eBook Packages: Computer ScienceComputer Science (R0)