Abstract
Complexity of shallow (one-hidden-layer) networks representing finite multivariate mappings is investigated. Lower bounds are derived on growth of numbers of network units and sizes of output weights in terms of variational norms of mappings to be represented. Probability distributions of mappings whose computations require large networks are described. It is shown that due to geometrical properties of high-dimensional Euclidean spaces, representation of almost any randomly chosen function on a sufficiently large domain by a shallow network with perceptrons requires untractably large network. Concrete examples of such functions are constructed using Hadamard matrices.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Fine, T.L.: Feedforward Neural Network Methodology. Springer, Heidelberg (1999)
Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation 18, 1527–1554 (2006)
Bengio, Y.: Learning deep architectures for AI. Foundations and Trends in Machine Learning 2, 1–127 (2009)
Ito, Y.: Finite mapping by neural networks and truth functions. Mathematical Scientist 17, 69–77 (1992)
Micchelli, C.A.: Interpolation of scattered data: Distance matrices and conditionally positive definite functions. Constructive Approximation 2, 11–22 (1986)
Kůrková, V., Savický, P., Hlaváčková, K.: Representations and rates of approximation of real-valued Boolean functions by neural networks. Neural Networks 11, 651–659 (1998)
Kainen, P.C., Kůrková, V., Vogt, A.: A Sobolev-type upper bound for rates of approximation by linear combinations of heaviside plane waves. Journal of Approximation Theory 147, 1–10 (2007)
Kainen, P.C., Kůrková, V., Sanguineti, M.: Complexity of Gaussian radial-basis networks approximating smooth functions. J. of Complexity 25, 63–74 (2009)
Kainen, P.C., Kůrková, V., Sanguineti, M.: Dependence of computational models on input dimension: Tractability of approximation and optimization tasks. IEEE Trans. on Information Theory 58, 1203–1214 (2012)
Kůrková, V.: Complexity estimates based on integral transforms induced by computational units. Neural Networks 33, 160–167 (2012)
Maiorov, V.: On best approximation by ridge functions. J. of Approximation Theory 99, 68–94 (1999)
Maiorov, V., Pinkus, A.: Lower bounds for approximation by MLP neural networks. Neurocomputing 25, 81–91 (1999)
Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numerica 8, 143–195 (1999)
Roychowdhury, V., Siu, K.Y., Orlitsky, A.: Neural models and spectral methods. In: Roychowdhury, V., Siu, K., Orlitsky, A. (eds.) Theoretical Advances in Neural Computation and Learning, pp. 3–36. Springer, New York (1994)
Kůrková, V.: Representations of highly-varying functions by one-hidden-layer networks. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014, Part I. LNCS (LNAI), vol. 8467, pp. 67–76. Springer, Heidelberg (2014)
Kůrková, V., Sanguineti, M.: Complexity of shallow networks representing functions with large variations. In: Wermter, S., Weber, C., Duch, W., Honkela, T., Koprinkova-Hristova, P., Magg, S., Palm, G., Villa, A.E.P. (eds.) ICANN 2014. LNCS, vol. 8681, pp. 331–338. Springer, Heidelberg (2014)
Cover, T.: Geometrical and statistical properties of systems of linear inequailities with applictions in pattern recognition. IEEE Trans. on Electronic Computers 14, 326–334 (1965)
Schläfli, L.: Theorie der vielfachen Kontinuität. Zürcher & Furrer, Zürich (1901)
Sylvester, J.: Thoughts on inverse orthogonal matrices, simultaneous sign successions, and tessellated pavements in two or more colours, with applications to Newton’s rule, ornamental tile-work, and the theory of numbers. Philosophical Magazine 34, 461–475 (1867)
Erdös, P., Spencer, J.H.: Probabilistic Methods in Combinatorics. Academic Press (1974)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Kůrková, V. (2015). Complexity of Shallow Networks Representing Finite Mappings. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2015. Lecture Notes in Computer Science(), vol 9119. Springer, Cham. https://doi.org/10.1007/978-3-319-19324-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-19324-3_4
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-19323-6
Online ISBN: 978-3-319-19324-3
eBook Packages: Computer ScienceComputer Science (R0)