Abstract
Lets≥1 be an integer andW be the class of all functions having integrable partial derivatives on [0, 1]s. We are interested in the minimum number of neurons in a neural network with a single hidden layer required in order to provide a mean approximation order of a preassignedε>0 to each function inW. We prove that this number cannot be\(\mathcal{O}( \in ^{ - s} log(1/ \in ))\) if a spline-like localization is required. This cannot be improved even if one allows different neurons to evaluate different activation functions, even depending upon the target function. Nevertheless, for anyδ>0, a network with\(\mathcal{O}( \in ^{ - s - \delta } )\) neurons can be constructed to provide this order of approximation, with localization. Analogous results are also valid for otherL p norms.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
A.R. Barron, Universal approximation bounds for superposition of a sigmoidal function, IEEE Trans. Information Theory 39, 1993, 930–945.
A.R. Barron and R.L. Barron, Statistical learning networks: a unified view, in:Symposium on the Interface: Statistics and Computing Science, Reston, Virginia, April, 1988.
D.S. Broomhead and D. Lowe, Multivariable functional interpolation and adaptive networks, Complex Systems 2, 1988, 321–355.
C.K. Chui, X. Li and H.N. Mhaskar, Localized approximation by neural networks, Mathematics of Computation 63, 1994, 607–623.
G. Cybenko, Approximation by superposition of sigmoidal functions, Mathematics of Control, Signal and Systems 2, 1989, 303–314.
R. DeVore, R. Howard and C.A. Micchelli, Optimal nonlinear approximation, Manuscripta Mathematica 63, 1989, 469–478.
F. Girosi, M. Jones and T. Poggio, Regularization theory and neural networks architectures, Neural Computation 7, 1995, 219–269.
K. Hornik, M. Stinchcombe and H. White, Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks, Neural Networks 3, 1990, 551–560.
M. Leshno, V. Lin, A. Pinkus and S. Schocken, Multilayer feedforward networks with a non-polynomial activation function can approximate any function, Neural Networks 6, 1993, 861–867.
H.N. Mhaskar, Approximation properties of a multilayered feedforward artificial neural network, Advances in Computational Mathematics 1, 1993, 61–80.
H.N. Mhaskar, Neural networks for localized approximation of real functions, in:Neural Networks for Signal Processing, III, eds. C.A. Kamm, G.M. Kuhn, B. Yoon, R. Chellappa and S.Y. Kung, IEEE, New York, 1993, pp. 190–196.
H.N. Mhaskar, Neural networks for optimal approximation of smooth and analytic functions, Neural Computation 8, 1996, 164–177.
H.N. Mhaskar, Approximation of real functions using neural networks, in:Proc. of Int. Conf. on Computational Mathematics, New Delhi, India, 1993, eds. H.P. Dikshit and C.A. Micchelli, World Scientific Press, 1994, pp. 267–278.
H.N. Mhaskar and C.A. Micchelli, Approximation by superposition of a sigmoidal function and radial basis functions, Advances in Applied Mathematics 13, 1992, 350–373.
H.N. Mhaskar and C.A. Micchelli, Dimension independent bounds on the degree of approximation by neural networks, IBM Journal of Research and Development 38, 1994, 277–284.
H.N. Mhaskar and C.A. Micchelli, Degree of approximation by neural and translation networks with a single hidden layer, Advances in Applied Mathematics 16, 1995, 151–183.
J. Moody and C. Darken, Fast learning in networks of locally tuned processing units, Neural Computations 1, 1989, 282–294.
T. Poggio and F. Girosi, Networks for approximation and learning, Proceedings of the IEEE 78, 1990.
T. Poggio, F. Girosi and M. Jones, From regularization to radial, tensor and additive splines, in:Neural Networks for Signal Processing, III, 1993, eds. C.A. Kamm, G.M. Buhn, B. Yoon, R. Chellappa and S.Y. Kung, IEEE, New York, 1993, pp. 3–10.
L.L. Schumaker,Spline Functions: Basic Theory, Wiley, New York, 1981.
E.M. Stein,Singular Integrals and Differentiability Properties of Functions, Princeton Univ. Press, Princeton, 1970.
Author information
Authors and Affiliations
Additional information
The research of this author was supported by NSF Grant # DMS 92-0698.
The research of this author was supported, in part, by AFOSR Grant #F49620-93-1-0150 and by NSF Grant #DMS 9404513.
Rights and permissions
About this article
Cite this article
Chui, C.K., Li, X. & Mhaskar, H.N. Limitations of the approximation capabilities of neural networks with one hidden layer. Adv Comput Math 5, 233–243 (1996). https://doi.org/10.1007/BF02124745
Issue Date:
DOI: https://doi.org/10.1007/BF02124745