Skip to main content
SpringerLink
Log in

On the near optimality of the stochastic approximation of smooth functions by neural networks

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

We consider the problem of approximating the Sobolev class of functions by neural networks with a single hidden layer, establishing both upper and lower bounds. The upper bound uses a probabilistic approach, based on the Radon and wavelet transforms, and yields similar rates to those derived recently under more restrictive conditions on the activation function. Moreover, the construction using the Radon and wavelet transforms seems very natural to the problem. Additionally, geometrical arguments are used to establish lower bounds for two types of commonly used activation functions. The results demonstrate the tightness of the bounds, up to a factor logarithmic in the number of nodes of the neural network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R.A. Adams, Sobolev Spaces (Academic Press, New York, 1975).

    MATH  Google Scholar 

  2. A.R. Barron, Universal approximation bounds for superposition of sigmoidal function, IEEE Trans. Inform. Theory 39 (1993) 930–945.

    Article  MATH  MathSciNet  Google Scholar 

  3. T. Chen, H. Chen and R. Liu, Approximation capability in C( _Rn) by multilayer feedforward networks and related problems, IEEE Trans. Neural Networks 6(1) (1995) 25–30.

    Article  MATH  Google Scholar 

  4. C.K. Chui, X. Li and H.N. Mhaskar, Some limitations of neural networks with one hidden layer, Adv. Comput. Math. 5 (1996) 233–244.

    Article  MATH  MathSciNet  Google Scholar 

  5. G. Cybenko, Approximation by superposition of sigmoidal functions, Math. Control Signals Systems 2 (1989) 303–314.

    MATH  MathSciNet  Google Scholar 

  6. R.A. DeVore, R. Howard and C.A. Micchelli, Optimal nonlinear approximation, Manuscripta Math. 63 (1989) 469–478.

    Article  MathSciNet  Google Scholar 

  7. R.A. DeVore, K. Oskolkov, P. Petrushev, Approximation by feed-forward neural networks, Ann. Numer. Math. 4 (1997) 261–287.

    MATH  MathSciNet  Google Scholar 

  8. I. Daubechies, Ten Lectures on Wavelets (SIAM Press, 1992).

  9. B. Delyon, A. Juditsky and A. Benveniste, Accuracy analysis for wavelet approximations, IEEE Trans. Neural Networks 6 (1995) 332–348.

    Article  Google Scholar 

  10. F. Girosi, Regularization theory, radial basis functions and networks, in: From Statistics to Neural Networks, eds. V. Cherkassy, J.H. Friedman and H. Wechsler (Springer, 1994) pp. 166–187.

  11. P. Hall and C.C. Heyde, Martingale Limit Theory and Its Applications (Academic Press, Orlando, 1980).

    Google Scholar 

  12. E. Hernandez and G.L. Weiss, First Course on Wavelets (CRC Press, 1996).

  13. S. Helgason, The Radon Transform (Birkh¨auser, Boston, 1980).

    MATH  Google Scholar 

  14. M. Karpinski and A.J. Macintyre, Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks, J. Comput. System Sci. 54 (1997) 1600–176.

    Article  MathSciNet  Google Scholar 

  15. V.Ya. Lin and A. Pinkus, Fundamentality of ridge functions, J. Approx. Theory 75 (1993) 295–311.

    Article  MATH  MathSciNet  Google Scholar 

  16. V. Maiorov, On best approximation by ridge functions, J. Approx. Theory 99 (1999) 68–94.

    Article  MATH  MathSciNet  Google Scholar 

  17. V. Maiorov and J. Ratsaby, On the degree of approximation using manifolds of finite pseudodimension, J. Construct. Approx. 15 (1999) 291–300.

    Article  MATH  MathSciNet  Google Scholar 

  18. V. Maiorov and A. Pinkus, Lower bounds for approximation by MLP neural networks, Neurocomputing 25 (1998) 81–91.

    Article  Google Scholar 

  19. R. Meir and V. Maiorov, On the optimality of neural network approximation using incremental algorithms, Technical Report CC-257, Department of Electrical Engineering, Technion, Israel, October 1998. To appear in IEEE Trans. Neural Networks (2000).

    Google Scholar 

  20. H.N. Mhaskar, Neural networks for optimal approximation of smooth and analytic functions, Neural Comput. 8 (1996) 164–177.

    Google Scholar 

  21. H.N. Mhaskar and C.A. Micchelli, Approximation by superposition of a sigmoidal function and radial basis functions, Adv. Appl. Math. 16 (1992) 350–373.

    Article  MathSciNet  Google Scholar 

  22. H.N. Mhaskar and C.A. Micchelli, Dimension independent bounds on the degree of approximation by neural networks, IBM J. Res. Develop. 38 (1994) 277–284.

    Article  MATH  Google Scholar 

  23. O.P. Misra and J.L. Lavoine, Transform Analysis of Generalized Functions, North-Holland Mathematics Studies, Vol. 119 (North-Holland, Amsterdam, 1986).

    Google Scholar 

  24. P.P. Petrushev, Approximation by ridge functions and neural networks, SIAM J. Math. Anal. 30 (1998) 155–189.

    Article  MathSciNet  Google Scholar 

  25. A. Pinkus, n-Widths in Approximation Theory (Springer, Berlin, 1985).

    MATH  Google Scholar 

  26. E. Stein and G. Weiss, Introduction to Fourier Analysis on Euclidean Spaces (Princeton Univ. Press, Princeton, NJ, 1971).

    MATH  Google Scholar 

  27. H. Triebel, Interpolation Theory of Function Spaces and Differential Operators (VEB Verlag, Berlin, 1978).

    Google Scholar 

  28. M. Vidyasagar, A Theory of Learning and Generalization (Springer, London, 1997).

    MATH  Google Scholar 

  29. V. Vapnik, The Nature of Statistical Learning Theory (Springer, New York, 1995).

    MATH  Google Scholar 

  30. A.G. Vitushkin, Estimation of the Complexity of the Tabulation Problem (Fizmatgiz, Moscow, 1959).

    Google Scholar 

  31. H.E. Warren, Lower bounds for approximation by nonlinear manifold, Trans. Amer. Math. Soc. 133 (1968) 167–178.

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Technion, Haifa, 32000, Israel

    V.E. Maiorov

  2. Department of Electrical Engineering, Technion, Haifa, 32000, Israel

    R. Meir

Authors
  1. V.E. Maiorov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. R. Meir
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Maiorov, V., Meir, R. On the near optimality of the stochastic approximation of smooth functions by neural networks. Advances in Computational Mathematics 13, 79–103 (2000). https://doi.org/10.1023/A:1018993908478

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1018993908478

Search

Navigation