Elsevier

Neural Networks

Volume 9, Issue 6, August 1996, Pages 947-956
Neural Networks

CONTRIBUTED ARTICLE
An Integral Representation of Functions Using Three-layered Networks and Their Approximation Bounds

https://doi.org/10.1016/0893-6080(96)00000-7Get rights and content

Abstract

Neural networks are widely known to provide a method of approximating nonlinear functions. In order to clarify its approximation ability, a new theorem on an integral transform of ridge functions is presented. By using this theorem, an approximation bound, which evaluates the quantitative relationship between the approximation accuracy and the number of elements in the hidden layer, can be obtained. This result shows that the approximation accuracy depends on the smoothness of target functions. It also shows that the approximation methods which use ridge functions are free from the “curse of dimensionality”. Copyright © 1996 Elsevier Science Ltd.

Section snippets

INTRODUCTION

In the middle of the 1980s, computational research on neural networks was revitalized by the works of the parallel distributed processing (PDP) group (Rumelhart et al., 1986). In this movement, multi-layered networks having sigmoidal functions together with back-propagation learning played an important role. The numerous examples provided by the PDP group attracted the interest of many other researchers, and a large number of subsequent computer simulations have shown that the multi-layered

INTEGRAL TRANSFORM USING RIDGE FUNCTIONS

First, define ridge functions.

Definition 1: When a function F : Rm → R is written as:F(x)=G(a·x−b) with a vector a ϵ Rm, a real number b ∈ R and an appropriate function G : R → R, it is called a ridge function.

In other words, a ridge function takes the same value on certain hyper-planes in Rm whose normal vectors are parallel to a (see for example Fig. 1). Clearly, the input-output relation of a neuron in a conventional artificial neural network, i.e., weighted sum and a sigmoidal activation

APPLICATION TO THREE-LAYERED NETWORKS

From the previous result, we can evaluate some aspects of approximating functions using three-layered networks.

CONCLUSION

We found an integral representation of functions using ridge functions. Especially when a bell-shaped function is used as the activation function of the hidden units of a three-layered network, there exists a clear and direct correspondence between the integral representation and the three-layered network. It gives an intuitive interpretation about the structure of the three-layered networks.

Based on this interpretation and the random coding technique, we gave an approximation bound of

Acknowledgements

The author would like to give very special thanks to the review for his useful comments simplifying the proofs and his careful examination of the manuscript. The author would like to thank Professor S. Amari, Professor S. Yoshizawa, Dr K. R. Müller and D. Harada for helpful suggestions and discussions. The present work is supported in part by Grant-in-Aid for Scientific Research in Priority Areas on Higher-Order Brain Information Processing from the Ministry of Education, Science and Culture of

References (11)

There are more references available in the full text version of this article.

Cited by (0)

View full text
Copyright © 1996 Elsevier Science Ltd. All rights reserved.