Abstract
Any number of linearly independent plane waves can be obtained by scaling, shifting and/or rotating a plane wave created from an activation function. We investigate the condition which ensures linear independence of the plane waves. In the case all the three procedures are combined, the linear independence can be proved by ad hoc methods for most of commonly used activation functions. The result can be used for proving structural uniqueness of neural networks and for implementing finite mapping by neural networks.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
E.K. Blum and L.K. Li, Approximation theory and feedforward networks, Neural Networks 4, 1991, 511–515.
G. Cybenko, Approximation by superpositions of a sigmoidal function, Math. Control Signal Systems 2, 1989, 303–314.
K. Funahashi, On the approximate realization of continuous mappings by neural networks, Neural Networks 2, 1989, 183–192.
L.M. Gel'fand and G.E. Shilov,Generalized Functions 2 (English translation), Academic Press, 1968.
K. Hornik, M. Stinchcombe and H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2, 1989, 359–366.
K. Hornik, M. Stinchcombe and H. White, Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks, Neural Networks 3, 1990, 551–560.
Y. Ito, Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory, Neural Networks 4, 1991, 385–394.
Y. Ito, Approximation of functions on a compact set by finite sums of a sigmoid function without scaling, Neural Networks 4, 1991, 817–826.
Y. Ito, Finite mapping by neural networks and truth functions, Math. Sci. 17, 1992, 69–77.
Y. Ito, Approximation of continuous functions on Rd by linear combinations of shifted rotations of a sigmoid function with and without scaling, Neural Networks 5, 1992, 105–115.
Y. Ito, Approximations of differentiable functions and their derivatives on compact sets by neural networks, Math. Sci. 18, 1993, 11–19.
Y. Ito, Approximation capability of layered neural networks with sigmoid units on two layers, Neural Comp. 6, 1994, 1233–1243.
Y. Ito, Differentiable approximation by means of the Radon transform and its applications to neural networks, J. Comput. Appl. Math. 55, 1994, 31–50.
Y. Ito and K. Saito, Superposition of linearly independent functions and finite mappings by neural networks, Math. Sci. 19, 1995, to appear.
W.S. McCulloch and W. Pitts, A logical calculus of the ideas immanent in nervous activity, Bull. Math. Biophys. 5, 1943, 115–133.
D.E. Rumelhart, G.E. Hinton and R.J. William, Learning representations by back-propagating errors, Nature 323, 1986, 533–536.
L. Schwartz,Théorie des Distributions, Hermann, Paris, 1966.
H.J. Sussmann, Uniqueness of the weights for minimal feedforward nets with a given input-output map, Neural Networks 5, 1992, 589–593.
Author information
Authors and Affiliations
Additional information
This paper was supported in part by Grant 07252213 from Ministry of Education and Culture, Japan.
Rights and permissions
About this article
Cite this article
Ito, Y. Nonlinearity creates linear independence. Adv Comput Math 5, 189–203 (1996). https://doi.org/10.1007/BF02124743
Issue Date:
DOI: https://doi.org/10.1007/BF02124743