Multivariate neural network operators with sigmoidal activation functions
Introduction
Neural networks (NNs) with one hidden layer can be represented as where, for , the ’s, , are the threshold values, the ’s, , are the weights, and the ’s are the coefficients. Here is the inner product in , and is the activation function of the network, see Chui and Li (1992), Costarelli and Spigler (submitted for publication), Jones (1988), Lenze (1992), Li (1996), Li and Micchelli (2000), Light (1993), Makovoz (1998), Mhaskar and Micchelli (1995) and Pinkus (1999). The activation function usually is a sigmoidal function. Neural networks are extensively used in Approximation Theory (Barron, 1993, Chen, 1993, Costarelli and Spigler, 2013b, Cybenko, 1989, Gao and Xu, 1993, Girosi and Anzellotti, 1993, Gnecco, 2012, Gnecco and Sanguineti, 2011, Hahm and Hong, 2002, Kainen and Kurková, 2009, Kurková, 2012, Lewicki and Marino, 2003, Lewicki and Marino, 2004, Mhaskar and Micchelli, 1992).
Constructive multivariate approximation algorithms based on sigmoidal functions are important since they play a central role in typical applications of neurocomputing processes concerning high-dimensional data. Applications of NNs with sigmoidal functions in Numerical Analysis, for instance, to the numerical solution of Volterra integral and integro-differential equations by suitable collocation methods were shown in Costarelli and Spigler, in press-a, Costarelli and Spigler, in press-b.
Anastassiou (1997) was the first to establish NN approximations for continuous functions, providing estimates for the rate of convergence, using NN operators of the Cardaliagnet–Euvrard type. He used the modulus of continuity of the function being approximated, to produce Jackson type inequalities. Subsequently, Anastassiou studied NN operators activated by the hyperbolic tangent as well as the logistic function, in both, the univariate and the multivariate case, see Anastassiou, 2011a, Anastassiou, 2011b, Anastassiou, 2011c, Anastassiou, 2011d, Anastassiou, 2012.
In this paper, we study the convergence, as well as the order of approximation, of a family of linear positive multivariate neural network operators, activated by sigmoidal functions. This work represents the extension to the multivariate case of the results established in Costarelli and Spigler (2013a) for univariate functions. We treat by a unified approach all cases studied by Anastassiou, and moreover, we can apply our theory to other useful sigmoidal functions, such as, for instance, the ramp function (Cao and Chen, 2012, Cheang, 2010), and many others (Costarelli & Spigler, submitted for publication).
We first study pointwise and uniform convergence for functions defined on bounded intervals of . In addition, we study the order of approximation for functions belonging to certain Lipschitz classes by means of our NN operators, following a moment-type approach. In particular, we exploit the finiteness of a few discrete absolute moments of certain density functions, , defined by the sigmoidal functions . The approximation error is considered in connection to both, the weights and the number of neurons of the network, in the sup-norm. In this framework, a remarkable result is that the order of approximation achieved when we approximate -functions by our operators with logistic or hyperbolic tangent sigmoidal functions, is higher than that obtained in Anastassiou, 2011b, Anastassiou, 2011c.
At the end, the case of quasi-interpolation operators constructed with sigmoidal functions is also considered in this paper, aimed at approximating functions defined on the whole space , see Anastassiou, 2011a, Anastassiou, 2011b, Anastassiou, 2011c, Anastassiou, 2011d, Anastassiou, 2012, Cao and Chen (2009) and Costarelli and Spigler (2013a), e.g.
The paper is organized as follows. In Section 2 we recall some preliminary results given in Costarelli and Spigler (2013a), and prove some new lemmas useful to establish the main results of the paper. In Section 3 the convergence and order of approximation theorems are proved. Moreover, many examples of sigmoidal functions satisfying the hypothesis of our theory are presented, and a discussion of results obtained is given. Finally, in Section 4 some final remarks are summarized and the case of quasi-interpolation operators constructed with sigmoidal functions is analyzed.
Section snippets
Preliminary results
In this section, we establish some preliminary results that will be useful in the rest of the paper. We recall that a measurable function is called a sigmoidal function if and only if and . In what follows, we consider non-decreasing functions , such that , and satisfying all the following assumptions:
is an odd function;
is concave for ;
as , for some .
For
The main results
In what follows, we denote by the -dimensional interval , and by and the spaces of all continuous real-valued functions defined on and , respectively, equipped with the sup-norm . Let us now define the operators that will be studied in this section. Definition 3.1 Let be a bounded function, and such that for every . The linear positive multivariate NN operators , activated by the sigmoidal function , and acting on , are
Final remarks and conclusions
In this paper, we study pointwise and uniform convergence, as well as the order of approximation, of a family of multivariate NN operators, activated by certain sigmoidal functions. Our approach allows us to extend some previous results. The order of approximation achieved using our operators is studied through a moment-type approach.
In the present context, we can use sigmoidal functions also to study convergence and order of approximation of a class of quasi-interpolation operators, defined
Acknowledgments
This work was supported, in part, by the GNAMPA and the GNFM of the Italian INdAM.
References (39)
Rate of convergence of some neural network operators to the unit-univariate case
Journal of Mathematical Analysis and Applications
(1997)Multivariate hyperbolic tangent neural network approximation
Computers & Mathematics with Applications
(2011)Multivariate sigmoidal neural network approximation
Neural Networks
(2011)Univariate hyperbolic tangent neural network approximation
Mathematical and Computer Modelling
(2011)Approximation with neural networks activated by ramp sigmoids
Journal of Approximation Theory
(2010)- et al.
Approximation by ridge functions and neural networks with one hidden layer
Journal of Approximation Theory
(1992) - et al.
Approximation results for neural network operators activated by sigmoidal functions
Neural Networks
(2013) - et al.
Univariant approximation by superpositions of a sigmoidal function
Journal of Mathematical Analysis and Applications
(1993) Complexity estimates based on integral transforms induced by computational units
Neural Networks
(2012)- et al.
Estimates of covering numbers of convex sets with slowly decaying orthogonal subsets
Discrete Applied Mathematics
(2007)
Approximation of functions of finite variation by superpositions of a sigmoidal function
Applied Mathematics Letters
Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer
Neurocomputing
Random approximants and neural networks
Journal of Approximation Theory
Uniform approximation by neural networks
Journal of Approximation Theory
Approximation by superposition of sigmoidal and radial basis functions
Advances in Applied Mathematics
Degree of approximation by neural and translation networks with a single hidden layer
Advances in Applied Mathematics
Univariate sigmoidal neural network approximation
Journal of Computational Analysis and Applications
Universal approximation bounds for superpositions of a sigmoidal function
IEEE Transactions on Information Theory
Cited by (113)
Some density results by deep Kantorovich type neural network operators
2024, Journal of Mathematical Analysis and ApplicationsIntegrating multivariate fuzzy neural networks into fuzzy inference system for enhanced decision making
2023, Fuzzy Sets and SystemsApproximation error for neural network operators by an averaged modulus of smoothness
2023, Journal of Approximation TheoryNeural network operators with hyperbolic tangent functions
2023, Expert Systems with ApplicationsGeneral sigmoid based Banach space valued neural network multivariate approximations
2024, Journal of Computational Analysis and ApplicationsParametrized Gudermannian function induced Banach space valued ordinary and fractional neural networks approximations
2024, Journal of Computational Analysis and Applications