Multivariate neural network operators with sigmoidal activation functions

Abstract

In this paper, we study pointwise and uniform convergence, as well as order of approximation, of a family of linear positive multivariate neural network (NN) operators with sigmoidal activation functions. The order of approximation is studied for functions belonging to suitable Lipschitz classes and using a moment-type approach. The special cases of NN operators, activated by logistic, hyperbolic tangent, and ramp sigmoidal functions are considered. Multivariate NNs approximation finds applications, typically, in neurocomputing processes. Our approach to NN operators allows us to extend previous convergence results and, in some cases, to improve the order of approximation. The case of multivariate quasi-interpolation operators constructed with sigmoidal functions is also considered.

Introduction

Neural networks (NNs) with one hidden layer can be represented as

$$N_n\left(x\right)=\sum _{j=0}^n{c}_j\sigma ({\underset{¯}{a}}_j\cdot \underset{¯}{x}+b_j)\text{,}\underset{¯}{x}\in {\mathbb{R}}^s,s\in {\mathbb{N}}^{+}\text{,}$$

where, for $0\le j\le n$, the $b_j$’s, $b_j\in \mathbb{R}$, are the threshold values, the ${\underset{¯}{a}}_j$’s, ${\underset{¯}{a}}_j\in {\mathbb{R}}^s$, are the weights, and the $c_j$’s are the coefficients. Here ${\underset{¯}{a}}_j\cdot \underset{¯}{x}$ is the inner product in ${\mathbb{R}}^s$, and $\sigma$ is the activation function of the network, see Chui and Li (1992), Costarelli and Spigler (submitted for publication), Jones (1988), Lenze (1992), Li (1996), Li and Micchelli (2000), Light (1993), Makovoz (1998), Mhaskar and Micchelli (1995) and Pinkus (1999). The activation function usually is a sigmoidal function. Neural networks are extensively used in Approximation Theory (Barron, 1993, Chen, 1993, Costarelli and Spigler, 2013b, Cybenko, 1989, Gao and Xu, 1993, Girosi and Anzellotti, 1993, Gnecco, 2012, Gnecco and Sanguineti, 2011, Hahm and Hong, 2002, Kainen and Kurková, 2009, Kurková, 2012, Lewicki and Marino, 2003, Lewicki and Marino, 2004, Mhaskar and Micchelli, 1992).

Constructive multivariate approximation algorithms based on sigmoidal functions are important since they play a central role in typical applications of neurocomputing processes concerning high-dimensional data. Applications of NNs with sigmoidal functions in Numerical Analysis, for instance, to the numerical solution of Volterra integral and integro-differential equations by suitable collocation methods were shown in Costarelli and Spigler, in press-a, Costarelli and Spigler, in press-b.

Anastassiou (1997) was the first to establish NN approximations for continuous functions, providing estimates for the rate of convergence, using NN operators of the Cardaliagnet–Euvrard type. He used the modulus of continuity of the function being approximated, to produce Jackson type inequalities. Subsequently, Anastassiou studied NN operators activated by the hyperbolic tangent as well as the logistic function, in both, the univariate and the multivariate case, see Anastassiou, 2011a, Anastassiou, 2011b, Anastassiou, 2011c, Anastassiou, 2011d, Anastassiou, 2012.

In this paper, we study the convergence, as well as the order of approximation, of a family of linear positive multivariate neural network operators, activated by sigmoidal functions. This work represents the extension to the multivariate case of the results established in Costarelli and Spigler (2013a) for univariate functions. We treat by a unified approach all cases studied by Anastassiou, and moreover, we can apply our theory to other useful sigmoidal functions, such as, for instance, the ramp function (Cao and Chen, 2012, Cheang, 2010), and many others (Costarelli & Spigler, submitted for publication).

We first study pointwise and uniform convergence for functions defined on bounded intervals of ${\mathbb{R}}^s$. In addition, we study the order of approximation for functions belonging to certain Lipschitz classes by means of our NN operators, following a moment-type approach. In particular, we exploit the finiteness of a few discrete absolute moments of certain density functions, ${\varphi }_{\sigma }$, defined by the sigmoidal functions $\sigma$. The approximation error is considered in connection to both, the weights and the number of neurons of the network, in the sup-norm. In this framework, a remarkable result is that the order of approximation achieved when we approximate $C^1$-functions by our operators with logistic or hyperbolic tangent sigmoidal functions, is higher than that obtained in Anastassiou, 2011b, Anastassiou, 2011c.

At the end, the case of quasi-interpolation operators constructed with sigmoidal functions is also considered in this paper, aimed at approximating functions defined on the whole space ${\mathbb{R}}^s$, see Anastassiou, 2011a, Anastassiou, 2011b, Anastassiou, 2011c, Anastassiou, 2011d, Anastassiou, 2012, Cao and Chen (2009) and Costarelli and Spigler (2013a), e.g.

The paper is organized as follows. In Section  2 we recall some preliminary results given in Costarelli and Spigler (2013a), and prove some new lemmas useful to establish the main results of the paper. In Section  3 the convergence and order of approximation theorems are proved. Moreover, many examples of sigmoidal functions satisfying the hypothesis of our theory are presented, and a discussion of results obtained is given. Finally, in Section  4 some final remarks are summarized and the case of quasi-interpolation operators constructed with sigmoidal functions is analyzed.