A random-weighted plane-Gaussian artificial neural network

Abstract

Multilayer perceptron (MLP) and radial basis function network (RBFN) have received considerable attentions in data classification and regression. As a bridge between MLP and RBFN, plane-Gaussian (PG) network is capable of exhibiting globality and locality simultaneously by so-called PG activation function. Due to tuning network weight values by back propagation or clustering method in the training phase, they all confront with slow convergence rate, time-consuming, and easily dropping in local minima. To speed training networks, random projection technologies, for instance, extreme learning machine (ELM), have brightened up in recent decades. In this paper, we propose a random-weighted PG network, termed as RwPG. Instead of plane clustering in PG network, our RwPG adopts random values as network weight, and then analytically calculates network output by matrix inversion. Compared to PG and ELM, the advantages of the proposed RwPG list in fourfold: (1) It will be proved that the RwPG is also a universal approximator. (2) It inherits the geometrical interpretation of PG network, and is also suitable for capturing linearity in data, especially for plane distribution cases. (3) It has comparable training speed for ELM, but significantly faster than that of PG network. (4) Owing to random-weighted technology, RwPG is probably capable of breaking through local extremum problems. Finally, experiments on artificial and benchmark datasets will show its superiorities.

Appendix

Theorem 2

It is given that a non-constant, bounded, and monotone-increasing continuous function$G$described in Eq. (4) is dense in$C(\varvec{I}_{d} )$. That is, for any$f \in C(\varvec{I}_{d} )$and$\varepsilon > 0$, there exists a set of$(\varvec{w}_{i} ,\gamma_{i} )$, such that

$$ \left| {\sum\limits_{i = 1}^{c} {\varvec{u}_{i} G(\varvec{w}_{i} ,\gamma_{i} ,\varvec{x}) - f(\varvec{x})} } \right| \le \varepsilon $$

(6)

Proof for theorem 2

Since $f \in C(\varvec{I}_{d} )$, it can be described by a limit-integral representation

$$ f = \mathop {\lim }\limits_{L \to r} \int_{V} {T[f(\lambda )]} G_{\lambda ,L} (x){\text{d}}\lambda $$

(10)

where $L$ and $\lambda$ are one- and d-dimensional (corresponding to foresaid input space) parameters of the activation function $G$, respectively. $T$ is an operator defined on $C(\varvec{I}_{d} )$, $r$ is a finite or infinite real number, and $V$ is the integral domain of the $\lambda$.

Two stages of approximation can be used to approximate the function $f$, as described below. The first stage is to approximate the limit value by the integral

$$ f \approx \int_{V} {T[f(\lambda )]} G_{\lambda ,l} (x){\text{d}}\lambda $$

(11)

where $l \in N(r,\varepsilon )$($\varepsilon$ is a fully small positive, and $N(r,\varepsilon )$ denotes an $\varepsilon$ neighbor domain of $r$. For convenience, noted it by $l \approx r$).

The second stage is to obtain an estimate of multivariable integral with random method, typically, Monte Carlo. The given set of c random values $\lambda = (\lambda_{1} ,\lambda_{2} , \cdots ,\lambda_{c} )$, i.i.d, (independently and identically distributed), is drawn from the uniform distribution on $V$. That is,

$$ f \approx \sum\limits_{k = 1}^{c} {a_{k} G_{{\lambda_{k} ,l}} (x)} $$

(12)

where $a_{k} = (|V|/n)T[f(\lambda_{k} )]$, and $|V|$ denotes the volume of integral domain.When n tends to infinite, the approximation error of Monte Carlo method is bounded by $C/\sqrt n$, where the constant C is, not independent of d, determined by the variance of the integral [7, 27].

Considering random terms of activation function $G$, parameter pairs $(\varvec{w}_{i} ,\gamma_{i} )$, we redefine the foresaid random term $\lambda$ in the form of components $\lambda_{i} = (w_{i1} , \ldots ,w_{id} ,b_{i} ) \in S_{c} (\varOmega ,\alpha )$, where $S_{c} (\varOmega ,\alpha )$ denotes probabilistic space, and c denotes the number of hidden neurons, $i = 1,2, \ldots ,c$. Then, Eq. (12) becomes

$$ f \approx \sum\limits_{k = 1}^{c} {a_{k} G(\varvec{w}_{k} ,b_{k} ,\varvec{x}} ) $$

(13)

Next, we expect the following expression holds, when c tends to infinite.

$$ \rho_{K} = \sqrt {E\int_{K} {[f(\varvec{x}) - \sum\limits_{k = 1}^{c} {a_{k} G(\varvec{w}_{k} ,b_{k} ,\varvec{x})} ]^{2} {\text{d}}\varvec{x}} \mathop \to \limits_{c \to \infty } 0} $$

(14)

where $K( \subset I^{d} )$ denotes a dense set (compact set), and $E$ is the expectation w.r.t. $S_{c} (\varOmega ,\alpha )$.

From the definition of the PGF function defined in Eq. (3), the absolute value term $\left| {\varvec{w}^{T} \varvec{x} - \gamma } \right|^{2}$ is not only used for interpreting geometrical meaning under the constraint of $||\varvec{w}|| = 1$, but making the activation function bounded, even when $\varvec{w}$ tends to infinite. (When $\varvec{w} \to \infty$, the value of PGF tends to zero.) Additionally, here it equals to $(\varvec{w}^{T} \varvec{x} - \gamma )^{2}$. Thus, in fact the PGF, $\exp ( - (\varvec{w}^{T} \varvec{x} - \gamma )^{2} /2\sigma^{2} )$, is a continuous and high-order differentiable. Conveniently, noted it by $g$ as

$$ g(\varvec{w},\gamma ,\varvec{x}) = \exp ( - (\varvec{w}^{T} \varvec{x} - \gamma )^{2} /2\sigma^{2} ) $$

(15)

Since the parameter pairs $(\varvec{w}_{i} ,\gamma_{i} )$ in $g$ is i.i.d, the expression in (14) can be taken in the form

$$ \begin{aligned} \sum\limits_{k = 1}^{c} {a_{k} g(\varvec{w}_{k} ,b_{k} ,\varvec{x})} = & \sum\limits_{k = 1}^{c} {a_{k} \prod\limits_{i = 1}^{d} {g(\varvec{w}_{ki} ,\gamma_{k} ,x_{i} )} } \\ = & \sum\limits_{k = 1}^{c} {a_{k} \prod\limits_{i = 1}^{d} {\exp ( - (w_{ki} x_{i} - \gamma_{k} )^{2} /2\sigma^{2} )} } \\ \end{aligned} $$

(16)

where $w_{ki}$ and $x_{i}$ denote the ith components of vector $\varvec{w}_{k}$ and $\varvec{x}$, respectively. So, for any dense set $K$, it is easy to know that the PGF $g$ satisfies

$$ \begin{aligned} \int_{R} {g^{2} (w_{ki} ,\gamma_{k} ,x_{i} ){\text{d}}x_{i} } \hfill \\ \quad = \int_{R} {[\exp ( - (w_{ki} x_{i} - \gamma_{k} )^{2} /2\sigma^{2} ,)]^{2} {\text{d}}x_{i} } < \infty \hfill \\ \end{aligned} $$

(17)

According to Theorem 1 in [27], there exist a sequence of $(\varvec{w},\gamma )$ in the probability measure space $S_{c} (\varOmega ,\alpha )$, such that (14) holds. That is, $\rho_{K}$ will converge to zero, when $c$ tends to infinite.

Here, we do not detail the rest proof, the similar description can be found in Appendix section of Ref [27].$\square$