Averaged learning equations of error-function-based multilayer perceptrons

Abstract

The multilayer perceptrons (MLPs) have strange behaviors in the learning process caused by the existing singularities in the parameter space. A detailed theoretical or numerical analysis of the MLPs is difficult due to the non-integrability of the traditional log-sigmoid activation function which leads to difficulties in obtaining the averaged learning equations (ALEs). In this paper, the error function is suggested as the activation function of the MLPs. By solving the explicit expressions of two important expectations, we obtain the averaged learning equations which make it possible for further analysis of the learning dynamics in MLPs. The simulation results also indicate that the ALEs play a significant role in investigating the singular behaviors of MLPs.

Appendix

From Eq. (1), we have

$$y-f_0({\user2{x}})=\varepsilon \sim {{\mathcal{N}}}(0,1),$$

(26)

then

$$\frac{1}{\sqrt{2\pi}}\int\limits_{-\infty}^{+\infty}\exp\left(-\frac{1}{2}(y-f_0({\user2{x}}))^2\right)\hbox{d}y =\frac{1}{\sqrt{2\pi}}\int\limits_{-\infty}^{+\infty}\exp\left(-{\frac{\varepsilon^2}{2}}\right)\hbox{d}\varepsilon=1.$$

(27)

$P_1({\user2{s}},{\user2{v}})$ and $P_2({\user2{s}},{\user2{v}})$ can be rewritten as

$$\begin{aligned} P_1({\user2{s}},{\user2{v}})&=\left(2\pi\right)^{-\frac{n}{2}}\int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{\infty}\phi({\user2{s}}^T{\user2{x}})\phi({\user2{v}}^T{\user2{x}}) \exp{(-\frac{1}{2}\|{\user2{x}}\|^2)}\times \frac{1}{\sqrt{2\pi}}\exp{\left(-\frac{1}{2}\left(y-f_0({\user2{x}})\right)^2\right)}\hbox{d}y\hbox{d}{\user2{x}}\\ &=\left(2\pi\right)^{-\frac{n}{2}}\int\limits_{-\infty}^{\infty}\phi({\user2{s}}^T{\user2{x}})\phi({\user2{v}}^T{\user2{x}}) \exp{(-\frac{1}{2}\|{\user2{x}}\|^2)}\hbox{d}{\user2{x}}, \end{aligned}$$

(28)

$$\begin{aligned} P_2({\user2{s}},{\user2{v}})&=\left(2\pi\right)^{-\frac{n}{2}}\int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{\infty}\phi({\user2{s}}^T{\user2{x}})\frac{\partial\phi({\user2{v}}^T{\user2{x}})}{\partial{\user2{v}}} \exp{(-\frac{1}{2}\|{\user2{x}}\|^2)}\times \frac{1}{\sqrt{2\pi}}\exp{\left(-\frac{1}{2}\left(y-f_0({\user2{x}})\right)^2\right)}\hbox{d}y\hbox{d}{\user2{x}}\\ &=\left(2\pi\right)^{-\frac{n}{2}}\int\limits_{-\infty}^{\infty}\phi({\user2{s}}^T{\user2{x}})\frac{\partial\phi({\user2{v}}^T{\user2{x}})}{\partial{\user2{v}}} \exp{(-\frac{1}{2}\|{\user2{x}}\|^2)}\hbox{d}{\user2{x}}. \end{aligned}$$

(29)

Then we have:

$$\begin{aligned} P_2({\user2{s}},{\user2{v}})&=\left(2\pi\right)^{-\frac{n}{2}}\int\limits_{-\infty}^{\infty}\phi ({\user2{s}}^T{\user2{x}})\frac{\partial\phi({\user2{v}}^T{\user2{x}})}{\partial{\user2{v}}} \exp{(-\frac{1}{2}\|{\user2{x}}\|^2)}\hbox{d}{\user2{x}}\\ &=\left(2\pi\right)^{-\frac{n+1}{2}}\int\limits_{-\infty}^{\infty}\phi({\user2{s}}^T{\user2{x}}) {\user2{x}}\exp(-\frac{1}{2}({\user2{v}}^T{\user2{x}})^2)\exp(-\frac{1}{2}\|{\user2{x}}\|^2)\hbox{d}{\user2{x}}\\ &=\left(2\pi\right)^{-\frac{n+1}{2}}\int\limits_{-\infty}^{\infty}{\user2{x}}\phi({\user2{s}}^T{\user2{x}}) \exp(-\frac{1}{2}(\|{\user2{x}}\|^2+({\user2{v}}^T{\user2{x}})^2))\hbox{d}{\user2{x}}\\ &=\frac{1}{2\pi}\sqrt{\det\left({\user2{B}}^{-1}\right)}{\user2{A}}^{-1}{\user2{s}}, \end{aligned}$$

(30)

where

$${\user2{A}}={\bf I}_n+{\user2{v}}{\user2{v}}^T,$$

(31)

$${\user2{B}}={\user2{A}}+{\user2{s}}{\user2{s}}^T.$$

(32)

According to Sherman–Morrison formula, we have:

$${\user2{A}}^{-1}=\left({\bf I}_n+{\user2{v}}{\user2{v}}^T\right)^{-1}={\bf I}_n-\frac{{\user2{v}}{\user2{v}}^T}{1+\|{\user2{v}}\|^2},$$

(33)

$${\user2{B}}^{-1}=\left({\user2{A}}+{\user2{s}}{\user2{s}}^T\right)^{-1} =\left({\bf I}_n-\frac{{\user2{A}}^{-1}{\user2{s}}{\user2{s}}^T}{1+{\user2{s}}^T{\user2{A}}^{-1}{\user2{s}}}\right){\user2{A}}^{-1}.$$

(34)

By using Sylvester’s determinant theorem, we have:

$$\det\left({\user2{A}}^{-1}\right)=1-\frac{\|{\user2{v}}\|^2}{1+\|{\user2{v}}\|^2}=\frac{1}{1+\|{\user2{v}}\|^2},$$

(35)

$$\det\left({\user2{B}}^{-1}\right)=\frac{1}{\left(1+\|{\user2{s}}\|^2\right)\left(1+\|{\user2{v}}\|^2\right)-({\user2{s}}^T{\user2{v}})^2}.$$

(36)

According to the Leibniz integral rule, the following equation holds:

$$P_2({\user2{s}},{\user2{v}})=\frac{\partial P_1({\user2{s}},{\user2{v}})}{\partial{\user2{v}}}.$$

(37)

From (37), we can get P ₁ by integrating P ₂ respective to ${\user2{v}}$, so the expression of P ₁ is of the following equation:

$$\begin{aligned} P_1({\user2{s}},{\user2{v}})&=\int\limits_{-\infty}^{{\user2{v}}}P_2({\user2{s}},{\user2{v}})\hbox{d}{\user2{v}}\\ &=\frac{1}{2\pi}\int\limits_{-\infty}^{{\user2{v}}}\frac{{\user2{A}}^{-1}{\user2{s}}}{\sqrt{(1+\|{\user2{s}}\|^2)(1+\|{\user2{v}}\|^2)-({\user2{s}}^T{\user2{v}})^2}}\hbox{d}{\user2{v}}\\ &=\frac{1}{2\pi}\left(\arcsin\frac{{\user2{s}}^T{\user2{v}}}{\sqrt{1+\|{\user2{s}}\|^2}\sqrt{1+\|{\user2{v}}\|^2}}+C\right), \end{aligned}$$

(38)

where C is a constant.

From (28), we know that $P_1({\varvec{0}},{\varvec{0}})={\frac{1}{4}}$, then we have $C={\frac{\pi}{2}}$. Finally, we get

$$P_1({\user2{s}},{\user2{v}})=\frac{1}{2\pi}\arcsin\frac{{\user2{s}}^T{\user2{v}}}{\sqrt{1+\|{\user2{s}}\|^2}\sqrt{1+\|{\user2{v}}\|^2}}+\frac{1}{4}.$$

(39)