Piecewise Polynomial Activation Functions for Feedforward Neural Networks

Abstract

Since the origins of artificial neural network research, many models of feedforward networks have been proposed. This paper presents an algorithm which adapts the shape of the activation function to the training data, so that it is learned along with the connection weights. The activation function is interpreted as a piecewise polynomial approximation to the distribution function of the argument of the activation function. An online learning procedure is given, and it is formally proved that it makes the training error decrease or stay the same except for extreme cases. Moreover, the model is computationally simpler than standard feedforward networks, so that it is suitable for implementation on FPGAs and microcontrollers. However, our present proposal is limited to two-layer, one-output-neuron architectures due to the lack of differentiability of the learned activation functions with respect to the node locations. Experimental results are provided, which show the performance of the proposal algorithm for classification and regression applications.

Appendix

Proof of Proposition 2

Let us assume that $u_{r,i}\in \left[ q_{i,k},q_{i,k+1}\right)$. Therefore,

$$\begin{aligned} g_{i}\left( u_{r,i}\right) =\frac{k+\delta _{i,k}\left( u_{r,i}\right) }{m} \end{aligned}$$

(63)

where $\delta _{i,k}\left( u_{r,i}\right) \in \left[ 0,1\right]$. The exact form of $\delta \left( u_{r,i}\right)$ depends on the order of the polynomials $\gamma$.

If $w_{2,i}\left( y_{r}-z_{r}\right) <0$ and $\left| \frac{2w_{2,i}}{m}\right| <\lambda \left| y_{r}-z_{r}\right|$, then the update Eq. (36) implies that:

$$\begin{aligned} q_{i,k}\le \bar{q}_{i,k+1}\le u_{r,i} \end{aligned}$$

(64)

Therefore,

$$\begin{aligned} \bar{g}_{i}\left( u_{r,i}\right) =\frac{k+1+\bar{\delta }_{i,k+1}\left( u_{r,i}\right) }{m} \end{aligned}$$

(65)

where the bars correspond to the values obtained after executing the update. Moreover, from (4):

$$\begin{aligned} y_{r}-\bar{y}_{r}= & {} \sum _{s=1}^{L}w_{2,s}g_{s}\left( u_{r,s}\right) -\sum _{s=1}^{L}w_{2,s}\bar{g}_{s}\left( u_{r,s}\right) \nonumber \\= & {} w_{2,i}g_{i}\left( u_{r,i}\right) -w_{2,i}\bar{g}_{i}\left( u_{r,i}\right) \end{aligned}$$

(66)

Then from (63), (65) and (66):

$$\begin{aligned} y_{r}-\bar{y}_{r}= & {} w_{2,i}\left( \frac{k+\delta _{i,k}\left( u_{r,i}\right) }{m}-\frac{k+1+\bar{\delta }_{i,k+1}\left( u_{r,i}\right) }{m}\right) \nonumber \\= & {} w_{2,i}\frac{\delta _{i,k}\left( u_{r,i}\right) -\bar{\delta }_{i,k+1}\left( u_{r,i}\right) -1}{m} \end{aligned}$$

(67)

Since $w_{2,i}\left( y_{r}-z_{r}\right) <0$ , there are two possible cases: (a) $\left( w_{2,i}>0\right) \wedge \left( y_{r}-z_{r}<0\right)$; (b) $\left( w_{2,i}<0\right) \wedge \left( y_{r}-z_{r}>0\right)$ .

For case (a), since $\delta _{i,k}\left( u_{r,i}\right) ,\bar{\delta }_{i,k+1}\left( u_{r,i}\right) \in \left[ 0,1\right]$, from (67) we obtain:

$$\begin{aligned} -\frac{2w_{2,i}}{m}\le y_{r}-\bar{y}_{r}\le 0 \end{aligned}$$

(68)

On the other hand, since $y_{r}-z_{r}<0$, $w_{2,i}>0$, $\lambda \in \left( 0,1\right]$ and $\left| \frac{2w_{2,i}}{m}\right| <\lambda \left| y_{r}-z_{r}\right|$ we have:

$$\begin{aligned} y_{r}-z_{r}\le -\frac{2w_{2,i}}{m} \end{aligned}$$

(69)

From (68) and (69):

$$\begin{aligned}&\displaystyle y_{r}-z_{r}\le y_{r}-\bar{y}_{r}\le 0 \end{aligned}$$

(70)

$$\begin{aligned}&\displaystyle -z_{r}\le -\bar{y}_{r}\le -y_{r} \end{aligned}$$

(71)

$$\begin{aligned}&\displaystyle y_{r} \le \bar{y}_{r}\le z_{r} \end{aligned}$$

(72)

$$\begin{aligned}&\displaystyle \bar{E}_{r} \le E_{r} \end{aligned}$$

(73)

That is, the new squared error $\bar{E}_{r}$ is lower than or equal to the old squared error $E_{r}$, as required.

For case (b), since $\delta _{i,k}\left( u_{r,i}\right) ,\bar{\delta }_{i,k-1}\left( u_{r,i}\right) \in \left[ 0,1\right]$, from (67) we obtain:

$$\begin{aligned} 0\le y_{r}-\bar{y}_{r}\le -\frac{2w_{2,i}}{m} \end{aligned}$$

(74)

On the other hand, since $y_{r}-z_{r}>0$, $w_{i}^{2}<0$, $\lambda \in \left( 0,1\right]$ and $\left| \frac{2w_{2,i}}{m}\right| <\lambda \left| y_{r}-z_{r}\right|$ we have:

$$\begin{aligned} -\frac{2w_{2,i}}{m}\le y_{r}-z_{r} \end{aligned}$$

(75)

From (74) and (75):

$$\begin{aligned}&\displaystyle 0\le y_{r}-\bar{y}_{r}\le y_{r}-z_{r} \end{aligned}$$

(76)

$$\begin{aligned}&\displaystyle -y_{r}\le -\bar{y}_{r}\le -z_{r} \end{aligned}$$

(77)

$$\begin{aligned}&\displaystyle z_{r} \le \bar{y}_{r}\le y_{r} \end{aligned}$$

(78)

$$\begin{aligned}&\displaystyle \bar{E}_{r} \le E_{r} \end{aligned}$$

(79)

And again the new squared error $\bar{E}_{r}$ is lower than or equal to the old squared error $E_{r}$, as required. $\square$