Investigating the Role of Astrocyte Units in a Feedforward Neural Network

Abstract

Current research in neuroscience has begun to shift perspective from neurons as sole information processors to including the astrocytes as equal and cooperating units in this function. Recent evidence sheds new light on astrocytes and presents them as important regulators of neuronal activity and synaptic plasticity. In this paper, we present a multi-layer perceptron (MLP) with artificial astrocyte units which listen to and regulate hidden neurons based on their activity. We test the behavior and performance of this bio-inspired model on two classification tasks, N-parity problem and the two-spirals problem and show that proposed models outperform the standard MLP. Interestingly, we have also discovered multiple regimes of astrocyte activity depending on the complexity of the problem.

Appendix: Derivation of the update formula

Here we derive formula for stochastic (online) update of astrocyte weights $\alpha _i$ in models A-MLP($\alpha$) and A-MLP($\alpha ,\theta ,\gamma$). The goal is to minimize the loss function $E(w) = 1/2 (d - y(x))^2$, by moving the astrocytic weights along the negative gradient, i.e. $\varDelta \alpha _i = - \partial E(w)/ \partial \alpha _i$. Since E is differentiable with respect to $\alpha _i$, we can write using the chain rule,

$$\begin{aligned} \varDelta \alpha _i = -\frac{\partial E}{\partial y} \frac{\partial y}{\partial net_y} \frac{\partial net_y}{\partial h_i} \frac{\partial h_i}{\partial net_{hi}} \frac{\partial net_{hi}}{\partial \alpha _i} \end{aligned}$$

(9)

$$\begin{aligned} \varDelta \alpha _i = -\overbrace{(d - y(x))y(x)(1-y(x))}^{\delta _y}w_{yh_i} h_i(1-h_i)\psi _i \end{aligned}$$

(10)

$$\begin{aligned} \varDelta \alpha _i = -\overbrace{\delta _y w_{yh_i}h_i(1-h_i)}^{\delta _i}\psi _i \end{aligned}$$

(11)

which yields the final formula:

$$\begin{aligned} \varDelta \alpha _i = -\delta _i\psi _i \end{aligned}$$

(12)