Influence of Number of Neurons in Time Delay Recurrent Networks with Stochastic Weight Update on Backpropagation Through Time

Abstract

We will examine the various modifications of backpropagation through time algorithm (BPTT) done by stochastic update in the recurrent neural networks (RCNN) including the influence of the different numbers of recurrent neurons. The general introduction involving the stochasticity into neural network was provided by Salvetti and Wilamowski in 1994 in order to improve probability of convergence and speed of convergence. The implementation is simple for arbitrary network topology. In stochastic update scenario, constant number of weights and neurons (neurons selected before starting learning phase) are randomly selected and updated. This is in contrast to classical ordered update, where always all weights or neurons are updated. Stochastic update is suitable to replace classical ordered update without any penalty on implementation complexity and with good chance without penalty on quality of convergence. We have provided first experiments with stochastic modification on backpropagation algorithm (BP) used for artificial feed-forward neural network (FFNN) in detail described in our paper [1]. We will present experiment results on simple toy-task data of time shifted and skewed signal as a verification of our implementation of different algorithm modifications.