The capacity of time-delay recurrent neural network for storing spatio-temporal sequences

Abstract

We investigate the capacity of a type of discrete-time recurrent neural network, called time-delay recurrent neural network, for storing spatio-temporal sequences. By introducing the order of a spatio-temporal sequence, the match law between a time-delay recurrent neural network and a spatio-temporal sequence has been established. It has been proved that the full order time-delay recurrent neural network of l-step feedback is able to learn and memorize (or store) any bipolar (or binary) spatio-temporal sequence of the order k if k⩽l, and that the asymptotic memory capacity of the first-order time-delay recurrent neural network of one-step feedback is not less than C₁(n)=n+1, where n is the number of processing neurons in the network. Moreover, we substantiate the theoretical results by simulation experiments.

Introduction

In temporal information processing, it is often necessary to store and recognize spatio-temporal sequences which are fundamentally different from static patterns. Actually, many approaches have been made to build a temporal system which can learn and memorize (or store) spatio-temporal sequences effectively. Recent developments in the field of neural networks have brought out certain good temporal models and methods to meet this demand. One major method is to incorporate feedback into static or mapping neural networks, making them recurrent. This leads to the so-called recurrent neural network [13]. In fact, the recurrent neural network contains many types of neural network, such as Hopfield network [2] as well as its generalized version [4], simplex memory neural network [5], continuous-time recurrent neural network [8] and discrete-time or real-time recurrent neural network [10], [15]. Moreover, the recurrent neural network has been widely applied to temporal information processing, such as speech recognition, prediction, and system identification and control (e.g., [7], [9], [10], [11], [14]).

However, little progress has been made on theoretical study of the capacity of the recurrent neural network to learn and memorize spatio-temporal sequences. In literature, we can only find a theoretic result given by Amari [1] that the memory capacity of the so-called autoassociative memory model with the sum-of-outer product scheme on a simple spatio-temporal sequence, is about 0.27n, where n is the number of the processing neurons. Actually, the autoassociative memory model is just a special form of discrete-time recurrent neural network such that it consists of n fully connected binary or bipolar neurons, with its output being the vector of states of n neurons evolving from an initial pattern as the only input signal. Certainly, this model can be generalized and improved considerably if multi-step time-delay feedback of the output is allowed to take part in the computation of the neurons to learn and memorize spatio-temporal sequences. We will refer to such an extension of the autoassociative memory model as time-delay recurrent neural network or TDRNN for short. In the present paper, we will investigate the capacity of time-delay recurrent neural network for storing spatio-temporal sequences.

In sequel, we give a brief description of a TDRNN and establish a match law between a TDRNN and a spatio-temporal sequence in Section 2. We then analyze the capacity of the high-ordered TDRNN for learning and memorizing a bipolar (or binary) spatio-temporal sequence in Section 3. We further analyze the asymptotic memory capacity of the first-order TDRNN of one-step feedback, i.e., the autoassociative memory model, under the perceptron learning algorithm in Section 4. In Section 5, simulation experiments are conducted to substantiate our theoretical results. Finally, we conclude in Section 6.