The capacity of time-delay recurrent neural network for storing spatio-temporal sequences☆
Introduction
In temporal information processing, it is often necessary to store and recognize spatio-temporal sequences which are fundamentally different from static patterns. Actually, many approaches have been made to build a temporal system which can learn and memorize (or store) spatio-temporal sequences effectively. Recent developments in the field of neural networks have brought out certain good temporal models and methods to meet this demand. One major method is to incorporate feedback into static or mapping neural networks, making them recurrent. This leads to the so-called recurrent neural network [13]. In fact, the recurrent neural network contains many types of neural network, such as Hopfield network [2] as well as its generalized version [4], simplex memory neural network [5], continuous-time recurrent neural network [8] and discrete-time or real-time recurrent neural network [10], [15]. Moreover, the recurrent neural network has been widely applied to temporal information processing, such as speech recognition, prediction, and system identification and control (e.g., [7], [9], [10], [11], [14]).
However, little progress has been made on theoretical study of the capacity of the recurrent neural network to learn and memorize spatio-temporal sequences. In literature, we can only find a theoretic result given by Amari [1] that the memory capacity of the so-called autoassociative memory model with the sum-of-outer product scheme on a simple spatio-temporal sequence, is about 0.27n, where n is the number of the processing neurons. Actually, the autoassociative memory model is just a special form of discrete-time recurrent neural network such that it consists of n fully connected binary or bipolar neurons, with its output being the vector of states of n neurons evolving from an initial pattern as the only input signal. Certainly, this model can be generalized and improved considerably if multi-step time-delay feedback of the output is allowed to take part in the computation of the neurons to learn and memorize spatio-temporal sequences. We will refer to such an extension of the autoassociative memory model as time-delay recurrent neural network or TDRNN for short. In the present paper, we will investigate the capacity of time-delay recurrent neural network for storing spatio-temporal sequences.
In sequel, we give a brief description of a TDRNN and establish a match law between a TDRNN and a spatio-temporal sequence in Section 2. We then analyze the capacity of the high-ordered TDRNN for learning and memorizing a bipolar (or binary) spatio-temporal sequence in Section 3. We further analyze the asymptotic memory capacity of the first-order TDRNN of one-step feedback, i.e., the autoassociative memory model, under the perceptron learning algorithm in Section 4. In Section 5, simulation experiments are conducted to substantiate our theoretical results. Finally, we conclude in Section 6.
Section snippets
The TDRNN and a match law
We begin with a brief description of a TDRNN of l-step (time-delay) feedback. As shown in Fig. 1, it consists of n processing neurons which form the single processing or output layer. The state of processing neuron i at time t is denoted by xi(t) which is generally a real number. Then, the state vector of the TDRNN at time t is denoted by X(t)=[x1(t),x2(t),…,xn(t)]T. Additionally, there is an input layer with an n×l array of input neurons. We let Ii,j denote the input neuron at the ith row and
The higher order TDRNNs
We begin with the order of a TDRNN. For the storage of bipolar spatio-temporal sequences, we use perceptrons as the processing neurons in the network. Then, the input pattern for each of these perceptrons is just the state of the array of input neurons. Mathematically, a perceptron becomes of higher order if some higher order terms of the components of the input pattern are involved in the computation. It is natural to define the order of a TDRNN as the order of these perceptrons, i.e., the
Memory capacity of the first-order TDRNN of one-step feedback
We first give two basic properties of the first-order TDRNN of one-step feedback on learning and memorizing a bipolar spatio-temporal sequence. Theorem 4 IfP1,P2,…,Pm−1are linearly independent, the first-order TDRNN of one-step feedback is able to learn and memorize the bipolar spatio-temporal sequenceP1P2⋯Pm. Proof According to the above definition, the first-order TDRNN of one-step feedback is uniquely defined by the weight matrix W and the threshold vector θ, where wi,j is the weight on the feedback line from
Simulation results
In this section, simulation experiments are carried out to demonstrate the performance of the TDRNN on learning and storing spatio-temporal sequences by the perceptron learning algorithm as well as its memory capacity. Moreover, we compare the object perceptron learning algorithm with the sum-of-outer product scheme on the first order TDRNN of one-step feedback.
Conclusion
We have investigated the capacity of time-delay recurrent neural network (TDRNN) for learning and memorizing spatio-temporal sequences. By introducing the order of a spatio-temporal sequence, we have established the match law between a TDRNN and a spatio-temporal sequence. It has been further proved that the full order TDRNN of l-step feedback is able to learn and memorize any bipolar (or binary) spatio-temporal sequence of the order k when k⩽l. Furthermore, we have obtained a lower bound of
Acknowledgements
The author wishes to thank Prof. Shiyi Shen who first gave the proof of Theorem 2(i) and made some good suggestions on the paper. The author also thanks Mr. Yang Zhao for his support of the experiments.
Jinwen Ma received the Master of Science degree in applied mathematics from Xi'an Jiaotong University in 1988 and the Ph.D. degree in probability theory and statistics from Nankai University in 1992. From July 1992 to November 1999, he was a Lecturer or Associate professor at Department of Mathematics, Shantou University. From December 1999, he worked as a full professor at Institute of Mathematic, Shantou University. In September 2001, he was transferred to the Department of Information
References (15)
The stability of the generalized Hopfield networks in randomly asynchronous mode
Neural Networks
(1997)Simplex memory neural network
Neural Networks
(1997)Dynamics and architecture for neural computation
J. Complexity
(1988)Associative memory and its statistical-neurodynamical analysis
Springer Ser. Synergetics
(1988)Neural networks and physical systems with collective computational abilities
Proc. Natl. Acad. Sci. USA
(1982)On the determinant of (0,1) matrices
Stud. Sci. Math. Hung
(1967)The object perceptron learning algorithm on generalised Hopfield networks
Neural Comput. Appl
(1999)
Cited by (2)
Proto-predictive representation of states with simple recurrent temporal-difference networks
2009, Proceedings of the 26th International Conference On Machine Learning, ICML 2009Proto-predictive representation of states with simple recurrent temporal-difference networks
2009, ACM International Conference Proceeding Series
Jinwen Ma received the Master of Science degree in applied mathematics from Xi'an Jiaotong University in 1988 and the Ph.D. degree in probability theory and statistics from Nankai University in 1992. From July 1992 to November 1999, he was a Lecturer or Associate professor at Department of Mathematics, Shantou University. From December 1999, he worked as a full professor at Institute of Mathematic, Shantou University. In September 2001, he was transferred to the Department of Information Science at the School of Mathematical Sciences, Peking University. During 1995 and 2003, he also visited and studied several times at Department of Computer Science and Engineering, the Chinese University of Hong Kong as a Research Associate or Fellow. He has published more than 40 academic papers on neural networks, pattern recognition, artificial intelligence, and information theory.
- ☆
This work was supported by Natural Science Foundation of China (19701022,60071004) and the Natural Science Foundation of Guangdong Province (970377).