Resonant spatiotemporal learning in large random recurrent networks

Abstract.

 Taking a global analogy with the structure of perceptual biological systems, we present a system composed of two layers of real-valued sigmoidal neurons. The primary layer receives stimulating spatiotemporal signals, and the secondary layer is a fully connected random recurrent network. This secondary layer spontaneously displays complex chaotic dynamics. All connections have a constant time delay. We use for our experiments a Hebbian (covariance) learning rule. This rule slowly modifies the weights under the influence of a periodic stimulus. The effect of learning is twofold: (i) it simplifies the secondary-layer dynamics, which eventually stabilizes to a periodic orbit; and (ii) it connects the secondary layer to the primary layer, and realizes a feedback from the secondary to the primary layer. This feedback signal is added to the incoming signal, and matches it (i.e., the secondary layer performs a one-step prediction of the forthcoming stimulus). After learning, a resonant behavior can be observed: the system resonates with familiar stimuli, which activates a feedback signal. In particular, this resonance allows the recognition and retrieval of partial signals, and dynamic maintenence of the memory of past stimuli. This resonance is highly sensitive to the temporal relationships and to the periodicity of the presented stimuli. When we present stimuli which do not match in time or space, the feedback remains silent. The number of different stimuli for which resonant behavior can be learned is analyzed. As with Hopfield networks, the capacity is proportional to the size of the second, recurrent layer. Moreover, the high capacity displayed allows the implementation of our model on real-time systems interacting with their environment. Such an implementation is reported in the case of a simple behavior-based recognition task on a mobile robot. Finally, we present some functional analogies with biological systems in terms of autonomy and dynamic binding, and present some hypotheses on the computational role of feedback connections.