Elsevier

Neurocomputing

Volume 64, March 2005, Pages 25-38
Neurocomputing

Input space bifurcation manifolds of recurrent neural networks

https://doi.org/10.1016/j.neucom.2004.11.030Get rights and content

Abstract

We derive analytical expressions of local codimension-1 bifurcations for a fully connected, additive, discrete-time recurrent neural network (RNN), where we regard the external inputs as bifurcation parameters. The complexity of the bifurcation diagrams obtained increases exponentially with the number of neurons. We show that a three-neuron cascaded network can serve as a universal oscillator, whose amplitude and frequency can be completely controlled by input parameters.

Introduction

An important approach to the understanding of the complex dynamical behaviour of recurrent neural networks (RNNs) is the study of their bifurcation manifolds. These manifolds separate regions in parameter space, which exhibit qualitatively different dynamical behaviour. Knowledge of these manifolds on the one hand deepens the understanding of RNNs and on the other hand allows to directly choose parameter sets which cause a specific dynamical behaviour. In particular small networks, which already exhibit all kinds of dynamical behaviour including chaos [14], have been used as basic pattern generators within more complex networks [11], as submodules in classification networks for dynamic signals [10], or as neural controllers in robotics systems [9]. These applications are motivated by the higher-order information processing capabilities of recurrent networks due to resonance and synchronisation effects between their components or with respect to time-dependent inputs.

Some authors have studied bifurcation manifolds in discrete- and continuous-time neural networks before, employing numerical methods only and restricting their analysis to simplified connection matrices [1], [8], [13]. The first attempt to compute bifurcation manifolds of RNNs analytically was made by Hoppensteadt and Izhikevich [4], who employed specific properties of the Fermi function to derive bifurcation curves of continuous-time networks. Using the same approach, we already computed analytically the bifurcation curves of discrete-time two-neuron networks [3].

While it is commonly known that weight parameters directly influence the dynamical behaviour of RNNs, we rather consider the external inputs as main bifurcation parameters and present an approach to compute their manifolds. The method is in principle applicable to networks of arbitrary size and with arbitrary activation, though the expressions become especially simple for the hyperbolic tangent. Thus, our method goes far beyond numerical continuation techniques [6], which can be used to compute one-dimensional bifurcation manifolds only [1]. It turns out that more complex networks possess in general a heavily cluttered input parameter space, which is divided by so many bifurcation manifolds that numerical analysis and visualisation are not feasible any more. For this reason, we state the expressions of the bifurcation manifolds for three-neuron networks and show that these small nets already can serve as universal oscillators, whose amplitude and frequency can be completely controlled with inputs (within a certain range determined by the weights).

After introducing saddle-node, period-doubling and Neimark–Sacker bifurcations of fixed points in Section 2, we derive analytical expressions for the bifurcation manifolds of these bifurcation types in Section 3. In Section 4, we consider numerical issues in solving and visualising the equations, and finally in Section 5 discuss the dynamical behaviour and control of oscillations of a cascaded network.

Section snippets

Codimension-1 bifurcations in RNNs

Naturally the dynamics of a RNN changes, if its parameters like weights and inputs are varied. A qualitative change of the dynamical behaviour, e.g. a transition from a unique stable fixed point to an oscillation or vice versa, is called bifurcation and the corresponding critical parameter set is called bifurcation point. Bifurcation points form manifolds in the parameter space, which separate parameter sets of different dynamical behaviour. If we visualise this partition of the parameter space

Derivation of Bifurcation Manifolds

To derive analytical expressions for bifurcation manifolds, the system of nonlinear equations—composed of the fixed-point equations (2) and the appropriate test condition of Table 1—has to be solved for the external inputs uRn yielding an (n-1)-dimensional manifold in input space. To this end, we assume that the network's weights are constant or at least varying on a slow time scale.

The key idea is to split the solution process into two stages: First, we solve the test condition within the

Numerical Computation and Visualisation

A satisfactory visualisation of the bifurcation manifolds poses another challenging task—mostly due to the large number of separate solution branches in input space. The visualisation of a bifurcation diagrams requires to sample the solution manifold in ψ-space along a rectangular (n-1)-dimensional grid and transfer the obtained point set to the input space employing Eq. (5). To achieve a uniform sampling of data points in the input parameter space, it is necessary to vary the grid spacing Δψk

Example: A cascaded three-neuron RNN

As an example we study the dynamical properties of the cascaded three-neuron RNN, shown in Fig. 4. It is composed of a two-neuron network, which exhibits stable oscillatory behaviour within a circular region in the u1u2-plane and which projects onto a third neuron, which exhibits a hysteresis domain of two stable fixed points in a certain input interval [u3-,u3+] as long as the self-feedback weight satisfies c>1 [8], [7].

Due to the cascaded nature of the network, its qualitative dynamical

Discussion

We present analytical expressions for bifurcation manifolds in the input space of general discrete-time RNNs, which we solve for the case of three-neuron networks. Employing these expressions, we can visualise the complex partitioning of the input space into regions of different dynamical behaviour. This allows us to choose directly input control variables in order to provoke a specific dynamical behaviour. Due to the real-time computation of the bifurcation diagrams it is additionally possible

Acknowledgement

This work was supported by the DFG Grants GK-231.

Robert Haschke received the diploma in Computer Science from Bielefeld University, Germany, in 1999. He obtained a 3-year scholarship within the graduate program ‘Strukturbildungsprozesse’ (‘Structure forming phenomena’) and received the PhD degree from Bielefeld University in 2004 with a thesis on oscillating recurrent neural networks. He is currently employed in the project “D6 - Architectures for Learning by Imitation” of the CRC 360 at Bielefeld University, working within the

References (14)

There are more references available in the full text version of this article.

Cited by (0)

Robert Haschke received the diploma in Computer Science from Bielefeld University, Germany, in 1999. He obtained a 3-year scholarship within the graduate program ‘Strukturbildungsprozesse’ (‘Structure forming phenomena’) and received the PhD degree from Bielefeld University in 2004 with a thesis on oscillating recurrent neural networks. He is currently employed in the project “D6 - Architectures for Learning by Imitation” of the CRC 360 at Bielefeld University, working within the Neuroinformatics Group. His fields of research include recurrent neural networks, virtual and real robotics, and grasping with multifingered dextrous hands.

Jochen J. Steil (www.jsteil.de) received the diploma in mathematics from the University of Bielefeld, Germany, in 1993. Since then he has been a member of the Neuroinformatics Group at the University of Bielefeld, interrupted by one year at the St. Petersburg Electrotechnical University, Russia under support of a German Academic Exchange Foudation (DAAD) grant. In 1999, he received the PhD. Degree with a Dissertation on “Input-Output Stability of Recurrent Neural Networks”, since 2002 he has been appointed tenured senior researcher and teacher (Akad. Rat). J.J. Steil is staff member of the special research unit 360 “Situated Artifical Communicators” and the Graduate Program ”Task Oriented Communication” and heads projects on robot learning and intelligent systems. Main research interests of J.J. Steil are analysis, stability, and control of recurrent dynamics and learning as well as the development of learning architectures for complex cognitive robots suited for multimodal Human-Machine communication, interaction, and instruction of grasping. He is member of the ENNS and the IEEE computational intelligence society.

View full text