Analyzing the weight dynamics of recurrent learning algorithms

Abstract

We provide insights into the organization and dynamics of recurrent online training algorithms by comparing real time recurrent learning (RTRL) with a new continuous-time online algorithm. The latter is derived in the spirit of a recent approach introduced by Atiya and Parlos (IEEE Trans. Neural Networks 11 (3) (2000) 697), which leads to non-gradient search directions. We refer to this approach as Atiya–Parlos learning (APRL) and interpret it with respect to its strategy to minimize the standard quadratic error. Simulations show that the different approaches of RTRL and APRL lead to qualitatively different weight dynamics. A formal analysis of the one-output behavior of APRL further reveals that the weight dynamics favor a functional partition of the network into a fast output layer and a slower dynamical reservoir, whose rates of weight change are closely coupled.

Introduction

Recurrent neural networks (RNNs) are attractive tools for tasks of sequential nature like time-series prediction, sequence generation, speech recognition, or adaptive control. Many architectures ranging from fully connected to partially or locally RNNs have been developed. Although their application is successful in practice, the dynamical behavior of the networks leads to high complexity of the algorithms. The credit assignment problem and the potentially rich dynamical properties of the networks make it difficult to devise efficient recurrent learning schemes. Contemporary approaches frequently employ regularization techniques to tackle these problems [2]. The analysis of dynamical properties of recurrent networks and the corresponding learning algorithms nevertheless remains a challenging field of research.

Some encouraging results are available with respect to formulate a common unifying framework for gradient-based techniques like real time recurrent learning (RTRL) or backpropagation through time (BPTT). Atiya and Parlos [1] have shown that these can be derived from a constrained optimization problem which combines the quadratic error with constraints reflecting the network dynamics. They also introduced a new strategy to minimize the standard error which employs search directions different from the gradient. It considers the states as control variables and computes weight updates to achieve a targeted change in the state variables. We reference to this strategy as Atiya–Parlos learning (APRL). In [1], an $\mathcal{O}(n^2)$-efficient APRL algorithm was given for discrete networks, while we will introduce below an APRL algorithm for continuous-time networks. We also show that APRL can be interpreted as a truncated “one-step-backward” propagation of the instantaneous error combined with a momentum term providing the necessary dynamic memory.

The existence of such different approaches as RTRL and APRL to minimize the standard quadratic error function along different search directions motivates further investigations on whether and how the resulting weight dynamics differ. This is especially interesting for online trajectory learning, because in this case the input data are highly correlated. In this case, the gradient direction becomes a well-founded heuristics because it does not implement a stochastic gradient descent any more and rather the combined dynamics of error function and weights determine the learning success. The obvious problem is that direct access to the dynamics of online learning on the high-dimensional error surface is not possible for two main reasons: because of the multiple constraints acting through the network dynamics and the complex dependency of the time-varying error surface on the network parameters.

Therefore, we use a comparative approach to investigate the Atiya–Parlos methodology and its weight dynamics as opposed to results for RTRL [8] on two typical tasks. Simulations and theoretical results show that the two algorithms behave quite different and yield evidence that their different ways to minimize the error lead to unrelated weight dynamics. This point of view is further supported by a formal treatment of the one-output case of APRL which reveals a functional division of the network which resembles more the “echo state network” [4], [5] and “liquid state machine” approaches [6], [7] than classical backpropagation.

The remainder of this work is organized as follows: In Section 2, we give a continuous-time online algorithm based on the APRL paradigm [11] and interpret the algorithm with respect to its strategy to minimize the error function. In Section 3, we present simulation results from APRL and RTRL and analyze the weight dynamics of the algorithms. In Section 4, we turn to the one-output case and prove that APRL leads to a very special weight dynamics that results in a functional partition of the network. In Section 5, we summarize the results and discuss the insights gained hereby.