Encoding-based memory for recurrent neural networks

Abstract

Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study memorization from the point of view of the design and training of recurrent neural networks. We study how to maximize the short-term memory of recurrent units, an objective difficult to achieve using backpropagation. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences. Additionally, we provide a specialized training algorithm that initializes the memory to efficiently encode the hidden activations of the network. Experimental results on synthetic and real-world datasets show that the chosen encoding mechanism is superior to static encodings such as orthogonal models and the delay line. The method also outperforms RNN and LSTM units trained using stochastic gradient descent. Experiments on symbolic music modeling show that the training algorithm specialized for the memorization component improves the final performance compared to stochastic gradient descent.

Introduction

Sequential data are ubiquitous and characterize as the primary and most effective representation for several information domains, including music, speech, text, videos. Recurrent neural networks are possibly the most popular learning models for sequential data, with impacting applications in natural language processing [1], speech recognition [2] or time series analysis [3]. Nonetheless, the adaptive processing of sequential data still poses high challenges mainly due to its dynamical nature, which requires maintaining an appropriate short-term memory of the elements of the sequence to succeed in the downstream tasks.

When it comes to recurrent neural models, such challenges have been well materialized by the exploding and vanishing gradient problem [4] which affects the capability of the recurrent models to learn long-term dependencies between elements of the sequences. Successive studies [5] have further deepened the understanding of such gradient issues from a geometric and dynamical system perspective, promoting a whole body of research focusing on attempts to solve these problems with gated architectures [6], [7], [8] or through orthogonal models [9], [10]. Although from different perspectives, both approaches attempt to circumvent the instability of the gradient from the same perspective, by attacking the underlying numerical problem, either through an architectural solution or by leveraging spectral properties of the weight matrices. A second line of research focuses on characterizing the short-term memory capacity of recurrent networks to gain insights allowing to surpass the limitations of dynamical memories in recurrent models [11]. A notable attempt, in this sense, is the reservoir computing design paradigm [12], [13], [14], which advocates the use of untrained dynamic memories with well-defined guarantees on their short-term memory capacity, again leveraging spectral considerations on recurrent weight matrices.

In this paper, we propose a novel approach to address memorization challenges in recurrent neural models, which puts forward a third way between the random encoding in the reservoir paradigm and the vanishing-gradient prone approach of fully-trained recurrent neural networks [4]. The objective is to train memorization units to maximize their short-term memory capacity, a difficult objective to achieve with backpropagation. Our contribution founds on an original intuition concerning the nature of supervised sequential tasks. We claim that solving sequential tasks entails addressing two separate subproblems: (i) the extraction of an input representation that is informative and effective for successfully addressing the target tasks; (ii) the memorization of relevant and task-effective information using limited resources [12], [15]. In the remainder of this paper, we will simply refer to the short-term memory and memorization component as memory for brevity. Notice that, this separation is not strictly necessary to train recurrent models, however, we believe that such separation may be helpful to devise novel and more effective models and algorithmic solutions. In fact, we show how such intuition can be mapped to a new design principle for recurrent neural networks which explicitly considers the conceptual separation of sequential problems into two subtasks:

• functional subtask, that is the extraction of task-relevant features, and therefore the mapping from the previous input subsequence and the current timestep into a set of abstract task-dependent features.

• memorization subtask, that is responsible for the update of the internal state of the model, and therefore the memorization of the new task-relevant features.

Throughout this work, we describe how such conceptual separation principles can be translated to a practical architectural pattern for recurrent models, which allows simplifying their design and defining specialised training algorithm to optimize their memorization abilities. Fig. 1 provides a first high-level overview of our architectural design: note how under our separation principle, only the memorization component is recurrent (however, the recurrence still affects the functional component indirectly). The paper further provides a concretisation of the proposed architectural design by introducing a novel recurrent neural network, dubbed Linear Memory Network (LMN), that addresses the memorization subtask through a simple short-term memory component based on a linear autoencoder for sequences (LAES) [16]. The linear coding provided by the LAES allows to efficiently encode the sequence of features, improving the short-term memory capacity of the model. We show how such a simple memory can be leveraged to provide a natural solution to the memorization problem where the autoencoder can be trained to memorize the entire sequence of task-relevant features, while the functional component can leverage an expressive non-linear model to realize the input-output mapping. In addition, we propose a novel training algorithm that builds on our reference architecture by attempting to solve the two subproblems separately, in different stages. Finally, the proposed training approach integrates the two components by an end-to-end fine tuning phase, whose aim is to bring in the advantage of parallel associative information processing needed to optimize the final solution. The last contribution of this work is an empirical characterization of the effectiveness of the LAES memory along with a comparative analysis against related approaches from the literature, dealing with the design of dynamical memories for recurrent neural models.

This paper extends a previous conference paper [17] which has introduced the LMN concept in a limited empirical setting. This work extends the original paper by:

• defining the conceptual framework based on the separation between the short-term memory and functional components and leveraging it to discuss short-term memory capacity and learnability of different recurrent architectures;

• extending the analysis of the LMN memory component and the formalization of its training algorithms, characterizing the LAES memorization capabilities compared to recent relevant works from literature;

• providing an empirical characterization of the architectural bias given by different recurrent autoencoders providing insights on how they allocate their short-term memory capacity by preferring recent or distant elements in the sequence. To the best of our knowledge, this paper is the first to provide an analysis of the short-term memory capacity for adaptive recurrent autoencoders.

The paper is organized as follows: Section 2 introduces background material, including the notation and the LAES. Section 2.4 gives a review of RNN architectures and their ability to solve the memorization and functional subtasks, and the linear autoencoder for sequences. Section 3 describes the LMN architecture. Section 3.2 presents the specialized training algorithm used to train the memorization component of the LMN. Section 4 shows the experimental results of the autoencoders on the reconstruction of synthetic and real-world data and a comparison of recurrent architectures on symbolic music modeling. Finally, we draw the conclusions in Section 5 and highlight possible avenues for future work.