Small, narrow, and parallel recurrent neural networks for sentence representation in extractive text summarization

Abstract

Recurrent Neural Networks (RNN) and their variants like Gated Recurrent Units (GRUs) have been the de-facto method in Natural Language Processing (NLP) for solving a range of NLP problems, including extractive text summarization. However, for certain sequential data with multiple temporal dependencies like the human text data, using a single RNN over the whole sequence might prove to be inadequate. Transformer models that use multiheaded attention have shown that human text contains multiple dependencies. Supporting networks like attention layers are needed to augment the RNNs to capture the numerous dependencies in text. In this work, we propose a novel combination of RNNs, called Parallel RNNs (PRNN), where small and narrow RNN units work on a sequence, in parallel and independent of each other, for the task of extractive text summarization. These PRNNs, without the need for any attention layers, capture various dependencies present in the sentence and document sequences. Our model achieved a 10% gain in ROUGE-2 score over the single RNN model on the popular CNN/Dailymail dataset. The boost in performance indicates that such an ensemble arrangement of RNNs improves the performance compared to the standard single RNNs, which alludes to the fact that constituent units of the PRNN learn various input sequence dependencies. Hence, the sequence is represented better using the combined representation from the constituent RNNs.