What is the best RNN-cell structure to forecast each time series behavior?

Highlights

• A taxonomy of time series behaviors was proposed.

• A taxonomy of Recurrent Neural Network (RNN) cell structures was proposed.

• A set of 31 RNN cell structures was evaluated.

• The utility of each component in the Long–Short Term Memory cell was evaluated.

• The best cell structure for forecasting each time series behavior was provided.

Abstract

It is unquestionable that time series forecasting is of paramount importance in many fields. The most used machine learning models to address time series forecasting tasks are Recurrent Neural Networks (RNNs). Typically, those models are built using one of the three most popular cells, ELMAN, Long–Short Term Memory (LSTM), or Gated Recurrent Unit (GRU) cells, each cell has a different structure and implies a different computational cost. However, it is not clear why and when to use each RNN-cell structure. Actually, there is no comprehensive characterization of all the possible time series behaviors and no guidance on what RNN cell structure is the most suitable for each behavior. The objective of this study is two-fold: it presents a comprehensive taxonomy of all possible time series behaviors (deterministic, random-walk, nonlinear, long-memory, and chaotic), and provides insights into the best RNN cell structure for each time series behavior. We conducted two experiments: (1) The first experiment evaluates and analyzes the role of each component in the LSTM-Vanilla cell by creating 11 variants based on one alteration in its basic architecture (removing, adding, or substituting one cell component). (2) The second experiment evaluates and analyzes the performance of 20 possible RNN-cell structures. To evaluate, compare, and select the best model, different statistical metrics were used: error-based metrics, information criterion-based metrics, naïve-based metric, and direction change-based metric. To further improve our confidence in the models’ interpretation and selection, Friedman Wilcoxon–Holm signed-rank test was used.

Our results advocate the usage and the exploration of the newly created RNN variant, named SLIM, in time series forecasting thanks to its high ability to accurately predict the different time series behaviors as well as its simple structural design that does not require expensive temporal and computing resources.

Introduction

Many real-world prediction problems involve a temporal dimension and typically require the estimation of numerical sequential data referred to as time series forecasting. Time series forecasting is one of the major stones in data science playing a pivotal role in almost all domains, including meteorology (Murat, Malinowska, Gos, & Krzyszczak, 2018), natural disasters control (Erdelj, Król, & Natalizio, 2017), energy (Bourdeau, Zhai, Nefzaoui, Guo, & Chatellier, 2019), manufacturing (Wang & Chen, 2018), finance (Liu, 2019), econometrics (Siami-Namini & Namin, 2018), telecommunication (Maeng, Kim, & Shin, 2020), healthcare (Khaldi, E. Afia, & Chiheb, 2019b) to name a few. Accurate time series forecasting requires robust forecasting models.

Currently, Recurrent Neural Network (RNN) models are one of the most popular machine learning models in sequential data modeling, including natural language, image/video captioning, and forecasting (Chimmula and Zhang, 2020, Sutskever et al., 2014, Vinyals et al., 2015). Such RNN models are built as a sequence of the same cell structure, for example, ELMAN cell, Long–Short Term Memory (LSTM) cell or Gated Recurrent Unit (GRU) cell. The simplest RNN cell is ELMAN, it includes one layer of hidden neurons. While, LSTM and GRU cells incorporate a gating mechanism, three gates in LSTM and two gates in GRU, where each gate is a layer of hidden neurons. Many other cell structures have been introduced in the literature (Lu and Salem, 2017, Mikolov et al., 2014, Pulver and Lyu, 2017, Zhou et al., 2016). However, to solve time series forecasting tasks, the building of RNN models is typically limited to the three aforementioned cell structures (Alkhayat and Mehmood, 2021, Liu et al., 2021, Rajagukguk et al., 2020, Runge and Zmeureanu, 2021, Sezer et al., 2020), as they provide very good accuracy (Runge and Zmeureanu, 2021, Sezer et al., 2020).

Nevertheless, building robust RNN models for time series forecasting is still a challenging task as there does not exist yet a clear understanding of times series data itself and hence there exist very little knowledge about what cell structure is the most appropriate for each data type. In general, when facing a new problem, practitioners select one of the most popular cells, usually LSTM, and use it as a building block for the RNN model without any guarantee on the appropriateness of this cell to the current data. The objective of this work is two-fold. It presents a comprehensive characterization of time series behaviors and provides guidelines on the best RNN cell structure for each behavior. As far as we know, this is the first work providing such insights. The main contributions of this study can be summarized as follows:

• To provide a better understanding of times series data by presenting a comprehensive characterization of their behaviors.

• To determine the most appropriate cell structure for each time series behavior (i.e., whether a specific cell structure should be avoided for certain behaviors).

• To identify differences in predictability between behaviors (i.e., whether certain behaviors are easier or harder to predict across all cell models).

• To provide useful guidelines that can assist decision-makers and scholars in the process of selecting the most suitable RNN-cell structure from both, a computational and performance point of view.

The remainder of this study is organized as follows: Section 2 states the related works. Section 3 presents a taxonomy of time series behaviors. Section 4 presents a taxonomy of RNN cells. Section 5 describes the experiment. Section 6 exhibits and discusses the obtained results. Finally, the last section concludes the findings and spots light on future research directions.