An investigation of neural networks for linear time-series forecasting

Abstract

This study examines the capability of neural networks for linear time-series forecasting. Using both simulated and real data, the effects of neural network factors such as the number of input nodes and the number of hidden nodes as well as the training sample size are investigated. Results show that neural networks are quite competent in modeling and forecasting linear time series in a variety of situations and simple neural network structures are often effective in modeling and forecasting linear time series.

Scope and purpose

Neural network capability for nonlinear modeling and forecasting has been established in the literature both theoretically and empirically. The purpose of this paper is to investigate the effectiveness of neural networks for linear time-series analysis and forecasting. Several research studies on neural network capability for linear problems in regression and classification have yielded mixed findings. This study aims to provide further evidence on the effectiveness of neural network with regard to linear time-series forecasting. The significance of the study is that it is often difficult in reality to determine whether the underlying data generating process is linear or nonlinear. If neural networks can compete with traditional forecasting models for linear data with noise, they can be used in even broader situations for forecasting researchers and practitioners.

Introduction

Time-series modeling and forecasting continues to be an important area in both academic research and practical application. Historical observations on the item to be forecast are collected and analyzed to specify a model to help capture the underlying data generating process. Then the model is used to predict the future. There are two different approaches to modeling time series depending on the theory or assumption about the relationship in the data. Traditional methods such as the time-series regression, exponential smoothing and autoregressive integrated moving average (ARIMA) are based on linear models. That is, they assume that the future value of a time series is linearly related to the past observations. In particular, the ARIMA model is representative of linear models and has achieved great popularity since the publication of Box–Jenkins’ classic book: Time-Series Analysis: Forecasting and Control [1]. On the other hand, a number of nonlinear time-series models have been developed in the last two decades based on the beliefs that most real-world problems are nonlinear and the linear approximation to the complex real situation may not be appropriate.

Traditionally, linear statistical forecasting methods have been widely used in many real-world situations. Linear models are easy to develop and implement. They are also simple to understand and interpret. For situations where a large number of items need to be forecast and/or forecast accuracy is not a demanding requirement, these characteristics are especially attractive. Linear models, however, do have the limitation in that many real-world problems are nonlinear [2]. Using linear models to approximate complex nonlinear problems is often not satisfactory particularly when the forecasting horizon is relatively long. Results from several large-scale forecasting competitions such as the M-competition [3] show that there is no single linear method which uniquely dominates for all data sets across all situations. One possible reason is that there is a varying degree of nonlinearity in the data which may not be handled adequately by linear statistical methods.

A variety of nonlinear time-series models have been proposed aiming to improve the forecasting performance for nonlinear systems. Among them, the bilinear model [4] the threshold autoregressive (TAR) model [5], the smoothing transition autoregressive (STAR) model [6], the autoregressive conditional heteroscedastic (ARCH) model [7] and generalized autoregressive conditional heterosecdastic (GARCH) model [8] receive the most attention. These ‘second-generation time-series models’ [9] are useful in both understanding the behavior of some nonlinear systems and solving real problems. The problem with these nonlinear parametric models is that they are developed specifically for particular problems without general applicability for other situations. For example, the basic ARCH and GARCH models are proposed to deal exclusively with the nonconstant conditional variance of the process. The pre-specified model forms also restrict the usefulness of these models since there are many possible nonlinear patterns and one specific form may not capture all the nonlinearieties in the data. Only limited success or gain has been found during the last two decades in using nonlinear models [10]. In addition, the formulation of an appropriate nonlinear model to a particular data set is a difficult task compared to building a linear model as “there are more possibilities, many more parameters and thus more mistakes can be made” [11].

Recently, artificial neural networks have been proposed as a promising alternative approach to time-series forecasting. A large number of successful applications have shown that neural networks can be a very useful tool for time-series modeling and forecasting [12]. Neural networks are basically data-driven methods with few priori assumptions about the underlying models. Instead they let data speak for themselves and have the capability to identify the underlying functional relationship in the data.

Neural networks belong to a generalized nonlinear modeling family. Theoretically, Cybenko [13], Hornik et al. [14], and Hornik [15] have established that neural networks are universal functional approximators and can approximate any nonlinear function with arbitrary accuracy. This is a very important advance for neural networks since the number of possible nonlinear patterns is huge for real-world problems and a good model should be able to approximate them all well [11]. Empirically, neural networks have been shown to be effective in modeling and forecasting nonlinear time-series with or without noises [16], [17], [18]. Many comparisons have been made between neural networks and traditional (linear) methods on time-series forecasting performance. While most researchers find that neural networks can outperform linear methods under a variety of situations, the conclusions are not consistent [12]. Although it is expected in theory that neural networks are suitable for problems with nonlinear structure [19], it is often difficult in reality to determine whether a problem under study is linear or nonlinear.

The purpose of this study is to explore the effectiveness of neural networks for linear time-series modeling and forecasting. The motivation comes from the question: what happens if neural networks are applied to an inherently linear process. This is a nontrivial question since as mentioned, it is not an easy task to determine whether the underlying data generating process of a real problem is linear or nonlinear. Forecasters often have a difficult time deciding whether a linear or a nonlinear model should be used for their problems. Although several nonlinear tests are now available, these tests are developed against specific forms of nonlinear patterns and do not have general capability to detect unknown nonlinear relationships. In addition, since almost all real data contain random errors or noise, the assumptions of the traditional linear methods may not be all satisfied even though the data come from a linear process. Hence, the estimation as well as the forecasting from linear models may be biased. It is of interest to know the capability of neural networks in modeling linear process when there is certain degree of noise as well as the effects sample size and neural network structure may have on their comparative performance. Furthermore, it is argued that using a flexible nonlinear model such as neural networks to model an essentially linear process is unnecessarily cumbersome and overfitting that will lead to a loss in forecast accuracy [20]. However, there has been no formal investigation on the effect of doing so in the context of time-series analysis and forecasting. Mixed findings have been reported in terms of the neural network capability for linear problems such as regression and classification. In a simulation study conducted by Markham and Rakes [21], the performance of neural networks was compared with that of linear regression on simple linear regression problems with varying sample sizes and noise levels. It was found that for linear regression problems with different levels of error variance and sample size, neural networks and linear regression models performed differently. At lower variance levels, regression models were better while at higher levels of variance, neural networks performed better. Experimenting with simulated data for linear regression problems, Denton [22] showed that, under ideal conditions with all assumptions satisfied, there was little difference in performance between neural networks and regression models. However, under less ideal conditions such as outliers, multicollinearity, and model mis-specification, neural networks performed better. Subramanian et al. [23] compared neural network models with classical classification models for problems ideal for traditional methods. They found that even under ideal conditions for the classical models, neural networks were still quite competitive.

In this paper, our focus is on the effects of neural network architecture on its performance for linear time-series forecasting. Traditional ARIMA models are employed to serve as a baseline for comparison. The results may have practical implications. If neural networks can compete with linear methods for linear problems, then it will be a great advantage for this technique to be used in even broader situations no matter whether the underlying relationship is linear or nonlinear. On the other hand, if neural networks fail to give good performance with linear problems, care should be excised in choosing an appropriate model for a particular situation.

The paper is organized as follows. The next section describes general features and applications of neural networks for time series forecasting. Research design and methodology as well as the data are outlined in Section 3. Results are discussed in Section 4. Finally, Section 5 summarizes the main findings and conclusions.