An investigation of neural networks for linear time-series forecasting
Introduction
Time-series modeling and forecasting continues to be an important area in both academic research and practical application. Historical observations on the item to be forecast are collected and analyzed to specify a model to help capture the underlying data generating process. Then the model is used to predict the future. There are two different approaches to modeling time series depending on the theory or assumption about the relationship in the data. Traditional methods such as the time-series regression, exponential smoothing and autoregressive integrated moving average (ARIMA) are based on linear models. That is, they assume that the future value of a time series is linearly related to the past observations. In particular, the ARIMA model is representative of linear models and has achieved great popularity since the publication of Box–Jenkins’ classic book: Time-Series Analysis: Forecasting and Control [1]. On the other hand, a number of nonlinear time-series models have been developed in the last two decades based on the beliefs that most real-world problems are nonlinear and the linear approximation to the complex real situation may not be appropriate.
Traditionally, linear statistical forecasting methods have been widely used in many real-world situations. Linear models are easy to develop and implement. They are also simple to understand and interpret. For situations where a large number of items need to be forecast and/or forecast accuracy is not a demanding requirement, these characteristics are especially attractive. Linear models, however, do have the limitation in that many real-world problems are nonlinear [2]. Using linear models to approximate complex nonlinear problems is often not satisfactory particularly when the forecasting horizon is relatively long. Results from several large-scale forecasting competitions such as the M-competition [3] show that there is no single linear method which uniquely dominates for all data sets across all situations. One possible reason is that there is a varying degree of nonlinearity in the data which may not be handled adequately by linear statistical methods.
A variety of nonlinear time-series models have been proposed aiming to improve the forecasting performance for nonlinear systems. Among them, the bilinear model [4] the threshold autoregressive (TAR) model [5], the smoothing transition autoregressive (STAR) model [6], the autoregressive conditional heteroscedastic (ARCH) model [7] and generalized autoregressive conditional heterosecdastic (GARCH) model [8] receive the most attention. These ‘second-generation time-series models’ [9] are useful in both understanding the behavior of some nonlinear systems and solving real problems. The problem with these nonlinear parametric models is that they are developed specifically for particular problems without general applicability for other situations. For example, the basic ARCH and GARCH models are proposed to deal exclusively with the nonconstant conditional variance of the process. The pre-specified model forms also restrict the usefulness of these models since there are many possible nonlinear patterns and one specific form may not capture all the nonlinearieties in the data. Only limited success or gain has been found during the last two decades in using nonlinear models [10]. In addition, the formulation of an appropriate nonlinear model to a particular data set is a difficult task compared to building a linear model as “there are more possibilities, many more parameters and thus more mistakes can be made” [11].
Recently, artificial neural networks have been proposed as a promising alternative approach to time-series forecasting. A large number of successful applications have shown that neural networks can be a very useful tool for time-series modeling and forecasting [12]. Neural networks are basically data-driven methods with few priori assumptions about the underlying models. Instead they let data speak for themselves and have the capability to identify the underlying functional relationship in the data.
Neural networks belong to a generalized nonlinear modeling family. Theoretically, Cybenko [13], Hornik et al. [14], and Hornik [15] have established that neural networks are universal functional approximators and can approximate any nonlinear function with arbitrary accuracy. This is a very important advance for neural networks since the number of possible nonlinear patterns is huge for real-world problems and a good model should be able to approximate them all well [11]. Empirically, neural networks have been shown to be effective in modeling and forecasting nonlinear time-series with or without noises [16], [17], [18]. Many comparisons have been made between neural networks and traditional (linear) methods on time-series forecasting performance. While most researchers find that neural networks can outperform linear methods under a variety of situations, the conclusions are not consistent [12]. Although it is expected in theory that neural networks are suitable for problems with nonlinear structure [19], it is often difficult in reality to determine whether a problem under study is linear or nonlinear.
The purpose of this study is to explore the effectiveness of neural networks for linear time-series modeling and forecasting. The motivation comes from the question: what happens if neural networks are applied to an inherently linear process. This is a nontrivial question since as mentioned, it is not an easy task to determine whether the underlying data generating process of a real problem is linear or nonlinear. Forecasters often have a difficult time deciding whether a linear or a nonlinear model should be used for their problems. Although several nonlinear tests are now available, these tests are developed against specific forms of nonlinear patterns and do not have general capability to detect unknown nonlinear relationships. In addition, since almost all real data contain random errors or noise, the assumptions of the traditional linear methods may not be all satisfied even though the data come from a linear process. Hence, the estimation as well as the forecasting from linear models may be biased. It is of interest to know the capability of neural networks in modeling linear process when there is certain degree of noise as well as the effects sample size and neural network structure may have on their comparative performance. Furthermore, it is argued that using a flexible nonlinear model such as neural networks to model an essentially linear process is unnecessarily cumbersome and overfitting that will lead to a loss in forecast accuracy [20]. However, there has been no formal investigation on the effect of doing so in the context of time-series analysis and forecasting. Mixed findings have been reported in terms of the neural network capability for linear problems such as regression and classification. In a simulation study conducted by Markham and Rakes [21], the performance of neural networks was compared with that of linear regression on simple linear regression problems with varying sample sizes and noise levels. It was found that for linear regression problems with different levels of error variance and sample size, neural networks and linear regression models performed differently. At lower variance levels, regression models were better while at higher levels of variance, neural networks performed better. Experimenting with simulated data for linear regression problems, Denton [22] showed that, under ideal conditions with all assumptions satisfied, there was little difference in performance between neural networks and regression models. However, under less ideal conditions such as outliers, multicollinearity, and model mis-specification, neural networks performed better. Subramanian et al. [23] compared neural network models with classical classification models for problems ideal for traditional methods. They found that even under ideal conditions for the classical models, neural networks were still quite competitive.
In this paper, our focus is on the effects of neural network architecture on its performance for linear time-series forecasting. Traditional ARIMA models are employed to serve as a baseline for comparison. The results may have practical implications. If neural networks can compete with linear methods for linear problems, then it will be a great advantage for this technique to be used in even broader situations no matter whether the underlying relationship is linear or nonlinear. On the other hand, if neural networks fail to give good performance with linear problems, care should be excised in choosing an appropriate model for a particular situation.
The paper is organized as follows. The next section describes general features and applications of neural networks for time series forecasting. Research design and methodology as well as the data are outlined in Section 3. Results are discussed in Section 4. Finally, Section 5 summarizes the main findings and conclusions.
Section snippets
Neural networks for time-series forecasting
Neural networks are computing models for information processing. They are particularly useful for identifying the fundamental functional relationship or pattern in the data. Fig. 1 is a popular neural network model – the feedforward multi-layer network. It is composed of several layers of basic processing units called neurons or nodes. Here the network model has one input layer, one hidden layer, and one output layer. The nodes in the input layer are used to receive information from the data.
Research design
The major research questions we try to address in this study are summarized as follows:
- •
Can neural networks approximate and forecast well the underlying structure of linear time series?
- •
What is the relative performance of neural networks compared to traditional ARIMA methods for linear time-series forecasting?
- •
What are the effects of neural network architecture on the in-sample modeling and out-of-sample forecasting ability of the network models?
To answer these questions, we conduct a
Results
This section reports the results of the simulation study and the real data applications. For the simulation study, the effect of three factors of input nodes, hidden nodes and training sample size is investigated with the SAS ANOVA procedure. Duncan's multiple range test is used to examine the main effect of the number of input nodes, the number of hidden nodes along with three different sample sizes. Three test sets with different time horizons of and 80 are employed to examine the
Summary and conclusions
Artificial neural networks have been widely used for various forecasting problems ranging from engineering to business. Their flexible nonlinear modeling capability is particularly useful for many complex real-world problems. A large number of simulation studies as well as real applications in the literature have established that neural networks are a valuable tool for nonlinear time-series analysis and forecasting.
This study investigates the effectiveness of neural networks for linear
Acknowledgements
I would like to thank two anonymous reviewers for their constructive comments and helpful suggestions.
Guoqiang Peter Zhang is Assistant Professor of Decision Sciences at Georgia State University. He received his Ph.D. in Operations Management/Operations Research from Kent State University. His current research interests include neural networks and time series forecasting. His research has appeared in Computers & Operations Research, Decision Sciences, European Journal of Operational Research, International Journal of Forecasting, International Journal of Production Economics, OMEGA, and others.
References (58)
Generalised autoregressive conditional heteroscedasticity
Journal of Econometrics
(1986)- et al.
Some recent developments in non-linear time series modelling, testing, and forecasting
International Journal of Forecasting
(1992) - et al.
Forecasting with artificial neural networks: The state of the art
International Journal of Forecasting
(1998) - et al.
Multilayer feedforward networks are universal approximators
Neural Networks
(1989) Approximation capability of multilayer feedforward networks
Neural Networks
(1991)Nonlinear prediction of noisy time series with feedforward networks
Physics Letters A
(1994)Research prospective on neural network forecasting
International Journal of Forecasting
(1994)- et al.
The effect of sample size and variability of data on the comparative performance of artificial neural networks and regression
Computers & Operations Research
(1998) - et al.
An experimental evaluation of neural networks for classification
Computers & Operations Research
(1993) Model-free forecasting for nonlinear time series (with application to exchange rates)
Computational Statistics & Data Analysis
(1995)
Artificial neural networks for forecasting and decision making
International Journal of Forecasting
Stock price prediction using neural networks: a project report
Neurocomputing
Training neural networks with the GRG2 nonlinear optimizer
European Journal of Operational Research
Error measures for generalizing about forecasting methods: empirical comparisons
International Journal of Forecasting
The evaluation of extrapolative forecasting methods
International Journal of Forecasting
Time series analysis: forecasting and control
Modelling nonlinear economic relationships
The accuracy of extrapolation (time series) methods: results of a forecasting competition
Journal of Forecasting
An introduction to bilinear time series models
Threshold autoregression, limit cycles and cyclical data
Journal of Royal Statistical Society, B
On tests for non-linearity in time series analysis
Journal of Forecasting
Autoregressive conditional heteroscedasticity with estimates of the variance of U.K. inflation
Econometrica
Second-generation time-series models: a comment on ‘Some advances in non-linear and adaptive modelling in time-series analysis’ by Tiao and Tsay
Journal of Forecasting
Strategies for modelling nonlinear time-series relationships
The Economic Record
Approximation by superpositions of a sigmoidal function
Mathematical Control Signals Systems
Nonlinear time series analysis by neural networks: a case study
International Journal of Neural Systems
The efficacy of neural networks in predicting returns on stock and bond indices
Decision Sciences
Cited by (162)
Improved dynamic programming algorithms for unconstrained two-dimensional guillotine cutting
2024, Computers and Operations ResearchFuzzy Autoregressive Distributed Lag model-based forecasting
2023, Fuzzy Sets and SystemsInfluence of exogenous factors on water demand forecasting models during the COVID-19 period
2023, Engineering Applications of Artificial IntelligenceFrom predictive to prescriptive analytics: A data-driven multi-item newsvendor model
2020, Decision Support SystemsA hybrid optimized error correction system for time series forecasting
2020, Applied Soft Computing JournalDeep learning-based short-term water demand forecasting in urban areas: A hybrid multichannel model
2024, Aqua Water Infrastructure, Ecosystems and Society
Guoqiang Peter Zhang is Assistant Professor of Decision Sciences at Georgia State University. He received his Ph.D. in Operations Management/Operations Research from Kent State University. His current research interests include neural networks and time series forecasting. His research has appeared in Computers & Operations Research, Decision Sciences, European Journal of Operational Research, International Journal of Forecasting, International Journal of Production Economics, OMEGA, and others.