Elsevier

Applied Soft Computing

Volume 77, April 2019, Pages 605-621
Applied Soft Computing

Deep belief network-based AR model for nonlinear time series forecasting

https://doi.org/10.1016/j.asoc.2019.02.006Get rights and content

Highlights

  • The DBN-AR model combines the advantage of the DBN and SD-AR model.

  • The DBN-AR model is driven by the state signal.

  • The initial target values of the DBN are determined by pseudo inverse matrix.

  • The whole DBN-AR model is fine tuned by using a BP algorithm.

  • The results show that the DBN-AR model is superior to some existing models.

Abstract

For a class of nonlinear time series whose dynamic behavior smoothly changes with the system state, a state-dependent auto-regressive (SD-AR) model is proposed to characterize the nonlinear time series. A set of deep belief networks (DBNs) is used to build the state-dependent functional coefficients of the SD-AR model, and the proposed model is called DBN-AR model, which combines the advantage of DBN in function approximation and the merit of SD-AR model in nonlinear dynamics description. The DBN-AR model is driven by the state signal changing with time. Based on the least squares solution with minimum norm and the pseudo inverse matrix approach, the initial target values of the DBNs are determined in pre-training stage. In fine tuning stage, all parameters of DBN-AR model is finally tuned by the back propagation (BP) algorithm designed for fine-tuning of DBN-AR model. Through experiment and comparative study on the sunspot data, the electricity load demand data sets from Australian Energy Market Operator (AEMO), the weekly British Pound/US dollar (GBP/USD) exchange rate data and the daily electricity generation data of the Three Gorges dam right bank power station, it is shown that the DBN-AR model is superior to some existing models or methods in prediction accuracy.

Introduction

Time series forecasting has become a very important research field because of its important applications in many fields [1]. For example, forecasting daily electricity generation is helpful to the power department; forecasting traffic condition helps people arrange travel reasonably [2]. However, time series data have different characteristics. It is difficult to use a same model or method to predict the different time series. In the past few decades, many time series models have been developed, an introduction to the time series forecasting models can be found in [3].

In general, a model may be categorized as parametric model or non-parametric model. From the point of view of modeling methods, the most frequently used time series prediction models may be divided into three types: statistical models, artificial intelligence (AI) models and hybrid models [4], [5]. In the first type, the autoregressive (AR) model, autoregressive moving average (ARMA) model, random walk (RW) and autoregressive integrated moving average (ARIMA) model are the widely used statistical models, in which the signal’s future values are modeled by a linear function of the past values. In literature, ARIMA model was applied to the different time series modeling, such as paddy production [6], short-term city electricity load [7] and daily wind speed [8]. ARMA model was used to forecast tourism demand [9], exchange rate [10] and electricity price [11], and the experimental results showed that the model has certain prediction accuracy. The AR type models above can well model linear parts of time series, but they are not suitable for nonlinear time series modeling [12].

In the second type, a large number of nonlinear artificial intelligence (AI) models have been used in different forecasting fields, such as artificial neural networks (ANNs), support vector machine (SVM), radial basis function (RBF) networks, and deep learning (DL) networks. A popular topic in modern data analysis has been ANNs [13], [14], [15], [16]. Many literatures used AI model to handle time series data for modeling and forecasting. Feedforward neural networks (FFNNs) are the most popular neural network paradigms in the prediction of time series [10]. For example, Palani et al. forecasted water quality using ANN [17]. Mandal et al. proposed a novel approach to forecasting electricity price for PJM using ANN and the similar days method [18]. Zhou et al. [19] proposed a SVM for short-term wind speed forecasting. The RBF neural network offers an alternative to traditional models because of its simple structure and strong learning capability [13]. The ANN models have been used in some classical time series and real life time series modeling, and the results showed that the ANN model has superior performance compared to some existing methods [20], [21], [22]. However, over fitting and the learning process to stop at local optima are two problems that ANN needs to avoid [23], [24]. To solve those two problems, Hinton et al. [25] proposed a deep belief network (DBN) model with multiple Restricted Boltzmann Machines (RBMs). DBN now is widely used in time series forecasting [12], [23], [26], [27], [28], and the results have shown DBN’s superiority over linear AR type model and conventional back propagation neural network (BPNN). For example, Kuremoto et al. [23] proposed a DBN model for time series forecasting, and the DBN’s superiority over conventional multilayer perceptron (MLP) model and ARIMA model has been verified by the CATS benchmark data modeling. The long short term memory (LSTM) neural network is a novel architecture of neural networks, which has good prediction performance and can overcome the issue of back-propagated error decay through memory blocks [29].

However, the traditional single model may not accurately represent the complex relations existed in the nonlinear and non-stationary time series [4], [30], [31], [32], [33], [34], [35]. Thus, in recent years, researchers studied some hybrid models that combine neural networks with other models. Hybrid model aims to further improve the performance of forecasting methods by strategically combining multiple algorithms, so this kind of hybrid models may achieve higher prediction accuracy [36]. For example, Ref. [37] showed that the empirical mode decomposition (EMD) based hybrid methods normally outperform the corresponding single structure models for time series forecasting, and nine benchmark methods have compared to verify the effectiveness of the EMD-DBN method, which are Persistence [37], Ensemble DBN (EDBN) [38], support vector regression (SVR) [39], ANN [40], DBN [25], random forest (RF) [41], EMD-SVR [42], EMD-ANN [43] and EMD-RF [37]. Schimbinschi et al. [44] proposed a learning topology-regularized universal vector autoregression (TRU-VAR) model for traffic forecasting, and the results showed that the proposed method scales well and can be trained efficiently with low generalization error. Li. et al. [45] combined traditional ANN with wavelet networks, and established a hybrid model for short-term power load forecasting, and the numerical testing showed that the proposed method can obtain better forecasting results in comparison with other standard and state of the art methods. In [46], the multiscale deep feature learning with hybrid model was used for daily reservoir inflow forecasting. In [12], DBN, ARIMA model and particle swarm optimization (PSO) method were combined for red tide time series forecasting. Although the hybrid model integrated some single models may improve the forecasting ability to some extent [47], [48], [49], [50], a single model often may not thoroughly deal with nonlinearity and non-stationarity of time series. State-dependent AR (SD-AR) model [51] can characterize nonlinear and non-stationary time series. On the basis of SD-AR modeling framework, Vesin [52] proposed the RBF-AR model that uses a set of RBF nets to approximate the functional coefficients of SD-AR model. Shi et al. [53] studied the estimation of RBF-AR model that was further developed by Peng et al. [54] to RBF-ARX model. Gan et al. [13] presented a modeling approach to nonlinear time series, which uses a set of locally linear RBF networks (LLRBF) to approximate the functional coefficients of SD-AR model. The RBF-ARX model has been applied to various model-based predictive control problems [55].

In general, as seen in [13], [54], [56], [57], [58], [59], the structure of radial basis function (RBF) network consists of only three layers: an input layer, a hidden layer and an output layer. Indeed, by adding more hidden layers the RBF network will be a deep network, so the input signals flow consecutively through the more hidden layers from the input to the output layer [60], [61], [62]. However, this type of deep network may be unstable as the network layer goes deeper, and its parameters estimation process is easy to fall into local minimum [60], [61], [62], which has a marked impact on the accuracy of the results. But as aforementioned, DBN model may solve the problem falling into local minimum [25]. Therefore, a type of SD-AR model, called DBN-AR model is proposed in this paper, which uses a set of DBNs to approximate the functional coefficients of SD-AR model.

The motivation to study DBN-AR model in this paper is to improve the RBF-AR modeling method [13], [54], [56] for nonlinear system modeling. The RBF-AR model is a state-dependent AR model [51] that can characterize nonlinear time series. However, The RBF-AR model’s coefficients are composed of the RBF nets with one hidden layer, which are not deep belief networks. Because the representing capability of the RBF net with one hidden layer to nonlinear behavior is usually weaker than that of deep belief network, we study the DBN-AR modeling problem in this paper by using DBN to replace the RBF net with one hidden layer in the RBF-AR for getting better nonlinear time series modeling performance. The DBN-AR model has the advantages of DBN in function approximation and SD-AR model in nonlinear dynamics description. Benefiting from the deeper layers of DBN and the SD-AR modeling framework, the proposed DBN-AR model may have a desirable modeling performance. The DBN-AR model is driven by system state signals to reflect the dynamic characteristics of time series. To estimate the DBN-AR model, a pseudo inverse matrix is designed to determine the initial target values of the DBNs in the DBN-AR model at the pre-training stage of the DBNs, and at fine tuning stage, all parameters of the DBN-AR model is finally tuned by the back propagation (BP) algorithm designed for fine-tuning of the DBN-AR model.

The DBN-AR model includes a DBN as one of its components, which may be regarded as a more general nonlinear model compared with a single DBN, and the structure of the DBN in the model may be much simpler than that of a single DBN model, because the DBN-AR model partially disperses the complexity of the model into the AR part. In addition, the DBN-AR model is trained in a greedy manner, for example, layer-wise pre-training, which permits training deeper network and alleviate trapping into local minimum [25]. On the other hand, a linear AR model with constant or time-varying coefficients is parametric model, and DBN is a non-parametric model. The DBN-AR model is basically a non-parametric model; however, if one regards it as a time-varying AR model with state-dependent DBN-type time-varying coefficients, then the DBN-AR model may be also regarded as a parametric model with time-varying coefficients. The DBN-AR model is actually a hybrid model combined of non-parametric and parametric approaches. In this paper, the proposed DBN-AR model is applied to characterize the sunspot time series [63], the electricity load demand data sets from Australian Energy Market Operator (AEMO) [64], the weekly British Pound/US dollar (GBP/USD) exchange rate data [10] and the daily electricity generation data from the Three Gorges dam right bank power station. The modeling results show that the DBN-AR model exhibits much better prediction accuracy compared with some existing modeling methods.

The remainder of this paper is organized as follows. Section 2 introduces the DBN-AR model. In Section 3, the estimation method for the DBN-AR model is presented. In Section 4, some performance measures are given to evaluate the validity of the estimated model. Results of the experimental investigation of the proposed DBN-AR model and its comparison with other modeling methods are described in Section 5. We give the future study and conclude this paper in Section 6. Finally, the fine tuning procedure of the DBN-AR model is presented in Appendix.

Section snippets

State-dependent AR (SD-AR) model

Without loss of generality, we consider one-step-ahead prediction for nonlinear time series. For a given nonlinear time series y(t),t=1,2,,N, the purpose of the time series modeling is to build a function with satisfactory prediction accuracy, f, and the form is defined as follows: yt=fyt1,yt2,,ytp+εtwhere f is the unknown nonlinear map, p is the order of the model, and εt is the Gaussian white noise.

Many types of functions have been applied to approximate the unknown nonlinear

Determining structure of DBN-AR model

Identification of DBN-AR model (4) mainly includes the selection of the order and the estimation of all the parameters when the model structure is determined. The Akaike Information Criterion (AIC) is used as the selection criteria of the DBN-AR model order and the architecture of the DBNs. AIC has been also used in the estimation of RBF-AR model [13], [54]. The AIC is defined as follows: AIC=Nlogδe2+2(s+1),Npwhere δe2 is the modeling residual variance under the chosen orders and DBN

Forecast error measures

To evaluate the modeling results of different models reasonably, under the same conditions, the modeling results can be analyzed by some evaluation criteria. In this paper, the mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), square sum of the error (SSE) and the normalized mean squared error (NMSE) are used as the evaluation criteria to analyze the modeling performance of different models or methods, which are given as

Case studies

To verify the validity of the proposed DBN-AR model, we provide a comprehensive experimental evaluation for the model. Modeling problems of the daily electricity generation of the right bank power station of the Three Gorges dam, the sunspot data [63], the electricity load data from Australian Energy Market Operator (AEMO) [64] and the weekly British Pound/US dollar (GBP/USD) exchange rate data [10] are studied in this section, and the modeling results are computed using the PC with Intel

Conclusions

Based on the deep belief network and the state-dependent AR model structure, the DBN-AR model was proposed to predict nonlinear time series in this paper. The DBN-AR model may combine the advantage of the state-dependent AR model in nonlinear dynamics description and the strong approximation capability of deep belief network for complex function. For estimating the DBN-AR model, we proposed an approach to generating the target values of the DBN modules of the model, which are used in the

Acknowledgments

The authors would like to thank the editors and the anonymous referees for their valuable comments and suggestions, which substantially improved the original manuscript. This work was supported by the National Natural Science Foundation of China (61773402, 51575167, 61540037).

References (67)

  • HamzaçebiC.

    Improving artificial neural networks’ performance in seasonal time series forecasting

    Inform. Sci.

    (2008)
  • ZhangG.P.

    A neural network ensemble method with jittered training data for time series forecasting

    Inform. Sci.

    (2007)
  • PalaniS. et al.

    An ANN application for water quality forecasting

    Mar. Pollut. Bull.

    (2008)
  • ZhouJ. et al.

    Fine tuning support vector machines for short-term wind speed forecasting

    Energy Convers. Manage.

    (2011)
  • AdhikariR.

    A neural network based linear ensemble framework for time series forecasting

    Neurocomputing

    (2015)
  • WangL. et al.

    Back propagation neural network with adaptive differential evolution algorithm for time series forecasting

    Expert Syst. Appl.

    (2015)
  • KuremotoT. et al.

    Time series forecasting using a deep belief network with restricted Boltzmann machines

    Neurocomputing

    (2014)
  • HongW.C.

    Chaotic particle swarm optimization algorithm in a support vector regression electric load forecasting model

    Energy Convers. Manage.

    (2009)
  • HuangH.B. et al.

    Evaluation of vehicle interior sound quality using a continuous restricted Boltzmann machine-based DBN

    Mech. Syst. Signal Process.

    (2017)
  • MaX. et al.

    Long short-term memory neural network for traffic speed prediction using remote microwave sensor data

    Transp. Res. C

    (2015)
  • ZhangG.P.

    Time series forecasting using a hybrid ARIMA and neural network model

    Neurocomputing

    (2003)
  • QiuX. et al.

    Ensemble incremental learning Random Vector Functional Link network for short-term electric load forecasting

    Knowl.-Based Syst.

    (2018)
  • QiuX. et al.

    Empirical mode decomposition based ensemble deep learning for load demand time series forecasting

    Appl. Soft Comput.

    (2017)
  • LiuH. et al.

    A hybrid model for wind speed prediction usingempirical mode decomposition and artificial neural networks

    Renew. Energy

    (2012)
  • SchimbinschiF. et al.

    Topology-regularized universal vector autoregression for traffic forecasting in large urban areas

    Expert Syst. Appl.

    (2017)
  • BaiY. et al.

    Daily reservoir inflow forecasting using multiscale deep feature learning with hybrid models

    J. Hydrol.

    (2016)
  • Ardalani-FarsaM. et al.

    Chaotic time series prediction with residual analysis method using hybrid Elman–NARX neural networks

    Neurocomputing

    (2010)
  • ChandraR. et al.

    Cooperative coevolution of Elman recurrent neural networks for chaotic time series prediction

    Neurocomputing

    (2012)
  • HanM. et al.

    Efficient clustering of radial basis perceptron neural network for pattern recognition

    Pattern Recognit.

    (2004)
  • BouzgouH. et al.

    Multiple architecture system for wind speed prediction

    Appl. Energy

    (2011)
  • DedinecA. et al.

    Deep belief network based electricity load forecasting: an analysis of Macedonian case

    Energy

    (2016)
  • Moreira-MatiasL. et al.

    Concept neurons-handling drift issues for real-time industrial data mining

  • Martinez-AlvarezF. et al.

    A survey on data mining techniques applied to electricity-related time series forecasting

    Energies

    (2015)
  • Cited by (0)

    View full text