Elsevier

Information Sciences

Volume 544, 12 January 2021, Pages 427-445
Information Sciences

A novel ensemble deep learning model with dynamic error correction and multi-objective ensemble pruning for time series forecasting

https://doi.org/10.1016/j.ins.2020.08.053Get rights and content

Abstract

In the past decade, deep learning models have shown to be promising tools for time series forecasting. However, owing to significant differences in the volatility characteristics among different types of time series data, it is difficult for an individual deep learning model to maintain robust forecasting performance. In this study, a novel ensemble deep learning model is proposed to achieve accurate and stable time series forecasting. First, a boosting deep learning method based on extended AdaBoost algorithm is proposed for generating various basic predictors. These basic predictors are further enhanced through a new dynamic error correction method. A stacking-based ensemble method that employs kernel ridge regression as the meta-predictor is then used to combine the basic predictors to produce the ultimate forecasting results. To increase forecasting accuracy and stability, an enhanced multi-population non-dominated sorting genetic algorithm-II is proposed for ensemble pruning. Finally, the forecasting performance of the proposed model is verified through the use of three different types of real-world time series data (i.e., PM2.5 concentration, wind speed, and electricity price). The experimental results showed that the proposed model is superior to other baseline models in dealing with time series forecasting tasks.

Introduction

Time series forecasting is the process of judging or predicting future phenomena by analyzing a series of historical observation data arranged in chronological order [24]. Accurate and stable time series forecasting plays an important role in various applications. For instance, air pollution forecasting can assist people in taking preventive measures to reduce the risk of disease caused by breathing unclean air; wind speed forecasting can improve the energy conversion efficiency of wind farms; and electricity price forecasting is helpful to power departments in coordinating the relationship between supply and demand of electricity and increasing related economic benefits. However, the inherent volatility and instability of time series data pose a significant challenge to the realization of high-precision time series forecasting.

In the past decade, deep learning models, such as the multi-layer perceptron neural network (MLP) [19], convolutional neural network (CNN) [8], long short-term memory network (LSTM) [2], and deep belief neural network (DBN) [35], have received considerable attention from researchers and have been widely used in various learning tasks, including image recognition and natural language processing. Deep learning models can transform original input from a level of representation to a more abstract level by combining simple but nonlinear modules layer by layer [32] to extract more comprehensive features from the data. Because of its superior feature extraction ability, deep learning has made a great contribution to improving the accuracy of time series forecasting [30]. Our previous research [3] also proposed a multifactor spatio-temporal correlation model for wind speed forecasting based on a CNN and LSTM, which were used to extract the spatial and temporal correlation features, respectively, between multiple sites.

However, deep learning models typically have high variances and low biases [15]. In a complicated and dynamic application environment, it is difficult for an individual deep learning model to maintain high forecasting accuracy and robustness. Ensemble learning, on the other hand, which uses several different individual models and certain ensemble strategies to improve the generalization performance of the entire model, has proved to be an effective method for overcoming this problem [31]. The implementation of ensemble learning includes mainly two stages: the construction of basic predictors, and the combination of basic predictors. The modeling and optimization strategies in each stage have a significant impact on the ultimate ensemble performance. The present study therefore focused on improving the ensemble performance from these two stages. Accordingly, in this study, a novel ensemble deep learning model with dynamic error correction and multi-objective ensemble pruning is proposed to achieve accurate and stable time series forecasting. The main contributions of this study are summarized as follows:

  • 1)

    A novel ensemble deep learning model based on MLP, CNN, and LSTM is proposed for time series forecasting. The model combines the advantages of deep learning and ensemble learning. It can enhance the generalization and robustness of the whole model on the premise of adaptive extraction of implicit features in the time series data.

  • 2)

    A boosting deep learning method (called EDNN) based on the extended AdaBoost algorithm is proposed to improve the diversity and accuracy of the basic predictors of the ensemble model. In the proposed method, the 2nd-order forecasting effectiveness (FE) metric [37] is leveraged to optimize the sample loss function and sample weight updating function of the AdaBoost algorithm [9]. Accordingly, each individual deep learning model can more effectively learn the feature information of different sample distributions under limited sample sizes.

  • 3)

    A new dynamic error correction (DEC) method is proposed to reduce the forecasting errors of the basic predictors. The proposed method can fully handle the relative trend of the error and decay characteristics of the time series for dynamically correcting the forecasting results of the basic predictors.

  • 4)

    A stacking-based ensemble method based on multi-objective ensemble pruning is proposed to produce time series forecasting results with high accuracy and strong stability. An enhanced multi-population non-dominated sorting genetic algorithm-II (MPNSGA-II) is used for ensemble pruning, for which an opposite population initialization method and a cross-population intercrossing operation are proposed to enhance the exploration and exploitation ability of the NSGA-II. Moreover, by employing kernel ridge regression (KRR) [23] as the meta-predictor in the stacking ensemble method, the influence of multicollinearity among the basic predictors in the ensemble model is alleviated and the forecasting performance is further improved.

  • 5)

    Two groups of experiments were conducted to verify the accuracy of each method involved in the proposed model. Six time series datasets of wind speed, PM2.5 concentration, and electricity price; 16 baseline models; and four evaluation metrics were used to compare and evaluate the performances of the models. The experimental results clearly showed that the proposed model can achieve more accurate and stable time series forecasting.

    The remainder of this paper is organized as follows. Section 2 reviews previous research on the construction and combination of basic predictors. Section 3 presents details on the methodology and the proposed forecasting model. Section 4 explains and discusses the implementation details and results of the experiments. Finally, Section 5 provides conclusions and recommendations for future research.

Section snippets

Related work

The model proposed in this study is intended to improve the performance of time series forecasting by optimizing the two stages (i.e., the construction and combination of basic predictors) of the ensemble learning process. In this section, some optimization methods and strategies related to these two stages are briefly reviewed.

Methodology

The framework of the proposed ensemble deep learning model for time series forecasting (noted as MPNSGA-II_KRR_EDNN_DEC) is presented in Fig. 1. The model consists of three stages: data preparation, basic predictor construction, and basic predictor combination. The model also includes two optimization modules: one for dynamic error correction, and one for multi-objective ensemble pruning. In the data preparation stage, the time series cross-validation method is used to partition the original

Experiments and analysis

To verify the accuracy, stability, and generalization ability of the proposed MPNSGA-II_KRR_EDNN_DEC model, three types of time series datasets, four widely used evaluation metrics, and 16 baseline models were used to conduct comparison experiments. To reasonably verify the generalization performance of the models on different datasets, the same experimental parameters were used in the experiments for each model, wherein the sequence length (time steps) of the input data was ten, and the other

Conclusion and future work

Accurate time series forecasting can provide convenient information to improve the productivity and other aspects of our everyday lives. In this study, a novel ensemble deep learning model with dynamic error correction and multi-objective ensemble pruning was proposed to achieve accurate and stable time series forecasting. Three different types of real-world time series data, including PM2.5 concentration, wind speed, and electricity price, were used to establish comparative experiments for

CRediT authorship contribution statement

Shuai Zhang: Conceptualization, Methodology, Writing - original draft, Writing - review & editing, Funding acquisition. Yong Chen: Methodology, Formal analysis, Conceptualization, Writing - original draft, Data curation, Software, Validation. Wenyu Zhang: Supervision, Methodology, Writing - review & editing, Funding acquisition, Project administration. Ruijun Feng: Software, Validation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The work has been supported by National Natural Science Foundation of China (No. 51975512, No. 51875503), and Zhejiang Natural Science Foundation of China (No. LZ20E050001).

References (49)

  • D. Liu et al.

    Short-term wind speed forecasting based on spectral clustering and optimised echo state networks

    Renewable Energy

    (2015)
  • H.Y. Luo et al.

    Research and application of a novel hybrid decomposition-ensemble learning paradigm with error correction for daily PM10 forecasting

    Atmos. Res.

    (2018)
  • B.B. Pan et al.

    Prediction of soft tissue deformations after CMF surgery with incremental kernel ridge regression

    Comput. Biol. Med.

    (2016)
  • S. Panigrahi et al.

    A hybrid ETS–ANN model for time series forecasting

    Eng. Appl. Artif. Intell.

    (2017)
  • A. Peimankar et al.

    Multi-objective ensemble forecasting with an application to power transformers

    Appl. Soft Comput.

    (2018)
  • A.S. Qureshi et al.

    Wind power prediction using deep neural network based meta regression and transfer learning

    Appl. Soft Comput.

    (2017)
  • H. Rong et al.

    Deep rolling: A novel emotion prediction model for a multi-participant communication context

    Inf. Sci.

    (2019)
  • H.D. Shao et al.

    A novel method for intelligent fault diagnosis of rolling bearings using ensemble deep auto-encoders

    Mech. Syst. Sig. Process.

    (2018)
  • Y. Son et al.

    Automated artifact elimination of physiological signals using a deep belief network: an application for continuously measured arterial blood pressure waveforms

    Inf. Sci.

    (2018)
  • G. Song et al.

    A novel double deep ELMs ensemble system for time series forecasting

    Knowl.-Based Syst.

    (2017)
  • J.J. Song et al.

    A novel combined model based on advanced optimization algorithm for short-term wind speed forecasting

    Appl. Energy

    (2018)
  • L.L. Wang et al.

    Short-term wind speed prediction using an extreme learning machine model with error correction

    Energy Convers. Manage.

    (2018)
  • Q.L. Wang et al.

    Enhancing intraday stock price manipulation detection by leveraging recurrent neural networks with ensemble learning

    Neurocomputing

    (2019)
  • D.H. Wolpert

    Stacked generalization

    Neural Networks

    (1992)
  • Cited by (83)

    View all citing articles on Scopus
    View full text