Abstract
Accurate demand forecasts are commonly needed in real life to deploy future schedules of economic activities, such as merchandise sales and electricity consumption predictions. Transformer network has been demonstrated to have potential for time series forecasting in recent studies, however, practical tasks generally require long sequence forecasting outputs with limited length of inputs, which leads to high time complexity and large memory consumption. In this research, we propose a stacking ensemble model for increasing long sequence time-series forecasting accuracy which is based on three Transformer networks. The base learners include Autoformer, Informer and Reformer, which have different improvements on Transformer that enable our approach to improve forecast performance. Then, the results from base learners are applied to train the meta learner (MLP), and finally the trained meta-learner is applied to predict future demand. Experiments on two overt datasets reveal that our proposed ensemble Transformer model possesses great prediction accuracy while requires minor amount of computation. This approach may provide a fresh solution to the demand forecasting domain.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bendaoud, N.M.M., Farah, N.: Using deep learning for short-term load forecasting. Neural Comput. Appl. 32(18), 15029–15041 (2020). https://doi.org/10.1007/s00521-020-04856-0
Choi, M., Kim, H., Han, B., Xu, N., Lee, K.M.: Channel attention is all you need for video frame interpolation. In: AAAI, pp. 10663–10671. AAAI Press (2020)
Feng, Y., Wang, S.: A forecast for bicycle rental demand based on random forests and multiple linear regression. In: ICIS, pp. 101–105. IEEE Computer Society (2017)
Huang, Y., Xu, C., Ji, M., Xiang, W., He, D.: Medical service demand forecasting using a hybrid model based on ARIMA and self-adaptive filtering method. BMC Medical Informatics Decis. Mak. 20(1), 237 (2020)
Jin, Y., Han, D.K., Ko, H.: Trseg: Transformer for semantic segmentation. Pattern Recognit. Lett. 148, 29–35 (2021)
Kitaev, N., Kaiser, L., Levskaya, A.: Reformer: The efficient transformer. In: ICLR, OpenReview.net (2020)
Li, R., Chen, X., Balezentis, T., Streimikiene, D., Niu, Z.: Multi-step least squares support vector machine modeling approach for forecasting short-term electricity demand with application. Neural Comput. Appl. 33(1), 301–320 (2020). https://doi.org/10.1007/s00521-020-04996-3
Lingelbach, K., Lingelbach, Y., Otte, S., Bui, M., Künzell, T., Peissner, M.: Demand forecasting using ensemble learning for effective scheduling of logistic orders. In: Ahram, T.Z., Karwowski, W., Kalra, J. (eds.) AHFE 2021. LNNS, vol. 271, pp. 313–321. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-80624-8_39
Liu, B., Tang, X., Cheng, J., Shi, P.: Traffic flow combination forecasting method based on improved LSTM and ARIMA. Int. J. Embed. Syst. 12(1), 22–30 (2020)
Liu, S., Ji, H., Wang, M.C.: Nonpooling convolutional neural network forecasting for seasonal time series with trends. IEEE Trans. Neural Networks Learn. Syst. 31(8), 2879–2888 (2020)
Mallick, A., Singh, S.N., Mohapatra, A.: Data driven day-ahead electrical load forecasting through repeated wavelet transform assisted SVM model. Appl. Soft Comput. 111, 107730 (2021)
Martínez, F., Frías, M.P., Pérez-Godoy, M.D., Rivera, A.J.: A methodology for applying k-nearest neighbor to time series forecasting. Artif. Intell. Rev. 52(3), 2019–2037 (2019)
Masum, S., Liu, Y., Chiverton, J.: Multi-step time series forecasting of electric load using machine learning models. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2018. LNCS (LNAI), vol. 10841, pp. 148–159. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91253-0_15
Parmar, N., et al.: Image transformer. In: ICML. Proceedings of Machine Learning Research, vol. 80, pp. 4052–4061. PMLR (2018)
Ponnoprat, D.: Short-term daily precipitation forecasting with seasonally-integrated autoencoder. Appl. Soft Comput. 102, 107083 (2021)
Qiu, X., Ren, Y., Suganthan, P.N., Amaratunga, G.A.J.: Empirical mode decomposition based ensemble deep learning for load demand time series forecasting. Appl. Soft Comput. 54, 246–255 (2017)
Siami-Namini, S., Tavakoli, N., Namin, A.S.: A comparison of ARIMA and LSTM in forecasting time series. In: ICMLA, pp. 1394–1401. IEEE (2018)
Sun, L., Xing, X., Zhou, Y., Hu, X.: Demand forecasting for petrol products in gas stations using clustering and decision tree. J. Adv. Comput. Intell. Intell. Informatics 22(3), 387–393 (2018)
Svetunkov, I., Boylan, J.E.: State-space ARIMA for supply-chain forecasting. Int. J. Prod. Res. 58(3), 818–827 (2020)
Teng, F., Teng, J., Qiao, L., Du, S., Li, T.: A multi-step forecasting model of online car-hailing demand. Inf. Sci. 587, 572–586 (2022)
Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)
Verma, M.: Revisiting linformer with a modified self-attention with linear complexity. CoRR abs/2101.10277 (2021)
Wang, Z., He, L., Zhao, Y.: Forecasting the seasonal natural gas consumption in the US using a gray model with dummy variables. Appl. Soft Comput. 113(Part), 108002 (2021)
Wu, H., Xu, J., Wang, J., Long, M.: Autoformer: decomposition transformers with auto-correlation for long-term series forecasting. In: NeurIPS, pp. 22419–22430 (2021)
Zhou, H., et al.: Informer: beyond efficient transformer for long sequence time-series forecasting. In: AAAI, pp. 11106–11115. AAAI Press (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Chu, J., Cao, J., Chen, Y. (2022). An Ensemble Deep Learning Model Based on Transformers for Long Sequence Time-Series Forecasting. In: Zhang, H., et al. Neural Computing for Advanced Applications. NCAA 2022. Communications in Computer and Information Science, vol 1638. Springer, Singapore. https://doi.org/10.1007/978-981-19-6135-9_21
Download citation
DOI: https://doi.org/10.1007/978-981-19-6135-9_21
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-6134-2
Online ISBN: 978-981-19-6135-9
eBook Packages: Computer ScienceComputer Science (R0)