Skip to main content
SpringerLink
Log in

A dual-stage attention-based Bi-LSTM network for multivariate time series prediction

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In the context of the big data era, time series data present the characteristics of high dimensionality and nonlinearity, which bring great challenges to the prediction of multivariate time series (MTS). In view of the problems of insufficient feature extraction of MTS data and insufficient long-term dependency characteristics, we propose a novel model for MTS prediction named a dual-stage attention-based bidirectional long short-term memory (DABi-LSTM). Specifically, the input attention mechanism gives important driving series greater weight, effectively extracting the features of the driving series. Secondly, the bidirectional long short-term memory network (Bi-LSTM) network extracts the MTS features in two directions. Furthermore, the combination of the attention mechanism, Bi-LSTM and long short-term memory (LSTM) network can effectively solve the problem of insufficient long-term dependence in MTS prediction. Experiments based on typical datasets of finance, environment and energy determine the optimal window size and hidden size of the model, and demonstrate that the model gets the state-of-the-art performance for single-step prediction and multistep prediction compared other methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Shah I, Bibi H, Ali S et al (2020) Forecasting one-day-ahead electricity prices for Italian electricity market using parametric and nonparametric approaches. IEEE Access 99:1. https://doi.org/10.1109/ACCESS.2020.3007189

    Article  Google Scholar 

  2. Chiewhawan T, Vateekul P (2020) Stock return prediction using dual-stage attention model with stock relation inference. pp 492–503. https://doi.org/10.1007/978-3-030-41964-6_42

  3. Rauf HT, Lali MI, Khan MA et al (2021) Time series forecasting of covid-19 transmission in Asia pacific countries using deep neural networks. Pers Ubiquit Comput. https://doi.org/10.1007/s00779-020-01494-0

    Article  Google Scholar 

  4. Jia P, Liu H, Wang S et al (2020) Research on a mine gas concentration forecasting model based on a GRU network. IEEE Access 8:38023–38031. https://doi.org/10.1109/ACCESS.2020.2975257

    Article  Google Scholar 

  5. Huang B, Liang Y, Qiu X (2021) Wind power forecasting using attention-based recurrent neural networks: a comparative study. IEEE Access 9:40432–40444. https://doi.org/10.1109/ACCESS.2021.3065502

    Article  Google Scholar 

  6. Peng H (2021) Time series forecasting model method based on neural network. In: 2021 International Conference on Applications and Techniques in Cyber Intelligence

  7. Daihong J, Sai Z, Lei D et al (2022) Multi-scale generative adversarial network for image super-resolution. Soft Comput 26(8):3631–3641. https://doi.org/10.1007/s00500-022-06822-5

    Article  Google Scholar 

  8. Xiao Y, Yin H, Duan T et al (2021) An Intelligent prediction model for UCG state based on dual-source LSTM. Int J Mach Learn Cybern 12(11):3169–3178. https://doi.org/10.1007/s13042-020-01210-7

    Article  Google Scholar 

  9. Box GEP, Jenkins GM (1970) Time series analysis forecasting and control. J Time Ser Anal. https://doi.org/10.2307/1912100

    Article  MATH  Google Scholar 

  10. Adhikari R, Agrawal RK (2013) An introductory study on time series modeling and forecasting[J]. [arXiv preprint] arXiv: 1302.6613

  11. Cao L (2003) Support vector machines experts for time series forecasting. Neurocomputing 51(1–4):321–339. https://doi.org/10.1016/S0925-2312(02)00577-5

    Article  Google Scholar 

  12. Wei X, Pu Z, Rong C et al (2018) A nonparametric Bayesian framework for short-term wind power probabilistic forecast. IEEE Trans Power Syst 1:371–379. https://doi.org/10.1109/TPWRS.2018.2858265

    Article  Google Scholar 

  13. Guo J, Wang J, Li Q et al (2018) Construction of prediction model of neural network railway bulk cargo floating price based on random forest regression algorithm. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3903-5

    Article  Google Scholar 

  14. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  15. Cho K, Merrienboer BV, Gulcehre C et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. Comput Sci. https://doi.org/10.3115/v1/D14-1179

    Article  Google Scholar 

  16. Qin Y, Song D, Chen H, et al (2017) A dual-stage attention-based recurrent neural network for time series prediction. https://doi.org/10.24963/ijcai.2017/366

  17. Sun Y, Leng B, Guan W (2015) A novel wavelet-SVM short-time passenger flow prediction in Beijing subway system. Neurocomputing 166(20):109–121. https://doi.org/10.1016/j.neucom.2015.03.085

    Article  Google Scholar 

  18. Das SP, Padhy S (2018) A novel hybrid model using teaching–learning-based optimization and a support vector machine for commodity futures index forecasting. Int J Mach Learn Cybern 9(1):97–111. https://doi.org/10.1007/s13042-015-0359-0

    Article  Google Scholar 

  19. Fan GF, Guo YH, Zheng JM et al (2020) A generalized regression model based on hybrid empirical mode decomposition and support vector regression with back-propagation neural network for mid-short-term load forecasting. J Forecast 39(5):737–756. https://doi.org/10.1002/for.2655

    Article  Google Scholar 

  20. Dulce-Chamorro E, Martinez-De-Pison FJ (2021) An advanced methodology to enhance energy efficiency in a hospital cooling-water system. J Build Eng 43:102839. https://doi.org/10.1016/j.jobe.2021.102839

    Article  Google Scholar 

  21. Li S, Wen J, Luo F et al (2018) Time-aware QoS prediction for cloud service recommendation based on matrix factorization. IEEE Access. https://doi.org/10.1109/ACCESS.2018.2883939

    Article  Google Scholar 

  22. Shi W, Zhu Y, Yu P et al (2017) Effective prediction of missing data on Apache Spark over multivariable time series. IEEE Trans Big Data. https://doi.org/10.1109/TBDATA.2017.2719703

    Article  Google Scholar 

  23. Mei J, Castro YD, Goude Y et al (2017) Nonnegative matrix factorization with side information for time series recovery and prediction. IEEE Trans Knowl Data Eng. https://doi.org/10.1109/TKDE.2018.2839678

    Article  Google Scholar 

  24. Sun H, Jin R, Luo Y (2019) Supervised subgraph augmented non-negative matrix factorization for interpretable manufacturing time series data analytics. IISE Trans 52:1–21. https://doi.org/10.1080/24725854.2019.1581389

    Article  Google Scholar 

  25. Mejia J, Ochoa-Zezzatti A, Cruz-Mejía O et al (2020) Prediction of time series using wavelet Gaussian process for wireless sensor networks. Wirel Netw. https://doi.org/10.1007/s11276-020-02250-1

    Article  Google Scholar 

  26. Yoo KM, Kil RM, Youn HY (2021) Time series prediction based on recursive update gaussian kernel function networks. In: 2021 15th International Conference on Ubiquitous Information Management and Communication (IMCOM)

  27. Hamidi O, Tapak L, Abbasi H et al (2017) Application of random forest time series, support vector regression and multivariate adaptive regression splines models in prediction of snowfall (a case study of Alvand in the middle Zagros, Iran). Theor Appl Climatol. https://doi.org/10.1007/s00704-017-2300-9

    Article  Google Scholar 

  28. Ahmadi A, Nabipour M, Mohammadi-Ivatloo B et al (2020) Long-term wind power forecasting using tree-based learning algorithms. IEEE Access 8:151511–151522. https://doi.org/10.1109/ACCESS.2020.3017442

    Article  Google Scholar 

  29. Li C, Lin S, Xu F et al (2018) Short-term wind power prediction based on data mining technology and improved support vector machine method: a case study in Northwest China. J Clean Prod 205:909–922. https://doi.org/10.1016/j.jclepro.2018.09.143

    Article  Google Scholar 

  30. Moon J, Park J, Hwang E et al (2018) Forecasting power consumption for higher educational institutions based on machine learning. J Supercomput 74(8):3778–3800. https://doi.org/10.1007/s11227-017-2022-x

    Article  Google Scholar 

  31. Zhou X, Ren J, An J et al (2021) Predicting open-plan office window operating behavior using the random forest algorithm. J Build Eng 42:102514. https://doi.org/10.1016/j.jobe.2021.102514

    Article  Google Scholar 

  32. Gong M, Wang J, Bai Y et al (2020) Heat load prediction of residential buildings based on discrete wavelet transform and tree-based ensemble learning. J Build Eng 32:101455. https://doi.org/10.1016/j.jobe.2020.101455

    Article  Google Scholar 

  33. Gensler A, Henze J, Sick B, et al (2017) Deep learning for solar power forecasting—an approach using AutoEncoder and LSTM neural networks. In: 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)

  34. Ji J, Hou J (2017) Forecast on bus trip demand based on ARIMA models and gated recurrent unit neural networks[C]//2017 International Conference on Computer Systems, Electronics and Control (ICCSEC). IEEE 2017:105–108

    Google Scholar 

  35. Li Y, Zhu Z, Kong D et al (2019) EA-LSTM: evolutionary attention-based LSTM for time series prediction. Knowl Based Syst 181:104785

    Article  Google Scholar 

  36. Cho K, Merrienboer BV, Bahdanau D et al (2014) On the properties of neural machine translation: encoder–decoder approaches. Comput Sci. https://doi.org/10.3115/v1/W14-4012

    Article  Google Scholar 

  37. Du S, Li T, Yang Y et al (2020) Multivariate time series forecasting via attention-based encoder–decoder framework. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.12.118

    Article  Google Scholar 

  38. Xiao Y, Yin H, Zhang Y et al (2021) A dual-stage attention-based Conv-LSTM network for spatio-temporal correlation and multivariate time series prediction. Int J Intell Syst 36(5):2036–2057. https://doi.org/10.1002/int.22370

    Article  Google Scholar 

  39. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. Comput Sci. https://doi.org/10.48550/arXiv.1409.0473

    Article  Google Scholar 

  40. Zamora-Martínez F, Romeu P, Botella-Rocamora P et al (2014) On-line learning of indoor temperature forecasting models towards energy efficiency. Energy Build 83:162–172. https://doi.org/10.1016/j.enbuild.2014.04.034

    Article  Google Scholar 

  41. Candanedo LM, Feldheim V, Deramaix D (2017) Data driven prediction models of energy use of appliances in a low-energy house. Energy Build 140:81–97. https://doi.org/10.1016/j.enbuild.2017.01.083

    Article  Google Scholar 

  42. Colak I, Sagiroglu S, Yesilbudak M, et al (2016) Multi-time series and -time scale modeling for wind speed and wind power forecasting part II: medium-term and long-term applications. In: 2015 International Conference on Renefwable Energy Research and Applications (ICRERA)

  43. Wang Q, Chen L, Zhao J et al (2020) A deep granular network with adaptive unequal-length granulation strategy for long-term time series forecasting and its industrial applications. Artif Intell Rev. https://doi.org/10.1007/s10462-020-09822-9

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116, China

    Qi Cheng, Yixin Chen, Yuteng Xiao, Hongsheng Yin & Weidong Liu

Authors
  1. Qi Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yixin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuteng Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongsheng Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weidong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yuteng Xiao or Weidong Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, Q., Chen, Y., Xiao, Y. et al. A dual-stage attention-based Bi-LSTM network for multivariate time series prediction. J Supercomput 78, 16214–16235 (2022). https://doi.org/10.1007/s11227-022-04506-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04506-3

Keywords

Search

Navigation