Skip to main content

Advertisement

MSDformer: an autocorrelation transformer with multiscale decomposition for long-term multivariate time series forecasting

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The improvement of performance and efficiency in long-term time series forecasting is significant for practical applications. However, while enhancing overall performance, existing time series forecasting methods often exhibit unsatisfactory capabilities in the restoration of details and prediction efficiency. To address these issues, an autocorrelation Transformer with multiscale decomposition (MSDformer) is proposed for long-term multivariate time series forecasting. Specifically, a multiscale decomposition (MSDecomp) module is designed, which identifies the temporal repeating patterns in time series with different scales to retain more historical details while extracting trend components. An Encoder layer is proposed based on the MSDecomp module and Auto-Correlation mechanism, which discovers the similarity of subsequences in a periodic manner and effectively captures the seasonal components to improve the degree of restoration of prediction details while extracting the residual trend components. Finally, unlike the traditional Transformer structure, the decoder structure is replaced by the proposed Autoregressive module to simplify the output mode of the decoder and enhance linear information. Compared to other advanced and representative models on six real-world datasets, the experimental results demonstrate that the MSDformer has a relative performance improvement of an average of 8.1%. MSDformer also has lower memory usage and temporal consumption, making it more advantageous for long-term time series forecasting.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

All the datasets used in this paper are publicly available and can be acquired from the above links. All data are available by contacting the corresponding author by reasonable request.

Notes

  1. https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams

  2. http://pems.dot.ca.gov

  3. https://www.bgc-jena.mpg.de/wetter/

  4. https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html

References

  1. Sbrana G, Antonetti P (2023) Persistence modeling for sales prediction: A simple, self-contained approach. J Bus Res 166:114103

    Article  MATH  Google Scholar 

  2. Ruan L, Bai Y, Li S, He S, Xiao L (2021) Workload time series prediction in storage systems: a deep learning based approach. Cluster Computing, 1–11

  3. Camastra F, Capone V, Ciaramella A, Riccio A, Staiano A (2022) Prediction of environmental missing data time series by support vector machine regression and correlation dimension estimation. Environ Model Softw 150:1043–1053

    Article  MATH  Google Scholar 

  4. Jassim MS, Coskuner G, Sultana N, Hossain SZ (2023) Forecasting domestic waste generation during successive covid-19 lockdowns by bidirectional lstm super learner neural network. Appl Soft Comput 133:1099–1108

    Article  Google Scholar 

  5. Callot L, Caner M, Önder AÖ, Ulaşan E (2021) A nodewise regression approach to estimating large portfolios. J Bus Econ Stat 39(2):520–531

    Article  MathSciNet  MATH  Google Scholar 

  6. Mohanty MK, Thakurta PKG, Kar S (2023) Agricultural commodity price prediction model: a machine learning framework. Neural Comput Appl 35(20):15109–15128

    Article  MATH  Google Scholar 

  7. Babu CN, Reddy BE (2014) Arima-based and multiapplication workload prediction with wavelet decomposition and savitzky–golay filter in clouds. Appl Soft Comput 23:27–38

    Article  MATH  Google Scholar 

  8. Smyl S (2020) A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. Int J Forecast 36(1):75–85

    Article  MATH  Google Scholar 

  9. Sun Y, Leng B, Guan W (2015) Exploiting pso-svm and sample entropy in bemd for the prediction of interval-valued time series and its application to daily pm2. 5 concentration forecasting. Neurocomput 166:109–121

    Article  MATH  Google Scholar 

  10. Yu H, Dai Q (2023) Ae-dil: A double incremental learning algorithm for non-stationary time series prediction via adaptive ensemble. Inf Sci 636:118916

    Article  MATH  Google Scholar 

  11. Vosoughi S, Mohsenvand MN, Roy D (2017) Ensemble conformalized quantile regression for probabilistic time series forecasting. ACM Trans knowl Discov Data (TKDD) 11(4):1–36

    Article  Google Scholar 

  12. Liu Y, Zhang Q, Song L, Chen Y (2019) Recurrent neural network-fitnets: improving early prediction of student performanceby time-series knowledge distillation. Comput Electron Agric 165:1049–1064

    MATH  Google Scholar 

  13. Hu J, Wang X, Zhang Y, Zhang D, Zhang M, Xue J (2020) Time series prediction method based on variant lstm recurrent neural network. Neural Process Lett 52:1485–1500

    Article  MATH  Google Scholar 

  14. Li Q, Xu Y (2019) Single-layer folded rnn for time series prediction and classification under a non-von neumann architecture. Appl Sci 9(15):30–41

    MATH  Google Scholar 

  15. Hewage P, Behera A, Trovati M, Pereira E, Ghahremani M, Palmieri F, Liu Y (2020) Temporal convolutional neural (tcn) network for an effective weather forecasting using time-series data from the local weather station. Soft Comput 24:16453–16482

    Article  Google Scholar 

  16. Zheng Y, Gao Z, Wang Y, Fu Q (2020) Mooc dropout prediction using fwts-cnn model based on fused feature weighting and time series. IEEE Access 8:225324–225335

    Article  Google Scholar 

  17. Gao C, Zhang N, Li Y, Bian F, Wan H (2020) Self-attention-based time-variant neural networks for multi-step time series forecasting. Neural Comput Appl 34(11):8737–8754

    Article  MATH  Google Scholar 

  18. Chen L, Chen D, Shang Z, Wu B, Zheng C, Wen B, Zhang W (2023) Multi-scale adaptive graph neural network for multivariate time series forecasting. IEEE Trans Knowl Data Eng 35(10):10748–10761

    Article  MATH  Google Scholar 

  19. Bai L, Yao L, Li C, Wang X, Wang C (2020) Adaptive graph convolutional recurrent network for traffic forecasting. Adv Neural Inf Process Syst 33:17804–17815

    MATH  Google Scholar 

  20. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30:1–11

    Google Scholar 

  21. Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W (2021) Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence 35:11106–11115

    Article  MATH  Google Scholar 

  22. Kitaev N, Kaiser L, Levskaya A (2020) Reformer: The efficient transformer. In: Proceedings of International Conference on Learning Representations, pp 1–12

  23. Liu S, Yu H, Liao C, Li J, Lin W, Liu AX, Dustdar S (2021) Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In: Proceedings of International Conference on Learning Representations, pp 1–20

  24. Geng Z, Chen Z, Meng Q, Han Y (2021) Novel transformer based on gated convolutional neural network for dynamic soft sensor modeling of industrial processes. IEEE Trans Ind Inf 18(3):1521–1529

    Article  MATH  Google Scholar 

  25. Wang Y, Peng J, Wang X, Zhang Z, Duan J (2024) Replacing self-attentions with convolutional layers in multivariate long sequence time-series forecasting. Appl Intell 54(1):522–543

    Article  MATH  Google Scholar 

  26. Song W, Fujimura S (2021) Capturing combination patterns of long-and short-term dependencies in multivariate time series forecasting. Neurocomput 464:72–82

    Article  MATH  Google Scholar 

  27. Bandara K, Bergmeir C, Hewamalage H (2020) Lstm-msnet: Leveraging forecasts on sets of related time series with multiple seasonal patterns. IEEE Trans Neural Netw Learn Syst 32(4):1586–1599

    Article  MATH  Google Scholar 

  28. Wang X, Liu H, Yang Z, Du J, Dong X (2023) Cnformer: a convolutional transformer with decomposition for long-term multivariate time series forecasting. Appl Intell 1–15

  29. Wu H, Xu J, Wang J, Long M (2021) Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Adv Neural Inf Process Syst 34:22419–22430

  30. Zhou T, Ma Z, Wen Q, Wang X, Sun L, Jin R (2022) Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In: Proceedings of International Conference on Machine Learning, pp 27268–27286. PMLR

  31. Han Z, Zhao J, Leung H, Ma KF, Wang W (2019) A review of deep learning models for time series prediction. IEEE Sensors J 21(6):7833–7848

    Article  MATH  Google Scholar 

  32. Natarajan YJ, Subramaniam Nachimuthu D (2020) New svm kernel soft computing models for wind speed prediction in renewable energy applications. Soft Comput 24:11441–11458

    Article  MATH  Google Scholar 

  33. Yu H, Dai Q (2022) Dwe-il: a new incremental learning algorithm for non-stationary time series prediction via dynamically weighting ensemble learning. Appl Intell 52(1):174–194

    Article  MATH  Google Scholar 

  34. Liu B (2023) Robust sequential online prediction with dynamic ensemble of multiple models: A review. Neurocomputing, 126553

  35. Mbuli N, Mathonsi M, Seitshiro M, Pretorius JHC (2020) Decomposition forecasting methods: A review of applications in power systems. Energy Rep 6:298–306

    Article  Google Scholar 

  36. Dudek G (2023) Std: a seasonal-trend-dispersion decomposition of time series. IEEE Trans Knowl Data Eng 35(10):10339–10350

    Article  MATH  Google Scholar 

  37. Zhou Y, Zhao J, Song Y, Sun J, Fu H, Chu M (2022) A seasonal-trend-decomposition-based voltage-source-inverter open-circuit fault diagnosis method. IEEE Trans Power Electr 37(12):15517–15527

    Article  Google Scholar 

  38. He X, Li Y, Tan J, Wu B, Li F (2023) Oneshotstl: One-shot seasonal-trend decomposition for online time series anomaly detection and forecasting. Proc VLDB Endowment 16(6):1399–1412

    Article  MATH  Google Scholar 

  39. Wang X, Chen G, Qian G, Gao P, Wei XY, Wang Y, Tian Y, Gao W (2023) Large-scale multi-modal pre-trained models: A comprehensive survey. Mach Intell Res 20(4):447–482

    Article  MATH  Google Scholar 

  40. Zerveas G, Jayaraman S, Patel D, Bhamidipaty A, Eickhoff C (2021) A transformer-based framework for multivariate time series representation learning. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp 2114–2124

  41. Wang X, Liu H, Du J, Yang Z, Dong X (2023) Clformer: Locally grouped auto-correlation and convolutional transformer for long-term multivariate time series forecasting. Eng Appl Artif Intell 121:106042

    Article  MATH  Google Scholar 

  42. Lai G, Chang W, Yang Y, Liu H (2018) Modeling long-and short-term temporal patterns with deep neural networks. In: Proceedings of The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp 95–104

  43. Abdulkadirov R, Lyakhov P, Nagornov N (2023) Survey of optimization algorithms in modern neural networks. Math 11(11):2466

    Article  MATH  Google Scholar 

Download references

Funding

This work is supported in part by National Key R&D program of China (Grant no. 2020YFC1523004).

Author information

Authors and Affiliations

Authors

Contributions

G.S. presented the innovation of paper, designed and carried out the experiments, analyzed the result of the experiments. Y.G. drafted the work or revised it critically for important intellectual content. All authors reviewed the manuscript.

Corresponding author

Correspondence to Yepeng Guan.

Ethics declarations

Ethical and informed consent for data used

This paper does not contain any studies involving humans or animals.

Conflict of interest/Competing interests

The authors have no competing financial interests to declare that are relevant to the content of this article. The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Su, G., Guan, Y. MSDformer: an autocorrelation transformer with multiscale decomposition for long-term multivariate time series forecasting. Appl Intell 55, 179 (2025). https://doi.org/10.1007/s10489-024-06105-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-06105-6

Keywords