Skip to main content
SpringerLink
Log in

CNformer: a convolutional transformer with decomposition for long-term multivariate time series forecasting

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Improving long-term time series forecasting accuracy and efficiency is of great value for real-world applications. The main challenge in the long-term forecasting of multivariate time series is to accurately capture the local dynamics and long-term dependencies of time series. Currently, most approaches capture temporal dependencies and inter-variable dependencies in intertwined temporal patterns, which are unreliable. Moreover, models based on time series decomposition methods are still unable to capture both short- and long-term dependencies well. In this paper, we propose an efficient multivariate time series forecasting model CNformer with three distinctive features. (1) The CNformer is a fully CNN-based time series forecasting model. (2) In the encoder, the stacked dilated convolution as a built-in block is combined with the time series decomposition to extract the seasonal component of the time series. (3) The convolution-based encoder-decoder attention mechanism refines seasonal patterns in the decoder and captures complex combinations between different related time series. Owing to these features, our CNformer has lower memory and time overhead than models based on self-attention and the Auto-Correlation mechanism. Experimental results show that our model achieves state-of-the-art performance on four real-world datasets, with a relative performance improvement of 20.29%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://pems.dot.ca.gov

  2. https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html

References

  1. Yamak PT, Yujian L, Gadosey PK (2019) A comparison between arima, lstm, and gru for time series forecasting. In: Proceedings of the 2019 2nd international conference on algorithms, computing and artificial intelligence, pp 49–55

  2. Smyl S (2020) A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. Int J Forecast 36(1):75–85

    Article  Google Scholar 

  3. Song W, Fujimura S (2021) Capturing combination patterns of long-and short-term dependencies in multivariate time series forecasting. Neurocomputing 464:72–82

    Article  Google Scholar 

  4. Yin C, Dai Q (2022) A deep multivariate time series multistep forecasting network. Appl Intell 52(8):8956–8974

    Article  Google Scholar 

  5. Lai G, Chang W-C, Yang Y, Liu H (2018) Modeling long-and short-term temporal patterns with deep neural networks. In: The 41st International ACM SIGIR conference on research & development in information retrieval, pp 95–104

  6. Salinas D, Flunkert V, Gasthaus J, Januschowski T (2020) Deepar: probabilistic forecasting with autoregressive recurrent networks. Int J Forecast 36(3):1181–1191

    Article  Google Scholar 

  7. Niu H, Xu K, Wang W (2020) A hybrid stock price index forecasting model based on variational mode decomposition and lstm network. Appl Intell 50(12):4296–4309

    Article  Google Scholar 

  8. Chang Y-Y, Sun F-Y, Wu Y-H, Lin S-D A memory-network based solution for multivariate time-series forecasting, arXiv:1809.02105

  9. Qin Y, Song D, Cheng H, Cheng W, Jiang G, Cottrell GW (2017) A dual-stage attention-based recurrent neural network for time series prediction. In: Proceedings of the 26th international joint conference on artificial intelligence, pp 2627–2633

  10. Shih S-Y, Sun F-K, Lee H-y (2019) Temporal pattern attention for multivariate time series forecasting. Mach Learn 108(8):1421–1441

    Article  MathSciNet  MATH  Google Scholar 

  11. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  12. Wang W, Xie E, Li X, Fan D-P, Song K, Liang D, Lu T, Luo P, Shao L (2021) Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 568–578

  13. Chu X, Tian Z, Wang Y, Zhang B, Ren H, Wei X, Xia H, Shen C (2021) Twins: Revisiting the design of spatial attention in vision transformers. Adv Neural Inf Process Syst 34:9355–9366

    Google Scholar 

  14. Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W (2021) Informer: Beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of AAAI

  15. Li S, Jin X, Xuan Y, Zhou X, Chen W, Wang Y-X, Yan X (2019) Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Adv Neural Inf Process Syst 32:5243–5253

    Google Scholar 

  16. Kitaev N, Kaiser L, Levskaya A (2019) Reformer: the efficient transformer. In: International conference on learning representations

  17. Bandara K, Bergmeir C, Hewamalage H (2020) Lstm-msnet: Leveraging forecasts on sets of related time series with multiple seasonal patterns. IEEE Trans Neural Netw Learn Syst 32(4):1586–1599

    Article  Google Scholar 

  18. Wu H, Xu J, Wang J, Long M (2021) Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Adv Neural Inf Process Syst 34:22419– 22430

    Google Scholar 

  19. He X, Shi S, Geng X, Xu L (2022) Information-aware attention dynamic synergetic network for multivariate time series long-term forecasting. Neurocomputing 500:143–154

    Article  Google Scholar 

  20. Bi H, Lu L, Meng Y (2022) Hierarchical attention network for multivariate time series long-term forecasting. Appl Intell :1–12

  21. Karanikola A, Liapis CM, Kotsiantis S (2022) A comparison of contemporary methods on univariate time series forecasting. In: Advances in machine learning/deep learning-based technologies, Springer, pp 143–168

  22. Hajmohammadi H, Heydecker B (2021) Multivariate time series modelling for urban air quality. Urban Clim 37:100834

    Article  Google Scholar 

  23. Fathian F, Fard AF, Ouarda TB, Dinpashoh Y, Nadoushani SM (2019) Modeling streamflow time series using nonlinear setar-garch models. J Hydrol 573:82–97

    Article  Google Scholar 

  24. Zhang M, Jiang X, Fang Z, Zeng Y, Xu K (2019) High-order hidden markov model for trend prediction in financial time series. Phys A Stat Mech Appl 517:1–12

    Article  Google Scholar 

  25. Rangapuram SS, Seeger M, Gasthaus J, Stella L, Wang Y, Januschowski T (2018) Deep state space models for time series forecasting. In: Proceedings of the 32nd international conference on neural information processing systems, pp 7796–7805

  26. Martínez F, Charte F, Frías MP, Martínez-Rodríguez AM (2022) Strategies for time series forecasting with generalized regression neural networks. Neurocomputing 491:509–521

    Article  Google Scholar 

  27. Chen W, Xu H, Chen Z, Jiang M (2021) A novel method for time series prediction based on error decomposition and nonlinear combination of forecasters. Neurocomputing 426:85–103

    Article  Google Scholar 

  28. Chen Z, Ma Q, Lin Z (2021) Time-aware multi-scale rnns for time series modeling. In: IJCAI

  29. Yang T, Yu X, Ma N, Zhao Y, Li H (2021) A novel domain adaptive deep recurrent network for multivariate time series prediction. Eng Appl Artif Intell 106:104498

    Article  Google Scholar 

  30. Yang Y, Fan C, Xiong H (2022) A novel general-purpose hybrid model for time series forecasting. Appl Intell 52(2):2212–2223

    Article  Google Scholar 

  31. Huang S, Wang D, Wu X, Tang A (2019) Dsanet: Dual self-attention network for multivariate time series forecasting. In: Proceedings of the 28th ACM international conference on information and knowledge management, pp 2129–2132

  32. Yazici I, Beyca OF, Delen D (2022) Deep-learning-based short-term electricity load forecasting: A real case application. Eng Appl Artif Intell 109:104645

    Article  Google Scholar 

  33. Bai S, Kolter JZ, Koltun V An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01271

  34. Fan J, Zhang K, Huang Y, Zhu Y, Chen B (2021) Parallel spatio-temporal attention-based TCN for multivariate time series prediction. Neural Comput Applic :1–10

  35. Kenton JDM-WC, Toutanova LK (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp 4171–4186

  36. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10012–10022

  37. Shen L, Wang Y (2022) Tcct: Tightly-coupled convolutional transformer on time series forecasting. Neurocomputing 480:131–145

    Article  Google Scholar 

  38. Lara-benítez P, Carranza-García M, Luna-Romera JM, Riquelme JC (2020) Temporal convolutional networks applied to energy-related time series forecasting. Appl Sci 10(7):2322

    Article  Google Scholar 

  39. Fang X, Yuan Z (2019) Performance enhancing techniques for deep learning models in time series forecasting. Eng Appl Artif Intell 85:533–542

    Article  Google Scholar 

  40. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L et al (2019) Pytorch: an imperative style, high-performance deep learning library. In: Proceedings of the 33rd international conference on neural information processing systems, pp 8026–8037

  41. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations

Download references

Funding

This work is partially supported by a grant from the National Natural Science Foundation of China (No. 62272368, No. 62032017), the Innovation Capability Support Program of Shaanxi (No. 2023-ZC-TD-0008), the Key Research and Development Program of Shaanxi (No. 2021ZDLGY03-09, No. 2021ZDLGY07-02, No. 2021ZDLGY07-03), Shaanxi Qinchuangyuan “scientists+engineers” team in 2023 (No. 41), and The Youth Innovation Team of Shaanxi Universities.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Xidian University, Xi’an, 710071, China

    Xingyu Wang, Hui Liu, Zhihan Yang, Junzhao Du & Xiyao Dong

Authors
  1. Xingyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhihan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junzhao Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiyao Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hui Liu or Junzhao Du.

Ethics declarations

Conflict of Interests

The authors have no competing financial interests to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, X., Liu, H., Yang, Z. et al. CNformer: a convolutional transformer with decomposition for long-term multivariate time series forecasting. Appl Intell 53, 20191–20205 (2023). https://doi.org/10.1007/s10489-023-04496-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04496-6

Keywords

Search

Navigation