Skip to main content
SpringerLink
Log in

Transformer network with decoupled spatial–temporal embedding for traffic flow forecasting

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Over the past few years, there has been significant research on applying Transformer models to time series prediction, yielding promising results. Simultaneously, researchers have begun exploring the utilization of Transformers for traffic prediction in order to mitigate the nonlinear spatial–temporal correlation inherent in traffic data. Some of these studies have attempted to characterize spatial–temporal features by incorporating embedding structures, with the goal of improving performance of the model. However, existing methods have not adequately addressed the issue of spatial–temporal correlation. To address these limitations, we propose the Transformer Network with Decoupled Spatial–Temporal Embedding (DSTET) model for traffic flow prediction. The key aspect of our model is its ability to effectively decouple the spatial and temporal embedding through the implementation of the Decoupled Spatial–Temporal Embedding structure. This structure enhances the characterization of spatial–temporal features, ultimately improving the performance of traffic prediction based on the Transformer model. Through experiments conducted on six real-world traffic datasets, our model consistently outperforms multiple baseline models, demonstrating its capability to address the identified problems. Moreover, we substantiate the efficacy of the suggested components via ablation experiments and furnish a thorough analysis of the attention weight matrix to clarify the functioning of the model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The datasets supporting the results of this study, including METR-LA and PEMS-Bay, can be accessed from reference [20], while the PEMS03, PEMS04, PEMS07, and PEMS08 datasets can be accessed from reference [45], subject to restrictions regarding their usage, as they are utilized with permission for this study.

References

  1. Bai L, Yao L, Li C, Wang X, Wang C (2020) Adaptive graph convolutional recurrent network for traffic forecasting. Adv Neural Inf Process Syst 33:17804–17815

    Google Scholar 

  2. Wu Z, Pan S, Long G, Jiang J, Chang X, Zhang C (2020) Connecting the dots: Multivariate time series forecasting with graph neural networks. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp 753–763

  3. Zheng C, Fan X, Wang C, Qi J (2020) Gman: A graph multi-attention network for traffic prediction. In: Proceedings of the AAAI conference on artificial intelligence. 34(01):1234–1241

  4. Feng A, Tassiulas L (2022) Adaptive graph spatial-temporal transformer network for traffic forecasting. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp 3933–3937

  5. Yan H, Ma X, Pu Z (2021) Learning dynamic and hierarchical traffic spatiotemporal features with transformer. IEEE Trans Intell Transp Syst 23(11):22386–22399

    Article  Google Scholar 

  6. Graves A, Graves A (2012) Long short-term memory. Supervised sequence labelling with recurrent neural networks, 37–45

  7. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Polosukhin I (2017) Attention is all you need. Adv Neur Inf Process Syst, 30

  8. Hu G, Xiao X, Shen M, Zhang B, Yan X, Liu Y (2023) TCGNN: Packet-grained network traffic classification via Graph Neural Networks. Eng Appl Artif Intell 123:106531

    Article  Google Scholar 

  9. Gong J, Zhao Y, Zhao J, Zhang J, Ma G, Zheng S, Tang J (2024) Personalized recommendation via inductive spatiotemporal graph neural network. Pattern Recognition 145:109884

    Article  Google Scholar 

  10. Van Der Voort M, Dougherty M, Watson S (1996) Combining Kohonen maps with ARIMA time series models to forecast traffic flow. Trans Res Part C Emerg Technol 4(5):307–318

    Article  Google Scholar 

  11. Li S, Corney J (2023) Multi-view expressive graph neural networks for 3D CAD model classification. Comput Ind 151:103993

    Article  Google Scholar 

  12. Chen Z, Li D, Liu M, Liu J (2023) Graph neural networks with molecular segmentation for property prediction and structure–property relationship discovery. Comput Chem Eng 179:108403

    Article  Google Scholar 

  13. Fu X, Peng J, Jiang G, Wang H (2022) Learning latent features with local channel drop network for vehicle re-identification. Eng Appl Artif Intell 107:104540

    Article  Google Scholar 

  14. Wang H, Yao M, Jiang G, Mi Z, Fu X (2023) Graph-collaborated auto-encoder hashing for multiview binary clustering. IEEE Trans Neur Netw Learn Sys

  15. Wang H, Wang Y, Zhang Z, Fu X, Zhuo L, Xu M, Wang M (2020) Kernelized multiview subspace analysis by self-weighted learning. IEEE Trans Multimed 23:3828–3840

    Article  Google Scholar 

  16. Wang H, Jiang G, Peng J, Deng R, Fu X (2022) Towards adaptive consensus graph: multi-view clustering via graph collaboration. IEEE Trans Multimed

  17. Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2020) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24

    Article  MathSciNet  Google Scholar 

  18. Ma X, Tao Z, Wang Y, Yu H, Wang Y (2015) Long short-term memory neural network for traffic speed prediction using remote microwave sensor data. Trans Res Part C Emerg Technol 54:187–197

    Article  Google Scholar 

  19. Gao S, Huang Y, Zhang S, Han J, Wang G, Zhang M, Lin Q (2020) Short-term runoff prediction with GRU and LSTM networks without requiring time step optimization during sample generation. J Hydrol 589:125188

    Article  Google Scholar 

  20. Huang Y, Weng Y, Yu S, Chen X (2019) Diffusion convolutional recurrent neural network with rank influence learning for traffic forecasting. In: 2019 18th IEEE International conference on trust, security and privacy in computing and communications/13th IEEE International conference on big data science and engineering (TrustCom/BigDataSE), pp. 678–685. IEEE

  21. Zhao L, Song Y, Zhang C, Liu Y, Wang P, Lin T, Li H (2019) T-gcn: A temporal graph convolutional network for traffic prediction. IEEE Trans Intell Trans Syst 21(9):3848–3858

    Article  Google Scholar 

  22. Yu B, Yin H, Zhu Z (2018) Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence 3634–3640

  23. Wu Z, Pan S, Long G, Jiang J, Zhang C (2019) Graph WaveNet for deep spatial-temporal graph modeling. In: The 28th International Joint Conference on Artificial Intelligence (IJCAI). International Joint Conferences on Artificial Intelligence Organization

  24. Song C, Lin Y, Guo S, Wan H (2020) Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In: Proceedings of the AAAI conference on artificial intelligence, 34(01):914–921

  25. Fang Z, Long Q, Song G, Xie K (2021) Spatial-temporal graph ode networks for traffic flow forecasting. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp 364–373

  26. Chen RT, Rubanova Y, Bettencourt J, Duvenaud DK (2018) Neural ordinary differential equations. Advances in neural information processing systems, 31

  27. Guo S, Lin Y, Wan H, Li X, Cong G (2021) Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting. IEEE Trans Knowl Data Eng 34(11):5415–5428

    Article  Google Scholar 

  28. Ye X, Fang S, Sun F, Zhang C, Xiang S (2022) Meta graph transformer: A novel framework for spatial–temporal traffic prediction. Neurocomputing 491:544–563

    Article  Google Scholar 

  29. Cai L, Janowicz K, Mai G, Yan B, Zhu R (2020) Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting. Trans GIS 24(3):736–755

    Article  Google Scholar 

  30. Fang Y, Zhao F, Qin Y, Luo H, Wang C (2022) Learning all dynamics: traffic forecasting via locality-aware spatio-temporal joint transformer. IEEE Trans Intell Transp Syst 23(12):23433–23446

    Article  Google Scholar 

  31. Chen Y, Zheng L, Liu W (2022) Spatio-temporal attention-based graph convolution networks for traffic prediction. In: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp 642–649. IEEE

  32. Zhang H, Zou Y, Yang X, Yang H (2022) A temporal fusion transformer for short-term freeway traffic speed multistep prediction. Neurocomputing 500:329–340

    Article  Google Scholar 

  33. Li Y, Moura JM (2020) Forecaster: A graph transformer for forecasting spatial and time-dependent data. In: ECAI 2020, pp 1293–1300. IOS Press

  34. Wang Q, He G, Lu P, Chen Q, Chen Y, Huang W (2022) Spatial-temporal graph-based transformer model for traffic flow forecasting. In: 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp 2806–2811. IEEE

  35. Li Y, Wang H, Li J, Liu C, Tan J (2022) ACT: Adversarial convolutional transformer for time series forecasting. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp 1–8. IEEE

  36. Li F, Yan H, Jin G, Liu Y, Li Y, Jin D (2022) Automated spatio-temporal synchronous modeling with multiple graphs for traffic prediction. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp 1084–1093

  37. Zheng Q, Zhang Y (2022) DSTAGCN: Dynamic spatial-temporal adjacent graph convolutional network for traffic forecasting. IEEE Trans Big Data 9(1):241–253

    Article  Google Scholar 

  38. Nie T, Qin G, Sun L, Wang Y, Sun J Nexus sine qua non: Essentially connected networks for traffic forecasting

  39. Ji J, Yu F, Lei M (2022) Self-supervised spatiotemporal graph neural networks with self-distillation for traffic prediction. IEEE Trans Intell Transp Syst 24(2):1580–1593

    Google Scholar 

  40. Sun Y, Jiang X, Hu Y, Duan F, Guo K, Wang B, Yin B (2022) Dual dynamic spatial-temporal graph convolution network for traffic prediction. IEEE Trans Intell Trans Syst 23(12):23680–23693

    Article  Google Scholar 

  41. Deng J, Chen X, Jiang R, Song X, Tsang IW (2021) St-norm: Spatial and temporal normalization for multi-variate time series forecasting. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp 269–278

  42. Shao Z, Zhang Z, Wang F, Wei W, Xu Y (2022) Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp 4454–4458

  43. Shan Y, Hoens TR, Jiao J, Wang H, Yu D, Mao JC (2016) Deep crossing: Web-scale modeling without manually crafted combinatorial features. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 255–262

  44. Jiang R, Wang Z, Yong J, Jeph P, Chen Q, Kobayashi Y, Suzumura T (2023) Spatio-temporal meta-graph learning for traffic forecasting. In: Proceedings of the AAAI Conference on Artificial Intelligence 37(7):8078–8086

    Article  Google Scholar 

  45. Guo S, Lin Y, Feng N, Song C, Wan H (2019) Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In: Proceedings of the AAAI conference on artificial intelligence. 33(01):922–929

Download references

Funding

No specific funding for this work was received by the authors.

Author information

Authors and Affiliations

  1. School of Information Engineering, Shanghai Maritime University, Shanghai, 201306, China

    Wei Sun, Rongzhang Cheng, Yingqi Jiao, Junbo Gao, Zhedian Zheng & Nan Lu

Authors
  1. Wei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rongzhang Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yingqi Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junbo Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhedian Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

W.S.: Writing-review & editing. R.C.: Writing-original draft. Y.J.: Supervision. J.G.: Supervision; Validation. Z.Z.: Supervision. N.L.: Supervision.

Corresponding author

Correspondence to Wei Sun.

Ethics declarations

Conflict of interests

The authors declare no conflicts of interest pertaining to the research, writing, and publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, W., Cheng, R., Jiao, Y. et al. Transformer network with decoupled spatial–temporal embedding for traffic flow forecasting. Appl Intell 53, 30148–30168 (2023). https://doi.org/10.1007/s10489-023-05126-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05126-x

Keywords

Search

Navigation