Abstract
Over the past few years, there has been significant research on applying Transformer models to time series prediction, yielding promising results. Simultaneously, researchers have begun exploring the utilization of Transformers for traffic prediction in order to mitigate the nonlinear spatial–temporal correlation inherent in traffic data. Some of these studies have attempted to characterize spatial–temporal features by incorporating embedding structures, with the goal of improving performance of the model. However, existing methods have not adequately addressed the issue of spatial–temporal correlation. To address these limitations, we propose the Transformer Network with Decoupled Spatial–Temporal Embedding (DSTET) model for traffic flow prediction. The key aspect of our model is its ability to effectively decouple the spatial and temporal embedding through the implementation of the Decoupled Spatial–Temporal Embedding structure. This structure enhances the characterization of spatial–temporal features, ultimately improving the performance of traffic prediction based on the Transformer model. Through experiments conducted on six real-world traffic datasets, our model consistently outperforms multiple baseline models, demonstrating its capability to address the identified problems. Moreover, we substantiate the efficacy of the suggested components via ablation experiments and furnish a thorough analysis of the attention weight matrix to clarify the functioning of the model.









Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The datasets supporting the results of this study, including METR-LA and PEMS-Bay, can be accessed from reference [20], while the PEMS03, PEMS04, PEMS07, and PEMS08 datasets can be accessed from reference [45], subject to restrictions regarding their usage, as they are utilized with permission for this study.
References
Bai L, Yao L, Li C, Wang X, Wang C (2020) Adaptive graph convolutional recurrent network for traffic forecasting. Adv Neural Inf Process Syst 33:17804–17815
Wu Z, Pan S, Long G, Jiang J, Chang X, Zhang C (2020) Connecting the dots: Multivariate time series forecasting with graph neural networks. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp 753–763
Zheng C, Fan X, Wang C, Qi J (2020) Gman: A graph multi-attention network for traffic prediction. In: Proceedings of the AAAI conference on artificial intelligence. 34(01):1234–1241
Feng A, Tassiulas L (2022) Adaptive graph spatial-temporal transformer network for traffic forecasting. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp 3933–3937
Yan H, Ma X, Pu Z (2021) Learning dynamic and hierarchical traffic spatiotemporal features with transformer. IEEE Trans Intell Transp Syst 23(11):22386–22399
Graves A, Graves A (2012) Long short-term memory. Supervised sequence labelling with recurrent neural networks, 37–45
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Polosukhin I (2017) Attention is all you need. Adv Neur Inf Process Syst, 30
Hu G, Xiao X, Shen M, Zhang B, Yan X, Liu Y (2023) TCGNN: Packet-grained network traffic classification via Graph Neural Networks. Eng Appl Artif Intell 123:106531
Gong J, Zhao Y, Zhao J, Zhang J, Ma G, Zheng S, Tang J (2024) Personalized recommendation via inductive spatiotemporal graph neural network. Pattern Recognition 145:109884
Van Der Voort M, Dougherty M, Watson S (1996) Combining Kohonen maps with ARIMA time series models to forecast traffic flow. Trans Res Part C Emerg Technol 4(5):307–318
Li S, Corney J (2023) Multi-view expressive graph neural networks for 3D CAD model classification. Comput Ind 151:103993
Chen Z, Li D, Liu M, Liu J (2023) Graph neural networks with molecular segmentation for property prediction and structure–property relationship discovery. Comput Chem Eng 179:108403
Fu X, Peng J, Jiang G, Wang H (2022) Learning latent features with local channel drop network for vehicle re-identification. Eng Appl Artif Intell 107:104540
Wang H, Yao M, Jiang G, Mi Z, Fu X (2023) Graph-collaborated auto-encoder hashing for multiview binary clustering. IEEE Trans Neur Netw Learn Sys
Wang H, Wang Y, Zhang Z, Fu X, Zhuo L, Xu M, Wang M (2020) Kernelized multiview subspace analysis by self-weighted learning. IEEE Trans Multimed 23:3828–3840
Wang H, Jiang G, Peng J, Deng R, Fu X (2022) Towards adaptive consensus graph: multi-view clustering via graph collaboration. IEEE Trans Multimed
Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2020) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24
Ma X, Tao Z, Wang Y, Yu H, Wang Y (2015) Long short-term memory neural network for traffic speed prediction using remote microwave sensor data. Trans Res Part C Emerg Technol 54:187–197
Gao S, Huang Y, Zhang S, Han J, Wang G, Zhang M, Lin Q (2020) Short-term runoff prediction with GRU and LSTM networks without requiring time step optimization during sample generation. J Hydrol 589:125188
Huang Y, Weng Y, Yu S, Chen X (2019) Diffusion convolutional recurrent neural network with rank influence learning for traffic forecasting. In: 2019 18th IEEE International conference on trust, security and privacy in computing and communications/13th IEEE International conference on big data science and engineering (TrustCom/BigDataSE), pp. 678–685. IEEE
Zhao L, Song Y, Zhang C, Liu Y, Wang P, Lin T, Li H (2019) T-gcn: A temporal graph convolutional network for traffic prediction. IEEE Trans Intell Trans Syst 21(9):3848–3858
Yu B, Yin H, Zhu Z (2018) Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence 3634–3640
Wu Z, Pan S, Long G, Jiang J, Zhang C (2019) Graph WaveNet for deep spatial-temporal graph modeling. In: The 28th International Joint Conference on Artificial Intelligence (IJCAI). International Joint Conferences on Artificial Intelligence Organization
Song C, Lin Y, Guo S, Wan H (2020) Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In: Proceedings of the AAAI conference on artificial intelligence, 34(01):914–921
Fang Z, Long Q, Song G, Xie K (2021) Spatial-temporal graph ode networks for traffic flow forecasting. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp 364–373
Chen RT, Rubanova Y, Bettencourt J, Duvenaud DK (2018) Neural ordinary differential equations. Advances in neural information processing systems, 31
Guo S, Lin Y, Wan H, Li X, Cong G (2021) Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting. IEEE Trans Knowl Data Eng 34(11):5415–5428
Ye X, Fang S, Sun F, Zhang C, Xiang S (2022) Meta graph transformer: A novel framework for spatial–temporal traffic prediction. Neurocomputing 491:544–563
Cai L, Janowicz K, Mai G, Yan B, Zhu R (2020) Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting. Trans GIS 24(3):736–755
Fang Y, Zhao F, Qin Y, Luo H, Wang C (2022) Learning all dynamics: traffic forecasting via locality-aware spatio-temporal joint transformer. IEEE Trans Intell Transp Syst 23(12):23433–23446
Chen Y, Zheng L, Liu W (2022) Spatio-temporal attention-based graph convolution networks for traffic prediction. In: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp 642–649. IEEE
Zhang H, Zou Y, Yang X, Yang H (2022) A temporal fusion transformer for short-term freeway traffic speed multistep prediction. Neurocomputing 500:329–340
Li Y, Moura JM (2020) Forecaster: A graph transformer for forecasting spatial and time-dependent data. In: ECAI 2020, pp 1293–1300. IOS Press
Wang Q, He G, Lu P, Chen Q, Chen Y, Huang W (2022) Spatial-temporal graph-based transformer model for traffic flow forecasting. In: 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp 2806–2811. IEEE
Li Y, Wang H, Li J, Liu C, Tan J (2022) ACT: Adversarial convolutional transformer for time series forecasting. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp 1–8. IEEE
Li F, Yan H, Jin G, Liu Y, Li Y, Jin D (2022) Automated spatio-temporal synchronous modeling with multiple graphs for traffic prediction. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp 1084–1093
Zheng Q, Zhang Y (2022) DSTAGCN: Dynamic spatial-temporal adjacent graph convolutional network for traffic forecasting. IEEE Trans Big Data 9(1):241–253
Nie T, Qin G, Sun L, Wang Y, Sun J Nexus sine qua non: Essentially connected networks for traffic forecasting
Ji J, Yu F, Lei M (2022) Self-supervised spatiotemporal graph neural networks with self-distillation for traffic prediction. IEEE Trans Intell Transp Syst 24(2):1580–1593
Sun Y, Jiang X, Hu Y, Duan F, Guo K, Wang B, Yin B (2022) Dual dynamic spatial-temporal graph convolution network for traffic prediction. IEEE Trans Intell Trans Syst 23(12):23680–23693
Deng J, Chen X, Jiang R, Song X, Tsang IW (2021) St-norm: Spatial and temporal normalization for multi-variate time series forecasting. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp 269–278
Shao Z, Zhang Z, Wang F, Wei W, Xu Y (2022) Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp 4454–4458
Shan Y, Hoens TR, Jiao J, Wang H, Yu D, Mao JC (2016) Deep crossing: Web-scale modeling without manually crafted combinatorial features. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 255–262
Jiang R, Wang Z, Yong J, Jeph P, Chen Q, Kobayashi Y, Suzumura T (2023) Spatio-temporal meta-graph learning for traffic forecasting. In: Proceedings of the AAAI Conference on Artificial Intelligence 37(7):8078–8086
Guo S, Lin Y, Feng N, Song C, Wan H (2019) Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In: Proceedings of the AAAI conference on artificial intelligence. 33(01):922–929
Funding
No specific funding for this work was received by the authors.
Author information
Authors and Affiliations
Contributions
W.S.: Writing-review & editing. R.C.: Writing-original draft. Y.J.: Supervision. J.G.: Supervision; Validation. Z.Z.: Supervision. N.L.: Supervision.
Corresponding author
Ethics declarations
Conflict of interests
The authors declare no conflicts of interest pertaining to the research, writing, and publication of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Sun, W., Cheng, R., Jiao, Y. et al. Transformer network with decoupled spatial–temporal embedding for traffic flow forecasting. Appl Intell 53, 30148–30168 (2023). https://doi.org/10.1007/s10489-023-05126-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-023-05126-x