Skip to main content
SpringerLink
Log in

An effective variational auto-encoder-based model for traffic flow imputation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Traffic flow imputation is crucial in modern intelligent transportation systems, due to frequent data missing caused by the failure of detectors or the influence of hostile external environment (e.g., signal strength). However, this work can be challenging for mainly three reasons: firstly, how to impute traffic flow that shows both universal complex spatio-temporal regularity and individual random authenticity is non-trivial. Secondly, though there are so many algorithms for traffic flow imputation, most of them either ignore different periodic dependencies or just deal with them independently. Lastly, the effectiveness of most deep learning models (etc., CNN, RNN) may be influenced by data missing in model training phase. To solve the problems above, an effective model traffic flow imputation variational auto-encoder (TFI-VAE) considering robust joint-periodic spatio-temporal features is proposed, which can impute missing value not only accurately but also realistically by introducing Gaussian mixture distribution enhanced VAE and normalization flows. Moreover, spatial missing oriented block and temporal missing oriented block are utilized in TFI-VAE to learn the spatial and temporal features of traffic flow data with ability to resist the negative effects of missing data. The experiments conducted on three real-world traffic flow datasets demonstrate that our TFI-VAE outperforms other classical imputation models from all aspects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

All data generated or analyzed during this study are included in this published article.

References

  1. Zhang S, Chen X, Chen J, Jiang Q, Huang H (2020) Anomaly detection of periodic multivariate time series under high acquisition frequency scene in IoT. In: International conference on data mining workshops (ICDMW), Sorrento Italy, pp 543–552

  2. Zhao N, Li Z, Li Y (2014) Improving the traffic data imputation accuracy using temporal and spatial information. In: International conference on intelligent computation technology and automation, Changsha China, pp 312–317

  3. Al-Deek HM, Venkata C, Ravi Chandra S (1867) New algorithms for filtering and imputation of real-time and archived dual-loop detector data in I-4 data warehouse. Transp Res Rec J Transp Res Board 116–126:2004. https://doi.org/10.3141/1867-14

    Article  Google Scholar 

  4. Qu L, Li L, Zhang Y, Hu J (2009) PPCA-based missing data imputation for traffic flow volume: a systematical approach. IEEE Trans Intell Transp Syst 10:512–522. https://doi.org/10.1109/tits.2009.2026312

    Article  Google Scholar 

  5. Xu J, Li X, Shi H (2010) Short-term traffic flow forecasting model under missing data. J Comput Appl 30:1117–1120. https://doi.org/10.3724/sp.j.1087.2010.0117

    Article  Google Scholar 

  6. Zhang J, Zheng Y, Qi D (2017) Deep spatio-temporal residual networks for citywide crowd flows prediction. In: Proceedings of the 31th AAAI conference on artificial intelligence, San Francisco USA, pp 1655–1661

  7. Chen Y, Lv Y, Wang FY (2020) Traffic flow imputation using parallel data and generative adversarial networks. IEEE Trans Intell Transp Syst 21:1624–1630. https://doi.org/10.1109/tits.2019.2910295

    Article  Google Scholar 

  8. Guo S, Lin Y, Feng N, Song C, Wan H (2019) Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In: Proceedings of the AAAI conference on artificial intelligence, Hawaii USA, pp 922–929

  9. Chen X, Cai Y, Ye Q, Chen L, Li Z (2018) Graph regularized local self-representation for missing value imputation with applications to on-road traffic sensor data. Neurocomputing 303:47–59. https://doi.org/10.1016/j.neucom.2018.04.029

    Article  Google Scholar 

  10. Guo S, Lin Y, Li S, Chen Z, Wan H (2019) Deep spatial-temporal 3D convolutional neural networks for traffic data forecasting. IEEE Trans Intell Transp Syst 20:3913–3926. https://doi.org/10.1109/tits.2019.2906365

    Article  Google Scholar 

  11. Rubin DB (1976) Inference and missing data. Biometrika 63(3):581–592

    Article  MathSciNet  Google Scholar 

  12. Gondara L, Wang K (2018) MIDA: multiple imputation using denoising auto-encoders. Adv Knowl Discov Data Min 10939:260–272. https://doi.org/10.1007/978-3-319-93040-4_21

    Article  Google Scholar 

  13. Zhong M, Sharma S, Lingras P (2004) Genetically designed models for accurate imputation of missing traffic counts. Transp Res Rec 1879:71–79. https://doi.org/10.3141/1879-09

    Article  Google Scholar 

  14. Elshenawy M, El-darieby M, Abdulhai B (2018) Automatic imputation of missing highway traffic volume data. In: IEEE international conference on pervasive computing and communications workshops, Athens Greece, pp 373–378

  15. Tak S, Woo S, Yeo H (2016) Data-driven imputation method for traffic data in sectional units of road links. IEEE Trans Intell Transp Syst 17(6):1762–1771. https://doi.org/10.1109/TITS.2016.2530312

    Article  Google Scholar 

  16. Li Y, Li Z, Li L (2014) Missing traffic data: comparison of imputation methods. IET Intell Transp Syst 8:51–57. https://doi.org/10.1049/iet-its.2013.0052

    Article  Google Scholar 

  17. Liu J, Musialski P, Wonka P, Ye J (2013) Tensor completion for estimating missing values in visual data. IEEE Trans Pattern Anal Mach Intell 35(1):208–220. https://doi.org/10.1109/TPAMI.2012.39

    Article  Google Scholar 

  18. Qiu X, Zhang Y (2019) A traffic speed imputation method based on self-adaption and clustering. In: 4th IEEE international conference on big data analytics (ICBDA), Ahmedabad India, pp 26–31

  19. Qu L, Zhang Y, Hu J, Jia L, Li L (2008) A BPCA based missing value imputing method for traffic flow volume data. In: IEEE intelligent vehicles symposium, pp 985–990. https://doi.org/10.1109/IVS.2008.4621153

  20. Li L, Li Y, Li Z (2013) Efficient missing data imputing for traffic flow by considering temporal and spatial dependence. Transp Res Part C Emerg Technol 34:108120. https://doi.org/10.1016/j.trc.2013.05.008

    Article  Google Scholar 

  21. Cao W, Wang D, Li J, Zhou H, Li L, Li Y (2018) BRITS: bidirectional recurrent imputation for time series. Adv Neural Inf Process Syst 31:6776–6786. https://doi.org/10.1007/978-3-030-89880-9_34

    Article  Google Scholar 

  22. Zhuang Y, Ke R, Wang Y (2019) Innovative method for traffic data imputation based on convolutional neural network. IET Intell Transp Syst 13:605–613. https://doi.org/10.1049/iet-its.2018.5114

    Article  Google Scholar 

  23. Asadi R, Regan A (2019) A convolution recurrent autoencoder for spatio-temporal missing data imputation. arXiv preprint, https://arxiv.org/abs/1904.12413

  24. Guo Z, Wan Y, Ye H (2019) A data imputation method for multivariate time series based on generative adversarial network. Neurocomputing 360:185–197. https://doi.org/10.1016/j.neucom.2019.06.007

    Article  Google Scholar 

  25. Luo YH, Zhang Y, Cai XR, Yuan XJ (2019) E2GAN: end-to-end generative adversarial network for multivariate time series imputation. In: Proceedings of the twenty-eighth international joint conference on artifificial intelligence, Hawaii USA, pp 3094–3100

  26. Xie C et al (2019) Image inpainting with learnable bidirectional attention maps. In: 2019 IEEE/CVF international conference on computer vision (ICCV), Seoul, Korea, pp 8857–8866

  27. Woo S, Park J, Lee JY, So Kweon I (2018) Cbam: convolutional block attention module. Comput Vis ECCV 11211:3–19. https://doi.org/10.1007/978-3-030-01234-2_1

    Article  Google Scholar 

  28. Kingma DP, Welling M (2013) Auto-encoding variational bayes. arXiv preprint, https://arxiv.org/abs/1312.6114

  29. Van Den Berg R, Hasenclever L, Tomczak JM, Welling M (2018) Sylvester normalizing flows for variational inference. In: 34th Conference on uncertainty in artificial intelligence, Monterey USA, pp 393–402

  30. Donovan B, Work D (2016) New York City taxi trip data (2010–2013). University of Illinois at Urbana-Champaign. https://doi.org/10.13012/J8PN93H8

  31. Miao D, Qin X, Wang W (2014) The periodic data traffic modelling based on multiplicative seasonal ARIMA model. In: 2014 6th international conference on wireless communications and signal processing, Hefei China, pp 1–5

  32. Hong H, Huang W, Zhou XB, Du SZ, Bian KG, Xie K (2015) Short-term traffic flow forecasting: Multi-metric KNN with related station discovery. In: 12th International conference on fuzzy systems and knowledge discovery, Zhangjiajie China, pp 1670–1675

  33. Ran B, Tan H, Wu Y, Jin PJ (2016) Tensor based missing traffic data completion with spatial–temporal correlation. Physica A Stat Mech Appl 446:54–63. https://doi.org/10.1016/j.physa.2015.09.105

    Article  Google Scholar 

  34. Li Z, Zheng H, Feng X (2018) 3D convolutional generative adversarial networks for missing traffic data completion. In: 10th International conference on wireless communications and signal processing, Hangzhou China, pp 1–6

  35. Lin Z, Feng J, Lu Z, Li Y, Jin D (2019) DeepSTN+: context-aware spatial-temporal neural network for crowd flow prediction in metropolis. In: Proceedings of the AAAI conference on artificial intelligence, Hawaii USA, pp 1020–1027

  36. Chen J, Zhang S, Chen X, Jiang Q, Huang H, Gu C (2021) Learning traffic as videos: a spatio-temporal VAE approach for traffic data imputation. In: International conference on artificial neural networks and machine learning, Bratislava Slovakia, pp 12895

  37. Duan Y, Lv Y, Liu YL, Wang FY (2016) An efficient realization of deep learning for traffic data imputation. Transp Res Part C Emerg Technol 72:168–181. https://doi.org/10.1016/j.trc.2016.09.015

    Article  Google Scholar 

  38. Ribeiro MVL, Aching Samatelo JL, Cetertich Bazzan AL (2022) A new microscopic approach to traffic flow classification using a convolutional neural network object detector and a multi-tracker algorithm. IEEE Trans Intell Transp Syst 23(4):3797–3801

    Article  Google Scholar 

  39. Shen G, Zhou W, Zhang W, Liu N, Liu Z, Kong X (2023) Bidirectional spatial–temporal traffic data imputation via graph attention recurrent neural network. Neurocomputing 531:151–162

    Article  Google Scholar 

  40. Kong X, Zhou W, Shen G, Zhang W, Liu N, Yang Y (2023) Dynamic graph convolutional recurrent imputation network for spatiotemporal traffic missing data. Knowl Based Syst 261:110188

    Article  Google Scholar 

  41. Yuan Y, Zhang Y, Wang B, Peng Y, Hu Y, Yin B (2023) STGAN: spatio-temporal generative adversarial network for traffic data imputation. IEEE Trans Big Data 9(1):200–211

    Article  Google Scholar 

  42. Wang P, Zhang T, Zheng Y, Hu T (2022) A multi-view bidirectional spatiotemporal graph network for urban traffic flow imputation. Int J Geograph Inf Sci 36(6):1231–1257

    Article  Google Scholar 

  43. Cuza CEM, Ho N, Zacharatou ET, Pedersen TB, Yang B (2022) Spatio-temporal graph convolutional network for stochastic traffic speed imputation. In: International conference on advances in geographic information systems, NY, USA, pp 1–12

  44. Ming J et al (2022) Multi-graph convolutional recurrent network for fine-grained lane-level traffic flow imputation. In: IEEE international conference on data mining, Orlando, FL, USA, pp 348–357

  45. Benkraouda O, Thodi BT, Yeo H, Menéndez M, Jabari SE (2020) Traffic data imputation using deep convolutional neural networks. IEEE Access 8:104740–104752

    Article  Google Scholar 

  46. Chen X, He Z, Sun L (2019) A Bayesian tensor decomposition approach for spatio-temporal traffic data imputation. Transp Res Part C Emerg Technol 98:73–84

    Article  Google Scholar 

  47. Cini A, Marisca I, Alippi C (2021) Filling the g_ap_s: multivariate time series imputation by graph neural networks. arXiv preprint https://arxiv.org/abs/2108.00298

Download references

Acknowledgements

This work is supported in part by Shenzhen Science and Technology Program Foundation of China under Grant 61732022, and in part by the Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies under Grant 2022B1212010005.

Author information

Authors and Affiliations

  1. Department of Computer Science, Harbin Institute of Technology, Shenzhen, China

    Shuo Zhang, Xingbang Hu, Jinyi Chen, Wenbo Zhang & Hejiao Huang

  2. Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Shenzhen, China

    Hejiao Huang

Authors
  1. Shuo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingbang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinyi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenbo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hejiao Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hejiao Huang.

Ethics declarations

Conflict of interest

The authors have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Hu, X., Chen, J. et al. An effective variational auto-encoder-based model for traffic flow imputation. Neural Comput & Applic 36, 2617–2631 (2024). https://doi.org/10.1007/s00521-023-09127-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09127-2

Keywords

Search

Navigation