Skip to main content
SpringerLink
Log in

A distributed WND-LSTM model on MapReduce for short-term traffic flow prediction

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Building data-driven intelligent transportation is a significant task for establishing data-centric smart cities, and exceptionally efficient and accurate traffic flow prediction (TFP) is a crucial technology in constructing intelligent transportation systems (ITSs). To address the computation and storage problems of processing traffic flow big data with the centralized model on a traditional mining platform, we propose a distributed long short-term memory weighted model combined with a time window and normal distribution based on a MapReduce parallel processing framework in this paper, named as WND-LSTM. More specifically, under the Hadoop distributed computing platform, a distributed modeling framework of forecasting traffic flow on MapReduce is developed to solve the existing issues of storage and calculation in handling large-scale traffic flow data with the stand-alone learning model. Moreover, a distributed WND-LSTM model is presented on the MapReduce-based distributed modeling framework to enhance the accuracy, efficiency, and scalability of short-term TFP. Finally, we forecast the traffic flow on the Sanlihe East Road of Beijing in China using the proposed WND-LSTM model with the real-world taxi trajectory big data. In particular, the extensively experimental results from a case study demonstrate that the MAPE value of WND-LSTM is 88.48%, 65.79%, 70.46%, 68.21%, 66.95%, 68.43%, and 70.41% lower than that of the autoregressive integrated moving average (ARIMA), logistical regression (LR), support vector regression (SVR), k-nearest neighbor (KNN), stacked autoencoders (SAEs), gated recurrent unit (GRU), and long short-term memory (LSTM), respectively, and achieves 71.25% accuracy improvement on average.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. http://www.datatang.com.

References

  1. Pan TL, Sumalee A, Zhong RX, Indra-Payoong N (2013) Short-term traffic state prediction based on temporal-spatial correlation. IEEE Trans Intell Transp Syst 14:1242–1254

    Google Scholar 

  2. Diaz G, Macia H, Valero V, Boubeta-Puig J, Cuartero F (2020) An intelligent transportation system to control air pollution and road traffic in cities integrating CEP and Colored Petri Nets. Neural Comput Appl 32:405–426

    Google Scholar 

  3. Xia D, Wang B, Li H, Li Y, Zhang Z (2016) A distributed spatial-temporal weighted model on MapReduce for short-term traffic flow forecasting. Neurocomputing 179:246–263

    Google Scholar 

  4. Abdi J, Moshiri B, Abdulhai B, Sedigh AK (2013) Short-term traffic flow forecasting: parametric and nonparametric approaches via emotional temporal difference learning. Neural Comput Appl 23:141–159

    Google Scholar 

  5. Castro-Neto M, Jeong YS, Jeong MK, Han LD (2009) Online-SVR for short-term traffic flow prediction under typical and atypical traffic conditions. Expert Syst Appl 36:6164–6173

    Google Scholar 

  6. Guo F, Krishnan R, Polak J (2013) A computationally efficient two-stage method for short-term traffic prediction on urban roads. Transp Plan Technol 36:62–75

    Google Scholar 

  7. Lippi M, Bertini M, Frasconi P (2013) Short-term traffic flow forecasting: an experimental comparison of time-series analysis and supervised learning. IEEE Trans Intell Transp Syst 14:871–882

    Google Scholar 

  8. Palivonaite R, Lukoseviciute K, Ragulskis M (2014) Short-term time series algebraic forecasting with mixed smoothing. Neurocomputing 127:161–171

    Google Scholar 

  9. Huang W, Song G, Hong H, Xie K (2014) Deep architecture for traffic flow prediction: deep belief networks with multitask learning. IEEE Trans Intell Transp Syst 15:2191–2201

    Google Scholar 

  10. Okutani I, Stephanedes YJ (1984) Dynamic prediction of traffic volume through Kalman Filtering theory. Transp Res Part B Methodol 18:1–11

    Google Scholar 

  11. Alghamdi T, Elgazzar K, Bayoumi M, Sharaf T, Shah S (2019) Forecasting traffic congestion using ARIMA modeling. In: 2019 15th International wireless communications & mobile computing conference (IWCMC), IEEE, pp 1227–1232. https://doi.org/10.1109/IWCMC.2019.8766698

  12. Dell’Acqua P, Bellotti F, Berta R, De Gloria A (2015) Time-aware multivariate nearest neighbor regression methods for traffic flow prediction. IEEE Trans Intell Transp Syst 16:3393–3402

    Google Scholar 

  13. Luo C, Huang C, Cao J, Lu J, Huang W, Guo J, Wei Y (2019) Short-term traffic flow prediction based on least square Support Vector Machine with hybrid optimization algorithm. Neural Process Lett 50:2305–2322

    Google Scholar 

  14. Chan KY, Dillon T, Chang E, Singh J (2012) Prediction of short-term traffic variables using intelligent swarm-based neural networks. IEEE Trans Control Syst Technol 21:263–274

    Google Scholar 

  15. Kong F, Li J, Jiang B, Song H (2019) Short-term traffic flow prediction in smart multimedia system for internet of vehicles based on deep belief network. Fut Generat Comput Syst 93:460–472

    Google Scholar 

  16. Liu B, Cheng J, Cai K, Shi P, Tang X (2017) Singular point probability improve LSTM network performance for long-term traffic flow prediction. In: Du D, Li L, Zhu E, He K (eds) Theoretical computer science. NCTCS 2017. Communications in computer and information science, vol 768. Springer, Singapore. https://doi.org/10.1007/978-981-10-6893-5_24

    Chapter  Google Scholar 

  17. Zhang N, Zhang Y, Lu H (2011) Seasonal autoregressive integrated moving average and support vector machine models: prediction of short-term traffic flow on freeways. Transp Res Rec 2215:85–92

    Google Scholar 

  18. Moretti F, Pizzuti S, Panzieri S, Annunziato M (2015) Urban traffic flow forecasting through statistical and neural network bagging ensemble hybrid modeling. Neurocomputing 167:3–7

    Google Scholar 

  19. Greff K, Srivastava RK, Koutník J, Steunebrink BR, Schmidhuber J (2016) LSTM: a search space odyssey. IEEE Trans Neural Netw Learn Syst 28:2222–2232

    MathSciNet  Google Scholar 

  20. Graves A, Liwicki M, Fernández S, Bertolami R, Bunke H, Schmidhuber J (2009) A novel connectionist system for unconstrained handwriting recognition. IEEE Trans Pattern Anal Mach Intell 31:855–868

    Google Scholar 

  21. Wang J, Zhang J, Wang X (2017) Bilateral LSTM: a two-dimensional long short-term memory model with multiply memory units for short-term cycle time forecasting in reentrant manufacturing systems. IEEE Trans Ind Inform 14:748–758

    Google Scholar 

  22. Mackenzie J, Roddick JF, Zito R (2018) An evaluation of HTM and LSTM for short-term arterial traffic flow prediction. IEEE Transa Intell Transp Syst 20:1847–1857

    Google Scholar 

  23. Chen J, Li D, Zhang G, Zhang X (2018) Localized space-time autoregressive parameters estimation for traffic flow prediction in urban road networks. Appl Sci 8:277

    Google Scholar 

  24. Li Y, Jiang X, Zhu H, He X, Peeta S, Zheng T, Li Y (2016) Multiple measures-based chaotic time series for traffic flow prediction based on Bayesian theory. Nonlinear Dyn 85:179–194

    MathSciNet  MATH  Google Scholar 

  25. Emami A, Sarvi M, Bagloee SA (2019) Using Kalman Filter algorithm for short-term traffic flow prediction in a connected vehicle environment. J Mod Transp 27:222–232

    Google Scholar 

  26. Ma M, Liang S, Guo H, Yang J (2017) Short-term traffic flow prediction using a self-adaptive two-dimensional forecasting method. Adv Mech Eng 9:1687814017719002

    Google Scholar 

  27. Safarinejadian B, Estahbanati ME (2015) Consensus filter-based distributed variational Bayesian algorithm for flow and speed density prediction with distributed traffic sensors. IEEE Syst J 11:2939–2948

    Google Scholar 

  28. Duan P, Mao G, Yue W, Wang S (2018) A unified STARIMA based model for short-term traffic flow prediction. In: 2018 21st International conference on intelligent transportation systems (ITSC), IEEE, pp 1652–1657. https://doi.org/10.1109/ITSC.2018.8569964

  29. Li M-W, Hong W-C, Kang H-G (2013) Urban traffic flow forecasting using Gauss-SVR with cat mapping, cloud model and PSO hybrid algorithm. Neurocomputing 99:230–240

    Google Scholar 

  30. Shang Q, Lin C, Yang Z, Bing Q, Zhou X (2016) Short-term traffic flow prediction model using particle swarm optimization-based combined kernel function-least squares support vector machine combined with chaos theory. Adv Mech Eng 8:1687814016664654

    Google Scholar 

  31. Duo M, Qi Y, Lina G, Xu E (2017) A short-term traffic flow prediction model based on EMD and GPSO-SVM. In: 2017 IEEE 2nd advanced information technology, electronic and automation control conference (IAEAC), IEEE, pp 2554–2558. https://doi.org/10.1109/IAEAC.2017.8054485

  32. Ling X, Feng X, Chen Z, Xu Y, Zheng H (2017) Short-term traffic flow prediction with optimized Multi-kernel Support Vector Machine. In: 2017 IEEE congress on evolutionary computation (CEC), IEEE, pp 294–300. https://doi.org/10.1109/CEC.2017.7969326

  33. Lv Y, Duan Y, Kang W, Li Z, Wang F-Y (2014) Traffic flow prediction with big data: a deep learning approach. IEEE Trans Intell Transp Syst 16:865–873

    Google Scholar 

  34. Zhao W, Gao Y, Ji T, Wan X, Ye F, Bai G (2019) Deep temporal convolutional networks for short-term traffic flow forecasting. IEEE Access 7:114496–114507

    Google Scholar 

  35. Zhao L, Zhou Y, Lu H, Fujita H (2019) Parallel computing method of deep belief networks and its application to traffic flow prediction. Knowl Based Syst 163:972–987

    Google Scholar 

  36. Li L, Qin L, Qu X, Zhang J, Wang Y, Ran B (2019) Day-ahead traffic flow forecasting based on a deep belief network optimized by the multi-objective particle swarm algorithm. Knowl Based Syst 172:1–14

    Google Scholar 

  37. Polson NG, Sokolov VO (2017) Deep learning for short-term traffic flow prediction. Transp Res Part C Emerg Technol 79:1–17

    Google Scholar 

  38. Deng S, Jia S, Chen J (2019) Exploring spatial-temporal relations via deep convolutional neural networks for traffic flow prediction with incomplete data. Appl Soft Comput 78:712–721

    Google Scholar 

  39. Mou L, Zhao P, Xie H, Chen Y (2019) T-LSTM: a long short-term memory neural network enhanced by temporal information for traffic flow prediction. IEEE Access 7:98053–98060

    Google Scholar 

  40. Zhang H, Wang X, Cao J, Tang M, Guo Y (2018) A multivariate short-term traffic flow forecasting method based on wavelet analysis and seasonal time series. Appl Intell 48:3827–3838

    Google Scholar 

  41. Tian X, Yu D, Xing X, Wang S, Wang Z (2019) Hybrid short-term traffic flow prediction model of intersections based on improved complete ensemble empirical mode decomposition with adaptive noise. Adv Mech Eng 11:1687814019841819

    Google Scholar 

  42. Zhao S, Zhao Q, Bai Y, Li S (2019) A traffic flow prediction method based on road crossing vector coding and a bidirectional recursive neural network. Electronics 8:1006

    Google Scholar 

  43. El-Sayed H, Sankar S, Daraghmi Y-A, Tiwari P, Rattagan E, Mohanty M, Puthal D, Prasad M (2018) Accurate traffic flow prediction in heterogeneous vehicular networks in an intelligent transport system using a supervised non-parametric classifier. Sensors 18:1696

    Google Scholar 

  44. Luo X, Niu L, Zhang S (2018) An algorithm for traffic flow prediction based on improved SARIMA and GA. KSCE J Eng 22:4107–4115

    Google Scholar 

  45. Shao H, Soong B-H (2016) Traffic flow prediction with long short-term memory networks (LSTMs). In: 2016 IEEE region 10 conference (TENCON), IEEE, pp 2986–2989. https://doi.org/10.1109/TENCON.2016.7848593

  46. Luo X, Li D, Yang Y, Zhang S (2019) Spatiotemporal traffic flow prediction with KNN and LSTM. J Adv Trans 2019:4145353

    Google Scholar 

  47. Wu T, Chen F, Wan Y (2018) Graph attention LSTM network: a new model for traffic flow forecasting. In: 2018 5th International conference on information science and control engineering (ICISCE), IEEE, pp 241–245. https://doi.org/10.1109/ICISCE.2018.00058

  48. Ma D, Sheng B, Jin S, Ma X, Gao P (2018) Short-term traffic flow forecasting by selecting appropriate predictions based on pattern matching. IEEE Access 6:75629–75638

    Google Scholar 

  49. Tian Y, Zhang K, Li J, Lin X, Yang B (2018) LSTM-based traffic flow prediction with missing data. Neurocomputing 318:297–305

    Google Scholar 

  50. Fu R, Zhang Z, Li L (2016) Using LSTM and GRU neural network methods for traffic flow prediction. In: 2016 31st Youth academic annual conference of Chinese association of automation (YAC), IEEE, pp 324–328. https://doi.org/10.1109/YAC.2016.7804912

  51. Duan Z, Yang Y, Zhang K, Ni Y, Bajgain S (2018) Improved deep hybrid networks for urban traffic flow prediction using trajectory data. IEEE Access 6:31820–31827

    Google Scholar 

  52. Ye Q, Szeto WY, Wong SC (2012) Short-term traffic speed forecasting based on data recorded at irregular intervals. IEEE Trans Intell Transp Syst 13:1727–1737

    Google Scholar 

  53. Xia D, Li H, Wang B, Li Y, Zhang Z (2016) A MapReduce-based nearest neighbor approach for big-data-driven traffic flow prediction. IEEE Access 4:2920–2934

    Google Scholar 

  54. Hong WC (2011) Traffic flow forecasting by seasonal SVR with chaotic simulated annealing algorithm. Neurocomputing 74:2096–2107

    Google Scholar 

  55. Huang ML (2015) Intersection traffic flow forecasting based on v-GSVR with a new hybrid evolutionary algorithm. Neurocomputing 147:343–349

    Google Scholar 

Download references

Acknowledgements

This work described in this paper was supported in part by the National Natural Science Foundation of China (Grant nos. 61762020, 61773321 and 61802082), the China Scholarship Council (Grant no. 201808525063), the High-level Innovative Talents Project of Guizhou (Grant no. QRLF201621), the Science and Technology Top-notch Talents Support Project of Colleges and Universities in Guizhou (Grant no. QJHKY2016065), the Science and Technology Foundation of Guizhou (Grant nos. QKHJC20161076, QKHJC20181083, QKHJC20181082 and QKHJC20191164), the National Statistical Science Research Project of China (Grant no. 2018LY66), the Science and Technology Talents Fund for Excellent Young of Guizhou (Grant no. QKHPTRC20195669), the Major Research Project of Innovative Groups in Colleges and Universities in Guizhou Province (Grant no. QJHKY2018018), and the Graduate Scientific Research Fund Project of Guizhou (Grant no. QJYHKYJJ201604). The authors would like to thank Datatang (Beijing) Technology Co., Ltd. for providing the experimental data.

Author information

Authors and Affiliations

  1. College of Data Science and Information Engineering, Guizhou Minzu University, Guiyang, 550025, China

    Dawen Xia, Maoting Zhang, Xiaobo Yan, Yu Bai & Yongling Zheng

  2. College of Computer Science, Chongqing University, Chongqing, 400044, China

    Yantao Li

  3. College of Electronic and Information Engineering, Southwest University, Chongqing, 400715, China

    Huaqing Li

Authors
  1. Dawen Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maoting Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaobo Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yongling Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yantao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Huaqing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huaqing Li.

Ethics declarations

Conflict of interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xia, D., Zhang, M., Yan, X. et al. A distributed WND-LSTM model on MapReduce for short-term traffic flow prediction. Neural Comput & Applic 33, 2393–2410 (2021). https://doi.org/10.1007/s00521-020-05076-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05076-2

Keywords

Search

Navigation