Skip to main content
Log in

Short-term traffic flow forecasting: parametric and nonparametric approaches via emotional temporal difference learning

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Information signal from real case and natural complex dynamical systems such as traffic flow are usually specified by irregular motions. Chaotic nonlinear dynamics approach is now the most powerful tool for scientists to deal with complexities in real cases, and neural networks and neuro-fuzzy models are widely used for their capabilities in nonlinear modeling of chaotic systems more than the traditional methods. As mentioned, the traffic flow conditions caused the forecasting values of traffic flow to lack robustness and accuracy. In this paper, the traffic flow forecasting is analyzed with emotional concepts and multi-agent systems (MASs) points of view as a new method in this field. The findings enabled the researchers to develop a newly object-oriented method of forecasting traffic flow. Its architecture is based on a temporal difference (TD) Q-learning with a neuro-fuzzy structure, which is the nonparametric approach. The performance of TD Q-learning is improved by emotional learning. The proposed method on the present conditions and the action of the system according to the criteria could forecast traffic signals so that the objectives are reached in minimum time. The ability of presented learning algorithm to prospect gains from future actions and obtain rewards from its past experiences allows emotional TD Q-learning algorithm to improve its decisions for the best possible actions. In addition, to study in a more practical situation, the neuro-fuzzy behaviors could be modeled by MAS. The proposed method (intelligent/nonparametric approach) is compared by parametric approach, autoregressive integrated moving average (ARIMA) method, which is implemented by multi-layer perceptron neural networks and called ARIMANN. Here, the ARIMANN is updated by backpropagation and temporal difference backpropagation for the first time. The simulation results revealed that the studied forecaster could discover the optimal forecasting by means of the Q-learning algorithm. Difficult to handle through parametric and classic methods, the real traffic flow signals used for fitting the algorithms is obtained from a two-lane street I-494 in Minnesota City.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. e i is an error signal between the forecasted signal and desired value; m is the number of each section (in this paper m = 3); c i is the critic signal; r i is the actual value of reward signal for every section; \( \hat{r}_{i} \) is the forecasted value of reward signal for every section, and \( \hat{r} \) is the forecasted value of reward signal for destination section. As it is shown, the emotional multi-critics are composed of the following functional elements:

    Critics: providing the emotional signals according to the error situation (This error may increase in peaks which are caused by such a non-recurring traffic incidents).

    Learning procedure: modeling the reinforcement signals that are eventually be obtained from the system for the current action determined by neuro-fuzzy forecaster.

    Fuzzy network: making decision for weights, \( w_{1} ,w_{2} , \ldots ,w_{m} \), for the m internal reinforcement signals according to m station status signals, \( e_{1} ,e_{2} , \ldots ,e_{m} \) in every iteration.

    Normalization unit: putting together the internal reinforcement signals decided by the TDs learning and assigning an appropriate reinforcement signal at each time-step. Consequently, the normalization unit can be combined with internal reinforcement signals at once.

References

  1. Abdi J, Fatourechi M, Lucas C, Sedigh AK, Famil Khalili GH (2004) Control of multivariable systems based on emotional temporal difference learning controller. Int J Eng Natl Centre Sci Res 17(4):363–376

    Google Scholar 

  2. Abdi J, Rashidi F, Lucas C, Sedigh AK (2005) The role of context in emotional temporal difference learning and its applications in control. J Sci Technol Sharif Univ Technol 30:13–21

    Google Scholar 

  3. Abdulhai B, Porwal H, Recker W (2002) Short term traffic-flow forecasting using neuro-genetic algorithms. J Intell Transp Syst Technol Plann Oper 7(1):3–41

    MATH  Google Scholar 

  4. Ahmed MS, Cook AR (1979) Analysis of freeway traffic time-series data by using Box-Jenkins techniques. Transp Res Rec 722:1–9

    Google Scholar 

  5. Al-Saba T, El-Amin I (1999) Artificial neural networks as applied to long-term demand forecasting. Artif Intell Eng 13:189–197

    Article  Google Scholar 

  6. Balkenius C, Moren J (2000) A computational model of context processing. In: Meyer JA, Berthoz A, Floreano D, Roitblat HL, Wilson SW (eds) From animals to animats 6: proceedings of the 6th international conference on the simulation of adaptive behaviour. MIT Press, Cambridge

    Google Scholar 

  7. Berenji HR, Khedkar P (1992) Learning and tuning fuzzy logic controller through reinforcements. IEEE Trans Neural Netw 3:724–740

    Article  Google Scholar 

  8. Box GEP, Jenkins GM (1976) Time series analysis: forecasting and control, 2nd edn. Holden Day, San Francisco

    MATH  Google Scholar 

  9. Cao L, Hong Y, Fang H, He G (1995) Predicting chaotic time series with wavelet networks. Physica D: Nonlinear Phenom 85(1–2):225–238

    Google Scholar 

  10. Cetin M, Comert G (2006) Short-term traffic flow prediction with regime switching models. Transp Res Rec 1965:23–31

    Article  Google Scholar 

  11. Chen HB, Grant-Muller S (2001) Use of sequential learning for short term traffic-flow predicting. Transp Res Part C 9(5):319–336

    Article  Google Scholar 

  12. Cheng CT, Ou CP, Chau KW (2002) Combining a fuzzy optimal model with a genetic algorithm to solve multiobjective rainfall-runoff model calibration. J Hydrol 268(1–4):72–86

    Article  Google Scholar 

  13. Denton JW (1995) How good are neural networks for causal forecasting. J Bus Forecast Methods Syst 14(2):17–20

    MathSciNet  Google Scholar 

  14. Dong H, Jia L, Sun X, Li C, Qin Y (2009) Road traffic-flow forecasting with a time-oriented ARIMA Model. In: Proceedings of the fifth international joint conference on INC, IMS and IDC, 1649–1652

  15. Faruk DO (2010) A hybrid neural network and ARIMA model for water quality time series prediction. Eng Appl Artif Intell 23:586–594

    Article  Google Scholar 

  16. Fatourechi M, Lucas C, Sedigh AK (2001) Reducing control effort by means of emotional learning. In: Proceedings of the 9th Iranian conference on electrical engineering, Iran, 41-1 to 41-8

  17. Fishman GS (1971) Estimating sample size in computing simulation experiments. Manage Sci 18(1):21–38

    Google Scholar 

  18. Gholipour A, Abbaspour A, Lucas C, Araabi BN, Fatourechi M (2003) Enhancing the performance of neuro-fuzzy predictors by emotional learning algorithm. Informatica 27(2):165–174

    Google Scholar 

  19. http://www.d.umn.edu/~tkwon/TDRLSoftware/TravelTimeMetroTC.msi (online)

  20. Hwang HP (2001) Insights into neural-network forecasting time series corresponding to ARMA (p; q) structures. Omega 29:273–289

    Article  Google Scholar 

  21. Hwang JN, Moon S (1991) Temporal difference method for multi-step prediction: application to power load forecasting. First International Forum on Applications of Neural Networks to Power Systems, USA, pp 41–45

    Google Scholar 

  22. Jaakkola T, Jordan MI, Singh SP (1994) On the convergence of stochastic iterative dynamic programming algorithms. Neural Comput 6:1185–1201

    Google Scholar 

  23. Jiang X, Adeli H (2005) Dynamic wavelet neural network model for traffic flow forecasting. J Transp Eng 131(10):771–779

    Article  Google Scholar 

  24. Kaastra I, Boyd M (1996) Designing a neural network for forecasting financial and economic time series. Neurocomputing 10(3):215–236

    Article  Google Scholar 

  25. Karlaftis MG, Vlahogianni EI (2009) Memory properties and fractional integration in transportation time-series. Transp Res Part C Emerg Technol 17(4):444–453

    Article  Google Scholar 

  26. Kentridge RW, Aggleton JP (1990) Emotion: sensory representations, reinforcement and the temporal lobe. Cogn Emot 4:191–208

    Article  Google Scholar 

  27. Li B, De Moor B (2002) Identification of influential observations on total least squares estimate. Linear Algebra Appl 348:23–39

    Article  MathSciNet  MATH  Google Scholar 

  28. Lin CT, Lee CSG (1991) Neural-network based fuzzy logic control and decision system. IEEE Trans Comput 40(12):1320–1327

    Article  MathSciNet  Google Scholar 

  29. Maes P (1991) Designing autonomous agents: theory and practice from biology to engineering and back. The MIT press, London

    Google Scholar 

  30. Markham IS, Rakes TR (1998) The effect of sample size and variability of data on the comparative performance of artificial neural networks and regression. Comput Oper Res 25:251–263

    Article  MATH  Google Scholar 

  31. Messer CJ (1993) Advanced freeway system ramp metering strategies for Texas. TTI Research Report 1232–23. Texas Transportation Institute, College Station

  32. Mitchell TM (1997) Machine learning. McGraw-Hill, New York

    MATH  Google Scholar 

  33. Okutani I, Stephanedes YJ (1984) Dynamic forecasting of traffic volume through Kalman filtering theory. Transp Res Part B 18(1):1–11

    Article  Google Scholar 

  34. Ong CS, Huang JJ, Tzeng GH (2005) Model identification of ARIMA family using genetic algorithms. Appl Math Comput 164(3):885–912

    Article  MathSciNet  MATH  Google Scholar 

  35. Pai PF, Lin CS (2005) A hybrid ARIMA and support vector machines model in stock price forecasting. Omega 33(6):497–505

    Article  Google Scholar 

  36. Park B (2002) Hybrid neuro-fuzzy application in short term freeway traffic volume predicting. Transp Res Rec 1802:190–196

    Article  Google Scholar 

  37. Rashidi F, Rashidi M, Hashemi-Hosseini A (2003) Emotional temporal difference learning based intelligent controller. In: Proceedings of the IEEE conference on control applications, 1:341–346

  38. Roger J (1993) ANFIS: adaptive-network-based fuzzy inference system. IEEE Trans Syst Man Cybern 23(3):665–685

    Article  Google Scholar 

  39. Rojas I, Valenzuela O, Rojas F, Guillen A, Herrera LJ, Pomares H, Marquez L, Pasadas M (2008) Soft-computing techniques and ARMA model for time series prediction. Neurocomputing 71:519–537

    Article  Google Scholar 

  40. Rolls ED (1998) The brain and emotion. Oxford University Press, Oxford

  41. Rummery GA, Niranjan M (1994) On-line q-learning using connectionist systems. Cambridge University Engineering Department Technical Report CUED/F-INFENG/TR 166

  42. Russel S, Norwig P (1995) A modern approach to artificial intelligence. Prentice-Hall, Englewood Cliffs

    Google Scholar 

  43. Russel S, Norwig P (2003) Artificial intelligence—a modern approach, 2nd edn. Pearson Education, Inc., New Jersey

    Google Scholar 

  44. SAS 9 (2004) Equations for the smoothing models, Jan. 2004. http://support.sas.com/91doc/getDoc/hpfug.hlp/hpfdet_sect7.htm

  45. Schriber TJ, Andrews RW (1984) ARMA-based confidence intervals for simulation output analysis. Am J Math Manage Sci 4(3–4):345–373

    MATH  Google Scholar 

  46. Sharda R, Patil RB (1990) Neural networks as forecasting experts: an empirical test. In: Proceedings of the International joint conference on neural networks, Washington, DC, 2:491–494

  47. Smith BL, Demetsky MJ (1994) Short-term traffic-flow prediction: neural network approach. Transp Res Rec 1453:98–104

    Google Scholar 

  48. Smith BL, Demetsky MJ (1997) Traffic-flow predicting: comparison of modeling approaches. J Trans Eng 123(4):261–266

    Article  Google Scholar 

  49. Smith BL, Williams BM, Oswald RK (2002) Comparison of parametric and nonparametric models for traffic-flow forecasting. Transp Res Part C 10:303–321

    Article  Google Scholar 

  50. Stathopoulos A, Karlaftis MG (2003) A multivariate state space approach for urban traffic flow modeling and prediction. Transp Res Part C Emerg Technol 11(2):121–135

    Article  Google Scholar 

  51. Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44

    Google Scholar 

  52. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. The MIT Press, England

    Google Scholar 

  53. Tan G, Deng Q, Tian Z, Yang J (2007) A Dish Parallel BP for Traffic-flow Forecasting. In: Proceedings of the international conference on computational intelligence and security, pp 546–549

  54. Tan MC, Feng LB, Xu JM (2007) Traffic flow prediction based on hybrid ARIMA and ANN model. Chin J Highway Transp 20(4):118–121

    Google Scholar 

  55. Tan G, Shi H, Wang F, Deng C (2009) Short-term traffic-flow forecasting based on parallel quasi-newton neural network. In: International conference on measuring technology and mechatronics automation, ICMTMA’09 3:305–308

  56. Tan MC, Wong SC, Xu JM, Guan ZR, Zhang P (2009) An aggregation approach to short-term traffic flow prediction. IEEE Trans Intell Transp Syst 10(1):60–69

    Article  Google Scholar 

  57. Tang Z, Fishwick PA (1993) Feedforward neural nets as models for time series forecasting. ORSA J Comput 5:374–385

    Article  MATH  Google Scholar 

  58. Tang Z, Almeida C, Fishwick PA (1991) Time series forecasting using neural networks vs Box–Jenkins methodology. Simulation 57:303–310

    Article  Google Scholar 

  59. Thomakos DD, Guerard JB, Naïve J (2004) ARIMA, nonparametric, transfer function and VAR models: a comparison of forecasting performance. Int J Forecast 20(1):53–67

    Article  Google Scholar 

  60. Valenzuela O, Rojas I, Rojas F, Pomares H, Herrera LJ, Guillen A, Marquez L, Pasadas M (2008) Hybridization of intelligent techniques and ARIMA models for time series prediction. Fuzzy Sets Syst 159:821–845

    Article  MathSciNet  MATH  Google Scholar 

  61. Van Der Voort M, Dougherty M, Watson S (1996) Combining Kohonen maps with ARIMA time series models to forecast traffic-flow. Transp Res Part C 4(5):307–318

    Article  Google Scholar 

  62. Vlahogianni EI (2009) Enhancing predictions in signalized arterials with information on short-term traffic flow dynamics. J Intell Transp Syst 13(2):73–84

    Article  Google Scholar 

  63. Vlahogianni EI, Karlaftis MG (2010) Advanced computational approaches for predicting tourist arrivals: the case of charter air-travel. In: Todd Evans (ed) Nonlinear dynamics, ISBN: 978-953-7619-61-9, Chap. 13, pp 309–324, INTECH, Croatia

  64. Vlahogianni EI, Golias JC, Karlaftis MG (2004) Short-term traffic forecasting: overview of objectives and methods. Transp Rev 24(5):533–557

    Article  Google Scholar 

  65. Vlahogianni EI, Karlaftis MG, Golias JC (2005) Optimized and meta-optimized neural networks for short-term traffic flow prediction: a genetic approach. Transp Res Part C 13:211–234

    Article  Google Scholar 

  66. Vlahogianni EI, Karlaftis MG, Golias JC (2006) Statistical methods for detecting nonlinearity and non-stationarity in univariate short-term time-series of traffic volume. Transp Res Part C Emerg Technol 14(5):351–367

    Article  Google Scholar 

  67. Vlahogianni EI, Karlaftis MG, Golias JC (2007) Spatio-temporal urban traffic volume forecasting using genetically-optimized modular networks. Computer-aided Civil Infrastruct Eng 22(5):317–325

    Article  Google Scholar 

  68. Vlahogianni EI, Karlaftis MG, Golias JC (2008) Temporal evolution of short-term urban traffic flow: a nonlinear dynamics approach. Computer-Aided Civil Infrastruct Eng 22(5):317–325

    Article  Google Scholar 

  69. Wang WC, Chau KW, Cheng CT, Qiu L (2009) A comparison of performance of several artificial intelligence methods for forecasting monthly discharge time series. J Hydrol 374:294–306

    Article  Google Scholar 

  70. Watkins PD (1992) Technical note Q-learning. Mach Learn 8:279–292

    MATH  Google Scholar 

  71. Wedding DK, Cios KJ (1996) Time series forecasting by combining RBF networks, certainty factors, and the Box–Jenkins model. Neurocomputing 10:149–168

    Article  MATH  Google Scholar 

  72. Wei CH, Lee Y (2007) Sequential forecast of incident duration using Artificial Neural Network models. J Accident Anal Prev 39:944–954

    Article  Google Scholar 

  73. Wooldridge M, Jennings N (1995) Intelligent agents: theory and practice. Knowl Eng Rev J 10(2):115–152

    Google Scholar 

  74. Xie JX, Cheng CT, Chau KW, Pei YZ (2006) A hybrid adaptive time-delay neural network model for multi-step-ahead prediction of sunspot activity. Int J Environ Pollut 28(3–4):364–381

    Article  Google Scholar 

  75. Zhang GP (2003) Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing 50:159–175

    Article  MATH  Google Scholar 

  76. Zhang X, Rice J (2001) Short-term travel time prediction using a time-varying coefficient linear model. Transp Res Part C 11:187–210

    Article  Google Scholar 

  77. Zhang Y, Ye Z (2008) Short term traffic-flow forecasting using fuzzy logic system methods. J Intell Transp Syst 12(3):102–112

    Article  MATH  Google Scholar 

  78. Zhang GP, Patuwo EB, Hu MY (2001) A simulation study of artificial neural networks for nonlinear time-series forecasting. Comput Oper Res 28:381–396

    Article  MATH  Google Scholar 

  79. Zhong M, Sharma S, Lingras P (2005) Short-term traffic prediction on different types of roads with genetically designed regression and time delay neural network models. J Comput Civil Eng 19(1):94–103

    Article  Google Scholar 

Download references

Acknowledgments

The author would like to express our sincere appreciations to Professor Caro Lucas for his invaluable contributions to the overall analysis of emotional learning theory in this project. We also would like to thank three anonymous referees for their helpful comments that greatly enhanced the article’s quality.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Javad Abdi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Abdi, J., Moshiri, B., Abdulhai, B. et al. Short-term traffic flow forecasting: parametric and nonparametric approaches via emotional temporal difference learning. Neural Comput & Applic 23, 141–159 (2013). https://doi.org/10.1007/s00521-012-0977-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-012-0977-3

Keywords

Navigation