Skip to main content

Advertisement

SpringerLink
Log in

UCRLF: unified constrained reinforcement learning framework for phase-aware architectures for autonomous vehicle signaling and trajectory optimization

  • Research Paper
  • Published:
Evolutionary Intelligence Aims and scope Submit manuscript

Abstract

Signaling and trajectory optimization work as contention and researchers have debated on what should be the best for the vehicle, but it seems that both components are complement to each other and there can be combined situations with bounds where maximum optimization can be achieved. This paper introduces a novel approach called Phase-Aware Deep Learning and Constrained Reinforcement Learning for optimization and constant improvement of signal and trajectory for autonomous vehicle operation modules for an intersection. It deals with all the components required for the signaling system to operate, communicate and also navigate the vehicle with proper trajectory so that it faces less waiting time and the overall system operates with minimum waiting time and comparable throughput rate. We have done analysis on the operating time and the vehicle movement as these are vital for pollution and energy consumption. Our methodologies are not only efficient in time and computation but also have incorporated highly optimized data representation to reduce the overhead of maintaining and accessing the data. This ensures very efficient time complexity and theoretical computation time and better lower bounds. Constrained Reinforcement Learning concept is the main contribution of this work and it helped in decreasing 84% of the waiting time for the vehicles.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. El-Tantawy S, Abdulhai B (2012) Multi-agent reinforcement learning for integrated network of adaptive TRAC signal controllers (MARLIN-ATSC). In: 2012 15th international IEEE conference on intelligent transportation systems (ITSC), pp 319–326. IEEE. 5, 18, 19

  2. Au T-C et al (2012) Evasion planning for autonomous vehicles at intersections. In: 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE

  3. Khamis MA, Gomaa W (2014) Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework. Eng Appl Artif Intell 29:134–151

    Article  Google Scholar 

  4. Garavello M, Piccoli B (2006) Traffic flow on networks, vol 1. American Institute of Mathematical Sciences, Springfield

    MATH  Google Scholar 

  5. Cesme B, Furth PG (2014) Self-organizing traffic signals using secondary extension and dynamic coordination. Transp Res Part C: Emerg Technol 48:1–15

    Article  Google Scholar 

  6. Bhaskar A, Tsubota T, Chung E (2014) Urban traffic state estimation: fusing point and zone based data. Transp Res Part C: Emerg Technol 48:120–142

    Article  Google Scholar 

  7. Ahmane M et al (2013) Modeling and controlling an isolated urban intersection based on cooperative vehicles. Transp Res Part C: Emerg Technol 28:44–62

    Article  Google Scholar 

  8. Fok C-L et al (2012) A platform for evaluating autonomous intersection management policies. In: Proceedings of the 2012 IEEE/ACM third international conference on cyber-physical systems, IEEE Computer Society

  9. VanMiddlesworth M, Dresner K, Stone P (2008) Replacing the stop sign: unmanaged intersection control for autonomous vehicles. In: Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 3, International Foundation for Autonomous Agents and Multiagent Systems

  10. Dresner K, Stone P (2005) Multiagent traffic management: an improved intersection control mechanism. In: Proceedings of the fourth international joint conference on autonomous agents and multiagent systems, ACM

  11. Cascone A, D’Apice C, Piccoli B, Rarità L (2008) Circulation of car traffic in congested urban areas. Commun Math Sci 6(3):765–784

    Article  MathSciNet  MATH  Google Scholar 

  12. Duwaer DA (2016) On deep reinforcement learning for data-driven traffic control. Student thesis, Master

  13. Zolfpour-Arokhlo M et al (2014) Modeling of route planning system based on Q-value-based dynamic programming with multi-agent reinforcement learning algorithms. Eng Appl Artif Intell 29:163–177

    Article  Google Scholar 

  14. Mousavi SS, et al (2017) Traffic light control using deep policy-gradient and value-function based reinforcement learning. arXiv preprint arXiv:1704.08883

  15. Dresner K, Stone P (2008) A multiagent approach to autonomous intersection management. J Artif Intell Res 31:591–656

    Article  Google Scholar 

  16. Hausknecht M, Au T-C, Stone P (2011) Autonomous intersection management: multi-intersection optimization. In: 2011 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE

  17. D’Apice C, Manzo R, Rarità L (2011) Splitting of traffic flows to control congestion in special events. Int J Math Math Sci. https://doi.org/10.1155/2011/563171

  18. Fajardo D et al (2011) Automated intersection control: performance of future innovation versus current traffic signal control. Transp Res Rec: J Transp Res Board 2259:223–232

    Article  Google Scholar 

  19. Au T-C, Zhang S, Stone P (2015) Autonomous intersection management for semi-autonomous vehicles. The Routledge handbook of transportation. Routledge, London, pp 88–104

    Google Scholar 

  20. Carlino D, Boyles SD, Stone P (2013) Auction-based autonomous intersection management. In: 2013 16th international IEEE conference on intelligent transportation systems-(ITSC), IEEE

  21. Wu X, Liu HX (2014) Using high-resolution event-based data for traffic modeling and control: an overview. Transp Res Part C: Emerg Technol 42:28–43

    Article  Google Scholar 

  22. Daganzo C, Daganzo CF (1997) Fundamentals of transportation and traffic operations, vol 30. Pergamon, Oxford

    Book  MATH  Google Scholar 

  23. Kerner BS (2004) The physics of traffic. Springer, New York

    Book  Google Scholar 

  24. Dresner K, Stone P (2006) Human-usable and emergency vehicle-aware control policies for autonomous intersection management. In: Fourth international workshop on agents in traffic and transportation (ATT), Hakodate, Japan

  25. Gilmore JF, Elibiary KJ, Abe N (1993) Traffic management applications of neural networks. In: Working notes, AAAI-93 Workshop on AI in intelligent vehicle highway systems

  26. Fagan D, Meier R (2009) Using context and behavioral patterns for intelligent traffic management. In: Proceedings of the 1st international workshop on context-aware middleware and services: affiliated with the 4th international conference on communication system software and middleware (COMSWARE 2009), ACM

  27. McKenney D, White T (2013) Distributed and adaptive traffic signal control within a realistic traffic simulation. Eng Appl Artif Intell 26(1):574–583

    Article  Google Scholar 

  28. Mirchandani P, Head L (2001) A real-time traffic signal control system: architecture, algorithms, and analysis. Transp Res Part C: Emerg Technol 9(6):415–432

    Article  Google Scholar 

  29. Zheng X, Recker W (2013) An adaptive control algorithm for traffic-actuated signals. Transp Res Part C: Emerg Technol 30:93–115

    Article  Google Scholar 

  30. De Oliveira LB, Camponogara E (2010) Multi-agent model predictive control of signaling split in urban traffic networks. Transp Res Part C: Emerg Technol 18(1):120–139

    Article  Google Scholar 

  31. Viti F, Van Zuylen HJ (2010) A probabilistic model for traffic at actuated control signals. Transp Res Part C: Emerg Technol 18(3):299–310

    Article  Google Scholar 

  32. Guler SI, Menendez M, Meier L (2014) Using connected vehicle technology to improve the efficiency of intersections. Transp Res Part C: Emerg Technol 46:121–131

    Article  Google Scholar 

  33. Kim J, Mahmassani HS (2014) A finite mixture model of vehicle-to-vehicle and day-to-day variability of traffic network travel times. Transp Res Part C: Emerg Technol 46:83–97

    Article  Google Scholar 

  34. Eichler M, Daganzo CF (2006) Bus lanes with intermittent priority: strategy formulae and an evaluation. Transp Res Part B: Methodol 40(9):731–744

    Article  Google Scholar 

  35. Guler SI, Menendez M (2014) Analytical formulation and empirical evaluation of pre-signals for bus priority. Transp Res Part B: Methodol 64:41–53

    Article  Google Scholar 

  36. Negenborn RR, Schutter BD, Hellendoorn J (2008) Multi-agent model predictive control for transportation networks: serial versus parallel schemes. Eng Appl Artif Intell 21(3):353–366

    Article  Google Scholar 

  37. Wen W (2008) A dynamic and automatic traffic light control expert system for solving the road congestion problem. Expert Syst Appl 34(4):2370–2381

    Article  Google Scholar 

  38. Araghi S, Khosravi A, Creighton D (2015) A review on computational intelligence methods for controlling traffic signal timing. Expert Syst Appl 42(3):1538–1550

    Article  Google Scholar 

  39. Scattolini R (2009) Architectures for distributed and hierarchical model predictive control—a review. J Process Control 19(5):723–731

    Article  Google Scholar 

  40. Dotoli M, Fanti MP, Meloni C (2006) A signal timing plan formulation for urban traffic control. Control Eng Pract 14(11):1297–1311

    Article  Google Scholar 

  41. Maslekar N et al (2013) CATS: an adaptive traffic signal system based on car-to-car communication. J Netw Comput Appl 36(5):1308–1315

    Article  Google Scholar 

  42. Ma X, Jin J, Lei W (2014) Multi-criteria analysis of optimal signal plans using microscopic traffic models. Transp Res Part D: Transp Environ 32:1–14

    Article  Google Scholar 

  43. Schepperle H., Böhm K. (2008) Auction-based traffic management: towards effective concurrent utilization of road intersections. In: 2008 10th IEEE conference on E-commerce technology and the fifth IEEE conference on enterprise computing, E-Commerce and E-Services, IEEE

  44. Yan F, Dridi M, El Moudni A (2013) Autonomous vehicle sequencing problem for a multi-intersection network: a genetic algorithm approach. In: 2013 international conference on advanced logistics and transport (ICALT), IEEE

  45. Lv Y et al (2015) Traffic flow prediction with big data: a deep learning approach. IEEE Trans Intell Transp Syst 16(2):865–873

    Google Scholar 

  46. Chevion D, Shehory O, Shimony Y (2009) Automated collaboration among communicating semiautonomous vehicles. Technical Report

  47. Dever C et al (2006) Nonlinear trajectory generation for autonomous vehicles via parameterized maneuver classes. J Guid Control Dyn 29(2):289–302

    Article  Google Scholar 

  48. Paden B et al (2016) A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans Intell Veh 1(1):33–55

    Article  Google Scholar 

  49. Au T-C, Quinlan M, Stone P (2012) Setpoint scheduling for autonomous vehicle controllers. In: 2012 IEEE international conference on robotics and automation (ICRA), IEEE

  50. Mnih V, et al (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602

  51. Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–285

    Article  Google Scholar 

  52. Hu J, Wellman MP (1998) Multiagent reinforcement learning: theoretical framework and an algorithm. In: ICML, vol 98

  53. Wiering MA (2000) Multi-agent reinforcement learning for traffic light control. In: Machine learning: proceedings of the seventeenth international conference (ICML’2000)

  54. Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8(3–4):279–292

    MATH  Google Scholar 

  55. Bi Y et al (2014) Type-2 fuzzy multi-intersection traffic signal control with differential evolution optimization. Expert Syst Appl 41(16):7338–7349

    Article  Google Scholar 

  56. Galvin R (2017) Energy consumption effects of speed and acceleration in electric vehicles: laboratory case studies and implications for drivers and policymakers. Transp Res Part D: Transp Environ 53:234–248

    Article  Google Scholar 

  57. He Q, Head KL, Ding J (2014) Multi-modal traffic signal control with priority, signal actuation and coordination. Transp Res Part C: Emerg Technol 46:65–82

    Article  Google Scholar 

  58. Yu X-H, Recker WW (2006) Stochastic adaptive control model for traffic signal systems. Transp Res Part C: Emerg Technol 14(4):263–282

    Article  Google Scholar 

  59. Abdulhai B, Pringle R, Karakoulas GJ (2003) Reinforcement learning for true adaptive traffic signal control. J Transp Eng 129(3):278–285

    Article  Google Scholar 

  60. Myrvoll TA, Soong FK (2003) On divergence based clustering of normal distributions and its application to HMM adaptation. In: Eighth European conference on speech communication and technology

  61. Murphy Kevin P (2006) Naive Bayes classifiers. University of British Columbia, Vancouver

    Google Scholar 

  62. Suykens JAK, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300

    Article  Google Scholar 

  63. Liaw A, Wiener M (2002) Classification and regression by randomForest. R news 2(3):18–22

    Google Scholar 

  64. Menard S (2002) Applied logistic regression analysis, vol 106. Sage, Beverly Hills

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer and Information Science and Engineering Department, University of Florida, Gainesville, USA

    Chiranjib Sur

Authors
  1. Chiranjib Sur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chiranjib Sur.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sur, C. UCRLF: unified constrained reinforcement learning framework for phase-aware architectures for autonomous vehicle signaling and trajectory optimization. Evol. Intel. 12, 689–712 (2019). https://doi.org/10.1007/s12065-019-00278-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12065-019-00278-7

Keywords

Search

Navigation