Skip to main content
SpringerLink
Log in

An Experimental Review of Reinforcement Learning Algorithms for Adaptive Traffic Signal Control

  • Chapter
  • First Online:
Book cover Autonomic Road Transport Support Systems

Part of the book series: Autonomic Systems ((ASYS))

Abstract

Urban traffic congestion has become a serious issue, and improving the flow of traffic through cities is critical for environmental, social and economic reasons. Improvements in Adaptive Traffic Signal Control (ATSC) have a pivotal role to play in the future development of Smart Cities and in the alleviation of traffic congestion. Here we describe an autonomic method for ATSC, namely, reinforcement learning (RL). This chapter presents a comprehensive review of the applications of RL to the traffic control problem to date, along with a case study that showcases our developing multi-agent traffic control architecture. Three different RL algorithms are presented and evaluated experimentally. We also look towards the future and discuss some important challenges that still need to be addressed in this field.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Horizon 2020: http://ec.europa.eu/programmes/horizon2020/en (2014)

  2. Traas: Traci as a service. http://traas.sourceforge.net/cms/ (2014)

  3. Abdoos, M., Mozayani, N., Bazzan, A.: Traffic light control in non-stationary environments based on multi agent q-learning. In: 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2011, pp. 1580–1585 (2011). doi:10.1109/ITSC.2011.6083114

  4. Abdoos, M., Mozayani, N., Bazzan, A.: Hierarchical control of traffic signals using q-learning with tile coding. Appl. Intell. 40(2), 201–213 (2014). doi:10.1007/s10489-013-0455-3

    Article  Google Scholar 

  5. Abdulhai, B., Kattan, L.: Reinforcement learning: introduction to theory and potential for transport applications. Can. J. Civ. Eng. 30(6), 981–991 (2003). doi:10.1139/l03-014

    Article  Google Scholar 

  6. Abdulhai, B., Pringle, R., Karakoulas, G.: Reinforcement learning for true adaptive traffic signal control. J. Transp. Eng. 129(3), 278–285 (2003). doi:10.1061/(ASCE)0733-947X(2003)129:3(278)

    Article  Google Scholar 

  7. Arel, I., Liu, C., Urbanik, T., Kohls, A.: Reinforcement learning-based multi-agent system for network traffic signal control. IET Intell. Transp. Syst. 4(2), 128–135 (2010). doi:10.1049/iet-its.2009.0070

    Article  Google Scholar 

  8. Bakker, B.: Cooperative multi-agent reinforcement learning of traffic lights. In: ACM Transactions on Multimedia Computing, Communications, and Applications (2005)

    Google Scholar 

  9. Bakker, B., Whiteson, S., Kester, L., Groen, F.: Traffic light control by multiagent reinforcement learning systems. In: Babuška, R., Groen, F. (eds.) Interactive Collaborative Information Systems. Studies in Computational Intelligence, vol. 281, pp. 475–510. Springer, Berlin/Heidelberg (2010). doi:10.1007/978-3-642-11688-9_18

    Chapter  Google Scholar 

  10. Balaji, P., German, X., Srinivasan, D.: Urban traffic signal control using reinforcement learning agents. IET Intell. Transp. Syst. 4(3), 177–188 (2010). doi:10.1049/iet-its.2009.0096

    Article  Google Scholar 

  11. Bazzan, A.L.C.: Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Auton. Agent. Multi Agent Syst. 18(3), 342–375 (2009). doi:10.1007/s10458-008-9062-9

    Article  Google Scholar 

  12. Bazzan, A.L.C., Klügl, F.: A review on agent-based technology for traffic and transportation. Knowl. Eng. Rev. 29, 375–403 (2014). doi:10.1017/S0269888913000118

    Article  Google Scholar 

  13. Brys, T., Pham, T.T., Taylor, M.E.: Distributed learning and multi-objectivity in traffic light control. Connect. Sci. 26(1), 65–83 (2014). doi:10.1080/09540091.2014.885282

    Article  Google Scholar 

  14. Busoniu, L., Babuška, R., Schutter, B.: Multi-agent reinforcement learning: an overview. In: Srinivasan, D., Jain, L. (eds.) Innovations in Multi-agent Systems and Applications - 1. Studies in Computational Intelligence, vol. 310, pp. 183–221. Springer, Berlin/Heidelberg (2010). doi:10.1007/978-3-642-14435-6_7

    Chapter  Google Scholar 

  15. Chen, B., Cheng, H.: A review of the applications of agent technology in traffic and transportation systems. IEEE Trans. Intell. Transp. Syst. 11(2), 485–497 (2010). doi:10.1109/TITS.2010.2048313

    Article  Google Scholar 

  16. Chin, Y.K., Bolong, N., Yang, S.S., Teo, K.: Exploring q-learning optimization in traffic signal timing plan management. In: 2011 Third International Conference on Computational Intelligence, Communication Systems and Networks (CICSyN), pp. 269–274 (2011). doi:10.1109/CICSyN.2011.64

  17. Cools, S.B., Gershenson, C., D’Hooghe, B.: Self-organizing traffic lights: a realistic simulation. In: Prokopenko, M. (ed.) Advances in Applied Self-organizing Systems, Advanced Information and Knowledge Processing, pp. 41–50. Springer, London (2008). doi:10.1007/978-1-84628-982-8_3

    Chapter  Google Scholar 

  18. Dresner, K., Stone, P.: Multiagent traffic management: opportunities for multiagent learning. In: Tuyls, K., Hoen, P., Verbeeck, K., Sen, S. (eds.) Learning and Adaption in Multi-agent Systems. Lecture Notes in Computer Science, vol. 3898, pp. 129–138. Springer, Berlin/Heidelberg (2006). doi:10.1007/11691839_7

    Chapter  Google Scholar 

  19. El-Tantawy, S., Abdulhai, B.: An agent-based learning towards decentralized and coordinated traffic signal control. In: 13th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2010, pp. 665–670 (2010). doi:10.1109/ITSC.2010.5625066

  20. El-Tantawy, S., Abdulhai, B.: Multi-agent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc). In: 15th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2012, pp. 319–326 (2012). doi:10.1109/ITSC.2012.6338707

  21. El-Tantawy, S., Abdulhai, B., Abdelgawad, H.: Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale application on downtown Toronto. IEEE Trans. Intell. Transp. Syst. 14(3), 1140–1150 (2013). doi:10.1109/TITS.2013.2255286

    Article  Google Scholar 

  22. Houli, D., Zhiheng, L., Yi, Z.: Multiobjective reinforcement learning for traffic signal control using vehicular ad hoc network. EURASIP J. Adv. Signal Process. 2010, 7:1–7:7 (2010). doi:10.1155/2010/724035

  23. Isa, J., Kooij, J., Koppejan, R., Kuijer, L.: Reinforcement learning of traffic light controllers adapting to accidents. In: Design and Organisation of Autonomous Systems (2006)

    Google Scholar 

  24. Khamis, M., Gomaa, W.: Enhanced multiagent multi-objective reinforcement learning for urban traffic light control. In: 11th International Conference on Machine Learning and Applications (ICMLA), 2012, vol. 1, pp. 586–591 (2012). doi:10.1109/ICMLA.2012.108

  25. Khamis, M.A., Gomaa, W.: Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework. Eng. Appl. Artif. Intell. 29, 134–151 (2014). doi:10.1016/j.engappai.2014.01.007

    Article  Google Scholar 

  26. Khamis, M., Gomaa, W., El-Shishiny, H.: Multi-objective traffic light control system based on Bayesian probability interpretation. In: 15th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2012, pp. 995–1000 (2012). doi:10.1109/ITSC.2012.6338853

  27. Krajzewicz, D., Erdmann, J., Behrisch, M., Bieker, L.: Recent development and applications of SUMO - Simulation of Urban MObility. Int. J. Adv. Syst. Meas. 5(3&4), 128–138 (2012)

    Google Scholar 

  28. Kuyer, L., Whiteson, S., Bakker, B., Vlassis, N.: Multiagent reinforcement learning for urban traffic control using coordination graphs. In: Daelemans, W., Goethals, B., Morik, K. (eds.) Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Science, vol. 5211, pp. 656–671. Springer, Berlin/Heidelberg (2008). doi:10.1007/978-3-540-87479-9_61

    Chapter  Google Scholar 

  29. Liu, Z.: A survey of intelligence methods in urban traffic signal control. Int. J. Comput. Sci. Netw. Secur. 7(7), 105–112 (2007)

    Google Scholar 

  30. Lu, S., Liu, X., Dai, S.: Incremental multistep q-learning for adaptive traffic signal control based on delay minimization strategy. In: 7th World Congress on Intelligent Control and Automation, 2008. WCICA 2008, pp. 2854–2858 (2008). doi:10.1109/WCICA.2008.4593378

    Google Scholar 

  31. Nair, R., Varakantham, P., Tambe, M., Yokoo, M.: Networked distributed POMDPs: a synthesis of distributed constraint optimization and POMDPs. In: Proceedings of the 20th National Conference on Artificial Intelligence, AAAI’05, vol. 1, pp. 133–139. AAAI Press, Pittsburgh (2005)

    Google Scholar 

  32. Ono, N., Fukumoto, K.: A modular approach to multi-agent reinforcement learning. In: Weiß, G. (ed.) Distributed Artificial Intelligence Meets Machine Learning in Multi-Agent Environments. Lecture Notes in Computer Science, vol. 1221, pp. 25–39. Springer, Berlin/Heidelberg (1997). doi:10.1007/3-540-62934-3_39

    Chapter  Google Scholar 

  33. Pham, T., Brys, T., Taylor, M.E.: Learning coordinated traffic light control. In: Proceedings of the Adaptive and Learning Agents workshop (at AAMAS-13) (2013)

    Google Scholar 

  34. Prashanth, L., Bhatnagar, S.: Reinforcement learning with average cost for adaptive control of traffic lights at intersections. In: 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2011, pp. 1640–1645 (2011). doi:10.1109/ITSC.2011.6082823

  35. Prashanth, L., Bhatnagar, S.: Reinforcement learning with function approximation for traffic signal control. IEEE Trans. Intell. Transp. Syst. 12(2), 412–421 (2011). doi:10.1109/TITS.2010.2091408

    Article  Google Scholar 

  36. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st edn. Wiley, New York (1994)

    Book  MATH  Google Scholar 

  37. Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems. Tech. Rep. 166, Cambridge University Engineering Department (1994)

    Google Scholar 

  38. Salkham, A., Cahill, V.: Soilse: a decentralized approach to optimization of fluctuating urban traffic using reinforcement learning. In: 13th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2010, pp. 531–538 (2010). doi:10.1109/ITSC.2010.5625145

  39. Salkham, A., Cunningham, R., Garg, A., Cahill, V.: A collaborative reinforcement learning approach to urban traffic control optimization. In: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, 2008. WI-IAT ’08, vol. 2, pp. 560–566 (2008). doi:10.1109/WIIAT.2008.88

  40. Singh, S., Jaakkola, T., Littman, M., Szepesvári, C.: Convergence results for single-step on-policy reinforcement-learning algorithms. Mach. Learn. 38(3), 287–308 (2000). doi:10.1023/A:1007678930559

    Article  MATH  Google Scholar 

  41. Steingröver, M., Schouten, R., Peelen, S., Nijhuis, E., Bakker, B.: Reinforcement learning of traffic light controllers adapting to traffic congestion. In: Proceedings of the Belgium-Netherlands Artificial Intelligence Conference, BNAIC ’05 (2005)

    Google Scholar 

  42. Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)

    Google Scholar 

  43. Thorpe, T.L., Anderson, C.W.: Traffic light control using sarsa with three state representations. Tech. rep., IBM Corporation (1996)

    Google Scholar 

  44. Watkins, C.J.C.H.: Learning from delayed rewards. Ph.D. thesis, King’s College (1989)

    Google Scholar 

  45. Watkins, C., Dayan, P.: Technical note: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992). doi:10.1023/A:1022676722315

    MATH  Google Scholar 

  46. Webster, F.V.: Traffic signal settings. Road Research Technical Paper No. 39, Road Research Laboratory, London, published by HMSO (1958)

    Google Scholar 

  47. Wen, K., Qu, S., Zhang, Y.: A stochastic adaptive control model for isolated intersections. In: IEEE International Conference on Robotics and Biomimetics, 2007. ROBIO 2007, pp. 2256–2260 (2007). doi:10.1109/ROBIO.2007.4522521

    Google Scholar 

  48. Wiering, M.: Multi-agent reinforcement learning for traffic light control. In: Proceedings of the Seventeenth International Conference on Machine Learning, ICML ’00, pp. 1151–1158. Morgan Kaufmann, San Francisco (2000)

    Google Scholar 

  49. Wiering, M., van Otterlo, M. (eds.): Reinforcement Learning: State-of-the-Art. Springer, Heidelberg (2012)

    Google Scholar 

  50. Wiering, M., Vreeken, J., van Veenen, J., Koopman, A.: Simulation and optimization of traffic in a city. In: IEEE Intelligent Vehicles Symposium, pp. 453–458 (2004). doi:10.1109/IVS.2004.1336426

  51. Woolridge, M.: Introduction to Multiagent Systems. Wiley, New York (2001)

    Google Scholar 

  52. Xu, L.H., Xia, X.H., Luo, Q.: The study of reinforcement learning for traffic self-adaptive control under multiagent markov game environment. Math. Probl. Eng. 2013 (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National University of Ireland, Galway, Ireland

    Patrick Mannion, Jim Duggan & Enda Howley

Authors
  1. Patrick Mannion
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jim Duggan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Enda Howley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick Mannion .

Editor information

Editors and Affiliations

  1. University of Huddersfield, School of Computing and Engineering, Huddersfield, West Yorkshire, United Kingdom

    Thomas Leo McCluskey

  2. School of Engineering and Computing Scie, Durham University, Co Durham, United Kingdom

    Apostolos Kotsialos

  3. Department of Informatics, TU Clausthal, Clausthal-Zellerfeld, Niedersachsen, Germany

    Jörg P. Müller

  4. Örebro University, School of Science and Technology, Sweden

    Franziska Klügl

  5. School of Computer Science and Informati, Cardiff University, Cardiff, United Kingdom

    Omer Rana

  6. Institute of Information Systems, HES-SO Valais-Wallis, Sierre, Switzerland

    René Schumann

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Mannion, P., Duggan, J., Howley, E. (2016). An Experimental Review of Reinforcement Learning Algorithms for Adaptive Traffic Signal Control. In: McCluskey, T., Kotsialos, A., Müller, J., Klügl, F., Rana, O., Schumann, R. (eds) Autonomic Road Transport Support Systems. Autonomic Systems. Birkhäuser, Cham. https://doi.org/10.1007/978-3-319-25808-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-25808-9_4

  • Published:

  • Publisher Name: Birkhäuser, Cham

  • Print ISBN: 978-3-319-25806-5

  • Online ISBN: 978-3-319-25808-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation