Skip to main content

Advertisement

Log in

CM3-VSL: Cooperative Multi-goal Multi-stage Multi-agent VSL Traffic Control

  • Published:
International Journal of Intelligent Transportation Systems Research Aims and scope Submit manuscript

Abstract

Variable speed limit (VSL) Systems play a crucial role in proactively optimizing traffic control. As a matter of fact, many countries have deployed VSL Systems to improve road safety and resolve traffic breakdown. Most of smart VSL strategies are deployed to optimize traffic flow within a single road segment only, while real-world scenarios often involve complex bottleneck situations arising from multiple ramps. In response, we introduce a novel Cooperative Multi-goal Multi-stage Multi-agent VSL (CM3-VSL) framework where a diverse set of VSL agents collaboratively work towards both individualized local goals and shared global objectives, addressing the complexities of real-world traffic scenarios. The VSL agents are trained using micro-simulations on a real-world Moroccan highway network. Employing a cooperative strategy, each VSL agent pursues both individual and collective goals. Evaluation against a baseline no-VSL scenario and a single-agent multi-objective Reinforcement Learning VSL demonstrates that CM3-VSL achieves superior performance, contributing to advancements in intelligent traffic control systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of supporting data

Not Applicable

References

  1. Lee, C., Hellinga, B., Saccomanno, F.: Evaluation of variable speed limits to improve traffic safety. Transp. Res. C: Emerg. Technol. 14(3), 213–228 (2006). https://doi.org/10.1016/j.trc.2006.06.002

    Article  Google Scholar 

  2. Pu, Z., Li, Z., Jiang, Y., Wang, Y.: Full bayesian before-after analysis of safety effects of variable speed limit system. IEEE Trans. Intell. Transp. Syst. 22(2), 964–976 (2021). https://doi.org/10.1109/TITS.2019.2961699

    Article  Google Scholar 

  3. Zhu, F., Ukkusuri, S.V.: Accounting for dynamic speed limit control in a stochastic traffic environment: A reinforcement learning approach. Transp. Res. C: Emerg. Technol. 41, 30–47 (2014). https://doi.org/10.1016/j.trc.2014.01.014

    Article  Google Scholar 

  4. Asmae, R., Zineb, E.A.: Learning-based Variable speed limit control strategies: RL, DRL and MARL. MAIDSS’24: Proceedings of the 2024 International Conference on Modern Artificial Intelligence and Data Science Systems (2024). https://doi.org/10.1007/978-3-031-65038-3_45

  5. Kušić, K., Ivanjko, E., Gregurić, M., Miletić, M.: An overview of reinforcement learning methods for variable speed limit control. Appl. Sci. 10(14), 4917 (2020). https://doi.org/10.3390/app10144917

    Article  Google Scholar 

  6. Li, Z., Xu, C., Guo, Y., Liu, P., Pu, Z.: Reinforcement learning-based variable speed limits control to reduce crash risks near traffic oscillations on freeways. IEEE Intell. Transp. Syst. Mag. 13(4), 64–70 (2021). https://doi.org/10.1109/MITS.2019.2907631

    Article  Google Scholar 

  7. Adler, J.L., Satapathy, G., Manikonda, V., Bowles, B., Blue, V.J.: A multi-agent approach to cooperative traffic management and route guidance. Transp. Res. B Methodol. 39(4), 297–318 (2005). https://doi.org/10.1016/j.trb.2004.03.005

    Article  Google Scholar 

  8. Hamidi, H., Kamankesh, A.: An approach to intelligent traffic management system using a multi-agent system. Int. J. Intell. Transp. Syst. Res. 16(2), 112–124 (2018). https://doi.org/10.1007/s13177-017-0142-6

    Article  Google Scholar 

  9. Hirankitti, V., Krohkaew, J., Hogger, C.: A multi-agent approach for intelligent traffic-light control (2007)

  10. Ikidid, A., Abdelaziz, E.F., Sadgal, M.: Multi-agent and fuzzy inference-based framework for traffic light optimization. Int. J. Interact. Multimed. Artif. Intell. 8(2), 88 (2023). https://doi.org/10.9781/ijimai.2021.12.002

    Article  Google Scholar 

  11. Cetin, N., Nagel, K., Raney, B., Voellmy, A.: Large-scale multi-agent transportation simulations. Comput. Phys. Commun. 147(1–2), 559–564 (2002). https://doi.org/10.1016/S0010-4655(02)00353-3

    Article  Google Scholar 

  12. Tao, C., Huang, S.: An extensible multi-agent based traffic simulation system. In: 2009 International Conference on Measuring Technology and Mechatronics Automation, pp. 713–716. IEEE, Zhangjiajie, Hunan, China (2009). https://doi.org/10.1109/ICMTMA.2009.42 . http://ieeexplore.ieee.org/document/5203302/ Accessed 04 Dec 2023

  13. Buşoniu, L., Babuška, R., De Schutter, B.: Multi-agent reinforcement learning: An Overview. In: Kacprzyk, J., Srinivasan, D., Jain, L.C. (eds.) Innovations in Multi-Agent Systems and Applications - 1 vol. 310, pp. 183–221. Springer, Berlin, Heidelberg (2010)

  14. Nguyen, T.T., Nguyen, N.D., Nahavandi, S.: Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE Trans. Cybern. 50(9), 3826–3839 (2020). https://doi.org/10.1109/TCYB.2020.2977374

    Article  Google Scholar 

  15. Perrusquía, A., Yu, W., Li, X.: Multi-agent reinforcement learning for redundant robot control in task-space. Int. J. Mach. Learn. Cybern. 12(1), 231–241 (2021). https://doi.org/10.1007/s13042-020-01167-7

    Article  Google Scholar 

  16. OroojlooyJadid, A., Hajinezhad, D.: A review of cooperative multi-agent deep reinforcement learning. arXiv (2021)

  17. Sharma, P.K., Zaroukian, E.G., Fernandez, R., Basak, A., Asher, D.E.: Survey of recent multi-agent reinforcement learning algorithms utilizing centralized training. In: Pham, T., Solomon, L., Hohil, M.E. (eds.) Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, p. 84. SPIE, Online Only, United States (2021). https://doi.org/10.1117/12.2585808

  18. Wang, C., Zhang, J., Xu, L., Li, L., Ran, B.: A new solution for freeway congestion: Cooperative speed limit control using distributed reinforcement learning. IEEE Access 7, 41947–41957 (2019). https://doi.org/10.1109/ACCESS.2019.2904619

    Article  Google Scholar 

  19. Kušić, K., Ivanjko, E., Vrbanić, F., Gregurić, M., Dusparic, I.: Spatial-temporal traffic flow control on motorways using distributed multi-agent reinforcement learning. Mathematics 9(23), 3081 (2021). https://doi.org/10.3390/math9233081

    Article  Google Scholar 

  20. Zheng, S., Li, M., Ke, Z., Li, Z.: Coordinated variable speed limit control for consecutive bottlenecks on freeways using multiagent reinforcement learning. J. Adv. Transp. 2023, 1–19 (2023). https://doi.org/10.1155/2023/4419907

    Article  Google Scholar 

  21. Fang, X., Péter, T., Tettamanti, T.: Variable speed limit control for the motorway-urban merging bottlenecks using multi-agent reinforcement learning. Sustainability 15(14), 11464 (2023). https://doi.org/10.3390/su151411464

    Article  Google Scholar 

  22. Zhang, Y., Quinones-Grueiro, M., Zhang, Z., Wang, Y., Barbour, W., Biswas, G., Work, D.: MARVEL: Multi-agent reinforcement-learning for large-scale variable speed limits. arXiv (2023)

  23. Yang, J., Nakhaei, A., Isele, D., Fujimura, K., Zha, H.: CM3: Cooperative multi-goal multi-stage multi-agent reinforcement learning. arXiv (2020)

  24. Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: Machine learning proceedings 1994, pp. 157–163. Elsevier (1994). https://doi.org/10.1016/B978-1-55860-335-6.50027-1. https://linkinghub.elsevier.com/retrieve/pii/B9781558603356500271. Accessed 29 Sept 2023

  25. Mushtaq, A., Haq, I.U., Sarwar, M.A., Khan, A., Khalil, W., Mughal, M.A.: Multi-agent reinforcement learning for traffic flow management of autonomous vehicles. Sensors 23(5), 2373 (2023). https://doi.org/10.3390/s23052373. Accessed 29 Sept 2023

  26. Yang, Y., Wang, J.: An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv. arXiv:2011.00583 [cs] (2021). http://arxiv.org/abs/2011.00583. Accessed 13 Nov 2023

  27. Tan, M.: Multi-agent reinforcement learning: Independent versus cooperative agents. In: International conference on machine learning (1997). https://api.semanticscholar.org/CorpusID:266591061

  28. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv (2020)

  29. Ryu, H., Shin, H., Park, J.: Multi-agent actor-critic with generative cooperative policy network (2018). https://doi.org/10.48550/ARXIV.1810.09206

  30. Mao, H., Zhang, Z., Xiao, Z., Gong, Z.: Modelling the dynamic joint policy of teammates with attention multi-agent DDPG (2018). https://doi.org/10.48550/ARXIV.1811.07029 . Publisher: arXiv Version Number: 1

  31. Wang, R.E., Everett, M., How, J.P.: R-MADDPG for partially observable environments and limited communication (2020). https://doi.org/10.48550/ARXIV.2002.06684. Publisher: arXiv Version Number: 2

  32. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual multi-agent policy gradients. arXiv (2017)

  33. Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., Tuyls, K., Graepel, T.: Value-decomposition networks for cooperative multi-agent learning (2017). https://doi.org/10.48550/ARXIV.1706.05296

  34. Rashid, T., Samvelyan, M., Witt, C.S., Farquhar, G., Foerster, J., Whiteson, S.: QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning (2018). https://doi.org/10.48550/ARXIV.1803.11485. Publisher: arXiv Version Number: 2

  35. Macua, S.V., Tukiainen, A., Hernández, D.G.-O., Baldazo, D., Cote, E.M., Zazo, S.: Diff-DAC: Distributed actor-critic for average multitask deep reinforcement learning (2017). https://doi.org/10.48550/ARXIV.1710.10363

  36. Zhang, K., Yang, Z., Liu, H., Zhang, T., Başar, T.: Fully decentralized multi-agent reinforcement learning with networked agents (2018). https://doi.org/10.48550/ARXIV.1802.08757

  37. Foerster, J.N., Assael, Y.M., Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning (2016). https://doi.org/10.48550/ARXIV.1605.06676

  38. Jorge, E., Kågebäck, M., Johansson, F.D., Gustavsson, E.: Learning to Play Guess Who? and inventing a grounded language as a consequence (2016). https://doi.org/10.48550/ARXIV.1611.03218

  39. Le, N.-T.-T.: Multi-agent reinforcement learning for traffic congestion on one-way multi-lane highways. J. Inf. Telecommun. 7(3), 255–269 (2023). https://doi.org/10.1080/24751839.2023.2182174

    Article  Google Scholar 

  40. Calvo, J.A., Dusparic, I.: Heterogeneous multi-agent deep reinforcement learning for traffic lights control (2018)

  41. Wang, X., Ke, L., Qiao, Z., Chai, X.: Large-scale traffic signal control using a novel multiagent reinforcement learning. IEEE Trans. Cybern. 51(1), 174–187 (2021). https://doi.org/10.1109/TCYB.2020.3015811

    Article  Google Scholar 

  42. Puterman, M.L.: Markov decision processes: Discrete stochastic dynamic programming, 1st edn. Wiley series in probability and statistics. Wiley, (1994). https://doi.org/10.1002/9780470316887

  43. Asmae, R., Zineb, E.A.: Multi-objective deep reinforcement learning for variable speed limit control. In: Computing Machinery, A. (ed.) ICMLC ’24: Proceedings of the 2024 16th International Conference on Machine Learning and Computing, New York (2024). https://doi.org/10.1145/3651671.3651719

  44. Mahmud, S.M.S., Ferreira, L., Hoque, M.S., Tavassoli, A.: Application of proximal surrogate indicators for safety evaluation: A review of recent developments and research needs. IATSS Research 41(4), 153–163 (2017). https://doi.org/10.1016/j.iatssr.2017.02.001

    Article  Google Scholar 

Download references

Acknowledgements

Not Applicable

Funding

Not Applicable

Author information

Authors and Affiliations

Authors

Contributions

Not Applicable

Corresponding author

Correspondence to Asmae Rhanizar.

Ethics declarations

Ethical Approval

Not Applicable

Competing interests

Not Applicable

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rhanizar, A., El Akkaoui, Z. CM3-VSL: Cooperative Multi-goal Multi-stage Multi-agent VSL Traffic Control. Int. J. ITS Res. 22, 720–734 (2024). https://doi.org/10.1007/s13177-024-00426-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13177-024-00426-z

Keywords

Navigation