Abstract
Variable speed limit (VSL) Systems play a crucial role in proactively optimizing traffic control. As a matter of fact, many countries have deployed VSL Systems to improve road safety and resolve traffic breakdown. Most of smart VSL strategies are deployed to optimize traffic flow within a single road segment only, while real-world scenarios often involve complex bottleneck situations arising from multiple ramps. In response, we introduce a novel Cooperative Multi-goal Multi-stage Multi-agent VSL (CM3-VSL) framework where a diverse set of VSL agents collaboratively work towards both individualized local goals and shared global objectives, addressing the complexities of real-world traffic scenarios. The VSL agents are trained using micro-simulations on a real-world Moroccan highway network. Employing a cooperative strategy, each VSL agent pursues both individual and collective goals. Evaluation against a baseline no-VSL scenario and a single-agent multi-objective Reinforcement Learning VSL demonstrates that CM3-VSL achieves superior performance, contributing to advancements in intelligent traffic control systems.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Availability of supporting data
Not Applicable
References
Lee, C., Hellinga, B., Saccomanno, F.: Evaluation of variable speed limits to improve traffic safety. Transp. Res. C: Emerg. Technol. 14(3), 213–228 (2006). https://doi.org/10.1016/j.trc.2006.06.002
Pu, Z., Li, Z., Jiang, Y., Wang, Y.: Full bayesian before-after analysis of safety effects of variable speed limit system. IEEE Trans. Intell. Transp. Syst. 22(2), 964–976 (2021). https://doi.org/10.1109/TITS.2019.2961699
Zhu, F., Ukkusuri, S.V.: Accounting for dynamic speed limit control in a stochastic traffic environment: A reinforcement learning approach. Transp. Res. C: Emerg. Technol. 41, 30–47 (2014). https://doi.org/10.1016/j.trc.2014.01.014
Asmae, R., Zineb, E.A.: Learning-based Variable speed limit control strategies: RL, DRL and MARL. MAIDSS’24: Proceedings of the 2024 International Conference on Modern Artificial Intelligence and Data Science Systems (2024). https://doi.org/10.1007/978-3-031-65038-3_45
Kušić, K., Ivanjko, E., Gregurić, M., Miletić, M.: An overview of reinforcement learning methods for variable speed limit control. Appl. Sci. 10(14), 4917 (2020). https://doi.org/10.3390/app10144917
Li, Z., Xu, C., Guo, Y., Liu, P., Pu, Z.: Reinforcement learning-based variable speed limits control to reduce crash risks near traffic oscillations on freeways. IEEE Intell. Transp. Syst. Mag. 13(4), 64–70 (2021). https://doi.org/10.1109/MITS.2019.2907631
Adler, J.L., Satapathy, G., Manikonda, V., Bowles, B., Blue, V.J.: A multi-agent approach to cooperative traffic management and route guidance. Transp. Res. B Methodol. 39(4), 297–318 (2005). https://doi.org/10.1016/j.trb.2004.03.005
Hamidi, H., Kamankesh, A.: An approach to intelligent traffic management system using a multi-agent system. Int. J. Intell. Transp. Syst. Res. 16(2), 112–124 (2018). https://doi.org/10.1007/s13177-017-0142-6
Hirankitti, V., Krohkaew, J., Hogger, C.: A multi-agent approach for intelligent traffic-light control (2007)
Ikidid, A., Abdelaziz, E.F., Sadgal, M.: Multi-agent and fuzzy inference-based framework for traffic light optimization. Int. J. Interact. Multimed. Artif. Intell. 8(2), 88 (2023). https://doi.org/10.9781/ijimai.2021.12.002
Cetin, N., Nagel, K., Raney, B., Voellmy, A.: Large-scale multi-agent transportation simulations. Comput. Phys. Commun. 147(1–2), 559–564 (2002). https://doi.org/10.1016/S0010-4655(02)00353-3
Tao, C., Huang, S.: An extensible multi-agent based traffic simulation system. In: 2009 International Conference on Measuring Technology and Mechatronics Automation, pp. 713–716. IEEE, Zhangjiajie, Hunan, China (2009). https://doi.org/10.1109/ICMTMA.2009.42 . http://ieeexplore.ieee.org/document/5203302/ Accessed 04 Dec 2023
Buşoniu, L., Babuška, R., De Schutter, B.: Multi-agent reinforcement learning: An Overview. In: Kacprzyk, J., Srinivasan, D., Jain, L.C. (eds.) Innovations in Multi-Agent Systems and Applications - 1 vol. 310, pp. 183–221. Springer, Berlin, Heidelberg (2010)
Nguyen, T.T., Nguyen, N.D., Nahavandi, S.: Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE Trans. Cybern. 50(9), 3826–3839 (2020). https://doi.org/10.1109/TCYB.2020.2977374
Perrusquía, A., Yu, W., Li, X.: Multi-agent reinforcement learning for redundant robot control in task-space. Int. J. Mach. Learn. Cybern. 12(1), 231–241 (2021). https://doi.org/10.1007/s13042-020-01167-7
OroojlooyJadid, A., Hajinezhad, D.: A review of cooperative multi-agent deep reinforcement learning. arXiv (2021)
Sharma, P.K., Zaroukian, E.G., Fernandez, R., Basak, A., Asher, D.E.: Survey of recent multi-agent reinforcement learning algorithms utilizing centralized training. In: Pham, T., Solomon, L., Hohil, M.E. (eds.) Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, p. 84. SPIE, Online Only, United States (2021). https://doi.org/10.1117/12.2585808
Wang, C., Zhang, J., Xu, L., Li, L., Ran, B.: A new solution for freeway congestion: Cooperative speed limit control using distributed reinforcement learning. IEEE Access 7, 41947–41957 (2019). https://doi.org/10.1109/ACCESS.2019.2904619
Kušić, K., Ivanjko, E., Vrbanić, F., Gregurić, M., Dusparic, I.: Spatial-temporal traffic flow control on motorways using distributed multi-agent reinforcement learning. Mathematics 9(23), 3081 (2021). https://doi.org/10.3390/math9233081
Zheng, S., Li, M., Ke, Z., Li, Z.: Coordinated variable speed limit control for consecutive bottlenecks on freeways using multiagent reinforcement learning. J. Adv. Transp. 2023, 1–19 (2023). https://doi.org/10.1155/2023/4419907
Fang, X., Péter, T., Tettamanti, T.: Variable speed limit control for the motorway-urban merging bottlenecks using multi-agent reinforcement learning. Sustainability 15(14), 11464 (2023). https://doi.org/10.3390/su151411464
Zhang, Y., Quinones-Grueiro, M., Zhang, Z., Wang, Y., Barbour, W., Biswas, G., Work, D.: MARVEL: Multi-agent reinforcement-learning for large-scale variable speed limits. arXiv (2023)
Yang, J., Nakhaei, A., Isele, D., Fujimura, K., Zha, H.: CM3: Cooperative multi-goal multi-stage multi-agent reinforcement learning. arXiv (2020)
Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: Machine learning proceedings 1994, pp. 157–163. Elsevier (1994). https://doi.org/10.1016/B978-1-55860-335-6.50027-1. https://linkinghub.elsevier.com/retrieve/pii/B9781558603356500271. Accessed 29 Sept 2023
Mushtaq, A., Haq, I.U., Sarwar, M.A., Khan, A., Khalil, W., Mughal, M.A.: Multi-agent reinforcement learning for traffic flow management of autonomous vehicles. Sensors 23(5), 2373 (2023). https://doi.org/10.3390/s23052373. Accessed 29 Sept 2023
Yang, Y., Wang, J.: An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv. arXiv:2011.00583 [cs] (2021). http://arxiv.org/abs/2011.00583. Accessed 13 Nov 2023
Tan, M.: Multi-agent reinforcement learning: Independent versus cooperative agents. In: International conference on machine learning (1997). https://api.semanticscholar.org/CorpusID:266591061
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv (2020)
Ryu, H., Shin, H., Park, J.: Multi-agent actor-critic with generative cooperative policy network (2018). https://doi.org/10.48550/ARXIV.1810.09206
Mao, H., Zhang, Z., Xiao, Z., Gong, Z.: Modelling the dynamic joint policy of teammates with attention multi-agent DDPG (2018). https://doi.org/10.48550/ARXIV.1811.07029 . Publisher: arXiv Version Number: 1
Wang, R.E., Everett, M., How, J.P.: R-MADDPG for partially observable environments and limited communication (2020). https://doi.org/10.48550/ARXIV.2002.06684. Publisher: arXiv Version Number: 2
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual multi-agent policy gradients. arXiv (2017)
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., Tuyls, K., Graepel, T.: Value-decomposition networks for cooperative multi-agent learning (2017). https://doi.org/10.48550/ARXIV.1706.05296
Rashid, T., Samvelyan, M., Witt, C.S., Farquhar, G., Foerster, J., Whiteson, S.: QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning (2018). https://doi.org/10.48550/ARXIV.1803.11485. Publisher: arXiv Version Number: 2
Macua, S.V., Tukiainen, A., Hernández, D.G.-O., Baldazo, D., Cote, E.M., Zazo, S.: Diff-DAC: Distributed actor-critic for average multitask deep reinforcement learning (2017). https://doi.org/10.48550/ARXIV.1710.10363
Zhang, K., Yang, Z., Liu, H., Zhang, T., Başar, T.: Fully decentralized multi-agent reinforcement learning with networked agents (2018). https://doi.org/10.48550/ARXIV.1802.08757
Foerster, J.N., Assael, Y.M., Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning (2016). https://doi.org/10.48550/ARXIV.1605.06676
Jorge, E., Kågebäck, M., Johansson, F.D., Gustavsson, E.: Learning to Play Guess Who? and inventing a grounded language as a consequence (2016). https://doi.org/10.48550/ARXIV.1611.03218
Le, N.-T.-T.: Multi-agent reinforcement learning for traffic congestion on one-way multi-lane highways. J. Inf. Telecommun. 7(3), 255–269 (2023). https://doi.org/10.1080/24751839.2023.2182174
Calvo, J.A., Dusparic, I.: Heterogeneous multi-agent deep reinforcement learning for traffic lights control (2018)
Wang, X., Ke, L., Qiao, Z., Chai, X.: Large-scale traffic signal control using a novel multiagent reinforcement learning. IEEE Trans. Cybern. 51(1), 174–187 (2021). https://doi.org/10.1109/TCYB.2020.3015811
Puterman, M.L.: Markov decision processes: Discrete stochastic dynamic programming, 1st edn. Wiley series in probability and statistics. Wiley, (1994). https://doi.org/10.1002/9780470316887
Asmae, R., Zineb, E.A.: Multi-objective deep reinforcement learning for variable speed limit control. In: Computing Machinery, A. (ed.) ICMLC ’24: Proceedings of the 2024 16th International Conference on Machine Learning and Computing, New York (2024). https://doi.org/10.1145/3651671.3651719
Mahmud, S.M.S., Ferreira, L., Hoque, M.S., Tavassoli, A.: Application of proximal surrogate indicators for safety evaluation: A review of recent developments and research needs. IATSS Research 41(4), 153–163 (2017). https://doi.org/10.1016/j.iatssr.2017.02.001
Acknowledgements
Not Applicable
Funding
Not Applicable
Author information
Authors and Affiliations
Contributions
Not Applicable
Corresponding author
Ethics declarations
Ethical Approval
Not Applicable
Competing interests
Not Applicable
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rhanizar, A., El Akkaoui, Z. CM3-VSL: Cooperative Multi-goal Multi-stage Multi-agent VSL Traffic Control. Int. J. ITS Res. 22, 720–734 (2024). https://doi.org/10.1007/s13177-024-00426-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13177-024-00426-z