Abstract
In multiple intersection environment, it is difficult to require all agents to collectively make a globally optimal decision, for which the dimension of action space increases exponentially with the number of agents. This paper is based on decentralized multi-agent deep reinforcement learning (decentralized MADRL), overcomes the scalability issue by distributing the global control to each local RL agent. In decentralized MADRL, the communication range between agents affects the performance and communication cost of the whole system. The adaptive subgraph reformulation algorithm is proposed to clarify the information acquisition range of agents. Our MADRL algorithm solves the trade-off problem of information interaction between various agents in large-scale urban traffic network. Each agent updates the decision based on local information and regional information extracted by subgraph reformulation to realize collaborative control of traffic signals. Through detailed simulation experiments, our algorithm comprehensively outperforms other RL algorithms when facing the real-world dataset with large number of intersections and complex road connections.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Nguyen, T.T., Nguyen, N.D., Nahavandi, S.: Deep reinforcement learning for multiagent systems: a review of challenges, solutions, and applications. IEEE Trans. Cybern. 99–113 (2020)
Wu, J.Q., Song, X.G.: Review on smart highways critical technology. J. Shandong Univ. (Eng. Sci.) 50(4), 52–69 (2020)
Alan, J.M.: Settings for fixed-cycle traffic signals. J. Oper. Res. Soc. 14(4), 373–386 (1963)
Webster, F.V.: Traffic signal settings. Road Research Technical Paper, 39 (1958)
Li, Z.L.: A differential game modeling approach to dynamic traffic assignment and traffic signal control. In: IEEE International Conference on Systems, Man, and Cybernetics. IEEE (2003)
Ceylan, H., Bell, M.G.: Traffic signal timing optimisation based on genetic algorithm approach, including drivers’ routing. Transport. Res. Part B-Methodol. 38(4), 1135–1145 (2004)
Wei, Y., Shao, Q., Hang, Y., et al.: Intersection signal control approach based on PSO and simulation. In: Proceedings of the 2008 Second International Conference on Genetic and Evolutionary Computing. IEEE, Jinzhou (2008)
Oroojlooy, A., Nazari, M., Hajinezhad, D., et al.: Attendlight: universal attention-based reinforcement learning model for trafficsignal control. In: Advances in Neural Information Processing Systems. NeurIPS, Online (2020)
Zheng, G., Xiong, Y., Zang, X., et al.: Learning phase competition for traffic signal control. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1963–1972. CIKM, Beijing (2019)
Zang, X., Yao, H., Zheng, G., et al.: Metalight: value-based meta-reinforcement learning for traffic signal control. In: AAAI Conference on Artificial Intelligence, pp. 1153–1160. AAAI, New York (2020)
Chen, C., Wei, H., Xu, N., et al.: Toward a thousand lights: decentralized deep reinforcement learning for large-scale traffic signal control. In: AAAI Conference on Artificial Intelligence, pp. 3414–3421. AAAI, New York (2020)
Yu, Z., Liang, S., Wei, L., et al.: Macar: urban traffic light control via active multi-agent communication and action rectification. In: International Joint Conference on Artificial Intelligence, pp. 2491–2497. IJCAI, Yokahama (2020)
Wang, X., Ke, L., Qiao, Z., et al.: Large-scale traffic signal control using a novel multiagent reinforcement learning. IEEE Trans. Cybern. 51, 174–187 (2020)
Aslani, M., Mesgari, M.S., Wiering, M.: Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events. Transport. Res. Part C: Emerg. Technol. 85, 732–752 (2017)
Das, S., Maurya, A.K.: Time headway analysis for four-lane and two-lane roads. Transport. Dev. Econ. 3, 1–18 (2017)
Wang, X., Yin, Y., Feng, Y., et al.: Learning the max pressure control for urban traffic networks considering the phase switching loss. Transport. Res. Part C: Emerg. Technol. 140, 103670 (2022)
Chu, T., Wang, J., Codecà, L., et al.: Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Trans. Intell. Transp. Syst. 21(3), 1086–1095 (2019)
Paczolay, G., Harmati, I.: A new advantage actor-critic algorithm for multi-agent environments. In: 23rd International Symposium on Measurement and Control in Robotics. ISMCR, Budapest (2020)
Devailly, F.X., Larocque, D., Charlin, L.: IG-RL: inductive graph reinforcement learning for massive-scale traffic signal control. IEEE Trans. Intell. Transp. Syst. 23(7), 7496–7507 (2021)
Acknowledgment
This research was funded by the Natural Science Foundation of Shandong Province for Key Project under Grant ZR2020KF006, the National Natural Science Foundation of China under Grant 62273164, the Development Program Project of Youth Innovation Team of Institutions of Higher Learning in Shandong Province.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Gong, K., Sun, Q., Zhong, X., Zhang, Y. (2023). Large-Scale Traffic Signal Control Based on Integration of Adaptive Subgraph Reformulation and Multi-agent Deep Reinforcement Learning. In: Huang, DS., Premaratne, P., Jin, B., Qu, B., Jo, KH., Hussain, A. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2023. Lecture Notes in Computer Science, vol 14086. Springer, Singapore. https://doi.org/10.1007/978-981-99-4755-3_65
Download citation
DOI: https://doi.org/10.1007/978-981-99-4755-3_65
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-4754-6
Online ISBN: 978-981-99-4755-3
eBook Packages: Computer ScienceComputer Science (R0)