Abstract
Traffic signal control is shifting from passive control to proactive control, which enables the controller to direct current traffic flow to reach its expected destinations. To this end, an effective prediction model is needed for signal controllers. What to predict, how to predict, and how to leverage the prediction for control policy optimization are critical problems for proactive traffic signal control. In this paper, we use an image that contains vehicle positions to describe intersection traffic states. Then, inspired by a model-based reinforcement learning method, DreamerV2, we introduce a novel learning-based traffic world model. The traffic world model that describes traffic dynamics in image form is used as an abstract alternative to the traffic environment to generate multi-step planning data for control policy optimization. In the execution phase, the optimized traffic controller directly outputs actions in real time based on abstract representations of traffic states, and the world model can also predict the impact of different control behaviors on future traffic conditions. Experimental results indicate that the traffic world model enables the optimized real-time control policy to outperform common baselines, and the model achieves accurate image-based prediction, showing promising applications in futuristic traffic signal control.
摘要
交通信号控制正从被动控制过渡到主动控制，以引导当前交通流按预期状态运行。一个有效的预测模型对主动交通信号控制至关重要；其中预测什么交通状态，如何高精度预测，以及如何利用预测优化控制策略是主动交通信号控制研究的关键问题。本文使用车辆位置图像描述路口交通状态，同时受基于模型的强化学习方法DreamerV2的启发，引入基于学习的交通世界模型。该世界模型以图像序列描述交通动态，并作为交通环境的抽象替代以生成多步预测样本用于控制策略优化。在执行阶段，优化后的交通信号控制器根据交通状态的抽象表示直接实时输出控制指令，同时世界模型能够预测不同控制行为对未来交通状态的影响。实验结果表明，基于交通世界模型优化的控制策略的性能优于一般基准，并且世界模型实现了基于图像的高精度预测；这些结果显示了世界模型在未来交通信号控制中的应用前景。
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abdoos M, Bazzan ALC, 2021. Hierarchical traffic signal optimization using reinforcement learning and traffic prediction with long-short term memory. Expert Syst Appl, 171:114580. https://doi.org/10.1016/j.eswa.2021.114580
Bertsekas D, 2021. Multiagent reinforcement learning: roll-out and policy iteration. IEEE/CAA J Autom Sin, 8(2):249–272. https://doi.org/10.1109/JAS.2021.1003814
Dai XY, Fu R, Zhao EM, et al., 2019. DeepTrend 2.0: a light-weighted multi-scale traffic prediction model using detrending. Transp Res Part C Emerg Technol, 103:142–157. https://doi.org/10.1016/j.trc.2019.03.022
Guo QQ, Li L, Ban XG, 2019. Urban traffic signal control with connected and automated vehicles: a survey. Transp Res Part C Emerg Technol, 101:313–334. https://doi.org/10.1016/j.trc.2019.01.026
Hafner D, Lillicrap T, Fischer I, et al., 2019. Learning latent dynamics for planning from pixels. Proc 36th Int Conf on Machine Learning, p.2555–2565.
Hafner D, Lillicrap TP, Norouzi M, et al., 2022. Mastering Atari with discrete world models. https://arxiv.org/abs/2010.02193
Hao ZZ, Boel R, Li ZW, 2018. Model based urban traffic control, part I: local model and local model predictive controllers. Transp Res Part C Emerg Technol, 97:61–81. https://doi.org/10.1016/j.trc.2018.09.026
Jin JC, Guo HF, Xu J, et al., 2021. An end-to-end recommendation system for urban traffic controls and management under a parallel learning framework. IEEE Trans Intell Transp Syst, 22(3):1616–1626. https://doi.org/10.1109/TITS.2020.2973736
Kim D, Jeong O, 2019. Cooperative traffic signal control with traffic flow prediction in multi-intersection. Sensors, 20(1):137. https://doi.org/10.3390/s20010137
Li L, Lv YS, Wang FY, 2016. Traffic signal timing via deep reinforcement learning. IEEE/CAA J Autom Sin, 3(3):247–254. https://doi.org/10.1109/JAS.2016.7508798
Li L, Lin YL, Zheng NN, et al., 2017. Parallel learning: a perspective and a framework. IEEE/CAA J Autom Sin, 4(3):389–395. https://doi.org/10.1109/JAS.2017.7510493
Li ZS, Xiong G, Tian YL, et al., 2022. A multi-stream feature fusion approach for traffic prediction. IEEE Trans Intell Transp Syst, 23(2):1456–1466. https://doi.org/10.1109/TITS.2020.3026836
Liang XY, Du XS, Wang GL, et al., 2019. A deep reinforcement learning network for traffic light cycle control. IEEE Trans Veh Technol, 68(2):1243–1253. https://doi.org/10.1109/TVT.2018.2890726
Liu CH, Zhu F, Liu Q, et al., 2021. Hierarchical reinforcement learning with automatic sub-goal identification. IEEE/CAA J Autom Sin, 8(10):1686–1696. https://doi.org/10.1109/JAS.2021.1004141
Lopez PA, Behrisch M, Bieker-Walz L, et al., 2018. Microscopic traffic simulation using SUMO. Proc 21st IEEE Int Conf on Intelligent Transportation Systems, p.2575–2582. https://doi.org/10.1109/ITSC.2018.8569938
Lv YS, Duan YJ, Kang WW, et al., 2014. Traffic flow prediction with big data: a deep learning approach. IEEE Trans Intell Transp Syst, 16(2):865–873. https://doi.org/10.1109/TITS.2014.2345663
Mao F, Li ZH, Li L, 2022. A comparison of deep reinforcement learning models for isolated traffic signal control. IEEE Intell Transp Syst Mag, early access. https://doi.org/10.1109/MITS.2022.3144797
Mei ZY, Tan Z, Zhang W, et al., 2019. Simulation analysis of traffic signal control and transit signal priority strategies under arterial coordination conditions. Simulation, 95(1):51–64. https://doi.org/10.1177/0037549718757651
Mnih V, Kavukcuoglu K, Silver D, et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533. https://doi.org/10.1038/nature14236
Newell GF, 1969. Properties of vehicle-actuated signals: I. one-way streets. Transp Sci, 3(1):30–52.
Nie J, Yan J, Yin HL, et al., 2021. A multimodality fusion deep neural network and safety test strategy for intelligent vehicles. IEEE Trans Intell Veh, 6(2):310–322. https://doi.org/10.1109/TIV.2020.3027319
Seng D, Lv FS, Liang ZY, et al., 2021. Forecasting traffic flows in irregular regions with multi-graph convolutional network and gated recurrent unit. Front Inform Technol Electron Eng, 22(9):1179–1193. https://doi.org/10.1631/FITEE.2000243
Sutton RS, Barto AG, 2018. Reinforcement Learning: an Introduction (2nd Ed.). The MIT Press, Cambridge, USA.
Varaiya P, 2013. Max pressure control of a network of signalized intersections. Transp Res Part C Emerg Technol, 36:177–195. https://doi.org/10.1016/j.trc.2013.08.014
Wang FY, 2010. Parallel control and management for intelligent transportation systems: concepts, architectures, and applications. IEEE Trans Intell Transp Syst, 11(3):630–638. https://doi.org/10.1109/TITS.2010.2060218
Wang HN, Liu N, Zhang YY, et al., 2020. Deep reinforcement learning: a survey. Front Inform Technol Electron Eng, 21(12):1726–1744. https://doi.org/10.1631/FITEE.1900533
Wang J, Li R, Wang J, et al., 2020. Artificial intelligence and wireless communications. Front Inform Technol Electron Eng, 21(10):1413–1425. https://doi.org/10.1631/FITEE.1900527
Webster FV, 1958. Traffic Signal Settings. Technical Report No. 39, Road Research Laboratory, UK.
Wei H, Xu N, Zhang HC, et al., 2019a. CoLight: learning network-level cooperation for traffic signal control. Proc 28th ACM Int Conf on Information and Knowledge Management, p.1913–1922. https://doi.org/10.1145/3357384.3357902
Wei H, Chen CC, Zheng GJ, et al., 2019b. PressLight: learning max pressure control to coordinate traffic signals in arterial network. Proc 25th ACM SIGKDD Int Conf on Knowledge Discovery & Data Mining, p.1290–1298. https://doi.org/10.1145/3292500.3330949
Wiering M, 2000. Multi-agent reinforcement learning for traffic light control. Proc 17th Int Conf on Machine Learning, p.1151–1158.
Xiao Y, Codevilla F, Gurram A, et al., 2022. Multimodal end-to-end autonomous driving. IEEE Trans Intell Transp Syst, 23(1):537–547. https://doi.org/10.1109/TITS.2020.3013234
Xiong G, Dong XS, Lu H, et al., 2020. Research progress of parallel control and management. IEEE/CAA J Autom Sin, 7(2):355–367. https://doi.org/10.1109/JAS.2019.1911792
Ye BL, Wu WM, Ruan KY, et al., 2019. A survey of model predictive control methods for traffic signal control. IEEE/CAA J Autom Sin, 6(3):623–640. https://doi.org/10.1109/JAS.2019.1911471
Yu ZX, Liang SX, Wei L, et al., 2020. MaCAR: urban traffic light control via active multi-agent communication and action rectification. Proc 29th Int Joint Conf on Artificial Intelligence, p.2491–2497. https://doi.org/10.24963/ijcai.2020/345
Zhang HC, Kafouros M, Yu Y, 2020. PlanLight: learning to optimize traffic signal control with planning and iterative policy improvement. IEEE Access, 8:219244–219255. https://doi.org/10.1109/ACCESS.2020.3041441
Zhang KQ, Yang ZR, Basar T, 2021. Decentralized multi-agent reinforcement learning with networked agents: recent advances. Front Inform Technol Electron Eng, 22(6):802–814. https://doi.org/10.1631/FITEE.1900661
Zhao YF, Gao H, Wang S, et al., 2017. A novel approach for traffic signal control: a recommendation perspective. IEEE Intell Transp Syst Mag, 9(3):127–135. https://doi.org/10.1109/MITS.2017.2709779
Zhu FH, Lv YS, Chen YY, et al., 2020. Parallel transportation systems: toward IoT-enabled smart urban traffic control and management. IEEE Trans Intell Transp Syst, 21(10):4063–4071. https://doi.org/10.1109/TITS.2019.2934991
Author information
Authors and Affiliations
Corresponding author
Additional information
Project supported by the National Natural Science Foundation of China (Nos. 62173329 and U1811463)
Contributors
Xingyuan DAI and Fei-Yue WANG designed the research. Xiao WANG and Yisheng LV contributed ideas for experiments and analysis. Chen ZHAO created the simulation platform. Xingyuan DAI and Yilun LIN performed simulations and analysis. Fei-Yue WANG managed the project. Xingyuan DAI and Chen ZHAO drafted the paper. Xiao WANG, Yisheng LV, and Fei-Yue WANG revised and finalized the paper.
Compliance with ethics guidelines
Xingyuan DAI, Chen ZHAO, Xiao WANG, Yisheng LV, Yilun LIN, and Fei-Yue WANG declare that they have no conflict of interest
Rights and permissions
About this article
Cite this article
Dai, X., Zhao, C., Wang, X. et al. Image-based traffic signal control via world models. Front Inform Technol Electron Eng 23, 1795–1813 (2022). https://doi.org/10.1631/FITEE.2200323
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2200323