Abstract
To improve the efficiency of the traffic signal control and alleviate traffic congestion, many researchers focus on applying deep reinforcement learning (DRL) for traffic signal control systems (TSCS). The TSCS consider all the vehicles’ waiting time around the intersection and decrease them so as to alleviate the traffic congestion. However, it has been confirmed that the DRL model is vulnerable to backdoor attacks. In this paper, we propose the first backdoor attack against DRL based TSCS. We define a special drive behavior as malicious input (called trigger). Once the trigger is activated via an attacker, the TSCS will only take into waiting time for the attacker’s vehicle at the intersection. Our empirical experiments show that our proposed backdoor attacks are effective with negligible impact on TSCS’s normal operation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of supporting data
Not applicable.
References
Lasley P (2021) 2021 urban mobility report
Chen C, Wei H, Xu N, Zheng G, Yang M, Xiong Y, Xu K, Li Z (2020) Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control. Proceedings of the AAAI Conference on Artificial Intelligence 34(04):3414–3421
Liang X, Du X, Wang G, Han Z (2019) A deep reinforcement learning network for traffic light cycle control. IEEE Trans Veh Technol 68(2):1243–1253
Lee J, Chung J, Sohn K (2019) Reinforcement learning for joint control of traffic signals in a transportation network. IEEE Trans Veh Technol 69(2):1375–1387
Wang Z, Zhu H, He M, Zhou Y, Luo X, Zhang N (2021) Gan and multi-agent drl based decentralized traffic light signal control. IEEE Trans Veh Technol 71(2):1333–1348
Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533
Ibarz J, Tan J, Finn C, Kalakrishnan M, Pastor P, Levine S (2021) How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research 40(4–5):698–721
Sallab AE, Abdou M, Perot E, Yogamani S (2017) Deep reinforcement learning framework for autonomous driving. Electronic Imaging 2017(19):70–76
Feng Y, Huang S, Chen QA, Liu HX, Mao ZM (2018) Vulnerability of traffic control system under cyberattacks with falsified data. Transp Res Rec 2672(1):1–11
Chen QA, Yin Y, Feng Y, Mao ZM, Liu HX (2018) Exposing congestion attack on emerging connected vehicle based traffic signal control. In: NDSS
Qu A, Tang Y, Ma W (2021) Attacking deep reinforcement learning-based traffic signal control systems with colluding vehicles. arXiv preprint arXiv:2111.02845
Haydari A, Zhang M, Chuah C-N (2021) Adversarial attacks and defense in deep reinforcement learning (drl)-based traffic signal controllers. IEEE Open Journal of Intelligent Transportation Systems 2:402–416
Wu C, Kreidieh AR, Parvate K, Vinitsky E, Bayen AM (2021) Flow: A modular learning framework for mixed autonomy traffic. IEEE Trans Robot
Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602
Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence 30(1)
Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971
Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
Kiourti P, Wardega K, Jha S, Li W (2020) Trojdrl: evaluation of backdoor attacks on deep reinforcement learning. In: 2020 57th ACM/IEEE Design Automation Conference (DAC). IEEE, pp 1–6
Wang Y, Sarkar E, Li W, Maniatakos M, Jabari SE (2021) Stop-and-go: Exploring backdoor attacks on deep reinforcement learning-based traffic congestion control systems. IEEE Trans Inf Forensics Secur 16:4772–4787
Wang L, Javed Z, Wu X, Guo W, Xing X, Song D (2021) Backdoorl: Backdoor attack against competitive reinforcement learning. arXiv preprint arXiv:2105.00579
Krajzewicz D, Erdmann J, Behrisch M, Bieker L (2012) Recent development and applications of sumo-simulation of urban mobility. International Journal on Advances in Systems and Measurements 5(3 & 4)
Liang E, Liaw R, Nishihara R, Moritz P, Fox R, Goldberg K, Gonzalez J, Jordan M, Stoica I (2018) Rllib: Abstractions for distributed reinforcement learning, in International Conference on Machine Learning. PMLR, pp 3053–3062
Wei H, Zheng G, Yao H, Li Z (2018) Intellilight: A reinforcement learning approach for intelligent traffic light control. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 2496–2505
Shabestary SMA, Abdulhai B (2022) Adaptive traffic signal control with deep reinforcement learning and high dimensional sensory inputs: Case study and comprehensive sensitivity analyses. IEEE Trans Intell Transp Syst
Yen C-C, Ghosal D, Zhang M, Chuah C-N, Chen H (2018) Falsified data attack on backpressure-based traffic signal control algorithms. In: 2018 IEEE Vehicular Networking Conference (VNC). IEEE, pp 1–8
Acknowledgements
We would like to show our gratitude to the editor and reviewers.
Funding
The work was partially supported by the National Natural Science Foundation of China under Grant 61873106, the Nature Science Foundation of Jiangsu Province for Distinguished Young Scholars under Grant BK20200049.
Author information
Authors and Affiliations
Contributions
Heng Zhang, Jun Gu, Yan Ren, Zhikun Zhang, and Linkang Du wrote the main manuscript text, and Jian Zhang and Hongran Li did the simulation. All authors reviewed the manuscript.
Corresponding author
Ethics declarations
Ethical approval and consent to participate
Not applicable.
Human and animal ethics
Not applicable.
Consent for publication
Not applicable.
Competing interests
Not applicable.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, H., Gu, J., Zhang, Z. et al. Backdoor attacks against deep reinforcement learning based traffic signal control systems. Peer-to-Peer Netw. Appl. 16, 466–474 (2023). https://doi.org/10.1007/s12083-022-01434-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12083-022-01434-0