Abstract
Decision making is an essential component of autonomous vehicle technology and received significant attention from academic and industry organizations. One of the promising approaches in designing a decision-making method is Reinforcement Learning (RL). To apply an RL algorithm to an autonomous driving problem, a feature representation of the state must first be chosen. The most commonly used representation is the spatial-temporal state feature. However, if the number or order of the surrounding vehicle changes, the feature representation will be affected. In this paper, we utilize time-to-collision (TTC) as the feature representation and propose a TTC-based safety check system. The action output by the RL controller would be replaced with a safer action chosen by the safety check system when an agent detects a potential collision, i.e., the TTC is below the time threshold. A ramp merging task is used to illustrate the effect. Simulation results show that the proposed method can effectively improve the arrival rate and reduce the collision rate, even in the case of dense traffic situations. Furthermore, we also conducted experiments to examine the performance of the safety check system with different time thresholds.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Riccardo S, Benjamin H (2014) Control concepts for facilitating motorway on-ramp merging using intelligent vehicles. Trans Rev 34(6):775–797
Wang X, Zhao D, Peng H, LeBlanc DJ (2017) Analysis of unprotected intersection left-turn conflicts based on naturalistic driving data. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pages 218–223. IEEE
Xuedong Y, Essam R (2007) Effect of restricted sight distances on driver behaviors during unprotected left-turn phase at signalized intersections. Transp Res Part F: Traffic Psychol Behav 10(4):330–344
Saad Y, Zaid N, Norgate Sarah H (2017) Narrow lanes and their effect on drivers’ behaviour at motorway roadworks. Transp Res Part F: Traffic Psychol Behav 47:86–100
Alexandra K, Hale David K, Mohamadamin A, Bastian S, Anxi J, Joe B (2019) Evaluating the operational effect of narrow lanes and shoulders for the highway capacity manual. Transp Res Record 2673(10):558–570
Paden B, Cáp M, Yong SZ, Yershov DS, Frazzoli E (2016) A survey of motion planning and control techniques for self-driving urban vehicles. CoRR. arXiv:abs/1604.07446
Martin T, Ansgar H, Dirk H (2000) Congested traffic states in empirical observations and microscopic simulations. Phys Rev E 62(2):1805–1824
Arne K, Martin T, Helbing D (1999) General lane-changing model mobil for car-following models. Transport Res Record 1:86–94
Gu S, Holly E, Lillicrap T, Levine S (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: 2017 IEEE international conference on robotics and automation (ICRA)
Jens K, Andrew BJ, Jan P (2013) Reinforcement learning in robotics: a survey. Int J Robot Res 32(11):1238–1274
Li Haoran, Zhang Q, Zhao D (2020) Deep reinforcement learning based automatic exploration for navigation in unknown environment. CoRR. arXiv:abs/2007.11808
Leurent E, Mercat J (2019) Social attention for autonomous decision-making in dense traffic. CoRR. arXiv:abs/1911.12250
Wang W, Wang L, Zhang C, Liu C, Sun L (2022) Social interactions for autonomous driving: a review and perspective
Jansson J (2005) Collision avoidance theory: with application to automotive collision mitigation. PhD thesis. Linköping University Electronic Press
Noh S, Han W-Y (2014) Collision avoidance in on-road environment for autonomous driving. In: 2014 14th International Conference on Control, Automation and Systems (ICCAS 2014), pages 884–889. IEEE
Baheri A, Nageshrao S, Tseng HE, Kolmanovsky I, Girard A, Filev D (2020) Deep reinforcement learning with enhanced safety for autonomous highway driving. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1550–1555
Nageshrao S, Tseng HE, Filev D (2019) Autonomous highway driving using deep reinforcement learning. In: 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pages 2326–2331
Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv:1707.06347
Van Der Horst R, Hogema J (1993) Time-to-collision and collision avoidance systems
Hayward JC (1972) Near miss determination through use of a scale of danger
Leurent E (2018) An environment for autonomous driving decision-making. https://github.com/eleurent/highway-env
Acknowledgements
This work was supported by Initiative for Realizing Diversity in the Research Environment (Specific Correspondence Type).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Xiaotong Nie is the presenter of this paper.
This work was submitted and accepted for the Journal Track of the joint symposium of the 28th International Symposium on Artificial Life and Robotics, the 8th International Symposium on BioComplexity, and the 6th International Symposium on Swarm Behavior and Bio-Inspired Robotics (Beppu, Oita, January 25-27, 2023).
About this article
Cite this article
Nie, X., Liang, Y. & Ohkura, K. Autonomous highway driving using reinforcement learning with safety check system based on time-to-collision. Artif Life Robotics 28, 158–165 (2023). https://doi.org/10.1007/s10015-022-00846-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10015-022-00846-8