Hebin Liang
ABSTRACT
In this paper, we focus on the decision-making challenge in autonomous driving, a central and intricate problem influencing the safety and practicality of autonomous vehicles. We propose an innovative hierarchical imitation learning framework that effectively alleviates the complexity of learning in autonomous driving decision-making problems by decoupling decision-making tasks into sub-problems. Specifically, the decision-making process is divided into two levels of sub-problems: the upper level directs the vehicle's lane selection and qualitative speed management, while the lower level implements precise control of the driving speed and direction. We harness Transformer-based models for solving each sub-problem, enabling overall hierarchical framework to comprehend and navigate diverse and various road conditions, ultimately resulting in improved decision-making. Through an evaluation in several typical driving scenarios within the SMARTS autonomous driving simulation environment, our proposed hierarchical decision-making framework significantly outperforms end-to-end reinforcement learning algorithms and behavior cloning algorithm, achieving an average pass rate of over 90%. Our framework's effectiveness is substantiated by its commendable achievements at the NeurIPS 2022 Driving SMARTS competition, where it secures dual track championships.
- Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. 2016. End to End Learning for Self-Driving Cars. arxiv: 1604.07316 [cs.CV]Google Scholar
- Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller. 2017. Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car. arxiv: 1704.07911 [cs.CV]Google Scholar
- Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. 2018. End-to-end driving via conditional imitation learning. In 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 4693--4700.Google ScholarDigital Library
- Serdar Coskun and Reza Langari. 2018. Predictive Fuzzy Markov Decision Strategy for Autonomous Driving in Highways. In 2018 IEEE Conference on Control Technology and Applications (CCTA). 1032--1039. https://doi.org/10.1109/CCTA.2018.8511369Google Scholar
- Christopher Diehl, Timo Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. 2022. UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning. arxiv: 2111.11097 [cs.RO]Google Scholar
- Andreas Folkers, Matthias Rick, and Christof Büskens. 2019. Controlling an Autonomous Vehicle with Deep Reinforcement Learning. In 2019 IEEE Intelligent Vehicles Symposium (IV). 2025--2031. https://doi.org/10.1109/IVS.2019.8814124Google ScholarDigital Library
- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. PMLR, 1861--1870.Google Scholar
- Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. 2022. Model-Based Imitation Learning for Urban Driving. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 20703--20716. https://proceedings.neurips.cc/paper_files/paper/2022/file/827cb489449ea216e4a257c47e407d18-Paper-Conference.pdfGoogle Scholar
- Zhiyu Huang, Haochen Liu, Jingda Wu, Wenhui Huang, and Chen Lv. 2023. Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving. arXiv preprint arXiv:2302.03939 (2023).Google Scholar
- Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative Q-Learning for Offline Reinforcement Learning. arxiv: 2006.04779 [cs.LG]Google Scholar
- Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature, Vol. 518, 7540 (2015), 529--533.Google Scholar
- Tung Nguyen, Qinqing Zheng, and Aditya Grover. 2023. Reliable Conditioning of Behavioral Cloning for Offline Reinforcement Learning. arxiv: 2210.05158 [cs.LG]Google Scholar
- Samyeul Noh. 2019. Decision-Making Framework for Autonomous Driving at Road Intersections: Safeguarding Against Collision, Overly Conservative Behavior, and Violation Vehicles. IEEE Transactions on Industrial Electronics, Vol. 66, 4 (2019), 3275--3286. https://doi.org/10.1109/TIE.2018.2840530Google ScholarCross Ref
- Alexandre Piche, Rafael Pardinas, David Vazquez, Igor Mordatch, and Chris Pal. 2022. Implicit Offline Reinforcement Learning via Supervised Learning. arXiv preprint arXiv:2210.12272 (2022).Google Scholar
- Zhiqian Qiao, Katharina Muelling, John Dolan, Praveen Palanisamy, and Priyantha Mudalige. 2018. POMDP and Hierarchical Options MDP with Continuous Actions for Autonomous Driving at Intersections. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). 2377--2382. https://doi.org/10.1109/ITSC.2018.8569400Google ScholarDigital Library
- Amir Rasouli, Randy Goebel, Matthew E Taylor, Iuliia Kotseruba, Soheil Alizadeh, Tianpei Yang, Montgomery Alban, Florian Shkurti, Yuzheng Zhuang, Adam Scibior, et al. 2022. NeurIPS 2022 Competition: Driving SMARTS. arXiv preprint arXiv:2211.07545 (2022).Google Scholar
- Moveh Samuel, Mohamed Hussein, and Maziah Binti Mohamad. 2016. A review of some pure-pursuit based path tracking techniques for control of autonomous vehicle. International Journal of Computer Applications, Vol. 135, 1 (2016), 35--38.Google ScholarCross Ref
- Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. 2023. Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer. In Proceedings of The 6th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 205), Karen Liu, Dana Kulic, and Jeff Ichnowski (Eds.). PMLR, 726--737. https://proceedings.mlr.press/v205/shao23a.htmlGoogle Scholar
- Yuan Tian. 2022. SMARTS_VCR. https://github.com/yuant95/SMARTS_VCR.Google Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arxiv: 1706.03762 [cs.CL]Google Scholar
- Pin Wang and Ching-Yao Chan. 2017. Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). 1--6. https://doi.org/10.1109/ITSC.2017.8317735Google ScholarDigital Library
- Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. 2017. End-To-End Learning of Driving Models From Large-Scale Video Datasets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
- Pan Zhao, Jiajia Chen, Yan Song, Xiang Tao, Tiejuan Xu, and Tao Mei. 2012. Design of a control system for an autonomous vehicle based on adaptive-pid. International Journal of Advanced Robotic Systems, Vol. 9, 2 (2012), 44.Google ScholarCross Ref
- Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et al. 2020. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint arXiv:2010.09776 (2020).Google Scholar
Index Terms
- A Hierarchical Imitation Learning-based Decision Framework for Autonomous Driving
Recommendations
Numerical Analysis of Tractor Accidents using Driving Simulator for Autonomous Driving Tractor
ICMRE'19: Proceedings of the 5th International Conference on Mechatronics and Robotics EngineeringAutonomous driving of automobiles is a hot research topic in recent years. The autonomous driving tractor also has been studied in the agricultural field as well as an autonomous driving automobile. On the other hand, tractor accidents frequently occur ...
Autonomous Driving: Investigating the Feasibility of Bimodal Take-Over Requests
Autonomous vehicles will need de-escalation strategies to compensate when reaching system limitations. Car-driver handovers can be considered one possible method to deal with system boundaries. The authors suggest a bimodal auditory and visual handover ...
Autonomous driving in framework of three-phase traffic theory
AbstractWe discuss a strategy for autonomous driving in the framework of three-phase traffic theory. The main reason for autonomous driving in the framework of the three-phase theory is that an autonomous driving vehicle should learn from driver behaviors ...
Comments