Abstract
Decision-making in autonomous driving is an emerging technology that has rapid progress over the last decade. In single-lane scenarios, autonomous vehicles should simultaneously optimize their velocity decisions and steering angle decisions to achieve safety, efficiency, comfort, small impacts on rear vehicles, and small offsets to the lane center line. Previous studies, however, have typically optimized these two decisions separately, ignoring the potential relationship between them. In this work, we propose a decision-making framework, named SMART (deciSion-Making frAmework based on ReinforcemenT learning), to optimize the velocity and steering angle of the autonomous vehicle in parallel. In order for the autonomous vehicle to effectively perceive the curvature of the lane and interactions with other vehicles, we adopt a graph attention mechanism to extract and fuse the features from different modalities (i.e., sensor-collected vehicle states and camera-collected lane information). Then a hybrid reward function takes into account aspects of safety, efficiency, comfort, impact, and lane centering to instruct the autonomous vehicle to make optimal decisions. Furthermore, our framework enables the autonomous vehicle to adaptively choose the duration of an action, which helps the autonomous vehicle pursue higher reward values. Extensive experiments evidence that SMART significantly outperforms the existing methods in multiple metrics.
Y. Xia and S. Liu—Both authors contribute equally to this paper.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Brenner, W., Herrmann, A.: An overview of technology, benefits and impact of automated and autonomous driving on the automotive industry. Digital marketplaces unleashed pp. 427–442 (2018)
Chang, M.F., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D., et al.: Argoverse: 3d tracking and forecasting with rich maps. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8748–8757 (2019)
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning. pp. 1–16 (2017)
Evans, L.: Traffic safety and the driver. Science Serving Society (1991)
Gao, J., Sun, C., Zhao, H., Shen, Y., Anguelov, D., Li, C., Schmid, C.: Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11525–11533 (2020)
Gupta, A., Anpalagan, A., Guan, L., Khwaja, A.S.: Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues. Array 10, 100057 (2021)
Gutiérrez, R., López-Guillén, E., Bergasa, L.M., Barea, R., Pérez, Ó., Gómez-Huélamo, C., Arango, F., Del Egido, J., López-Fernández, J.: A waypoint tracking controller for autonomous road vehicles using ros framework. Sensors 20(14), 4062 (2020)
Hsu, T.M., Wang, C.H., Chen, Y.R.: End-to-end deep learning for autonomous longitudinal and lateral control based on vehicle dynamics. In: Proceedings of the 2018 International Conference on Artificial Intelligence and Virtual Reality. pp. 111–114 (2018)
Jacobson, I., Richards, L., Kuhlthau, A.: Models of human comfort in vehicle environments. HUMAN FACTORS IN TRANSPORT RESEARCH EDITED BY DJ OBORNE, JA LEVIS 2 (1980)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Luo, W., Yang, B., Urtasun, R.: Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 3569–3577 (2018)
Milanés, V., Shladover, S.E.: Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data. Transportation Research Part C: Emerging Technologies 48, 285–300 (2014)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. nature 518(7540), 529–533 (2015)
Pérez-Gil, O., Barea, R., López-Guillén, E., Bergasa, L.M., Revenga, P.A., Gutiérrez, R., Díaz, A.: Dqn-based deep reinforcement learning for autonomous driving. In: Workshop of Physical Agents. pp. 60–76. Springer (2020)
Puterman, M.L.: Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons (2014)
Salfer-Hobbs, M.B.: Acceleration, Braking, and Steering Controller for Polaris GEM e2. Ph.D. thesis (2019)
Sun, L., Yan, Z., Mellado, S.M., Hanheide, M., Duckett, T.: 3dof pedestrian trajectory prediction learned from long-term autonomous mobile robot deployment data. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 5942–5948. IEEE (2018)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
Wolf, P., Hubschneider, C., Weber, M., Bauer, A., Härtl, J., Dürr, F., Zöllner, J.M.: Learning how to drive in a real world simulation with deep q-networks. In: 2017 IEEE Intelligent Vehicles Symposium (IV). pp. 244–250. IEEE (2017)
Xia, Y., Liu, S., Chen, X., Xu, Z., Zheng, K., Su, H.: Rise: A velocity control framework with minimal impacts based on reinforcement learning. CIKM ’22, New York, NY, USA (2022)
Xiao, L., Wang, M., Van Arem, B.: Realistic car-following models for microscopic simulation of adaptive and cooperative adaptive cruise control vehicles. Transportation Research Record 2623(1), 1–9 (2017)
Xu, Z., Liu, S., Wu, Z., Chen, X., Zeng, K., Zheng, K., Su, H.: Patrol: A velocity control framework for autonomous vehicle via spatial-temporal reinforcement learning. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. pp. 2271–2280 (2021)
Zhu, M., Wang, Y., Pu, Z., Hu, J., Wang, X., Ke, R.: Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving. Transportation Research Part C: Emerging Technologies 117, 102662 (2020)
Acknowledgment
This work is partially supported by NSFC (No. 61972069, 61836007, 61832017, 62272086), Shenzhen Municipal Science and Technology R &D Funding Basic Research Program (JCYJ20210324133607021), and Municipal Government of Quzhou under Grant (No. 2022D037, 2021D022).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Xia, Y. et al. (2023). SMART: A Decision-Making Framework with Multi-modality Fusion for Autonomous Driving Based on Reinforcement Learning. In: Wang, X., et al. Database Systems for Advanced Applications. DASFAA 2023. Lecture Notes in Computer Science, vol 13946. Springer, Cham. https://doi.org/10.1007/978-3-031-30678-5_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-30678-5_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30677-8
Online ISBN: 978-3-031-30678-5
eBook Packages: Computer ScienceComputer Science (R0)