Abstract
Simulation now plays an important role in the development of autonomous driving algorithms as it can significantly reduce the economical cost and ethical risk of real-world testing. However, building a high-quality driving simulator is not trivial as it calls for realistic interactive behaviors of road agents. Recently, several simulators employ interactive trajectory prediction models learnt in a data-driven manner. While they are successful in generating short-term interactive scenarios, the simulator quickly breaks down when the time horizon gets longer. We identify the reason behind: existing interactive trajectory predictors suffer from the out-of-domain (OOD) problem when recursively feeding predictions as the input back to the model. To this end, we propose to introduce a tailored model predictive control (MPC) module as a rescue into the state-of-the art interactive trajectory prediction model M2I, forming a new simulator named M\(^2\)Sim. Notably, M\(^2\)Sim can effectively address the OOD problem of long-term simulation by enforcing a flexible regularization that admits the replayed data, while still enjoying the diversity of data-driven predictions. We demonstrate the superiority of M\(^2\)Sim using both quantitative results and visualizations and release our data, code and models: https://github.com/0nhc/m2sim.
Sponsored by Baidu Inc. through Apollo-AIR Joint Research Center.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Tian, B., Liu, M., Gao, H.-A., Li, P., Zhao, H., Zhou, G.: Unsupervised road anomaly detection with language anchors. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7778–7785. IEEE (2023)
Li, P., et al.: LODE: locally conditioned Eikonal implicit scene completion from sparse LiDAR. arXiv preprint arXiv:2302.14052 (2023)
Jin, B., et al.: ADAPT: action-aware driving caption transformer. arXiv preprint arXiv:2302.00673 (2023)
Zheng, Y., et al.: Steps: joint self-supervised nighttime image enhancement and depth estimation. arXiv preprint arXiv:2302.01334 (2023)
Hu, Y., et al.: Planning-oriented autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17 853–17 862 (2023)
Ngiam, J., et al.: Scene transformer: a unified architecture for predicting future trajectories of multiple agents. In: International Conference on Learning Representations (2022)
Sun, Q., Huang, X., Gu, J., Williams, B.C., Zhao, H.: M2I: from factored marginal trajectory prediction to interactive prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6543–6552 (2022)
Liu, X., Wang, Y., Jiang, K., Zhou, Z., Nam, K., Yin, C.: Interactive trajectory prediction using a driving risk map-integrated deep learning method for surrounding vehicles on highways. IEEE Trans. Intell. Transp. Syst. 23(10), 19 076–19 087 (2022)
Zhang, K., Zhao, L., Dong, C., Wu, L., Zheng, L.: AI-TP: attention-based interaction-aware trajectory prediction for autonomous driving. IEEE Trans. Intell. Veh. (2022)
Shen, Z.: Towards out-of-distribution generalization: a survey. arXiv preprint arXiv:2108.13624 (2021)
Filos, A., Tigkas, P., McAllister, R., Rhinehart, N., Levine, S., Gal, Y.: Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In: International Conference on Machine Learning, pp. 3145–3153. PMLR (2020)
Kerrigan, E.C.: Predictive control for linear and hybrid systems [bookshelf]. IEEE Control Syst. Mag. 38(2), 94–96 (2018)
Falcone, P., et al.: Nonlinear model predictive control for autonomous vehicles (2007)
Carvalho, A., Gao, Y., Gray, A., Tseng, H.E., Borrelli, F.: Predictive control of an autonomous ground vehicle using an iterative linearization approach. In: 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), pp. 2335–2340. IEEE (2013)
Gao, Y., Lin, T., Borrelli, F., Tseng, E., Hrovat, D.: Predictive control of autonomous ground vehicles with obstacle avoidance on slippery roads. In: Dynamic Systems and Control Conference, vol. 44175, pp. 265–272 (2010)
Beal, C.E., Gerdes, J.C.: Model predictive control for vehicle stabilization at the limits of handling. IEEE Trans. Control Syst. Technol. 21(4), 1258–1269 (2012)
Levinson, J., et al.: Towards fully autonomous driving: systems and algorithms. In: 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 163–168. IEEE (2011)
Mohamed, A., Qian, K., Elhoseiny, M., Claudel, C.: Social-STGCNN: a social spatio-temporal graph convolutional neural network for human trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14 424–14 432 (2020)
Casas, S., Gulino, C., Suo, S., Luo, K., Liao, R., Urtasun, R.: Implicit latent variable model for scene-consistent motion forecasting. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12368, pp. 624–641. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58592-1_37
Kamra, N., Zhu, H., Trivedi, D.K., Zhang, M., Liu, Y.: Multi-agent trajectory prediction with fuzzy query attention. In: Advances in Neural Information Processing Systems, vol. 33, pp. 22 530–22 541 (2020)
Nayakanti, N., Al-Rfou, R., Zhou, A., Goel, K., Refaat, K.S., Sapp, B.: Wayformer: motion forecasting via simple & efficient attention networks. arXiv preprint arXiv:2207.05844 (2022)
Lopez, P.A.: Microscopic traffic simulation using sumo. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2575–2582. IEEE (2018)
Zhang, H.: CityFlow: a multi-agent reinforcement learning environment for large scale city traffic scenario. In: The World Wide Web Conference, pp. 3620–3624 (2019)
Althoff, M., Koschi, M., Manzinger, S.: CommonRoad: composable benchmarks for motion planning on roads. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 719–726. IEEE (2017)
Suo, S., Regalado, S., Casas, S., Urtasun, R.: TrafficSim: learning to simulate realistic multi-agent behaviors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10 400–10 409 (2021)
Bergamini, L., et al.: SimNet: learning reactive self-driving simulations from real-world observations. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 5119–5125. IEEE (2021)
Sun, Q., Huang, X., Williams, B.C., Zhao, H.: InterSim: interactive traffic simulation via explicit relation modeling. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11 416–11 423. IEEE (2022)
Zhou, J.: Exploring imitation learning for autonomous driving with feedback synthesizer and differentiable rasterization. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1450–1457. IEEE (2021)
Kong, J., Pfeiffer, M., Schildbach, G., Borrelli, F.: Kinematic and dynamic vehicle models for autonomous driving control design. In: 2015 IEEE Intelligent Vehicles Symposium (IV), pp. 1094–1099. IEEE (2015)
Cho, M., Lee, Y., Kim, K.-S.: Model predictive control of autonomous vehicles with integrated barriers using occupancy grid maps. IEEE Rob. Autom. Lett. (2023)
Kühner, T., Kümmerle, J.: Large-scale volumetric scene reconstruction using LiDAR. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 6261–6267. IEEE (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Han, Z. et al. (2024). Long-Term Interactive Driving Simulation: MPC to the Rescue. In: Fang, L., Pei, J., Zhai, G., Wang, R. (eds) Artificial Intelligence. CICAI 2023. Lecture Notes in Computer Science(), vol 14474. Springer, Singapore. https://doi.org/10.1007/978-981-99-9119-8_17
Download citation
DOI: https://doi.org/10.1007/978-981-99-9119-8_17
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-9118-1
Online ISBN: 978-981-99-9119-8
eBook Packages: Computer ScienceComputer Science (R0)