Abstract
Variants of accuracy and precision are the gold-standard by which the computer vision community measures progress of perception algorithms. One reason for the ubiquity of these metrics is that they are largely task-agnostic; we in general seek to detect zero false negatives or positives. The downside of these metrics is that, at worst, they penalize all incorrect detections equally without conditioning on the task or scene, and at best, heuristics need to be chosen to ensure that different mistakes count differently. In this paper, we revisit “Planning KL-Divergence”, a principled metric for 3D object detection specifically for the task of self-driving. The core idea behind PKL is to isolate the task of object detection and measure the impact the produced detections would induce on the downstream task of driving. We summarize functionality provided by our python package planning-centric-metrics that implements PKL. nuScenes is in the process of incorporating PKL into their detection leaderboard and we hope that the convenience of our implementation encourages other leaderboards to follow suit.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Nvidia drive constellation. https://www.nvidia.com/en-us/self-driving-cars/drive-constellation/. Accessed 14 Oct 2019
Bansal, M., Krizhevsky, A., Ogale, A.S.: Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. CoRR abs/1812.03079 (2018). http://arxiv.org/abs/1812.03079
Caesar, H., et al.: nuscenes: A multimodal dataset for autonomous driving. CoRR abs/1903.11027 (2019). http://arxiv.org/abs/1903.11027
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning, pp. 1–16 (2017)
Kar, A., et al.: Meta-Sim: learning to generate synthetic datasets. In: ICCV (2019). http://arxiv.org/abs/1904.11621
Manivasagam, S., et al.: LiDARsim: Realistic LiDAR simulation by leveraging the real world (2020)
Mao, H., Yang, X., Dally, W.J.: A delay metric for video object detection: What average precision fails to tell (2019)
Philion, J., Kar, A., Fidler, S.: Learning to evaluate perception models using planner-centric metrics (2020)
Szegedy, C., et al.: Intriguing properties of neural networks (2013)
Yang, Z., et al.: SurfelGAN: Synthesizing realistic sensor data for autonomous driving (2020)
Zemel, R., Wu, Y., Swersky, K., Pitassi, T., Dwork, C.: Learning fair representations. In: Dasgupta, S., McAllester, D. (eds.) ICML, Proceedings of Machine Learning Research, Vol. 28, pp. 325–333. PMLR, Atlanta, Georgia, USA (17–19 Jun 2013). http://proceedings.mlr.press/v28/zemel13.html
Zeng, W., et al.: End-to-end interpretable neural motion planner. In: CVPR (June 2019)
Zhu, B., Jiang, Z., Zhou, X., Li, Z., Yu, G.: Class-balanced grouping and sampling for point cloud 3d object detection (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Philion, J., Kar, A., Fidler, S. (2020). Implementing Planning KL-Divergence. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12540. Springer, Cham. https://doi.org/10.1007/978-3-030-65414-6_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-65414-6_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-65413-9
Online ISBN: 978-3-030-65414-6
eBook Packages: Computer ScienceComputer Science (R0)