Abstract
Object detection on Lidar point cloud data is a promising technology for autonomous driving and robotics which has seen a significant rise in performance and accuracy during recent years. Particularly uncertainty estimation is a crucial component for down-stream tasks and deep neural networks remain error-prone even for predictions with high confidence. Previously proposed methods for quantifying prediction uncertainty tend to alter the training scheme of the detector or rely on prediction sampling which results in vastly increased inference time. In order to address these two issues, we propose LidarMetaDetect (LMD), a light-weight post-processing scheme for prediction quality estimation. Our method can easily be added to any pre-trained Lidar object detector without altering anything about the base model and is purely based on post-processing, therefore, only leading to a negligible computational overhead. Our experiments show a significant increase of statistical reliability in separating true from false predictions. We propose and evaluate an additional application of our method leading to the detection of annotation errors. Explicit samples and a conservative count of annotation error proposals indicates the viability of our method for large-scale datasets like KITTI and nuScenes. On the widely-used nuScenes test dataset, 43 out of the top 100 proposals of our method indicate, in fact, erroneous annotations.
T. Riedlinger, M. Schubert, S. Penquitt and J.-M. Kezmann—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Caesar, H., et al.: nuScenes: a multimodal dataset for autonomous driving. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, pp. 11618–11628. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.01164. https://ieeexplore.ieee.org/document/9156412/
Chen, H., Huang, Y., Tian, W., Gao, Z., Xiong, L.: MonoRUn: monocular 3D object detection by reconstruction and uncertainty propagation. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, pp. 10374–10383. IEEE (2021). https://doi.org/10.1109/CVPR46437.2021.01024. https://ieeexplore.ieee.org/document/9578186/
Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. Technical report, Department of Computer Sciences and Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison (2006). https://minds.wisconsin.edu/bitstream/handle/1793/60482/TR1551.pdf
Feng, D., Rosenbaum, L., Dietmayer, K.: Towards safe autonomous driving: capture uncertainty in the deep neural network for lidar 3D vehicle detection. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 3266–3273 (2018). https://doi.org/10.1109/ITSC.2018.8569814. ISSN 2153-0017
Feng, D., Rosenbaum, L., Timm, F., Dietmayer, K.: Leveraging heteroscedastic aleatoric uncertainties for robust real-time LiDAR 3D object detection. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 1280–1287 (2019). https://doi.org/10.1109/IVS.2019.8814046. ISSN 2642-7214
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361 (2012). https://doi.org/10.1109/CVPR.2012.6248074. ISSN 1063-6919
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks (2017)
Havasi, M., et al.: Training independent subnetworks for robust prediction (2021). http://arxiv.org/abs/2010.06610 [cs, stat]
Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: PointPillars: fast encoders for object detection from point clouds. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 12689–12697. IEEE (2019). https://doi.org/10.1109/CVPR.2019.01298. https://ieeexplore.ieee.org/document/8954311/
Lin, W.H., Hauptmann, A.: Meta-classification: combining multimodal classifiers. In: Zaïane, O.R., Simoff, S.J., Djeraba, C. (eds.) PAKDD 2002. LNCS, vol. 2797, pp. 217–231. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-39666-6-14
Meyer, G.P., Laddha, A., Kee, E., Vallespi-Gonzalez, C., Wellington, C.K.: LaserNet: an efficient probabilistic 3D object detector for autonomous driving. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 12669–12678. IEEE (2019). https://doi.org/10.1109/CVPR.2019.01296. https://ieeexplore.ieee.org/document/8953739/
Meyer, G.P., Thakurdesai, N.: Learning an uncertainty-aware object detector for autonomous driving. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10521–10527 (2020). https://doi.org/10.1109/IROS45743.2020.9341623. ISSN 2153-0866
MMDetection3D Contributors: OpenMMLab’s Next-generation Platform for General 3D Object Detection (2020). https://github.com/open-mmlab/mmdetection3d, original-date: 2020-07-08T03:39:45Z
Naeini, M.P., Cooper, G.F., Hauskrecht, M.: Obtaining well calibrated probabilities using Bayesian binning. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)
Pitropov, M., Huang, C., Abdelzad, V., Czarnecki, K., Waslander, S.: LiDAR-MIMO: efficient uncertainty estimation for LiDAR-based 3D object detection. In: 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 813–820 (2022). https://doi.org/10.1109/IV51971.2022.9827244
Riedlinger, T., Rottmann, M., Schubert, M., Gottschalk, H.: Gradient-based quantification of epistemic uncertainty for deep object detectors. In: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, pp. 3910–3920. IEEE (2023). https://doi.org/10.1109/WACV56688.2023.00391. https://ieeexplore.ieee.org/document/10030773/
Rottmann, M., et al.: Prediction error meta classification in semantic segmentation: detection via aggregated dispersion measures of softmax probabilities. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–9 (2020). https://doi.org/10.1109/IJCNN48605.2020.9206659. ISSN 2161-4407
Rottmann, M., Maag, K., Chan, R., Hüger, F., Schlicht, P., Gottschalk, H.: Detection of false positive and false negative samples in semantic segmentation. In: 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1351–1356 (2020). https://doi.org/10.23919/DATE48585.2020.9116288. ISSN 1558-1101
Schubert, M., Kahl, K., Rottmann, M.: MetaDetect: uncertainty quantification and prediction quality estimates for object detection. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–10 (2021). https://doi.org/10.1109/IJCNN52387.2021.9534289. ISSN 2161-4407
Stanley, T.D., Jarrell, S.B.: Meta-regression analysis: a quantitative method of literature surveys. J. Econ. Surv. 19(3), 299–308 (2005). https://doi.org/10.1111/j.0950-0804.2005.00249.x. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.0950-0804.2005.00249.x
Yan, Y., Mao, Y., Li, B.: Second: sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)
Yang, B., Luo, W., Urtasun, R.: PIXOR: real-time 3D object detection from point clouds. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp. 7652–7660. IEEE (2018). https://doi.org/10.1109/CVPR.2018.00798. https://ieeexplore.ieee.org/document/8578896/
Yang, Q., Chen, H., Chen, Z., Su, J.: Uncertainty estimation for monocular 3D object detectors in autonomous driving. In: 2021 6th International Conference on Robotics and Automation Engineering (ICRAE), pp. 55–59 (2021). https://doi.org/10.1109/ICRAE53653.2021.9657820
Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3D object detection and tracking. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, pp. 11779–11788. IEEE (2021). https://doi.org/10.1109/CVPR46437.2021.01161. https://ieeexplore.ieee.org/document/9578166/
Acknowledgement
We gratefully acknowledge financial support by the state Ministry of Economy, Innovation and Energy of Northrhine Westphalia (MWIDE) and the European Fund for Regional Development via the FIS.NRW project BIT, grant no. EFRE-0400216, as well as “Projekt UnrEAL”, grant no. 01IS22069, funded by the German Federal Ministry of Education and Research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Riedlinger, T. et al. (2024). LMD: Light-Weight Prediction Quality Estimation for Object Detection in Lidar Point Clouds. In: Köthe, U., Rother, C. (eds) Pattern Recognition. DAGM GCPR 2023. Lecture Notes in Computer Science, vol 14264. Springer, Cham. https://doi.org/10.1007/978-3-031-54605-1_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-54605-1_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54604-4
Online ISBN: 978-3-031-54605-1
eBook Packages: Computer ScienceComputer Science (R0)