Skip to main content
SpringerLink
Log in
Book cover

International Conference on Computer Safety, Reliability, and Security

SAFECOMP 2022: Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops pp 316–329Cite as

Object Detection with Probabilistic Guarantees: A Conformal Prediction Approach

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13415))

Abstract

Providing reliable uncertainty quantification for complex visual tasks such as object detection is of utmost importance for safety-critical applications such as autonomous driving, tumor detection, etc. Conformal prediction methods offer simple yet practical means to build uncertainty estimations that come with probabilistic guarantees. In this paper we apply such methods to the task of object localization and illustrate our analysis on a pedestrian detection use-case. We highlight both theoretical and practical implications of our analysis.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    All the coordinates of the true box will be found inside the rectangle defined by the predicted bounding box of the object.

  2. 2.

    More complex variants exist. The typical process outlined here is more precisely known as split conformal prediction.

  3. 3.

    The errors are sometimes called “residuals” (hence the \(R^i\) notation).

  4. 4.

    Mathematically speaking, it is in fact sufficient that the calibration data and the data at inference time are exchangeable, conditionally on the training data.

  5. 5.

    The \(1-\alpha \) guarantee only holds on average over all calibration sets, see Sect. 6.

  6. 6.

    These coverage values include statistical error margins at level \(95\%\).

  7. 7.

    In fact, conformal guarantees work slightly beyond independence—under a so-called “exchangeability” assumption [1, 23, 37].

References

  1. Angelopoulos, A.N., Bates, S.: A gentle introduction to conformal prediction and distribution-free uncertainty quantification (2021). arXiv:2107.07511

  2. Azevedo, T., de Jong, R., Maji, P.: Stochastic-YOLO: efficient probabilistic object detection under dataset shifts. In: ML4AD Workshop, NeurIPS 2020 (2020)

    Google Scholar 

  3. Barber, R.F., Candes, E.J., Ramdas, A., Tibshirani, R.J.: Conformal prediction beyond exchangeability (2022). arXiv:2202.13415

  4. Bates, S., Angelopoulos, A., Lei, L., Malik, J., Jordan, M.I.: Distribution-free, risk-controlling prediction sets. J. ACM 68(6), 1–34 (2021)

    Google Scholar 

  5. Bickel, P.J., Doksum, K.A.: Mathematical Statistics: Basic Ideas and Selected Topics, vol. 1. Chapman and Hall/CRC, London (2015)

    Google Scholar 

  6. Bonnin, H., et al.: Can we reconcile safety objectives with machine learning performances? In: ERTS 2022 (2022)

    Google Scholar 

  7. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  8. Chih-Hong Cheng, T.S., Burton, S.: Logically sound arguments for the effectiveness of ML safety measures (2021). arXiv:2111.02649

  9. Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: CVPR 2016 (2016)

    Google Scholar 

  10. Deepshikha, K., Yelleni, S.H., Srijith, P.K., Mohan, C.K.: Monte Carlo DropBlock for modelling uncertainty in object detection (2021). arXiv:2108.03614

  11. Ducoffe, M., Gerchinovitz, S., Sen Gupta, J.: A high-probability safety guarantee for shifted neural network surrogates. In: SafeAI 2020 (2020)

    Google Scholar 

  12. Feng, D., Harakeh, A., Waslander, S.L., Dietmayer, K.: A review and comparative study on probabilistic object detection in autonomous driving. IEEE T-ITS, 1–20 (2021)

    Google Scholar 

  13. Girshick, R.B.: Fast R-CNN. In: ICCV 2015 (2015)

    Google Scholar 

  14. Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR 2014 (2014)

    Google Scholar 

  15. Harakeh, A., Smart, M., Waslander, S.L.: BayesOD: a Bayesian approach for uncertainty estimation in deep object detectors. In: ICRA 2020 (2020)

    Google Scholar 

  16. Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: NeurIPS 2017 (2017)

    Google Scholar 

  17. Kläs, M., Jöckel, L.: A framework for building uncertainty wrappers for AI/ML-based data-driven components. In: Casimiro, A., Ortmeier, F., Schoitsch, E., Bitsch, F., Ferreira, P. (eds.) SAFECOMP 2020. LNCS, vol. 12235, pp. 315–327. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55583-2_23

    Chapter  Google Scholar 

  18. Kläs, M., Sembach, L.: Uncertainty wrappers for data-driven models: increase the transparency of AI/ML-based models through enrichment with dependable situation-aware uncertainty estimates. In: WAISE 2019 (2019)

    Google Scholar 

  19. Kraus, F., Dietmayer, K.: Uncertainty estimation in one-stage object detection. In: ITSC 2019 (2019)

    Google Scholar 

  20. Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logistics Q. 2(1–2), 83–97 (1955)

    Google Scholar 

  21. Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: NeurIPS 2017 (2017)

    Google Scholar 

  22. Le, M.T., Diehl, F., Brunner, T., Knol, A.: Uncertainty estimation for deep neural object detectors in safety-critical applications. In: ITSC 2018 (2018)

    Google Scholar 

  23. Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R.J., Wasserman, L.: Distribution-free predictive inference for regression. JASA 113(523), 1094–1111 (2018)

    Google Scholar 

  24. Lin, T., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR 2017 (2017)

    Google Scholar 

  25. Lin, T., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. IEEE T-PAMI 42(02), 318–327 (2020)

    Google Scholar 

  26. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  27. Liu, W., et al.: SSD: single shot MultiBox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  28. Lyu, Z., Gutierrez, N., Rajguru, A., Beksi, W.J.: Probabilistic object detection via deep ensembles. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12540, pp. 67–75. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65414-6_7

    Chapter  Google Scholar 

  29. Miller, D., Dayoub, F., Milford, M., Sunderhauf, N.: Evaluating merging strategies for sampling-based uncertainty techniques in object detection. In: ICRA 2019 (2019)

    Google Scholar 

  30. Miller, D., Nicholson, L., Dayoub, F., Sünderhauf, N.: Dropout sampling for robust object detection in open-set conditions. In: ICRA 2018 (2018)

    Google Scholar 

  31. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: CVPR 2016 (2016)

    Google Scholar 

  32. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: CVPR 2017 (2017)

    Google Scholar 

  33. Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement (2018). arXiv:1804.02767

  34. Rucklidge, W.: Efficiently locating objects using the hausdorff distance. IJCV 24, 251–270 (1997)

    Google Scholar 

  35. Tibshirani, R.J., Barber, R.F., Candes, E.J., Ramdas, A.: Conformal prediction under covariate shift. In: NeurIPS 2019 (2019)

    Google Scholar 

  36. Schuster, T., Seferis, E., Burton, S., Cheng, C.H.: Unaligned but safe - formally compensating performance limitations for imprecise 2D object detection (2022). arXiv:2202.05123

  37. Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005). https://doi.org/10.1007/b106715

  38. Yu, F., et al.: BDD100K: a diverse driving dataset for heterogeneous multitask learning (2018). arXiv:1805.04687

Download references

Acknowledgements

This work has benefited from the AI Interdisciplinary Institute ANITI, which is funded by the French “Investing for the Future - PIA3” program under the Grant agreement ANR-19-P3IA-0004. The authors gratefully acknowledge the support of the DEEL project (https://www.deel.ai/).

Author information

Authors and Affiliations

  1. Thales AVS France SAS, Toulouse, France

    Florence de Grancey

  2. Renault, Toulouse, France

    Jean-Luc Adam

  3. Continental, Toulouse, France

    Lucian Alecu

  4. IRT Saint Exupéry, Toulouse, France

    Sébastien Gerchinovitz, Franck Mamalet & David Vigouroux

  5. Institut de Mathématiques de Toulouse, Toulouse, France

    Sébastien Gerchinovitz

Authors
  1. Florence de Grancey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jean-Luc Adam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lucian Alecu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sébastien Gerchinovitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Franck Mamalet
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. David Vigouroux
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sébastien Gerchinovitz .

Editor information

Editors and Affiliations

  1. Fraunhofer IKS, Munich, Germany

    Mario Trapp

  2. Austrian Institute of Technology, Vienna, Austria

    Erwin Schoitsch

  3. LAAS-CNRS, University of Toulouse, Toulouse, France

    Jérémie Guiochet

  4. Thales Deutschland GmbH, Ditzingen, Germany

    Friedemann Bitsch

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

de Grancey, F., Adam, JL., Alecu, L., Gerchinovitz, S., Mamalet, F., Vigouroux, D. (2022). Object Detection with Probabilistic Guarantees: A Conformal Prediction Approach. In: Trapp, M., Schoitsch, E., Guiochet, J., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops . SAFECOMP 2022. Lecture Notes in Computer Science, vol 13415. Springer, Cham. https://doi.org/10.1007/978-3-031-14862-0_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-14862-0_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-14861-3

  • Online ISBN: 978-3-031-14862-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation