Skip to main content
Springer Nature Link
Log in

Pixel-wise confidence estimation for segmentation in Bayesian Convolutional Neural Networks

  • ORIGINAL PAPER
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Bayesian convolutional neural networks represent an emerging state-of-the-art computer vision framework. This particular version of CNN allows the modeling of two types of uncertainties: epistemic uncertainty and aleatoric uncertainty. These uncertainties permit to evaluate the degree of confidence in predictions yielded by a model. Nevertheless, few studies automatically integrate uncertainty in an end-to-end prediction pipeline. Our research proposes a novel way to assess the degree of confidence in yielded predictions. To that end, a model is trained using the cross-entropy loss function. Thereafter, the learning of this model is resumed with a pixel weighting dynamic calculation designed to reduce the uncertainty of well-classified pixels and penalize wrong classifications. From the epistemic uncertainty measures provided by these two trainings, a histogram is calculated. A final neural network model is used to determine an interval of confidence for the predictions. This interval defines the pixels to be considered with caution at test time, depending on the uncertainty they yield. Validation is performed and shows that the uncertainty yielded by our sample weighting provides a better confidence interval than the regular, unweighted, cross-entropy loss function. Furthermore, our expected calibration error averaged over all datasets (0.07) is lower than available methods (0.1 for the state-of-the-art, 0.18 without calibration). Furthermore, the proposed uncertainty-based thresholding provides better accuracy than baseline uncertainty thresholding, while also minimizing the number of confident errors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)

    Article  Google Scholar 

  2. Amirkhani, A., Barshooi, A.H., Ebrahimi, A.: Enhancing the robustness of visual object tracking via style transfer. CMC-Comput. Mater. Continua 1, 981–997 (2022)

    Article  Google Scholar 

  3. Barshooi, A.H., Amirkhani, A.: A novel data augmentation based on Gabor filter and convolutional deep learning for improving the classification of COVID-19 chest X-ray images. Biomed. Signal Process. Control 72, 103326 (2022)

    Article  Google Scholar 

  4. Aladem, M., Rawashdeh, S.A.: A single-stream segmentation and depth prediction CNN for autonomous driving. IEEE Intell. Syst. 4, 79–85 (2020)

    Google Scholar 

  5. Guynn, J.: Google photos labeled black people ’gorillas’. USA Today (2015)

  6. National Highway Traffic Safety Administration: Tesla crash preliminary evaluation report. Technical report, U.S. Department of Transportation (2017)

  7. Konyushkova, K., Sznitman, R., Fua, P.: Geometry in active learning for binary and multi-class image segmentation. Comput. Vis. Image Underst. (2019). https://doi.org/10.1016/j.cviu.2019.01.007

    Article  Google Scholar 

  8. Kohli, P., Torr, P.H.: Measuring uncertainty in graph cut solutions. Comput. Vis. Image Underst. 1, 30–38 (2008)

    Article  Google Scholar 

  9. Der Kiureghian, A., Ditlevsen, O.: Aleatory or epistemic? does it matter? Struct. Saf. 2, 105–112 (2009)

    Article  Google Scholar 

  10. Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: representing model uncertainty in deep learning. In: International Conference on Machine Learning, pp. 1050–1059 (2016)

  11. Kononenko, I.: Bayesian neural networks. Biol. Cybern. 5, 361–370 (1989)

    Article  Google Scholar 

  12. Neal, R.M.: Bayesian learning for neural networks vol. 118, 10–14 (2012)

  13. Gal, Y., Hron, J., Kendall, A.: Concrete dropout. In: Advances in Neural Information Processing Systems, pp. 3584–3593 (2017)

  14. Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.: Learning uncertain convolutional features for accurate saliency detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 212– 221 (2017)

  15. Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems, pp. 5580–5590 (2017)

  16. Sensoy, M., Kaplan, L., Kandemir, M.: Evidential deep learning to quantify classification uncertainty. In: Advances in Neural Information Processing Systems, pp. 3183– 3193 (2018)

  17. Cheng, F., Zhang, H., Yuan, D., Sun, M.: Leveraging semantic segmentation with learning-based confidence measure. Neurocomputing (2019). https://doi.org/10.1016/j.neucom.2018.10.037

    Article  Google Scholar 

  18. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning. vol 70, pp. 1321–1330 (2017). JMLR. org

  19. Kurz, A., Hauser, K., Mehrtens, H.A., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Fröhling, S., von Kalle, C., Brinker, T.J., et al.: Uncertainty estimation in medical image classification: systematic review. JMIR Med. Inf. 10(8), e36427 (2022)

    Article  Google Scholar 

  20. Mehrtash, A., Wells, W.M., Tempany, C.M., Abolmaesumi, P., Kapur, T.: Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Trans. Med. Imaging 39(12), 3868–3878 (2020)

    Article  Google Scholar 

  21. Ding, Y., Liu, J., Xiong, J., Shi, Y.: Revisiting the evaluation of uncertainty estimation and its application to explore model complexityuncertainty trade-off. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 4–5 (2020)

  22. Fingscheidt, T., Gottschalk, H., Houben, S.: Deep neural networks and data for automated driving: Robustness, uncertainty quantification, and insights towards safety (2022)

  23. Niculescu-Mizil, A., Caruana, R.: Predicting good probabilities with supervised learning. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 625– 632 (2005). ACM

  24. Naeini, M.P., Cooper, G.F., Hauskrecht, M.: Obtaining well calibrated probabilities using bayesian binning. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. AAAI’15, pp. 2901–2907. AAAI Press, Austin, Texas (2015). http://dl.acm.org/citation.cfm?id=2888116.2888120

  25. Lin, G., Milan, A., Shen, C., Reid, I.: Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1925–1934 (2017)

  26. J´egou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: fully convolutional densenets for semantic segmentation. In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference On, pp. 1175–1183 (2017). IEEE

  27. Brostow, G.J., Fauqueur, J., Cipolla, R.: Semantic object classes in video: a highdefinition ground truth database. Pattern Recognition Letters (2008)

  28. Brostow, G.J., Shotton, J., Fauqueur, J., Cipolla, R.: Segmentation and recognition using structure from motion point clouds. In: ECCV (1), pp. 44–57 (2008)

  29. Nathan Silberman, P.K. Derek Hoiem, Fergus, R.: Indoor segmentation and support inference from RGBD images. In: ECCV (2012)

  30. Kavur, A.E., Selver, M.A., Dicle, O., Barıs, M., Gezer, N.S.: CHAOS - combined (CT-MR) healthy abdominal organ segmentation challenge data. Med. Image Anal. (2019). https://doi.org/10.5281/zenodo.3431873

Download references

Acknowledgements

This work was supported in part by NSERC Discovery grant. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X GPU used for this research.

Funding

This work was supported by Natural Sciences and Engineering Research Council.

Author information

Authors and Affiliations

  1. Department of Software and Information Technology Engineering, Ecole de Technologié Supérieure, 1100 Notre-Dame, Montréal, QC, H3C 1K3, Canada

    Rémi Martin & Luc Duong

Authors
  1. Rémi Martin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Luc Duong
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rémi Martin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Martin, R., Duong, L. Pixel-wise confidence estimation for segmentation in Bayesian Convolutional Neural Networks. Machine Vision and Applications 34, 19 (2023). https://doi.org/10.1007/s00138-022-01369-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-022-01369-9

Keywords