Skip to main content
SpringerLink
Log in

Monte Carlo Concrete DropPath for Epistemic Uncertainty Estimation in Brain Tumor Segmentation

  • Conference paper
  • First Online:
Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis (UNSURE 2021, PIPPI 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12959))

  • 1471 Accesses

Abstract

Well-calibrated uncertainty is crucial for medical imaging tasks. However, Monte Carlo (MC) Dropout - one of the most common methods for epistemic uncertainty estimation in deep neural networks (DNN), has been found ineffective for multi-path DNN, such as NASNet, and has been recently bypassed by DropPath and ScheduledDropPath.

In this work, we propose two novel model calibration frameworks for uncertainty estimation: MC ScheduledDropPath and MC Concrete DropPath. Particularly, MC ScheduledDropPath drops out paths in DNN cells during test-time, which has proven to improve the model calibration. At the same time, the MC Concrete DropPath method applies concrete relaxation for DropPath probability optimization, which was found to even better regularize and calibrate DNNs at scale. We further investigate both methods on the problem of brain tumour segmentation and demonstrate a significant Dice score improvement and better calibration ability as compared to state-of-the-art baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Al-Shamahy, H.: Prevalence of CNS tumors and Histo-logical recognition in the operated patients: 10 years experience. Ann. Clin. Med. Case Rep. 6(12), 1–8 (2021)

    Google Scholar 

  2. Ashukha, A., Lyzhov, A., Molchanov, D., Vetrov, D.: Pitfalls of in-domain uncertainty estimation and ensembling in deep learning (2019)

    Google Scholar 

  3. Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. Data 4(1), 1–13 (2017)

    Article  MathSciNet  Google Scholar 

  4. Bakas, S., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018)

  5. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

    Google Scholar 

  6. Eaton-Rosen, Z., Bragman, F., Bisdas, S., Ourselin, S., Cardoso, M.J.: Towards safe deep learning: accurately quantifying biomarker uncertainty in neural network predictions. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 691–699. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_78

    Chapter  Google Scholar 

  7. Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation. arXiv preprint arXiv:1506.02157 (2015)

  8. Gal, Y., Hron, J., Kendall, A.: Concrete dropout. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  9. Ghiasi, G., Lin, T.Y., Le, Q.V.: DropBlock: a regularization method for convolutional networks. arXiv preprint arXiv:1810.12890 (2018)

  10. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Improving neural networks by preventing co-adaptation of feature detectors. CoRR abs/1207.0580 (2012). http://arxiv.org/abs/1207.0580

  11. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  12. Isensee, F., Jäger, P.F., Full, P.M., Vollmuth, P., Maier-Hein, K.H.: nnU-Net for brain tumor segmentation. In: Crimi, A., Bakas, S. (eds.) BrainLes 2020. LNCS, vol. 12659, pp. 118–132. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72087-2_11

    Chapter  Google Scholar 

  13. Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 (2018)

  14. Jungo, A., Meier, R., Ermis, E., Herrmann, E., Reyes, M.: Uncertainty-driven sanity check: application to postoperative brain tumor cavity segmentation. arXiv preprint arXiv:1806.03106 (2018)

  15. Kaneko, S., Nomura, K., Yoshimura, T., et al.: Trend of brain tumor incidence by histological subtypes in Japan: estimation from the brain tumor registry of t. J. Neurooncol. 60, 61–69 (2002). https://doi.org/10.1023/A:1020239720852

    Article  Google Scholar 

  16. Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems, pp. 5574–5584 (2017)

    Google Scholar 

  17. Larsson, G., Maire, M., Shakhnarovich, G.: FractalNet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 (2016)

  18. Loquercio, A., Segu, M., Scaramuzza, D.: A general framework for uncertainty estimation in deep learning. IEEE Robot. Autom. Lett. 5(2), 3153–3160 (2020)

    Article  Google Scholar 

  19. Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: a continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)

  20. Mehta, R., Arbel, T.: RS-Net: regression-segmentation 3D CNN for synthesis of full resolution missing brain MRI in the presence of tumours. In: Gooya, A., Goksel, O., Oguz, I., Burgos, N. (eds.) SASHIMI 2018. LNCS, vol. 11037, pp. 119–129. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00536-8_13

    Chapter  Google Scholar 

  21. Mehta, R., Filos, A., Gal, Y., Arbel, T.: Uncertainty evaluation metric for brain tumour segmentation. arXiv preprint arXiv:2005.14262 (2020)

  22. Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2014)

    Article  Google Scholar 

  23. Naeini, M.P., Cooper, G., Hauskrecht, M.: Obtaining well calibrated probabilities using Bayesian binning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29 (2015)

    Google Scholar 

  24. Nalepa, J., Marcinkiewicz, M., Kawulok, M.: Data augmentation for brain-tumor segmentation: a review. Front. Comput. Neurosci. 13, 83 (2019)

    Article  Google Scholar 

  25. Northcutt, C.G., Athalye, A., Mueller, J.: Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749 (2021)

  26. Ovadia, Y., et al.: Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. arXiv preprint arXiv:1906.02530 (2019)

  27. Porter, K.R., McCarthy, B.J., Freels, S., Kim, Y., Davis, F.G.: Prevalence estimates for primary brain tumors in the united states by age, gender, behavior, and histology. Neuro Oncol. 12(6), 520–527 (2010)

    Article  Google Scholar 

  28. Rousseau, A.J., Becker, T., Bertels, J., Blaschko, M.B., Valkenborg, D.: Post training uncertainty calibration of deep networks for medical image segmentation. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1052–1056. IEEE (2021)

    Google Scholar 

  29. Seferbekov, S.: DSB 2018 [ods.ai] topcoders 1st place solution (2018). https://github.com/selimsef/dsb2018_topcoders

  30. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  31. Supplementary Material: The dice AUC, FTP AUC, and FTN AUC for deep ensemble and mc concrete droppath (2021). https://genome.ifmo.ru/files/papers_files/MICCAI2021/UNSURE/AUC.png

  32. Supplementary Material: The prediction and uncertainty visualization (2021). https://genome.ifmo.ru/files/papers_files/MICCAI2021/UNSURE/MRI_uncertainty.png

  33. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

    Google Scholar 

  34. Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 648–656 (2015)

    Google Scholar 

  35. Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T.: Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing 338, 34–45 (2019)

    Article  Google Scholar 

  36. Wang, G., Li, W., Ourselin, S., Vercauteren, T.: Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 178–190. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75238-9_16

    Chapter  Google Scholar 

  37. Wang, G., Li, W., Ourselin, S., Vercauteren, T.: Automatic brain tumor segmentation using convolutional neural networks with test-time augmentation. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 61–72. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_6

    Chapter  Google Scholar 

  38. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)

    Google Scholar 

  39. Zeineldin, R.A., Karar, M.E., Coburger, J., Wirtz, C.R., Burgert, O.: DeepSeg: deep neural network framework for automatic brain tumor segmentation using magnetic resonance flair images. Int. J. Comput. Assist. Radiol. Surg. 15(6), 909–920 (2020). https://doi.org/10.1007/s11548-020-02186-z

    Article  Google Scholar 

  40. Zhang, Z., Dalca, A.V., Sabuncu, M.R.: Confidence calibration for convolutional neural networks using structured dropout. arXiv preprint arXiv:1906.09551 (2019)

  41. Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)

    Google Scholar 

  42. Zou, K.H., et al.: Statistical validation of image segmentation quality based on a spatial overlap index1: scientific reports. Acad. Radiol. 11(2), 178–189 (2004)

    Article  Google Scholar 

Download references

Acknowledgments

This work is financially supported by National Center for Cognitive Research of ITMO University and Russian Science Foundation, Grant 19-19-00696.

The authors thank Ilya Osmakov and Pavel Ulyanov for useful ideas, Alex Farseev, Inna Anokhina, and Tatyana Polevaya for useful comments.

Author information

Authors and Affiliations

  1. Machine Learning Lab, ITMO University, 49 Kronverksky Pr, 197101, St. Petersburg, Russia

    Natalia Khanzhina, Maxim Kashirin & Andrey Filchenkov

Authors
  1. Natalia Khanzhina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maxim Kashirin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrey Filchenkov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Natalia Khanzhina .

Editor information

Editors and Affiliations

  1. University College London/King's College London, London, UK

    Carole H. Sudre

  2. Medical University of Vienna and TU Wien, Vienna, Austria

    Roxane Licandro

  3. University of Tübingen, Tübingen, Germany

    Christian Baumgartner

  4. King's College London, London, UK

    Andrew Melbourne

  5. Massachusetts General Hospital, Harvard Medical School, MIT, Cambridge, MA, USA

    Adrian Dalca

  6. King's College London, London, UK

    Jana Hutter

  7. Microsoft Research/University College London, London, UK

    Ryutaro Tanno

  8. Boston Children's Hospital, Boston, MA, USA

    Esra Abaci Turk

  9. Technical University Denmark, Kongens Lyngby, Denmark

    Koen Van Leemput

  10. Hewlett Packard, Barcelona, Spain

    Jordina Torrents Barrena

  11. Harvard Medical School/Brigham and Women's Hospital, Boston, MA, USA

    William M. Wells

  12. The Hospital For Sick Children, University of Toronto, Toronto, ON, Canada

    Christopher Macgowan

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Khanzhina, N., Kashirin, M., Filchenkov, A. (2021). Monte Carlo Concrete DropPath for Epistemic Uncertainty Estimation in Brain Tumor Segmentation. In: Sudre, C.H., et al. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis. UNSURE PIPPI 2021 2021. Lecture Notes in Computer Science(), vol 12959. Springer, Cham. https://doi.org/10.1007/978-3-030-87735-4_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87735-4_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87734-7

  • Online ISBN: 978-3-030-87735-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation