Skip to main content
SpringerLink
Log in

Uncertainty-guided transformer for brain tumor segmentation

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Multi-model data can enhance brain tumor segmentation for the rich information it provides. However, it also introduces some redundant information that interferes with the segmentation estimation, as some modalities may catch features irrelevant to the tissue of interest. Besides, the ambiguous boundaries and irregulate shapes of different grade tumors lead to a non-confidence estimate of segmentation quality. Given these concerns, we exploit an uncertainty-guided U-shaped transformer with multiple heads to construct drop-out format masks for robust training. Specifically, our drop-out masks are composed of boundary mask, prior probability mask, and conditional probability mask, which can help our approach focus more on uncertainty regions. Extensive experimental results show that our method achieves comparable or higher results than previous state-of-the-art brain tumor segmentation methods, achieving average dice coefficients of \(91.37\%\) and Hausdorff distance of 4.91 on the BraTS2021 dataset. Our code is freely available at https://github.com/chaineypung/BTS-UGT

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availibility

The data that support the findings of this study are openly available at https://www.kaggle.com/datasets/dschettler8845/brats-2021-task1

References

  1. Deo S, Sharma J, Kumar S (2022) Globocan 2020 report on global cancer burden: challenges and opportunities for surgical oncologists. Ann Surg Oncol 29(11):6497–6500

    Article  CAS  PubMed  Google Scholar 

  2. Farmanfarma K. K., M, Mohammadian Shahabinia Z, et al. (2019) “Brain cancer in the world: an epidemiological review,” World Cancer Research Journal 6(5),

  3. Hoover J.M, Morris J.M, and Meyer F.B, (2011) “Use of preoperative magnetic resonance imaging t1 and t2 sequences to determine intraoperative meningioma consistency,” Surg Neurol Int 2

  4. Zhang D, Huang G, Zhang Q et al (2021) Cross-modality deep feature learning for brain tumor segmentation. Pattern Recognition 110 107562

  5. Zhang Y, Yang J, Tian J, et al. (2021) “Modality-aware mutual learning for multi-modal medical image segmentation,” International Conference on Medical Image Computing and Computer-Assisted Intervention, 589–599

  6. Jin B, Cruz L, Gonçalves N (2020) Deep facial diagnosis: deep transfer learning from face recognition to facial diagnosis. IEEE Access 8:123649–123661

    Article  Google Scholar 

  7. Zheng Q, Zhao P, Li Y et al (2021) Spectrum interference-based two-level data augmentation method in deep learning for automatic modulation classification. Neural Comput Appl 33(13):7723-7745

    Article  Google Scholar 

  8. Zheng Q, Zhao P, Zhang D et al (2021) Mr-dcae: manifold regularization-based deep convolutional autoencoder for unauthorized broadcasting identification. Int J Intell Syst 36(12):7204–7238

    Article  Google Scholar 

  9. Xiao Z, Xu X, Xing H et al (2021) Rtfn: a robust temporal feature network for time series classification. Information sciences 571:65–86

    Article  Google Scholar 

  10. Zheng Q, Tian X, Yang M et al (2020) Pac-bayesian framework based drop-path method for 2d discriminative convolutional network pruning. Multidimens Syst Signal 31(3):793–827

    Article  Google Scholar 

  11. Jiang Z, Ding C, Liu M et al (2019) Two-stage cascaded u-net: 1st place solution to brats challenge 2019 segmentation task in International MICCAI brainlesion workshop, 231–241. Springer

  12. Isensee F, Jäger P.F., Full P.M., et al. (2020) “nnu-net for brain tumor segmentation,” International MICCAI Brainlesion Workshop, 118–132

  13. Isensee F, Jaeger PF, Kohl SA et al (2021) nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2):203–211

    Article  CAS  PubMed  Google Scholar 

  14. Wang W, Chen C, Ding M, et al. (2021) “Transbts: multimodal brain tumor segmentation using transformer,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, 109–119

  15. Hatamizadeh A, Tang Y, Nath V, et al. (2022) “Unetr: transformers for 3d medical image segmentation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 574–584

  16. Zheng Q, Zhao P, Wang H et al (2022) Fine-grained modulation classification using multi-scale radio transformer with dual-channel representation. IEEE Communications Letters 26(6):1298–1302

    Article  Google Scholar 

  17. Touvron H, Cord M, Douze M, et al. (2021) “Training data-efficient image transformers & distillation through attention,” in International Conference on Machine Learning, 10347–10357

  18. Zheng S, Lu J, Zhao H, et al. (2021) “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6881–6890

  19. Chen J, Lu Y, Yu Q, et al. (2021) “Transunet: transformers make strong encoders for medical image segmentation,” arXiv:2102.04306

  20. Liu Z, Lin Y, Cao Y, et al. (2021) “Swin transformer: hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference o Computer Vision, 10012–10022

  21. Goetz M, Weber C, Binczyk F et al (2015) Dalsa: domain adaptation for supervised learning from sparsely annotated mr images. IEEE transactions on medical imaging 35(1):184–196

    Article  PubMed  Google Scholar 

  22. McKinley R, Meier R, Wiest R (2018) Ensembles of densely-connected cnns with label-uncertainty for brain tumor segmentation, in International MICCAI Brainlesion Workshop, 456–465. Springer

    Google Scholar 

  23. Jungo A, McKinley R, Meier R et al (2017) Towards uncertainty-assisted brain tumor segmentation and survival prediction, in International MICCAI Brainlesion Workshop, 474–485. Springer

  24. Lakshminarayanan B, Pritzel A, and Blundell C, (2017) “Simple and scalable predictive uncertainty estimation using deep ensembles,” Advances in neural information processing systems 30

  25. Gal Y, and Ghahramani Z, (2016) “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning, 1050–1059, PMLR

  26. Amersfoort J. Van, Smith L, Teh Y. W, et al. (2020) “Uncertainty estimation using a single deep deterministic neural network,” in International conference on machine learning, 9690–9700, PMLR

  27. Sensoy M, Kaplan L, and Kandemir M, (2018) “Evidential deep learning to quantify classification uncertainty,” Advances in neural information processing systems 31

  28. McKinley R, Rebsamen M, Meier R et al (2020) Triplanar ensemble of 3d-to-2d cnns with label-uncertainty for brain tumor segmentation, in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 5th International Workshop, BrainLes 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019, Revised Selected Papers, Part I 5, 379–387. Springer

  29. Mehrtash A, Wells WM, Tempany CM et al (2020) Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE transactions on medical imaging 39(12):3868–3878

    Article  PubMed  PubMed Central  Google Scholar 

  30. Jungo A, Meier R, Ermis E et al (2018) On the effect of inter-observer variability for a reliable estimation of uncertainty of medical image segmentation, in Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16–20, 2018, Proceedings, Part I, 682–690. Springer

  31. Nair T, Precup D, Arnold DL et al (2020) Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical image analysis 59 101557

  32. Kohl S, Romera-Paredes B, Meyer C, et al. (2018) “A probabilistic u-net for segmentation of ambiguous images,” Adv Neural Inf Process 31

  33. Mukhoti J, Amersfoort J, van Torr P.H., et al. (2021) “Deep deterministic uncertainty for semantic segmentation,” arXiv:2111.00079

  34. Peiris H, Hayat M, Chen Z, et al. (2021) “A volumetric transformer for accurate 3d tumor segmentation,” arXiv:2111.13300

  35. Dosovitskiy A, Beyer L, Kolesnikov A, et al. (2020) “An image is worth 16x16 words: transformers for image recognition at scale,” arXiv:2010.11929

  36. Yang F, Zhai Q, Li X, et al. (2021) “Uncertainty-guided transformer reasoning for camouflaged object detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 4146–4155

  37. Myronenko A (2018) “3d mri brain tumor segmentation using autoencoder regularization,” in International MICCAI Brainlesion Workshop, 311–320

  38. Taha AA, Hanbury A (2015) Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC medical imaging 15(1):1–28

    Article  Google Scholar 

  39. Menze BH, Jakab A, Bauer S et al (2014) The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging 34(10):1993–2024

    Article  PubMed  PubMed Central  Google Scholar 

  40. Chang J, Zhang X, Ye M, et al. (2018) “Brain tumor segmentation based on 3d unet with multi-class focal loss,” in 2018 11th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics, 1–5

  41. Chen Z, Xie L, Chen Y et al. (2021) “Generative adversarial network based cerebrovascular segmentation for time-of-flight magnetic resonance angiography image,” Neurocomputing

  42. Chen C, Liu X, Ding M, et al. (2019) “3d dilated multi-fiber network for real-time brain tumor segmentation in mri,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, 184–192

  43. Shen H, Wang R, Zhang J, et al. (2017) “Boundary-aware fully convolutional network for brain tumor segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, 433–441, Springer

  44. Xiao Z, Zhang H, Tong H, et al. (2022) “An efficient temporal network with dual self-distillation for electroencephalography signal classification,” in 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 1759–1762, IEEE

  45. Xing H, Xiao Z, Zhan D et al (2022) Selfmatch: robust semisupervised time-series classification with self-distillation. Int J Intell Syst 37(11):8583–8610

    Article  Google Scholar 

  46. Cheng J, Liu J, Kuang H et al (2022) A fully automated multimodal mri-based multi-task learning for glioma segmentation and idh genotyping. IEEE Transactions on Medical Imaging 41(6):1520–1532

    Article  PubMed  Google Scholar 

  47. Tanno R, Saeedi A, Sankaranarayanan S et al. (2019) “Learning from noisy labels by regularized estimation of annotator confusion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11244–11253

Download references

Acknowledgements

This research was sponsored in part by the National Natural Science Foundation of China (Grant No. 62002327, 61976190, 62073294, U22A2040); Natural Science Foundation of Zhejiang Province (Grant No. LQ21F020017, LZ21F030003); The Key Technology Research and Development Program of Zhejiang Province (Grant No. 2020C03070)

Author information

Authors and Affiliations

  1. College of Information Engineering, Zhejiang University of Technology, Hangzhou, 310023, China

    Zan Chen, Chenxu Peng, Wenlong Guo, Lei Xie & Yuanjing Feng

  2. Paul C. Lauterbur Research Center for Biomedical Imaging, SIAT, CAS Shenzhen, 518055, China

    Shanshan Wang

  3. First Affilated Hospital of Wenzhou Medical University, Wenzhou, 325000, China

    Qichuan Zhuge & Caiyun Wen

  4. Zhejiang Provincial United Key Laboratory of Embedded Systems, Hangzhou, 310023, China

    Yuanjing Feng

Authors
  1. Zan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chenxu Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenlong Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shanshan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qichuan Zhuge
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Caiyun Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yuanjing Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Zan Chen: writing—original draft, methodology, investigation, validation, formal analysis, funding acquisition. Chenxu Peng: writing—original draft, methodology, software, validation, visualization. Wenlong Guo: software, validation, visualization. Lei Xie: investigation, validation, data curation. Qichuan Zhuge: resources, supervision, data curation. Caiyun Wen: investigation, resources, project administration. Yuanjing Feng: writing—review and editing, conceptualization, methodology, supervision, project administration, formal analysis, funding acquisition.

Corresponding author

Correspondence to Yuanjing Feng.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Z., Peng, C., Guo, W. et al. Uncertainty-guided transformer for brain tumor segmentation. Med Biol Eng Comput 61, 3289–3301 (2023). https://doi.org/10.1007/s11517-023-02899-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02899-8

Keywords

Search

Navigation