Skip to main content
SpringerLink
Log in

Multi-modal U-Nets with Boundary Loss and Pre-training for Brain Tumor Segmentation

  • Conference paper
  • First Online:
Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (BrainLes 2019)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11993))

Included in the following conference series:

Abstract

Gliomas are the most common primary brain tumors, and their manual segmentation is a time-consuming and user-dependent process. We present a two-step multi-modal U-Net-based architecture with unsupervised pre-training and surface loss component for brain tumor segmentation which allows us to seamlessly benefit from all magnetic resonance modalities during the delineation. The results of the experimental study, performed over the newest release of the BraTS test set, revealed that our method delivers accurate brain tumor segmentation, with the average DICE score of 0.72, 0.86, and 0.77 for the enhancing tumor, whole tumor, and tumor core, respectively. The total time required to process one study using our approach amounts to around 20 s.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aljabar, P., Heckemann, R., Hammers, A., Hajnal, J., Rueckert, D.: Multi-atlas based segmentation of brain images: atlas selection and its effect on accuracy. NeuroImage 46(3), 726–738 (2009)

    Article  Google Scholar 

  2. Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Nat. Sci. Data 4, 1–13 (2017). https://doi.org/10.1038/sdata.2017.117

    Article  Google Scholar 

  3. Bakas, S., et al.: Segmentation labels and radiomic features for the pre-operative scans of the TCGA-GBM collection (2017). the Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2017.KLXWJJ1Q

  4. Bakas, S., et al..: Segmentation labels and radiomic features for the pre-operative scans of the TCGA-LGG collection (2017). the Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2017.GJQ7R0EF

  5. Bakas, S., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. CoRR abs/1811.02629 (2018). http://arxiv.org/abs/1811.02629

  6. Bauer, S., Seiler, C., Bardyn, T., Buechler, P., Reyes, M.: Atlas-based segmentation of brain tumor images using a Markov random field-based tumor growth model and non-rigid registration. In: Proceedings of IEEE EMBC, pp. 4080–4083 (2010). https://doi.org/10.1109/IEMBS.2010.5627302

  7. Cabezas, M., Oliver, A., Lladó, X., Freixenet, J., Cuadra, M.B.: A review of atlas-based segmentation for magnetic resonance brain images. Comput. Methods Programs Biomed. 104(3), e158–e177 (2011)

    Article  Google Scholar 

  8. Chander, A., Chatterjee, A., Siarry, P.: A new social and momentum component adaptive PSO algorithm for image segmentation. Expert Syst. Appl. 38(5), 4998–5004 (2011)

    Article  Google Scholar 

  9. Dai, L., Li, T., Shu, H., Zhong, L., Shen, H., Zhu, H.: Automatic brain tumor segmentation with domain adaptation. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 380–392. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_34

    Chapter  Google Scholar 

  10. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11, 625–660 (2010). http://dl.acm.org/citation.cfm?id=1756006.1756025

    MathSciNet  MATH  Google Scholar 

  11. Fan, X., Yang, J., Zheng, Y., Cheng, L., Zhu, Y.: A novel unsupervised segmentation method for MR brain images based on fuzzy methods. In: Liu, Y., Jiang, T., Zhang, C. (eds.) CVBIA 2005. LNCS, vol. 3765, pp. 160–169. Springer, Heidelberg (2005). https://doi.org/10.1007/11569541_17

    Chapter  Google Scholar 

  12. Fang, L., He, H.: Three pathways U-Net for brain tumor segmentation. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries - 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, Pre-Conference Proceedings, pp. 119–126 (2018)

    Google Scholar 

  13. Geremia, E., Clatz, O., Menze, B.H., Konukoglu, E., Criminisi, A., Ayache, N.: Spatial decision forests for MS lesion segmentation in multi-channel magnetic resonance images. NeuroImage 57(2), 378–390 (2011)

    Article  Google Scholar 

  14. Ghafoorian, M., et al.: Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities. CoRR abs/1610.04834 (2016). http://arxiv.org/abs/1610.04834

  15. Ghafoorian, M., et al.: Transfer learning for domain adaptation in MRI: application in brain lesion segmentation. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 516–524. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_59

    Chapter  Google Scholar 

  16. Gholipour, A., Kehtarnavaz, N., Briggs, R., Devous, M., Gopinath, K.: Brain functional localization: a survey of image registration techniques. IEEE Trans. Med. Imaging 26(4), 427–451 (2007). https://doi.org/10.1109/TMI.2007.892508

    Article  Google Scholar 

  17. Havaei, M., Dutil, F., Pal, C., Larochelle, H., Jodoin, P.-M.: A convolutional neural network approach to brain tumor segmentation. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Handels, H. (eds.) BrainLes 2015. LNCS, vol. 9556, pp. 195–208. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30858-6_17

    Chapter  Google Scholar 

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, June 2016. https://doi.org/10.1109/CVPR.2016.90

  19. Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., Maier-Hein, K.H.: No new-net. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 234–244. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_21

    Chapter  Google Scholar 

  20. Ji, S., Wei, B., Yu, Z., Yang, G., Yin, Y.: A new multistage medical segmentation method based on superpixel and fuzzy clustering. Comput. Math. Methods Med. 747549:1–747549:13 (2014)

    Google Scholar 

  21. Kamnitsas, K., et al.: Ensembles of multiple models and architectures for robust brain tumour segmentation. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 450–462. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75238-9_38

    Chapter  Google Scholar 

  22. Kervadec, H., Bouchtiba, J., Desrosiers, C., Granger, E., Dolz, J., Ben Ayed, I.: Boundary loss for highly unbalanced segmentation. In: Cardoso, M.J., et al. (eds.) Proceedings of The 2nd International Conference on Medical Imaging with Deep Learning. Proceedings of Machine Learning Research, vol. 102, pp. 285–296. PMLR, London, 08–10 July 2019. http://proceedings.mlr.press/v102/kervadec19a.html

  23. Korfiatis, P., Kline, T.L., Erickson, B.J.: Automated segmentation of hyperintense regions in FLAIR MRI using deep learning. Tomogr.: J. Imaging Res. 2(4), 334–340 (2016). https://doi.org/10.18383/j.tom.2016.00166

    Article  Google Scholar 

  24. Ladgham, A., Torkhani, G., Sakly, A., Mtibaa, A.: Modified support vector machines for MR brain images recognition. In: Proceedings of CoDIT, pp. 032–035 (2013). https://doi.org/10.1109/CoDIT.2013.6689515

  25. Lorenzo, P.R., et al.: Segmenting brain tumors from FLAIR MRI using fully convolutional neural networks. Comput. Methods Programs Biomed. 176, 135–148 (2019). https://doi.org/10.1016/j.cmpb.2019.05.006. http://www.sciencedirect.com/science/article/pii/S0169260718315955

    Article  Google Scholar 

  26. Marcinkiewicz, M., Nalepa, J., Lorenzo, P.R., Dudzik, W., Mrukwa, G.: Segmenting brain tumors from MRI using cascaded multi-modal U-Nets. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 13–24. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_2

    Chapter  Google Scholar 

  27. McKinley, R., Meier, R., Wiest, R.: Ensembles of densely-connected CNNs with label-uncertainty for brain tumor segmentation. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 456–465. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_40

    Chapter  Google Scholar 

  28. Mei, P.A., de Carvalho Carneiro, C., Fraser, S.J., Min, L.L., Reis, F.: Analysis of neoplastic lesions in magnetic resonance imaging using self-organizing maps. J. Neurol. Sci. 359(1–2), 78–83 (2015)

    Article  Google Scholar 

  29. Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2015). https://doi.org/10.1109/TMI.2014.2377694

    Article  Google Scholar 

  30. Milletari, F., Navab, N., Ahmadi, S.: V-net: fully convolutional neural networks for volumetric medical image segmentation. CoRR abs/1606.04797 (2016). http://arxiv.org/abs/1606.04797

  31. Moeskops, P., Viergever, M.A., Mendrik, A.M., de Vries, L.S., Benders, M.J.N.L., Isgum, I.: Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans. Med. Imaging 35(5), 1252–1261 (2016). https://doi.org/10.1109/TMI.2016.2548501

    Article  Google Scholar 

  32. Myronenko, A.: 3D MRI brain tumor segmentation using autoencoder regularization. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 311–320. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_28

    Chapter  Google Scholar 

  33. Nalepa, J., et al.: Data augmentation via image registration. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 4250–4254, September 2019. https://doi.org/10.1109/ICIP.2019.8803423

  34. Park, M.T.M., et al.: Derivation of high-resolution MRI atlases of the human cerebellum at 3T and segmentation using multiple automatically generated templates. NeuroImage 95, 217–231 (2014)

    Article  Google Scholar 

  35. Pinto, A., Pereira, S., Correia, H., Oliveira, J., Rasteiro, D.M.L.D., Silva, C.A.: Brain tumour segmentation based on extremely rand. forest with high-level features. In: Proceedings of IEEE EMBC, pp. 3037–3040 (2015). https://doi.org/10.1109/EMBC.2015.7319032

  36. Pipitone, J., et al.: Multi-atlas segmentation of the whole hippocampus and subfields using multiple automatically generated templates. NeuroImage 101, 494–512 (2014)

    Article  Google Scholar 

  37. Rajendran, A., Dhanasekaran, R.: Fuzzy clustering and deformable model for tumor segmentation on MRI brain image: a combined approach. Procedia Eng. 30, 327–333 (2012)

    Article  Google Scholar 

  38. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. CoRR abs/1505.04597 (2015)

    Google Scholar 

  39. Saha, S., Bandyopadhyay, S.: MRI brain image segmentation by fuzzy symmetry based genetic clustering technique. In: Proceedings of IEEE CEC, pp. 4417–4424 (2007)

    Google Scholar 

  40. Sembiring, R.W., Zain, J.M., Embong, A.: Dimension reduction of health data clustering. CoRR abs/1110.3569 (2011). http://arxiv.org/abs/1110.3569

  41. Simi, V., Joseph, J.: Segmentation of glioblastoma multiforme from MR images - a comprehensive review. Egypt. J. Radiol. Nuclear Med. 46(4), 1105–1110 (2015)

    Article  Google Scholar 

  42. Soltaninejad, M., et al.: Automated brain tumour detection and segmentation using superpixel-based extremely randomized trees in FLAIR MRI. Int. J. Comput. Assist. Radiol. Surg. 12(2), 183–203 (2017). https://doi.org/10.1007/s11548-016-1483-3

    Article  Google Scholar 

  43. Taherdangkoo, M., Bagheri, M.H., Yazdi, M., Andriole, K.P.: An effective method for segmentation of MR brain images using the ant colony optimization algorithm. J. Digit. Imaging 26(6), 1116–1123 (2013). https://doi.org/10.1007/s10278-013-9596-5

    Article  Google Scholar 

  44. Verma, N., Cowperthwaite, M.C., Markey, M.K.: Superpixels in brain MR image analysis. In: Proc. IEEE EMBC. pp. 1077–1080 (2013). https://doi.org/10.1109/EMBC.2013.6609691

  45. Wadhwa, A., Bhardwaj, A., Verma, V.S.: A review on brain tumor segmentation of MRI images. Magn. Reson. Imaging 61, 247–259 (2019)

    Article  Google Scholar 

  46. Wu, W., Chen, A.Y.C., Zhao, L., Corso, J.J.: Brain tumor detection and segmentation in a CRF framework with pixel-pairwise affinity and superpixel-level features. Int. J. Comput. Assist. Radiol. Surg. 9(2), 241–253 (2014). https://doi.org/10.1007/s11548-013-0922-7

    Article  Google Scholar 

  47. Zhao, X., Wu, Y., Song, G., Li, Z., Zhang, Y., Fan, Y.: A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. CoRR abs/1702.04528 (2017)

    Google Scholar 

  48. Zhuge, Y., et al.: Brain tumor segmentation using holistically nested neural networks in MRI images. Med. Phys. 44, 1–10 (2017). https://doi.org/10.1002/mp.12481

    Article  Google Scholar 

  49. Zikic, D., et al.: Decision forests for tissue-specific segmentation of high-grade gliomas in multi-channel MR. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7512, pp. 369–376. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33454-2_46

    Chapter  Google Scholar 

Download references

Acknowledgments

This research was supported by the Silesian University of Technology (PRL: BKM-556/RAU2/2018, JN: 02/020/BKM19/0183, 02/020/RGH19/0185). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the computing resources used for this research.

This paper is in memory of Dr. Grzegorz Nalepa, an extraordinary scientist, pediatric hematologist/oncologist, and a compassionate champion for kids at Riley Hospital for Children, Indianapolis, USA, who helped countless patients and their families through some of the most challenging moments of their lives. JN thanks Dana K. Mitchell for lots of inspiring discussions on (not only) brain MRI analysis.

Author information

Authors and Affiliations

  1. NVIDIA, Warsaw, Poland

    Pablo Ribalta Lorenzo

  2. Silesian University of Technology, Gliwice, Poland

    Pablo Ribalta Lorenzo & Jakub Nalepa

  3. Netguru, Poznan, Poland

    Michal Marcinkiewicz

Authors
  1. Pablo Ribalta Lorenzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michal Marcinkiewicz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jakub Nalepa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jakub Nalepa .

Editor information

Editors and Affiliations

  1. University Hospital of Zurich, Zurich, Switzerland

    Alessandro Crimi

  2. University of Pennsylvania, Philadelphia, PA, USA

    Spyridon Bakas

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ribalta Lorenzo, P., Marcinkiewicz, M., Nalepa, J. (2020). Multi-modal U-Nets with Boundary Loss and Pre-training for Brain Tumor Segmentation. In: Crimi, A., Bakas, S. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2019. Lecture Notes in Computer Science(), vol 11993. Springer, Cham. https://doi.org/10.1007/978-3-030-46643-5_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-46643-5_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-46642-8

  • Online ISBN: 978-3-030-46643-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation