Skip to main content
SpringerLink
Log in

Multi-modality encoded fusion with 3D inception U-net and decoder model for brain tumor segmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With deep learning playing a crucial role in biomedical image segmentation, multi-modality fusion-based techniques have gained rapid growth. For any radiologist, identification and segmentation of brain tumor (gliomas) via multi-sequence 3D volumetric MRI scan for diagnosis, monitoring, and treatment, are complex and time-consuming tasks. The brain tumor segmentation (BraTS) challenge offers 3D volumes of high-graded gliomas (HGG), and low-graded gliomas (LGG) MRI scans with four modalities: T1, T1c, T2 and FLAIR. In this article, the tumor segmentation is performed on the preprocessed multi-modalities by proposed 3D deep neural network components: multi-modalities fusion, tumor extractor, and tumor segmenter. The multi-modalities fusion component uses the deep inception based encoding to merge the multi-modalities. Tumor extractor component is passed with the fused images to recognise the tumor patterns using the 3D inception U-Net model. Finally, tumor segmenter utilises the progressive approach to decode the extracted feature maps into the tumor regions. The architecture segments each lesion region into the whole tumor (WT), core tumor (CT), and enhancing tumor (ET) using the five target classes: background, necrosis, edema, enhancing tumor and non-enhancing tumor. To tackle the class imbalance problem, the weighted segmentation loss function is proposed based on the dice coefficient and the Jaccard index. This article illustrates the significance of each component on the BraTS 2017 and 2018 datasets by achieving better segmentation results. The performance of the proposed approach is also compared with the other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Avants BB, Tustison N, Song G (2009) Advanced normalization tools (ants). Insight J 2(365):1–35

    Google Scholar 

  2. Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby JS, Freymann JB, Farahani K, Davatzikos C (2017) Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Sci Data 4:170117

    Article  Google Scholar 

  3. Bakas S, Reyes M, Jakab A, Bauer S, Rempfler M, Crimi A, Shinohara RT, Berger C, Ha SM, Rozycki M, et al. (2018) Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv:1811.02629

  4. Brownlee J (2018) A gentle introduction to early stopping to avoid overtraining neural networks. https://machinelearningmastery.com/early-stopping-to-avoid-overtraining-neural-network-models/. [Online; accessed March 02, 2020]

  5. Chen C, Liu X, Ding M, Zheng J, Li J (2019) 3d dilated multi-fiber network for real-time brain tumor segmentation in mri. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 184–192

  6. Chen L, Wu Y, DSouza AM, Abidin AZ, Wismüller A, Xu C (2018) Mri tumor segmentation with densely connected 3d cnn. In: Medical imaging 2018: image processing. International society for optics and photonics, vol 10574, p 105741F

  7. Chollet F (2015) Keras: deep learning for humans. https://github.com/fchollet/keras. [Online; accessed March 02, 2020]

  8. Drozdzal M, Vorontsov E, Chartrand G, Kadoury S, Pal C (2016) The importance of skip connections in biomedical image segmentation. In: Deep learning and data labeling for medical applications. Springer, Berlin, pp 179–187

  9. Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, Cui C, Corrado G, Thrun S, Dean J (2019) A guide to deep learning in healthcare. Nat Med 25(1):24–29

    Article  Google Scholar 

  10. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  11. Guo Z, Li X, Huang H, Guo N, Li Q (2019) Deep learning-based image segmentation on multimodal medical imaging. IEEE Trans Radiat Plasma Med Sci 3(2):162–169

    Article  Google Scholar 

  12. Hatamizadeh A, Terzopoulos D, Myronenko A (2019) End-to-end boundary aware networks for medical image segmentation. In: International workshop on machine learning in medical imaging. Springer, Berlin, pp 187–194

  13. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580

  14. Isensee F, Kickingereder P, Wick W, Bendszus M, Maier-Hein KH (2018) No new-net. In: International MICCAI brainlesion workshop. Springer, Berlin, pp 234–244

  15. Juntu J, Sijbers J, Van Dyck D, Gielen J (2005) Bias field correction for mri images. In: Computer recognition systems. Springer, Berlin, pp 543–551

  16. Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Med Image Anal 36:61–78

    Article  Google Scholar 

  17. Ker J, Wang L, Rao J, Lim T (2017) Deep learning applications in medical image analysis. IEEE Access 6:9375–9389

    Article  Google Scholar 

  18. Lahat D, Adali T, Jutten C (2015) Multimodal data fusion: an overview of methods, challenges, and prospects. Proc IEEE 103(9):1449–1477

    Article  Google Scholar 

  19. Maas AL, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic models. In: Proc icml, vol 30, p 3

  20. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, et al. (2014) The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans Med Imaging 34(10):1993–2024

    Article  Google Scholar 

  21. Miotto R, Wang F, Wang S, Jiang X, Dudley JT (2018) Deep learning for healthcare: review, opportunities and challenges. Brief Bioinform 19 (6):1236–1246

    Article  Google Scholar 

  22. Myronenko A (2018) 3d mri brain tumor segmentation using autoencoder regularization. In: International MICCAI brainlesion workshop. Springer, Berlin, pp 311–320

  23. Preston DC (2006) Magnetic resonance imaging (mri) of the brain and spine: basics. MRI Basics, Case Med 30

  24. Punn NS, Agarwal S (2020) Inception u-net architecture for semantic segmentation to identify nuclei in microscopy cell images. ACM Trans Multimed Comput Commun Appl (TOMM) 16(1):1–15

    Article  Google Scholar 

  25. Qin Y, Kamnitsas K, Ancha S, Nanavati J, Cottrell G, Criminisi A, Nori A (2018) Autofocus layer for semantic segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 603–611

  26. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 234–241

  27. Ruder S (2016) An overview of gradient descent optimization algorithms. arXiv:1609.04747

  28. Sibi P, Jones SA, Siddarth P (2013) Analysis of different activation functions using back propagation neural networks. J Theor Appl Inf Technol 47 (3):1264–1268

    Google Scholar 

  29. Song J, Zhang Z (2019) Brain tissue segmentation and bias field correction of mr image based on spatially coherent fcm with nonlocal constraints. Computational and mathematical methods in medicine 2019

  30. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  31. Tompson J, Goroshin R, Jain A, LeCun Y, Bregler C (2015) Efficient object localization using convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 648–656

  32. Tustison NJ, Shrinidhi K, Wintermark M, Durst CR, Kandel BM, Gee JC, Grossman MC, Avants BB (2015) Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (simplified) with antsr. Neuroinformatics 13(2):209–225

    Article  Google Scholar 

  33. Ulyanov D, Vedaldi A, Lempitsky V (2016) Instance normalization: the missing ingredient for fast stylization. arXiv:1607.08022

  34. Wang G, Li W, Ourselin S, Vercauteren T (2017) Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. In: International MICCAI brainlesion workshop. Springer, Berlin, pp 178–190

  35. Yang HY, Yang J (2018) Automatic brain tumor segmentation with contour aware residual network and adversarial training. In: International MICCAI brainlesion workshop. Springer, Berlin, pp 267–278

  36. Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122

  37. Yu L, Yang X, Chen H, Qin J, Heng PA (2017) Volumetric convnets with mixed residual connections for automated prostate segmentation from 3d mr images. In: Thirty-first AAAI conference on artificial intelligence

  38. Zhou C, Ding C, Lu Z, Wang X, Tao D (2018) One-pass multi-task convolutional neural networks for efficient brain tumor segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 637–645

  39. Zhou X, Takayama R, Wang S, Hara T, Fujita H (2017) Deep learning of the sectional appearances of 3d ct images for anatomical structure segmentation based on an fcn voting method. Med Phys 44(10):5221–5233

    Article  Google Scholar 

Download references

Acknowledgements

We thank our institute, Indian Institute of Information Technology Allahabad (IIITA), India and Big Data Analytics (BDA) lab for allocating the centralised computing facility and other necessary resources to perform this research. We extend our thanks to our colleagues for their valuable guidance and suggestions.

Author information

Authors and Affiliations

  1. IIIT Allahabad, Prayagraj, 211015, India

    Narinder Singh Punn & Sonali Agarwal

Authors
  1. Narinder Singh Punn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sonali Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Narinder Singh Punn.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Punn, N.S., Agarwal, S. Multi-modality encoded fusion with 3D inception U-net and decoder model for brain tumor segmentation. Multimed Tools Appl 80, 30305–30320 (2021). https://doi.org/10.1007/s11042-020-09271-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09271-0

Keywords

Search

Navigation