Skip to main content

Advertisement

SpringerLink
Log in

MFU-Net: a deep multimodal fusion network for breast cancer segmentation with dual-layer spectral detector CT

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

With the development of medical imaging technologies, breast cancer segmentation remains challenging, especially when considering multimodal imaging. Compared to a single-modality image, multimodal data provide additional information, contributing to better representation learning capabilities. This paper applies these advantages by presenting a deep learning network architecture for segmenting breast cancer with multimodal computed tomography (CT) images based on fusing U-Net architectures that can learn richer representations from multimodal data. The multipath fusion architecture introduces an additional fusion module across different paths, enabling the model to extract features from different modalities at each level of the encoding path. This approach enhances segmentation performance and produces more robust results compared to using a single modality. The study reports experiments conducted on multimodal CT images from 36 patients for training, validation, and testing purposes. The results demonstrate that the proposed model ouperforms the U-Net architecture when considering different combinations of input image modalities. Specifically, when combining two distinct CT modalities, the ZE and IoNW input combination yields the highest Dice score of 0.8546.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

All requests for raw and analyzed data will be made available upon reasonable request for academic use and within the limitations of the provided informed consent by the corresponding author upon acceptance. Every request will be reviewed by the institutional review board of the School of Electrical Engineering of Southwest Jiaotong University and the affiliated hospital of Southwest Medical University.

Notes

  1. https://github.com/bigmb/Unet-Segmentation-Pytorch-Nest-of-Unets

  2. Github code will be made available for reference at https://github.com/AisenCD/dualCT.

  3. https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py

References

  1. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A (2018) Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA a cancer journal for clinicians 68(6):394–424

    Article  Google Scholar 

  2. Zou L, Yu S, Meng T, Zhang Z, Liang X, Xie Y (2019) A technical review of convolutional neural network-based mammographic breast cancer diagnosis. Comput Math Methods Med 2019

  3. Ye C, Wang W, Zhang S, Wang K (2019) Multi-depth fusion network for whole-heart ct image segmentation. IEEE Access 7:23421–23429

    Article  Google Scholar 

  4. Hua C-h, Shapira N, Merchant TE, Klahr P, Yagil Y (2018) Accuracy of electron density, effective atomic number, and iodine concentration determination with a dual-layer dual-energy computed tomography system. Med Phys 45(6):2486–2497

    Article  Google Scholar 

  5. Michael E, Ma H, Li H, Kulwa F, Li J (2021) Breast cancer segmentation methods: current status and future potentials. BioMed Res Int 2021

  6. Bai J, Posner R, Wang T, Yang C, Nabavi S (2021) Applying deep learning in digital breast tomosynthesis for automatic breast cancer detection: a review. Med Image Anal 71:102049

  7. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 234–241

  8. Salama WM, Aly MH (2021) Deep learning in mammography images segmentation and classification: automated cnn approach. Alex Eng J 60(5):4701–4709

  9. Ravitha Rajalakshmi N, Vidhyapriya R, Elango N, Ramesh N (2021) Deeply supervised u-net for mass segmentation in digital mammograms. Int J Imaging Syst Technol 31(1):59–71

    Article  Google Scholar 

  10. Abdelhafiz D, Bi J, Ammar R, Yang C, Nabavi S (2020) Convolutional neural network for automated mass segmentation in mammography. BMC bioinformatics 21(1):1–19

    Google Scholar 

  11. Hossain MS (2019) Microc alcification segmentation using modified u-net segmentation network from mammogram images. J King Saud Univ-Comput Inform Sci

  12. Li J, Dong D, Fang M, Wang R, Tian J, Li H, Gao J (2020) Dual-energy ct-based deep learning radiomics can improve lymph node metastasis risk prediction for gastric cancer. Eur Radiol 30(4):2324–2333

    Article  Google Scholar 

  13. An C, Li D, Li S, Li W, Tong T, Liu L, Jiang D, Jiang L, Ruan G, Hai N et al (2022) Deep learning radiomics of dual-energy computed tomography for predicting lymph node metastases of pancreatic ductal adenocarcinoma. Eur J Nucl Med Mol Imaging 49(4):1187–1199

    Article  Google Scholar 

  14. Wang Y-W, Chen C-J, Wang T-C, Huang H-C, Chen H-M, Shih J-Y, Chen J-S, Huang Y-S, Chang Y-C, Chang R-F (2022) Multi-energy level fusion for nodal metastasis classification of primary lung tumor on dual energy ct using deep learning. Comput Biol Med 141:105185

    Article  Google Scholar 

  15. Zhang W, Li R, Deng H, Wang L, Lin W, Ji S, Shen D (2015) Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage 108:214–224

    Article  Google Scholar 

  16. Dolz J, Ayed IB, Yuan J, Desrosiers C (2018) Isointense infant brain segmentation with a hyper-dense connected convolutional neural network. In: 2018 IEEE 15th International symposium on biomedical imaging (ISBI 2018), IEEE, pp 616–620

  17. Nie D, Wang L, Gao Y, Shen D (2016) Fully convolutional networks for multi-modality isointense infant brain image segmentation. In: 2016 IEEE 13Th International symposium on biomedical imaging (ISBI), IEEE, pp 1342–1345

  18. Dolz J, Desrosiers C, Ben Ayed I (2018) Ivd-net: intervertebral disc localization and segmentation in mri with a multi-modal unet. In: International workshop and challenge on computational methods and clinical applications for Spine imaging, Springer, pp 130–143

  19. Große Hokamp N, Lennartz S, Salem J, Pinto dos Santos D, Heidenreich A, Maintz D, Haneder S (2020) Dose independent characterization of renal stones by means of dual energy computed tomography and machine learning: an ex-vivo study. Eur Radiol 30:1397–1404

    Article  Google Scholar 

  20. Choi B, Choi IY, Yeom Cha SH, SK, Chung HH, Lee SH, Cha J, Lee J-H (2020) Feasibility of computed tomography texture analysis of hepatic fibrosis using dual-energy spectral detector computed tomography. Japan J Radiology 38:1179–1189

  21. Shapira N, Fokuhl J, Schultheiß M, Beck S, Kopp FK, Pfeiffer D, Dangelmaier J, Pahn G, Sauter AP, Renger B et al (2020) Liver lesion localisation and classification with convolutional neural networks: a comparison between conventional and spectral computed tomography. Biomedical Physics & Engineering Express 6(1):015038

    Article  Google Scholar 

  22. Alom MZ, Taha TM, Yakopcic C, Westberg S, Sidike P, Nasrin MS, Van Esesn BC, Awwal AAS, Asari VK (2018) The history began from alexnet: a comprehensive survey on deep learning approaches. arXiv:1803.01164

  23. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  24. Milletari F, Navab N, Ahmadi S-A (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth international conference on 3d vision (3DV), IEEE, pp 565–571

  25. Han X (2017) Automatic liver lesion segmentation using a deep convolutional neural network method. arXiv:1704.07239

  26. Yu S, Xiao D, Frost S, Kanagasingam Y (2019) Robust optic disc and cup segmentation with deep learning for glaucoma detection. Comput Med Imaging Grap 74:61–71

    Article  Google Scholar 

  27. Kong Z, Xiong F, Zhang C, Fu Z, Zhang M, Weng J, Fan M (2020) Automated maxillofacial segmentation in panoramic dental x-ray images using an efficient encoder-decoder network. IEEE Access 8:207822–207833

    Article  Google Scholar 

  28. Pham V-T, Tran T-T, Wang P-C, Chen P-Y, Lo M-T (2021) Ear-unet: a deep learning-based approach for segmentation of tympanic membranes from otoscopic images. Artif Intell Med 115:102065

  29. Gao S-H, Cheng M-M, Zhao K, Zhang X-Y, Yang M-H, Torr P (2019) Res2net: a new multi-scale backbone architecture. IEEE Trans Pattern Anal Mach Intell 43(2):652–662

  30. Qin J, Huang Y, Wen W (2020) Multi-scale feature fusion residual network for single image super-resolution. Neurocomputing 379:334–342

    Article  Google Scholar 

  31. Fu J, Li W, Du J, Huang Y (2021) A multiscale residual pyramid attention network for medical image fusion. Biomed Signal Process Control 66:102488

    Article  Google Scholar 

  32. Ma Y, Qi F, Wang P, Liang F, Lv H, Yu X, Li Z, Xue H, Wang J, Zhang Y (2020) Multiscale residual attention network for distinguishing stationary humans and common animals under through-wall condition using ultra-wideband radar. IEEE Access 8:121572–121583

    Article  Google Scholar 

  33. Zhang H, Wu C, Zhang Z, Zhu Y, Lin H, Zhang Z, Sun Y, He T, Mueller J, Manmatha R et al (2020) Resnest: split-attention networks. arXiv:2004.08955

  34. Zhang J, Zhang J, Hu G, Chen Y, Yu S (2019) Scalenet: a convolutional network to extract multi-scale and fine-grained visual features. IEEE Access 7:147560–147570

  35. Liu H, Cao H, Song E, Ma G, Xu X, Jin R, Jin Y, Hung C-C (2019) A cascaded dual-pathway residual network for lung nodule segmentation in ct images. Physica Medica 63:112–121

    Article  Google Scholar 

  36. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456 PMLR

  37. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034

  38. van Ommen F, Bennink E, Vlassenbroek A, Dankbaar JW, Schilham AM, Viergever MA, de Jong HW (2018) Image quality of conventional images of dual-layer spectral ct: a phantom study. Med Phys 45(7):3031–3042

  39. Ghaffari M, Sowmya A, Oliver R (2019) Automated brain tumor segmentation using multimodal brain scans: a survey based on models submitted to the brats 2012–2018 challenges. IEEE Rev Biomed Eng 13:156–168

    Article  Google Scholar 

  40. Taha AA, Hanbury A (2015) An efficient algorithm for calculating the exact hausdorff distance. IEEE Trans Pattern Anal Mach Intell 37(11):2153–2163

    Article  Google Scholar 

  41. Koonce B, Koonce B (2021) Mobilenetv3. Image recognition and dataset categorization, convolutional neural networks with Swift for Tensorflow, pp 125–144

  42. Ibtehaz N, Rahman MS (2020) Multiresunet: rethinking the u-net architecture for multimodal biomedical image segmentation. Neural Netw 121:74–87

  43. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

Download references

Acknowledgements

This work was financially aided by the Central Universities Pay for Basic Scientific Research (2682021ZTPY027), Natural Science Foundation of China (62173279, U1934221) and Sichuan Science and Technology Program under Grant 2022YFG0247, 2021JDJQ0012, 2020YFQ0057.

Author information

Authors and Affiliations

  1. Institute of Systems Science and Technology, School of Electrical Engineering, Southwest Jiaotong University, Chengdu, 611756, China

    Aisen Yang, Na Qin & Deqing Huang

  2. Department of Radiology, The Affiliated Hospital of Southwest Medical University, Luzhou, 10587, 646000, China

    Lulu Xu & Jian Shu

  3. State Key Laboratory of Air Traffic Management Systems, Nanjing, 210022, China

    Ziyi Liu

Authors
  1. Aisen Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lulu Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Na Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deqing Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ziyi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jian Shu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Na Qin or Jian Shu.

Ethics declarations

Conflicts of interests

All authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, A., Xu, L., Qin, N. et al. MFU-Net: a deep multimodal fusion network for breast cancer segmentation with dual-layer spectral detector CT. Appl Intell 54, 3808–3824 (2024). https://doi.org/10.1007/s10489-023-05090-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05090-6

Keywords

Search

Navigation