Skip to main content

Advertisement

SpringerLink
Log in

Preliminary study of generalized semiautomatic segmentation for 3D voxel labeling of lesions based on deep learning

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

The three-dimensional (3D) voxel labeling of lesions requires significant radiologists’ effort in the development of computer-aided detection software. To reduce the time required for the 3D voxel labeling, we aimed to develop a generalized semiautomatic segmentation method based on deep learning via a data augmentation-based domain generalization framework. In this study, we investigated whether a generalized semiautomatic segmentation model trained using two types of lesion can segment previously unseen types of lesion.

Methods

We targeted lung nodules in chest CT images, liver lesions in hepatobiliary-phase images of Gd-EOB-DTPA-enhanced MR imaging, and brain metastases in contrast-enhanced MR images. For each lesion, the 32 × 32 × 32 isotropic volume of interest (VOI) around the center of gravity of the lesion was extracted. The VOI was input into a 3D U-Net model to define the label of the lesion. For each type of target lesion, we compared five types of data augmentation and two types of input data.

Results

For all considered target lesions, the highest dice coefficients among the training patterns were obtained when using a combination of the existing data augmentation-based domain generalization framework and random monochrome inversion and when using the resized VOI as the input image. The dice coefficients were 0.639 ± 0.124 for the lung nodules, 0.660 ± 0.137 for the liver lesions, and 0.727 ± 0.115 for the brain metastases.

Conclusions

Our generalized semiautomatic segmentation model could label unseen three types of lesion with different contrasts from the surroundings. In addition, the resized VOI as the input image enables the adaptation to the various sizes of lesions even when the size distribution differed between the training set and the test set.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Giger ML, Chan HP, Boone J (2008) Anniversary paper: history and status of CAD and quantitative image analysis: the role of Medical Physics and AAPM. Med Phys 35(12):5799–5820

    Article  Google Scholar 

  2. van Ginneken B, Schaefer-Prokop CM, Prokop M (2011) Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology 261(3):719–732

    Article  Google Scholar 

  3. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  4. Yasaka K, Akai H, Kunimatsu A, Kiryu S, Abe O (2018) Deep learning with convolutional neural network in radiology. Jpn J Radiol 36(4):257–272

    Article  Google Scholar 

  5. Hu Z, Tang J, Wang Z, Zhang K, Zhang L, Sun Q (2018) Deep learning for image-based cancer detection and diagnosis—a survey. Pattern Recognit 83:134–149

    Article  Google Scholar 

  6. Kuhnigk JM, Dicken V, Bornemann L, Bakai A, Wormanns D, Krass S, Peitgen HO (2006) Morphological segmentation and partial volume analysis for volumetry of solid pulmonary lesions in thoracic CT scans. IEEE Trans Med Imaging 25(4):417–434

    Article  Google Scholar 

  7. Smeets D, Loeckx D, Stijnen B, De Dobbelaer B, Vandermeulen D, Suetens P (2010) Semi-automatic level set segmentation of liver tumors combining a spiral-scanning technique with supervised fuzzy pixel classification. Med Image Anal 14(1):13–20

    Article  Google Scholar 

  8. Farag AA, El Munim HE, Graham JH, Farag AA (2013) A novel approach for lung nodules segmentation in chest CT using level sets. IEEE Trans Image Process 22(12):5202–5213

    Article  Google Scholar 

  9. Lassen BC, Jacobs C, Kuhnigk JM, van Ginneken B, van Rikxoort EM (2015) Robust semi-automatic segmentation of pulmonary subsolid nodules in chest computed tomography scans. Phys Med Biol 60(3):1307–1323

    Article  CAS  Google Scholar 

  10. Li W (2015) Automatic segmentation of liver tumor in CT images with deep convolutional neural networks. J Comput Commun 3(11):146

    Article  Google Scholar 

  11. Kamnitsas K, Ledig C, Newcombe VFJ, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal 36:61–78

    Article  Google Scholar 

  12. Alex V, Vaidhya K, Thirunavukkarasu S, Kesavadas C, Krishnamurthi G (2017) Semisupervised learning using denoising autoencoders for brain lesion detection and segmentation. J Med Imaging 4(4):041311

    Article  Google Scholar 

  13. Liu Y, Stojadinovic S, Hrycushko B, Wardak Z, Lau S, Lu W, Yan Y, Jiang SB, Zhen X, Timmerman R, Nedzi L, Gu X (2017) A deep convolutional neural network-based automatic delineation strategy for multiple brain metastases stereotactic radiosurgery. PLoS ONE 12(10):e0185844

    Article  Google Scholar 

  14. Dolz J, Xu X, Rony J, Yuan J, Liu Y, Granger E, Desrosiers C, Zhang X, Ben Ayed I, Lu H (2018) Multiregion segmentation of bladder cancer structures in MRI with progressive dilated convolutional networks. Med Phys 45(12):5482–5493

    Article  Google Scholar 

  15. Liu H, Cao H, Song E, Ma G, Xu X, Jin R, Jin Y, Hung CC (2019) A cascaded dual-pathway residual network for lung nodule segmentation in CT images. Phys Med 63:112–121

    Article  Google Scholar 

  16. Usman M, Lee BD, Byon SS, Kim SH, Lee BI, Shin YG (2020) Volumetric lung nodule segmentation using adaptive ROI with multi-view residual learning. Sci Rep 10(1):12839

    Article  CAS  Google Scholar 

  17. Nomura Y, Miki S, Hayashi N, Hanaoka S, Sato I, Yoshikawa T, Masutani Y, Abe O (2020) Novel platform for development, training, and validation of computer-assisted detection/diagnosis software. Int J Comput Assist Radiol Surg 15(4):661–672

    Article  Google Scholar 

  18. Balaji Y, Sankaranarayanan S, Chellappa R (2018) Metareg: towards domain generalization using meta-regularization. Adv Neural Inf Process Syst 31:998–1008

    Google Scholar 

  19. Chen C, Bai W, Davies RH, Bhuva AN, Manisty CH, Augusto JB, Moon JC, Aung N, Lee AM, Sanghvi MM, Fung K, Paiva JM, Petersen SE, Lukaschuk E, Piechnik SK, Neubauer S, Rueckert D (2020) Improving the generalizability of convolutional neural network-based segmentation on CMR images. Front Cardiovasc Med 7:105

    Article  Google Scholar 

  20. Zhang L, Wang X, Yang D, Sanford T, Harmon S, Turkbey B, Wood BJ, Roth H, Myronenko A, Xu D, Xu Z (2020) Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Trans Med Imaging 39(7):2531–2540

    Article  Google Scholar 

  21. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. Proc ICCV 2017:2223–2232

    Google Scholar 

  22. Shin HC, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM (2016) Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 35(5):1285–1298

    Article  Google Scholar 

  23. Karani N, Chaitanya K, Baumgartner C, Konukoglu E (2018) A lifelong learning approach to brain MR segmentation across scanners and protocols. In: MICCAI 2018, LNCS, vol 11070, pp 476–484

  24. Bermúdez-Chacón R, Márquez-Neila P, Salzmann M, Fua P (2018) A domain-adaptive two-stream U-Net for electron microscopy image segmentation. In: Proceedings of the 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp 400–404

  25. Li K, Yu L, Wang S, Heng PA (2020) Towards cross-modality medical image segmentation with online mutual knowledge distillation. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, no 1, pp 775–783

  26. Dou Q, Ouyang C, Chen C, Chen H, Heng PA (2018) Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversarial loss. In: Proceedings of the 27th international joint conference on artificial intelligence, pp 691–697

  27. Cheng O, Konstantinos K, Carlo B, Jinming D, Daniel R (2019) Data efficient unsupervised domain adaptation for cross-modality image segmentation. In: MICCAI 2019, LNCS, vol 11765, pp 669–677

  28. Liu D, Zhang D, Song Y, Zhang F, O’Donnell L, Huang H, Chen M, Cai W (2021) PDAM: a panoptic-level feature alignment framework for unsupervised domain adaptive instance segmentation in microscopy images. IEEE Trans Med Imaging 40(1):154–165

    Article  Google Scholar 

  29. Chen C, Dou Q, Chen H, Qin J, Heng PA (2019) Synergistic image and feature adaptation: towards cross-modality domain adaptation for medical image segmentation. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, no 1, pp 865–872

  30. Chen C, Dou Q, Chen H, Qin J, Heng PA (2020) Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation. IEEE Trans Med Imaging 39(7):2494–2505

    Article  Google Scholar 

  31. Pham DD, Dovletov G, Pauli J (2020) Liver segmentation in CT with MRI data: zero-shot domain adaptation by contour extraction and shape priors. In: 2020 IEEE 17th international symposium on biomedical imaging (ISBI), pp 1538–1542

  32. Wang S, Yu L, Li K, Yang X, Fu CW, Heng PA (2020) DoFE: domain-oriented feature embedding for generalizable fundus image segmentation on unseen datasets. IEEE Trans Med Imaging 39(12):4237–4248

    Article  Google Scholar 

  33. Liu Q, Chen C, Qin J, Dou Q, Heng PA (2021) FedDG: federated domain generalization on medical image segmentation via episodic learning in continuous Frequency Space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR 2021), pp 1013–1023

  34. Xu Z, Liu D, Yang J, Raffel C, Niethammer M (2021) Robust and generalizable visual representation learning via random convolutions. In: ICLR 2021

  35. Wang S, Yu L, Li C, Fu CW, Heng PA (2020) Learning from extrinsic and intrinsic supervisions for domain generalization. In: ECCV 2020, LNCS, vol 12354, pp 159–176

  36. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. https://arxiv.org/arXiv:1505.04597

  37. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) PyTorch: an imperative style, high-performance deep learning library. https://arxiv.org/arXiv:1912.01703

  38. Nomura Y, Sato I, Hanawa T, Hanaoka S, Nakao T, Takenaga T, Hoshino T, Sekiya Y, Miki S, Yoshikawa T, Hayashi N, Abe O (2020) Development of training environment for deep learning with medical images on supercomputer system based on asynchronous parallel Bayesian optimization. J Supercomput 76:7315–7332

    Article  Google Scholar 

  39. Abraham N, Khan NM A (2019) Novel focal Tversky loss function with improved attention U-Net for lesion segmentation. In: 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019), pp 683–687

  40. Liu L, Jiang H, He P, Chen W, Liu X, Gao J, Han J (2019) On the variance of the adaptive learning rate and beyond. https://arxiv.org/arXiv:1908.03265

  41. Peng S, Chen W, Sun J, Liu B (2020) Multi-scale 3D U-Nets: an approach to automatic segmentation of brain tumor. Int J Imag Syst Technol 30(1):5–17

    Article  Google Scholar 

  42. Zheng S, Lin X, Zhang W, He B, Jia S, Wang P, Jiang H, Shi J, Jia F (2021) MDCC-Net: multiscale double-channel convolution U-Net framework for colorectal tumor segmentation. Comput Biol Med 130:104183

    Article  Google Scholar 

  43. Nomura Y, Hayashi N, Hanaoka S, Takenaga T, Nemoto M, Miki S, Yoshikawa T, Abe O (2019) Can the spherical gold standards be used as an alternative to painted gold standards for the computerized detection of lesions using voxel-based classification? Jpn J Radiol 37(3):264–273

    Article  CAS  Google Scholar 

  44. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, Lanczi L, Gerstner E, Weber MA, Arbel T, Avants BB, Ayache N, Buendia P, Collins DL, Cordier N, Corso JJ, Criminisi A, Das T, Delingette H, Demiralp Ç, Durst CR, Dojat M, Doyle S, Festa J, Forbes F, Geremia E, Glocker B, Golland P, Guo X, Hamamci A, Iftekharuddin KM, Jena R, John NM, Konukoglu E, Lashkari D, Mariz JA, Meier R, Pereira S, Precup D, Price SJ, Raviv TR, Reza SM, Ryan M, Sarikaya D, Schwartz L, Shin HC, Shotton J, Silva CA, Sousa N, Subbanna NK, Szekely G, Taylor TJ, Thomas OM, Tustison NJ, Unal G, Vasseur F, Wintermark M, Ye DH, Zhao L, Zhao B, Zikic D, Prastawa M, Reyes M, Van Leemput K (2015) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 34(10):1993–2024

    Article  Google Scholar 

  45. Philbrick KA, Weston AD, Akkus Z, Kline TL, Korfiatis P, Sakinis T, Kostandy P, Boonrod A, Zeinoddini A, Takahashi N, Erickson BJ (2019) RIL-contour: a medical imaging dataset annotation tool for and with deep learning. J Digit Imaging 32(4):571–581

    Article  Google Scholar 

  46. Yang Z, Liu H, Liu Y, Stojadinovic S, Timmerman R, Nedzi L, Dan T, Wardak Z, Lu W, Gu X (2020) A web-based brain metastases segmentation and labeling platform for stereotactic radiosurgery. Med Phys 47(8):3263–3276

    Article  Google Scholar 

  47. Armato SG 3rd, McLennan G, Bidaut L, McNitt-Gray MF, Meyer CR, Reeves AP, Zhao B, Aberle DR, Henschke CI, Hoffman EA, Kazerooni EA, MacMahon H, van Beeke EJ, Yankelevitz D, Biancardi AM, Bland PH, Brown MS, Engelmann RM, Laderach GE, Max D, Pais RC, Qing DP, Roberts RY, Smith AR, Starkey A, Batrah P, Caligiuri P, Farooqi A, Gladish GW, Jude CM, Munden RF, Petkovska I, Quint LE, Schwartz LH, Sundaram B, Dodd LE, Fenimore C, Gur D, Petrick N, Freymann J, Kirby J, Hughes B, Casteele AV, Gupte S, Sallamm M, Heath MD, Kuhn MH, Dharaiya E, Burns R, Fryd DS, Salganicoff M, Anand V, Shreter U, Vastagh S, Croft BY (2011) The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys 38(2):915–931

    Article  Google Scholar 

  48. Bilic P, Christ PF, Vorontsov E, Chlebus G, Chen H, Dou Q, Fu C-W, Han X, Heng P-A, Hesser J (2019) The liver tumor segmentation benchmark (lits). https://arxiv.org/arXiv:190104056

Download references

Acknowledgements

The Department of Computational Radiology and Preventive Medicine, The University of Tokyo Hospital, is sponsored by HIMEDIC Inc. and Siemens Healthcare K.K. This work was supported by Japan Society for the Promotion of Science (JSPS) KAKENHI Grant No. 18K12096. It was also supported by the Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures and High Performance Computing Infrastructure projects in Japan (Project IDs: jh190047-DAH, jh200042-DAH, and jh210011-DAH).

Author information

Authors and Affiliations

  1. Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan

    Yukihiro Nomura, Tomomi Takenaga, Takahiro Nakao, Hisaichi Shibata, Takeharu Yoshikawa & Naoto Hayashi

  2. Center for Frontier Medical Engineering, Chiba University, 1-33 Yayoi-cho, Inage-ku, Chiba, 263-8522, Japan

    Yukihiro Nomura

  3. Department of Radiology, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan

    Shouhei Hanaoka, Soichiro Miki, Takeyuki Watadani & Osamu Abe

Authors
  1. Yukihiro Nomura
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shouhei Hanaoka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tomomi Takenaga
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Takahiro Nakao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hisaichi Shibata
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Soichiro Miki
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Takeharu Yoshikawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Takeyuki Watadani
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Naoto Hayashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Osamu Abe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yukihiro Nomura.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1975 Helsinki Declaration, as revised in 2008(5). For this type of study, formal consent is not required.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nomura, Y., Hanaoka, S., Takenaga, T. et al. Preliminary study of generalized semiautomatic segmentation for 3D voxel labeling of lesions based on deep learning. Int J CARS 16, 1901–1913 (2021). https://doi.org/10.1007/s11548-021-02504-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-021-02504-z

Keywords

Search

Navigation