Skip to main content
SpringerLink
Log in

U2F-GAN: Weakly Supervised Super-pixel Segmentation in Thyroid Ultrasound Images

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Precise nodule segmentation in thyroid ultrasound images is important for clinical quantitative analysis and diagnosis. Fully supervised deep learning method can effectively extract representative features from nodules and background. Despite the great success, deep learning–based segmentation methods still face a critical hindrance: the difficulty in acquiring sufficient training data due to high annotation costs. To this end, we propose a weakly supervised framework called uncertainty to fine generative adversarial network (U2F-GAN) for nodule segmentation in thyroid ultrasound images that exploits only a handful of rough bounding box annotations to successfully generate reliable labels from these weak supervisions. Based on feature-matching GAN, the proposed method alternates between generating masks and learning a segmentation network in an adversarial manner. Super-pixel processing mechanism is adopted to reflect low-level image structure features for learning and inferring semantic segmentation, which largely improve the efficiency of training process. In addition, we introduce a similarity comparison module and a distributed loss function with constraints to effectively remove noise in localization annotations and enhance the generalization capability of the network, thus strengthen the overall segmentation performance. Compared to existing weakly supervised approaches, our proposed U2F-GAN yields a significant performance boost. The segmentation results are also comparable to fully supervised methods, but the annotation burden is much lower. Also, the training speed of the network model is much faster than other methods with weak supervisions, which enables the network to be updated in time, thus is beneficial to high-throughput medical image setting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Blankenship DR, Chin E, Terris DJ. Contemporary management of thyroid cancer. American Journal of Otolaryngology-Head and Neck Medicine and Surgery. 2005;26(4):249–60.

    Google Scholar 

  2. Zhao J, Zheng W, Zhang L, Tian H. Segmentation of ultrasound images of thyroid nodule for assisting fine needle aspiration cytology. Health Inf Sci Syst. 2013;1(1):1–12.

    Article  Google Scholar 

  3. Peng B, Zhang L, Zhang D, Yang J. Image segmentation by iterated region merging with localized graph cuts. Pattern Recognit. 2011;44(10–11):2527–38.

    Article  Google Scholar 

  4. Huang Q, Lee S, Liu L, Lu M, Li A. A robust graph-based segmentation method for breast nodules in ultrasound images. Ultrasonics. 2011;52(2):266–75.

    Article  Google Scholar 

  5. Yap M, Edirisinghe E, Bez H. Processed images in human perception: a case study in ultrasound breast imaging. Eur J Radiol. 2010;73(3):682–7.

    Article  Google Scholar 

  6. Bomeli S, Lebeau S, Ferris R. Evaluation of a thyroid nodule. Otolaryngol Clin North Am. 2010;43(2):229–38.

    Article  Google Scholar 

  7. Suha K, Seunghoon H, Bohyung H. Weakly supervised semantic segmentation using super-pixel pooling network. In: 31st AAAI Conference on Artificial Intelligence. 2017;4111–7.

  8. Dong M, Liu D, Xiong Z, et al. Instance segmentation from volumetric biomedical images without voxel-wise labeling. In: International Conference on Medical Image Computing & Computer-assisted Intervention. Springer, Cham. 2019;83–91.

  9. Arnab A, Torr P. Pixelwise instance segmentation with a dynamically instantiated network. In: 2017 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). 2017;879–88.

  10. Lin T, Maire M, Belongie S, et al. Microsoft coco: common objects in context. In: 11th European Conference on Computer Vision (ECCV). 2014;740–55.

  11. Yoo I, Yoo D, Paeng K. PseudoEdgeNet: Nuclei segmentation only with point annotations. arXiv: 1906.02924v1. 2019.

  12. Khoreva A, Benenson R, Hosang J, Hein M, Schiele B. Simple does it: weakly supervised instance and semantic segmentation. In: 2017 IEEE International Conference On Computer Vision and Pattern Recognition (CVPR). 2017;1665–74.

  13. Xue H, Liu C, Wan F, Jiao J. DANet: divergent activation for weakly supervised object localization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2019;6588–97.

  14. Bouget D, Allan M, Stoyanov D, Jannin P. Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal. 2017;35:633–54.

    Article  Google Scholar 

  15. Zhou Y, Zhu Y, Ye Q, Qiu Q, Jiao J. Weakly supervised instance segmentation using class peak response. In: 2018 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). 2018;3791–800.

  16. Zhang W, Zhang Q, Cheng J, Bai C, Hao P. End-to-end panoptic segmentation with pixel-level non-overlapping embedding. In: 2019 IEEE International Conference on Multimedia and Expo (ICME). 2019;976–81.

  17. Li Q, Arnab A, Torr P. Weakly- and semi-supervised panoptic segmentation. In: 15th European Conference on Computer Vision (ECCV). 2018;106–24.

  18. Hwang S, Kim H. Self-transfer learning for weakly supervised lesion localization. In: International Conference on Medical Image Computing & Computer-assisted Intervention. Springer, Cham. 2016;239–46.

  19. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers R. Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: 2017 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). 2017;2097–106.

  20. Feng X, Yang J, Laine A, Angelini E. Discriminative localization in cnns for weakly-supervised segmentation of pulmonary nodules. In: International Conference on Medical Image Computing & Computer-assisted Intervention. Springer, Cham. 2017;568–76.

  21. Yang X, Wang Z, Liu C, et al. Joint detection and diagnosis of prostate cancer in multi-parametric MRI based on multimodal convolutional neural networks. In: International Conference on Medical Image Computing & Computer-assisted Intervention. Springer, Cham. 2017;426–34.

  22. Demiray, B, Rackerseder, J, Bozhinoski, S, Navab N. Weakly-supervised white and grey matter segmentation in 3D brain ultrasound. arXiv: 1904.05191. 2019.

  23. Carneiro G, Peng T, Bayer C, Navab N. Automatic quantification of tumour hypoxia from multi-modal microscopy images using weakly-supervised learning methods. IEEE Trans Med Imaging. 2017;36(7):1405–17.

    Article  Google Scholar 

  24. Shin SY, Lee S, Yun I, Lee K. Joint weakly and semi-supervised deep learning for localization and classification of masses in breast ultrasound images. IEEE Trans Med Imaging. 2019;38(3):762–74.

    Article  Google Scholar 

  25. Yang L, Zhang Y, Chen J, Zhang S, Chen D. Suggestive annotation: a deep active learning framework for biomedical image segmentation. In: International Conference on Medical Image Computing & Computer-assisted Intervention. Springer, Cham. 2017;399–407.

  26. Zhao Z, Yang L, Zheng H, Guldner I, Zhang S, Chen D. Deep learning based instance segmentation in 3D biomedical images using weak annotation. In: International Conference on Medical Image Computing & Computer-assisted Intervention. Springer, Cham. 2018;352–60.

  27. Khan S, Shahin A, Villafruela J, Shen J, Shao L. Extreme points derived confidence map as a cue for class-agnostic interactive segmentation using deep neural network. In: International Conference on Medical Image Computing & Computer-assisted Intervention. Springer, Cham. 2019;66–73.

  28. Nishimura K, Ker D, Bise, R. Weakly supervised cell instance segmentation by propagating from detection response. arXiv: 1911.13077v1. 2019.

  29. Ganin Y, Ustinova E, Ajakan H, et al. Domain-adversarial training of neural networks. J Mach Learn Res. 2016;17(1):1–35.

    MathSciNet  MATH  Google Scholar 

  30. Luc P, Couprie C, Chintala S, Verbeek J. Semantic segmentation using adversarial networks. arXiv: 1611.08408. 2016.

  31. Kohl S, Bonekamp D, Schlemmer HP, et al. Adversarial networks for the detection of aggressive prostate cancer. arXiv: 1702.08014. 2017.

  32. Dai W, Doyle J, Liang X, et al. SCAN: structure correcting adversarial network for chest X-rays organ segmentation. arXiv: 1703.08770v1. 2017.

  33. Liu R, Zhou S, Guo Y, Wang Y, Chang C. Nodule localization in thyroid ultrasound images with a joint-training convolutional neural network. J Digit Imaging. 2020;33:1266–79.

    Article  Google Scholar 

  34. Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A, Chen X. Improved techniques for training GANs. Adv Neural Inf Process Syst. 2016;29:2234–42.

    Google Scholar 

  35. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing & Computer-assisted Intervention. Springer, Cham. 2015;234–41.

  36. Achanta R, Shaji A, Smith K, et al. SLIC Super-pixels compared to state-of-the-art super-pixel methods. IEEE Trans Pattern Anal Mach Intell. 2012;34(11):2274–82.

    Article  Google Scholar 

  37. Milletari F, Navab N, Ahmadi S. (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV). 2016;565–71.

  38. Ker J, Wang L, Rao J, Lim T. Deep learning applications in medical image analysis. IEEE Access. 2018;6(3):9375–89.

    Article  Google Scholar 

  39. Lin T, Goyal P, Girshick R, He K, Dollár P. Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell. 2020;42(2):318–27.

    Article  Google Scholar 

  40. Ma J, Wu F, Jiang T, Zhu J, Kong D. Cascade convolutional neural networks for automatic detection of thyroid nodules in ultrasound images. Med Phys. 2017;44(5):1678–91.

    Article  Google Scholar 

  41. Kingma D, Ba J. Adam: A method for stochastic optimization. arXiv: 1412.6980. 2015.

  42. Abadi M, Barham P, Chen J, et al. TensorFlow: a system for large-scale machine learning. In: Conference on Operating Systems Design and Implementation. 2016;265–83.

  43. Ma J, Wu F, Jiang T, et al. Ultrasound image-based thyroid nodule automatic segmentation using convolutional neural networks. Int J Comput Assist Radiol Surg. 2017;12:1895–910.

    Article  Google Scholar 

  44. Hu Y, Guo Y, Wang Y, et al. Automatic nodule segmentation in breast ultrasound images using a dilated fully convolutional network combined with an active contour model. Med Phys. 2018;46(1):215–28.

    Article  Google Scholar 

Download references

Funding

This work is supported by the National Natural Science Foundation of China (61871135, 81830058, 81627804) and the Science and Technology Commission of Shanghai Municipality (18511102904, 20DZ1100104).

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, School of Information Science and Technology, Fudan University, No. 220, Handan Road, Yangpu District, Shanghai, 200433, China

    Ruoyun Liu, Yi Guo & Yuanyuan Wang

  2. Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention (MICCAI) of Shanghai, Shanghai, 200032, China

    Ruoyun Liu, Yi Guo & Yuanyuan Wang

  3. Department of Ultrasound, Fudan University Shanghai Cancer Center, Shanghai, China

    Shichong Zhou & Cai Chang

  4. Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China

    Shichong Zhou & Cai Chang

Authors
  1. Ruoyun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shichong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuanyuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cai Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yi Guo or Yuanyuan Wang.

Ethics declarations

Ethical Approval

This article does not contain any studies with human participants performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ruoyun Liu and Shichong Zhou contributed equally to this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, R., Zhou, S., Guo, Y. et al. U2F-GAN: Weakly Supervised Super-pixel Segmentation in Thyroid Ultrasound Images. Cogn Comput 13, 1099–1113 (2021). https://doi.org/10.1007/s12559-021-09909-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-021-09909-7

Keywords

Search

Navigation