Skip to main content
SpringerLink
Log in

Annotation-efficient training of medical image segmentation network based on scribble guidance in difficult areas

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

The training of deep medical image segmentation networks usually requires a large amount of human-annotated data. To alleviate the burden of human labor, many semi- or non-supervised methods have been developed. However, due to the complexity of clinical scenario, insufficient training labels still causes inaccurate segmentation in some difficult local areas such as heterogeneous tumors and fuzzy boundaries.

Methods

We propose an annotation-efficient training approach, which only requires scribble guidance in the difficult areas. A segmentation network is initially trained with a small amount of fully annotated data and then used to produce pseudo labels for more training data. Human supervisors draw scribbles in the areas of incorrect pseudo labels (i.e., difficult areas), and the scribbles are converted into pseudo label maps using a probability-modulated geodesic transform. To reduce the influence of the potential errors in the pseudo labels, a confidence map of the pseudo labels is generated by jointly considering the pixel-to-scribble geodesic distance and the network output probability. The pseudo labels and confidence maps are iteratively optimized with the update of the network, and the network training is promoted by the pseudo labels and the confidence maps in turn.

Results

Cross-validation based on two data sets (brain tumor MRI and liver tumor CT) showed that our method significantly reduces the annotation time while maintains the segmentation accuracy of difficult areas (e.g., tumors). Using 90 scribble-annotated training images (annotated time: ~ 9 h), our method achieved the same performance as using 45 fully annotated images (annotation time: > 100 h) but required much shorter annotation time.

Conclusion

Compared to the conventional full annotation approaches, the proposed method significantly saves the annotation efforts by focusing the human supervisions on the most difficult regions. It provides an annotation-efficient way for training medical image segmentation networks in complex clinical scenario.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Kohli MD, Summers RM, Geis JR (2017) Medical image data and datasets in the era of machine learning-whitepaper from the 2016 C-MIMI meeting dataset session. J Digit Imaging 30:392–399. https://doi.org/10.1007/s10278-017-9976-3

    Article  PubMed  PubMed Central  Google Scholar 

  2. Li X, Yu L, Chen H, Fu CW, Heng PA (2020) Transformation-consistent self-ensembling model for semisupervised medical image segmentation. IEEE Trans Neural Netw Learn Syst pp. 1–12

  3. Luo XD, Chen JN, Song T, Wang GT, (2021) Assoc advancement artificial, I. Semi-supervised medical image segmentation through dual-task consistency. In: Proceedings of the 35th AAAI conference on artificial intelligence/33rd conference on innovative applications of artificial intelligence/11th symposium on educational advances in artificial intelligence, Electr Network, Feb 02–09, 2021; pp. 8801–8809

  4. Sedai S, Mahapatra D, Hewavitharanage S, Maetschke S, Garnavi R (2017) Semi-supervised segmentation of optic cup in retinal fundus images using variational autoencoder. In: 20th international conference on medical image computing and computer-assisted intervention, MICCAI 2017, Proceedings. LNCS 10434, pp. 75–82

  5. Dai C, Mo Y, Angelini E, Guo Y, Bai W (2019) Transfer learning from partial annotations for whole brain segmentation. In: Proceedings of the domain adaptation and representation transfer and medical image learning with less labels and imperfect data, Cham, 2019, pp. 199–206

  6. Chen XC, Yao LN, Zhou T, Dong JM, Zhang Y (2021) Momentum contrastive learning for few-shot COVID-19 diagnosis from chest CT images. Pattern Recogn. https://doi.org/10.1016/j.patcog.2021.107826

    Article  Google Scholar 

  7. Ganaye PA, Sdika M, Triggs B, Benoit-Cattin H (2019) Removing segmentation inconsistencies with semi-supervised non-adjacency constraint. Med Image Anal 58:101551. https://doi.org/10.1016/j.media.2019.101551

    Article  PubMed  Google Scholar 

  8. Wang L, Guo D, Wang GT, Zhang ST (2021) Annotation-efficient learning for medical image segmentation based on noisy pseudo labels and adversarial learning. IEEE Trans Med Imaging 40:2795–2807. https://doi.org/10.1109/tmi.2020.3047807

    Article  PubMed  Google Scholar 

  9. Zhou ZW, Sodha V, Pang JX, Gotway MB, Liang JM (2021) Models genesis. Med Image Anal. https://doi.org/10.1016/j.media.2020.101840

    Article  PubMed  PubMed Central  Google Scholar 

  10. Zhu J, Li Y, Hu Y, Ma K, Zhou SK, Zheng Y (2020) Rubik’s Cube+: A self-supervised feature learning framework for 3D medical image analysis. Med Image Anal 64:101746. https://doi.org/10.1016/j.media.2020.101746

    Article  PubMed  Google Scholar 

  11. Dong NQ, Kampffmeyer M, Voiculescu I (2021) Self-supervised multi-task representation learning for sequential medical images. in: Proceedings of the European conference on machine learning and principles and practice of knowledge discovery in databases (ECML PKDD), Electr Network, Sep 13–17, pp. 779–794

  12. Chen H, Ni D, Qin J, Li SL, Yang X, Wang TF, Heng PA (2015) Standard plane localization in fetal ultrasound via domain transferred deep neural networks. IEEE J Biomed Health Inform 19:1627–1636. https://doi.org/10.1109/jbhi.2015.2425041

    Article  PubMed  Google Scholar 

  13. Girshick R, Donahue J, Darrell T, Malik J (2016) Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans Pattern Anal Mach Intell 38:142–158. https://doi.org/10.1109/tpami.2015.2437384

    Article  PubMed  Google Scholar 

  14. Cui HJ, Wei D, Ma K, Gu S, Zheng YF (2021) A unified framework for generalized low-shot medical image segmentation with scarce data. IEEE Trans Med Imaging 40:2656–2671. https://doi.org/10.1109/tmi.2020.3045775

    Article  PubMed  Google Scholar 

  15. Lu YH, Zheng K, Li WJ, Wang YR, Harrison AP, Lin CH, Wang S, Xiao J, Lu L, Kuo CF, Miao S (2021) Contour transformer network for one-shot segmentation of anatomical structures. IEEE Trans Med Imaging 40:2672–2684. https://doi.org/10.1109/tmi.2020.3043375

    Article  PubMed  Google Scholar 

  16. Wang SS, Li C, Wang RP, Liu ZY, Wang MY, Tan HN, Wu YP, Liu XF, Sun H, Yang R, Liu X, Chen J, Zhou HH, Ben AI, Zheng HR (2021) Annotation-efficient deep learning for automatic medical image segmentation. Nat Commun. https://doi.org/10.1038/s41467-021-26216-9

    Article  PubMed  PubMed Central  Google Scholar 

  17. Hu Y, Soltoggio A, Lock R, Carter S (2019) A fully convolutional two-stream fusion network for interactive image segmentation. Neural Netw 109:31–42. https://doi.org/10.1016/j.neunet.2018.10.009

    Article  PubMed  Google Scholar 

  18. Luo X, Wang G, Song T, Zhang J, Aertsen M, Deprest J, Ourselin S, Vercauteren T, Zhang S (2021) MIDeepSeg: minimally interactive segmentation of unseen objects from medical images using deep learning. Med Image Anal 72:102102. https://doi.org/10.1016/j.media.2021.102102

    Article  PubMed  PubMed Central  Google Scholar 

  19. Wang G, Li W, Zuluaga MA, Pratt R, Patel PA, Aertsen M, Doel T, David AL, Deprest J, Ourselin S, Vercauteren T (2018) Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE Trans Med Imaging 37:1562–1573. https://doi.org/10.1109/TMI.2018.2791721

    Article  PubMed  Google Scholar 

  20. Wang GT, Zuluaga MA, Li WQ, Pratt R, Patel PA, Aertsen M, Doel T, David AL, Deprest J, Ourselin S, Vercauteren T (2019) DeeplGeoS: a deep interactive geodesic framework for medical image segmentation. IEEE Trans Pattern Anal Mach Intell 41:1559–1572. https://doi.org/10.1109/tpami.2018.2840695

    Article  PubMed  Google Scholar 

  21. Rajchl M, Lee MCH, Oktay O, Kamnitsas K, Passerat-Palmbach J, Bai W, Damodaram M, Rutherford MA, Hajnal JV, Kainz B, Rueckert D (2017) DeepCut: object segmentation from bounding box annotations using convolutional neural networks. IEEE Trans Med Imaging 36:674–683. https://doi.org/10.1109/TMI.2016.2621185

    Article  PubMed  Google Scholar 

  22. Criminisi A, Sharp T, Blake A (2008) GeoS: Geodesic Image Segmentation. Springer, Berlin, pp 99–112

    Google Scholar 

  23. Bai X, Sapiro G (2009) Geodesic matting: a framework for fast interactive image and video segmentation and matting. Int J Comput Vision 82:113–132. https://doi.org/10.1007/s11263-008-0191-z

    Article  Google Scholar 

  24. Wang G, Zuluaga MA, Li W, Pratt R, Patel PA, Aertsen M, Doel T, David AL, Deprest J, Ourselin S, Vercauteren T (2019) DeepIGeoS: a deep interactive geodesic framework for medical image segmentation. IEEE Trans Pattern Anal Mach Intell 41:1559–1572. https://doi.org/10.1109/TPAMI.2018.2840695

    Article  PubMed  Google Scholar 

  25. Isensee F, Jaeger PF, Kohl SAA, Petersen J, Maier-Hein KH (2021) nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18:203. https://doi.org/10.1038/s41592-020-01008-z

    Article  CAS  PubMed  Google Scholar 

  26. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, Lanczi L, Gerstner E, Weber MA, Arbel T, Avants BB, Ayache N, Buendia P, Collins DL, Cordier N, Corso JJ, Criminisi A, Das T, Delingette H, Demiralp Ç, Durst CR, Dojat M, Doyle S, Festa J, Forbes F, Geremia E, Glocker B, Golland P, Guo X, Hamamci A, Iftekharuddin KM, Jena R, John NM, Konukoglu E, Lashkari D, Mariz JA, Meier R, Pereira S, Precup D, Price SJ, Raviv TR, Reza SMS, Ryan M, Sarikaya D, Schwartz L, Shin HC, Shotton J, Silva CA, Sousa N, Subbanna NK, Szekely G, Taylor TJ, Thomas OM, Tustison NJ, Unal G, Vasseur F, Wintermark M, Ye DH, Zhao L, Zhao B, Zikic D, Prastawa M, Reyes M, Leemput KV (2015) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 34:1993–2024. https://doi.org/10.1109/TMI.2014.2377694

    Article  PubMed  Google Scholar 

  27. Antonelli M, Reinke A, Bakas S, Farahani K, Kopp-Schneider A, Landman BA, Litjens G, Menze B, Ronneberger O, Summers RM, van Ginneken B, Bilello M, Bilic P, Christ PF, Do RKG, Gollub MJ, Heckers SH, Huisman H, Jarnagin WR, McHugo MK, Napel S, Pernicka JSG, Rhode K, Tobon-Gomez C, Vorontsov E, Meakin JA, Ourselin S, Wiesenfarth M, Arbeláez P, Bae B, Chen S, Daza L, Feng J, He B, Isensee F, Ji Y, Jia F, Kim I, Maier-Hein K, Merhof D, Pai A, Park B, Perslev M, Rezaiifar R, Rippel O, Sarasua I, Shen W, Son J, Wachinger C, Wang L, Wang Y, Xia Y, Xu D, Xu Z, Zheng Y, Simpson AL, Maier-Hein L, Cardoso MJ (2022) The medical segmentation Decathlon. Nat Commun 13:4128. https://doi.org/10.1038/s41467-022-30695-9

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Key Research and Development Program Nos. 2020YFB1711500, 2020YFB1711501 and 2020YFB1711503, the general program of National Natural Science Fund of China (Nos. 81971693, 61971445 and 61971089), the funding of Dalian Engineering Research Center for Artificial Intelligence in Medical Imaging, Hainan Province Key Research and Development Plan ZDYF2021SHFZ244, the Fundamental Research Funds for the Central Universities (No. DUT22YG229), the funding of Liaoning Key Lab of IC & BME System and Dalian Engineering Research Center for Artificial Intelligence in Medical Imaging.

Author information

Authors and Affiliations

  1. School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China

    Mingrui Zhuang, Zhonghua Chen, Yuxin Yang & Hongkai Wang

  2. Faculty of Information Technology, University of Jyväskylä, 40100, Jyvaskyla, Finland

    Zhonghua Chen & Lauri Kettunen

  3. Liaoning Key Laboratory of Integrated Circuit and Biomedical Electronic System, Dalian, China

    Hongkai Wang

Authors
  1. Mingrui Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhonghua Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuxin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lauri Kettunen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongkai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongkai Wang.

Ethics declarations

Conflicts of interest

The authors have no conflicts of interest to declare.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Availability of data and material

Declarations.

Code availability

The source code and trained models are open source (https://github.com/DlutMedimgGroup/Scribble-Guided-Segmentation).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhuang, M., Chen, Z., Yang, Y. et al. Annotation-efficient training of medical image segmentation network based on scribble guidance in difficult areas. Int J CARS 19, 87–96 (2024). https://doi.org/10.1007/s11548-023-02931-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-023-02931-0

Keywords

Search

Navigation