Skip to main content
SpringerLink
Log in

A teacher-student framework for liver and tumor segmentation under mixed supervision from abdominal CT scans

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Liver and tumor segmentation from abdominal CT scans and an important step towards computer-assisted diagnosis or treatment planning for various hepatic diseases. Training convolutional neural networks for image segmentation demands a large number of pixel-wise labels which are inefficient to acquire. In order to leverage massive weak annotations, we developed a teacher-student framework using both pixel annotated dataset (strong dataset) and bounding box annotated dataset (weak dataset). A teacher annotator transfers the knowledge from the strong dataset to the weak one by refining its bounding box labels into pseudo pixel-wise labels. Motivated by the spatial layout of organ and tumor, we proposed a hierarchical organ-to-lesion (O2L) attention module to regularize the teacher annotator trained on the strong dataset. A student segmentor is trained with the mix of strong and refined weak datasets. A localization branch in the student network aggregates deep features to predict positions of organ and lesion, improving the segmentation of small objects. A comparative study with state-of-the-art methods demonstrates the proposed method strikes the balance between model performance and annotation efficiency. This model shows robustness to the quality of bounding box annotations. The model is also validated on kidney and tumor segmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. https://competitions.codalab.org/competitions/17094.

  2. https://kits19.grand-challenge.org/home/.

References

  1. Ahn J, Kwak S (2018) Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In: IEEE conference on computer vision and pattern recognition, pp 4981–4990

  2. Bearman A, Russakovsky O, Ferrari V, Fei-Fei L (2016) What’s the point: semantic segmentation with point supervision. In: European conference on computer vision. Springer, Berlin, pp 549–565

  3. Bhalgat Y, Shah M, Awate S (2018) Annotation-cost minimization for medical image segmentation using suggestive mixed supervision fully convolutional networks. arXiv preprint arXiv:181211302

  4. Bilic P, Christ PF, Vorontsov E, Chlebus G, Chen H, Dou Q, Fu CW, Han X, Heng PA, Hesser J et al (2019) The liver tumor segmentation benchmark (LiTS). arXiv preprint arXiv:190104056

  5. Can YB, Chaitanya K, Mustafa B, Koch LM, Konukoglu E, Baumgartner CF (2018) Learning to segment medical images with scribble-supervision alone. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Berlin, pp 236–244

  6. Chen Y, Yin X, Shi L, Shu H, Luo L, Coatrieux JL, Toumoulin C (2013) Improving abdomen tumor low-dose CT images using a fast dictionary learning based processing. Phys Med Biol 58(16):5803

    Article  Google Scholar 

  7. Chen Z, Wang S, Hu Y, Zhou H, Shen Y, Li X (2021) Cervical spondylotic myelopathy segmentation using shape-aware U-net. In: International conference on neural computing for advanced applications. Springer, Berlin, pp 671–681

  8. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 424–432

  9. Conze PH, Noblet V, Rousseau F, Heitz F, de Blasi V, Memeo R, Pessaux P (2017) Scale-adaptive supervoxel-based random forests for liver tumor segmentation in dynamic contrast-enhanced CT scans. Int J Comput Assist Radiol Surg 12(2):223–233

    Article  Google Scholar 

  10. Dai J, He K, Sun J (2015) BoxSup: exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In: IEEE International conference on computer vision, pp 1635–1643

  11. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. In: International conference on learning representations

  12. Dou Q, Yu L, Chen H, Jin Y, Yang X, Qin J, Heng PA (2017) 3D deeply supervised network for automated segmentation of volumetric medical images. Med Image Anal 41:40–54

    Article  Google Scholar 

  13. Ferlay J, Shin HR, Bray F, Forman D, Mathers C, Parkin DM (2010) Estimates of worldwide burden of cancer in 2008: Globocan 2008. Int J Cancer 127(12):2893–2917

    Article  Google Scholar 

  14. Heimann T, Van Ginneken B, Styner MA, Arzhaeva Y, Aurich V, Bauer C, Beck A, Becker C, Beichel R, Bekes G et al (2009) Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Trans Med Imaging 28(8):1251–1265

    Article  Google Scholar 

  15. Heller N, Sathianathen N, Kalapara A, Walczak E, Moore K, Kaluzniak H, Rosenberg J, Blake P, Rengel Z, Oestreich M, et al. (2019) The KiTS19 challenge data: 300 kidney tumor cases with clinical context, CT semantic segmentations, and surgical outcomes. arXiv preprint arXiv:190400445

  16. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:150302531

  17. Hu S, Zhang J, Xia Y (2020) Boundary-aware network for kidney tumor segmentation. In: International workshop on machine learning in medical imaging. Springer, Berlin, pp 189–198

  18. Huang W, Yang Y, Lin Z, Huang GB, Zhou J, Duan Y, Xiong W (2014) Random feature subspace ensemble based extreme learning machine for liver tumor detection and segmentation. In: Annual international conference of the IEEE engineering in medicine and biology society. IEEE, pp 4675–4678

  19. Ibrahim MS, Vahdat A, Ranjbar M, Macready WG (2020) Semi-supervised semantic image segmentation with self-correcting networks. In: IEEE conference on computer vision and pattern recognition, pp 12715–12725

  20. Khoreva A, Benenson R, Hosang J, Hein M, Schiele B (2017) Simple does it: weakly supervised instance and semantic segmentation. In: IEEE conference on computer vision and pattern recognition, pp 876–885

  21. Li X, Chen H, Qi X, Dou Q, Fu CW, Heng PA (2018) H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE Trans Med Imaging 37(12):2663–2674

    Article  Google Scholar 

  22. Mlynarski P, Delingette H, Criminisi A, Ayache N (2019) Deep learning with mixed supervision for brain tumor segmentation. J Med Imaging 6(3):034002

    Article  Google Scholar 

  23. Papandreou G, Chen LC, Murphy KP, Yuille AL (2015) Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: IEEE international conference on computer vision, pp 1742–1750

  24. Rajchl M, Lee MC, Oktay O, Kamnitsas K, Passerat-Palmbach J, Bai W, Damodaram M, Rutherford MA, Hajnal JV, Kainz B et al (2016) DeepCut: object segmentation from bounding box annotations using convolutional neural networks. IEEE Trans Med Imaging 36(2):674–683

    Article  Google Scholar 

  25. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 234–241

  26. Rother C, Kolmogorov V, Blake A (2004) “GrabCut’’ interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314

    Article  Google Scholar 

  27. Shah MP, Merchant S, Awate SP (2018) MS-Net: mixed-supervision fully-convolutional networks for full-resolution segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 379–387

  28. Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE conference on computer vision and pattern recognition, pp 1874–1883

  29. Song C, Huang Y, Ouyang W, Wang L (2019) Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation. In: IEEE/CVF conference on computer vision and pattern recognition, pp 3136–3145

  30. Soret M, Bacharach SL, Buvat I (2007) Partial-volume effect in pet tumor imaging. J Nucl Med 48(6):932–945

    Article  Google Scholar 

  31. Sun L, Ma W, Ding X, Huang Y, Liang D, Paisley J (2019) A 3D spatially weighted network for segmentation of brain tissue from MRI. IEEE Trans Med Imaging 39(4):898–909

    Article  Google Scholar 

  32. Tang W, Zou D, Yang S, Shi J, Dan J, Song G (2020) A two-stage approach for automatic liver segmentation with faster R-CNN and DeepLab. Neural Comput Appl 1–10

  33. Tang Y, Tang Y, Zhu Y, Xiao J, Summers RM (2020) E\(^2\) net: An edge enhanced network for accurate liver and tumor segmentation on CT scans. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 512–522

  34. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: IEEE conference on computer vision and pattern recognition, pp 3156–3164

  35. Wang D, Li M, Ben-Shlomo N, Corrales CE, Cheng Y, Zhang T, Jayender J (2019) Mixed-supervised dual-network for medical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 192–200

  36. Woo S, Park J, Lee JY, Kweon IS (2018) CBAM: convolutional block attention module. In: European conference on computer vision, pp 3–19

  37. Yu Q, Shi Y, Sun J, Gao Y, Zhu J, Dai Y (2019) Crossbar-net: a novel convolutional neural network for kidney tumor segmentation in CT images. IEEE Trans Image Process 28(8):4060–4074

    Article  MathSciNet  Google Scholar 

  38. Zhang J, Xie Y, Zhang P, Chen H, Xia Y, Shen C (2019) Light-weight hybrid convolutional network for liver tumor segmentation. In: International joint conference on artificial intelligence, vol 19, pp 4271–4277

  39. Zhou X, Wang D, Krähenbühl P (2019) Objects as points. arXiv preprint arXiv:190407850

  40. Zhu X, Cheng D, Zhang Z, Lin S, Dai J (2019) An empirical study of spatial attention mechanisms in deep networks. In: IEEE/CVF international conference on computer vision, pp 6688–6697

Download references

Acknowledgements

The study is supported partly by the National Key Research and Development Program of China (No. 2019YFC0118100), ZheJiang Province Key Research Development Program (No. 2020C03073), China Postdoctoral Science Foundation (No. 2021M702726), National Natural Science Foundation of China under Grants 82172033, 61971369, U19B2031, Science and Technology Key Project of Fujian Province (No. 2019HZ020009), Fundamental Research Funds for the Central Universities 20720200003, and Tencent Open Fund.

Author information

Authors and Affiliations

  1. School of Informatics, Xiamen University, Xiamen, China

    Liyan Sun, Xinghao Ding & Yue Huang

  2. School of Electronic Science and Engineering, Xiamen University, Xiamen, China

    Liyan Sun, Jianxiong Wu & Zhong Chen

  3. Department of Radiology, The Third Medical Centre, Chinese PLA General Hospital, Beijing, China

    Guisheng Wang

  4. Deepwise AI Laboratory, 100125, Beijing, China

    Yizhou Yu

Authors
  1. Liyan Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianxiong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinghao Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yue Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guisheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yizhou Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinghao Ding.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, L., Wu, J., Ding, X. et al. A teacher-student framework for liver and tumor segmentation under mixed supervision from abdominal CT scans. Neural Comput & Applic 34, 16547–16561 (2022). https://doi.org/10.1007/s00521-022-07240-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07240-2

Keywords

Search

Navigation