Skip to main content
SpringerLink
Log in

Confidence-weighted mutual supervision on dual networks for unsupervised cross-modality image segmentation

  • Research Paper
  • Special Topic: Deep Learning for Computer Vision
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

The unsupervised cross-modality image segmentation has gained much attention. Many methods attempt to align different modalities via adversarial learning. Recently, self-training with pseudo labels for the unsupervised target modality has also been widely used and achieved very promising results. The pseudo labels are usually obtained by selecting reliable predictions whose highest predicted probability is larger than an empirically set value. Such pseudo label generation inevitably has noise and training a segmentation model using incorrect pseudo labels could yield nontrivial errors for the target modality. In this paper, we propose a confidence-weighted mutual supervision on dual networks for unsupervised cross-modality image segmentation. Specifically, we independently initialize two networks with the same architecture, and propose a novel confidence-weighted Dice loss to mutually supervise the two networks using their predicted results for unlabeled data. In this way, we make full use of all predictions of unlabeled images and leverage the prediction confidence to alleviate the negative impact of noisy pseudo labels. Extensive experiments on three widely-used unsupervised cross-modality image segmentation datasets (i.e., MM-WHS 2017, Brats 2018, and Multi-organ segmentation) demonstrate that the proposed method achieves superior performance to some state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. 3431–3440

  2. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015. 234–241

  3. Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2881–2890

  4. Chen L C, Papandreou G, Kokkinos I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell, 2017, 40: 834–848

    Article  Google Scholar 

  5. Wang J H, Liu B, Xu K. Semantic segmentation of high-resolution images. Sci China Inf Sci, 2017, 60: 123101

    Article  Google Scholar 

  6. Geng Q C, Zhou Z, Cao X C. Survey of recent progress in semantic image segmentation with CNNs. Sci China Inf Sci, 2018, 61: 051101

    Article  MathSciNet  Google Scholar 

  7. Ma S, Pang Y W, Pan J, et al. Preserving details in semantics-aware context for scene parsing. Sci China Inf Sci, 2020, 63: 120106

    Article  Google Scholar 

  8. Zhang Z J, Pang Y W. CGNet: cross-guidance network for semantic segmentation. Sci China Inf Sci, 2020, 63: 120104

    Article  Google Scholar 

  9. Isensee F, Jaeger P F, Kohl S A A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods, 2021, 18: 203–211

    Article  Google Scholar 

  10. Xu Q, Xi X M, Meng X J, et al. Difficulty-aware bi-network with spatial attention constrained graph for axillary lymph node segmentation. Sci China Inf Sci, 2022, 65: 192102

    Article  Google Scholar 

  11. Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), 2017. 2223–2232

  12. Zou D, Zhu Q, Yan P. Unsupervised domain adaptation with dual-scheme fusion network for medical image segmentation. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence, 2020. 3291–3298

  13. Chen J, Zhang Z, Xie X, et al. Beyond mutual information: generative adversarial network for domain adaptation using information bottleneck constraint. IEEE Trans Med Imag, 2021, 41: 595–607

    Article  Google Scholar 

  14. Chen X, Lian C, Wang L, et al. Anatomy-regularized representation learning for cross-modality medical image segmentation. IEEE Trans Med Imag, 2020, 40: 274–285

    Article  Google Scholar 

  15. Liu J, Liu H, Gong S, et al. Automated cardiac segmentation of cross-modal medical images using unsupervised multi-domain adaptation and spatial neural attention structure. Med Image Anal, 2021, 72: 102135

    Article  Google Scholar 

  16. Jafari M, Francis S, Garibaldi J M, et al. LMISA: a lightweight multi-modality image segmentation network via domain adaptation using gradient magnitude and shape constraint. Med Image Anal, 2022, 81: 102536

    Article  Google Scholar 

  17. Xie Q, Li Y, He N, et al. Unsupervised domain adaptation for medical image segmentation by disentanglement learning and self-training. IEEE Trans Med Imag, 2022. doi: https://doi.org/10.1109/TMI.2022.3192303

  18. Zhou W, Wang Y, Chu J, et al. Affinity space adaptation for semantic segmentation across domains. IEEE Trans Image Process, 2020, 30: 2549–2561

    Article  Google Scholar 

  19. Toldo M, Maracani A, Michieli U, et al. Unsupervised domain adaptation in semantic segmentation: a review. Technologies, 2020, 8: 35

    Article  Google Scholar 

  20. French G, Mackiewicz M, Fisher M. Self-ensembling for visual domain adaptation. In: Proceedings of International Conference On Learning Representations, 2018

  21. Li J, Zhou K, Qian S H, et al. Feature re-representation and reliable pseudo label retraining for cross-domain semantic segmentation. IEEE Trans Pattern Anal Mach Intell, 2022. doi: https://doi.org/10.1109/TPAMI.2022.3154933

  22. Tranheden W, Olsson V, Pinto J, et al. DACS: domain adaptation via cross-domain mixed sampling. In: Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), 2021. 1379–1389

  23. Chen M, Xue H, Cai D. Domain adaptation for semantic segmentation with maximum squares loss. In: Proceedings of International Conference on Computer Vision, 2019. 2090–2099

  24. Spadotto T, Toldo M, Michieli U, et al. Unsupervised domain adaptation with multiple domain discriminators and adaptive self-training. In: Proceedings of International Conference on Pattern Recognition, 2021. 2845–2852

  25. Zou Y, Yu Z, Kumar B, et al. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Proceedings of European Conference on Computer Vision, 2018. 289–305

  26. Han B, Yao Q, Yu X, et al. Co-teaching: robust training of deep neural networks with extremely noisy labels. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018. 8536–8546

  27. Ouali Y, Hudelot C, Tami M. Semi-supervised semantic segmentation with cross-consistency training. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 12674–12684

  28. Sohn K, Berthelot D, Carlini N, et al. FixMatch: simplifying semi-supervised learning with consistency and confidence. In: Proceedings of Advances in Neural Information Processing Systems, 2020. 33: 596–608

    Google Scholar 

  29. Liu S, Niles-Weed J, Razavian N, et al. Early-learning regularization prevents memorization of noisy labels. In: Proceedings of Advances in Neural Information Processing Systems, 2020. 33: 20331–20342

    Google Scholar 

  30. Liu S, Liu K, Zhu W, et al. Adaptive early-learning correction for segmentation from noisy annotations. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2606–2616

  31. Zhang Y, Xiang T, Hospedales T M, et al. Deep mutual learning. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 4320–4328

  32. Chen X, Yuan Y, Zeng G, et al. Semi-supervised semantic segmentation with cross pseudo supervision. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2613–2622

  33. Yang Y, Soatto S. FDA: Fourier domain adaptation for semantic segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 4085–4095

  34. Chen X, Kuang T, Deng H, et al. Dual adversarial attention mechanism for unsupervised domain adaptive medical image segmentation. IEEE Trans Med Imag, 2022, 41: 3445–3453

    Article  Google Scholar 

  35. Vu T H, Jain H, Bucher M, et al. Advent: adversarial entropy minimization for domain adaptation in semantic segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2517–2526

  36. Truong T D, Duong C N, Le N, et al. BiMaL: Bijective maximum likelihood approach to domain adaptation in semantic scene segmentation. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 8548–8557

  37. Han X, Qi L, Yu Q, et al. Deep symmetric adaptation network for cross-modality medical image segmentation. IEEE Trans Med Imag, 2021, 41: 121–132

    Article  Google Scholar 

  38. Li C, Luo X, Chen W, et al. AttENT: domain-adaptive medical image segmentation via attention-aware translation and adversarial entropy minimization. In: Proceedings of IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2021. 952–959

  39. Chen C, Dou Q, Chen H, et al. Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation. IEEE Trans Med Imag, 2020, 39: 2494–2505

    Article  Google Scholar 

  40. Liu X, Xing F, Fakhri G E, et al. Self-semantic contour adaptation for cross modality brain tumor segmentation. In: Proceedings of IEEE 19th International Symposium on Biomedical Imaging (ISBI), 2022. 1–5

  41. Zeng G, Lerch T D, Schmaranzer F, et al. Semantic consistent unsupervised domain adaptation for cross-modality medical image segmentation. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021. 201–210

  42. Choi J, Kim T, Kim C. Self-ensembling with GAN-based data augmentation for domain adaptation in semantic segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 6830–6840

  43. Liu Z, Zhu Z, Zheng S, et al. Margin preserving self-paced contrastive learning towards domain adaptation for medical image segmentation. IEEE J Biomed Health Inform, 2022, 26: 638–647

    Article  Google Scholar 

  44. Fan J, Gao B, Jin H, et al. UCC: uncertainty guided cross-head co-training for semi-supervised semantic segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 9947–9956

  45. Wang Y, Wang H, Shen Y, et al. Semi-supervised semantic segmentation using unreliable pseudo-labels. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 4248–4257

  46. Gu X, Sun J, Xu Z. Unsupervised and semi-supervised robust spherical space domain adaptation. IEEE Trans Pattern Anal Mach Intell, 2022. doi: https://doi.org/10.1109/TPAMI.2022.3158637

  47. He N J, Fang L Y, Plaza A. Hybrid first and second order attention Unet for building segmentation in remote sensing images. Sci China Inf Sci, 2020, 63: 140305

    Article  Google Scholar 

  48. Zhou Z, Siddiquee M M R, Tajbakhsh N, et al. UNet+-+: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imag, 2020, 39: 1856–1867

    Article  Google Scholar 

  49. Gut D, Tabor Z, Szymkowski M, et al. Benchmarking of deep architectures for segmentation of medical images. IEEE Trans Med Imag, 2022, 41: 3231–3241

    Article  Google Scholar 

  50. Zhuang X, Shen J. Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med Image Anal, 2016, 31: 77–87

    Article  Google Scholar 

  51. Menze B H, Jakab A, Bauer S, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imag, 2014, 34: 1993–2024

    Article  Google Scholar 

  52. Kavur A E, Gezer N S, Barış M, et al. CHAOS Challenge-combined (CT-MR) healthy abdominal organ segmentation. Med Image Anal, 2021, 69: 101950

    Article  Google Scholar 

  53. Landman B, Xu Z, Igelsias J, et al. MICCAI multi-atlas labeling beyond the cranial vault-workshop and challenge. In: Proceedings of MICCAI Workshop, 2015. 12

  54. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 770–778

  55. Isola P, Zhu J Y, Zhou T, et al. Image-to-image translation with conditional adversarial networks. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1125–1134

  56. Kingma D P, Ba J. Adam: a method for stochastic optimization. In: Proceedings of International Conference on Learning Representations, 2015

  57. Jiang J, Hu Y C, Tyagi N, et al. PSIGAN: joint probabilistic segmentation and image distribution matching for unpaired cross-modality adaptation-based MRI segmentation. IEEE Trans Med Imag, 2020, 39: 4071–4084

    Article  Google Scholar 

  58. Deng J, Dong W, Socher R, et al. ImageNet: a large-scale hierarchical image database. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 248–255

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant Nos. 62061160490, 62122029, U20B2064).

Author information

Authors and Affiliations

  1. School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, 430074, China

    Yajie Chen & Xin Yang

  2. School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, 430074, China

    Xiang Bai

Authors
  1. Yajie Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiang Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yajie Chen, Xin Yang or Xiang Bai.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Yang, X. & Bai, X. Confidence-weighted mutual supervision on dual networks for unsupervised cross-modality image segmentation. Sci. China Inf. Sci. 66, 210104 (2023). https://doi.org/10.1007/s11432-022-3871-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-022-3871-0

Keywords

Search

Navigation