Skip to main content
SpringerLink
Log in

Unsupervised deep consistency learning adaptation network for cardiac cross-modality structural segmentation

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Deep neural networks have recently been succeessful in the field of medical image segmentation; however, they are typically subject to performance degradation problems when well-trained models are tested in another new domain with different data distributions. Given that annotated cross-domain images may inaccessible, unsupervised domain adaptation methods that transfer learnable information from annotated source domains to unannotated target domains with different distributions have attracted substantial attention. Many methods leverage image-level or pixel-level translation networks to align domain-invariant information and mitigate domain shift issues. However, These methods rarely perform well when there is a large domain gap. A new unsupervised deep consistency learning adaptation network, which adopts input space consistency learning and output space consistency learning to realize unsupervised domain adaptation and cardiac structural segmentation, is introduced in this paper The framework mainly includes a domain translation path and a cross-modality segmentation path. In domain translation path, a symmetric alignment generator network with attention to cross-modality features and anatomy is introduced to align bidirectional domain features. In the segmentation path, entropy map minimization, output probability map minimization and segmentation prediction minimization are leveraged to align the output space features. The model conducts supervised learning to extract source domain features and conducts unsupervised deep consistency learning to extract target domain features. Through experimental testing on two challenging cross-modality segmentation tasks, our method has robust performance compared to that of previous methods. Furthermore, ablation experiments are conducted to confirm the effectiveness of our framework.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data Availability

Data related to the current study are available from the corresponding author on reasonable request.

Code Availability

The codes used during the study are available from the First author by request.

References

  1. Bateson M, Dolz J, Kervadec H, Lombaert H, Ayed IB (2021) Constrained domain adaptation for image segmentation. IEEE Trans Med Imaging 40(7):1875–1887

    Article  CAS  PubMed  Google Scholar 

  2. Bernard O, Lalande A, Zotti C, Cervenansky F, Yang X, Heng PA, Cetin I, Lekadir K, Camara O, Ballester MAG et al (2018) Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans Med Imaging 37(11):2514–2525

    Article  PubMed  Google Scholar 

  3. Borgwardt KM, Gretton A, Rasch MJ, Kriegel HP, Schölkopf B, Smola AJ (2006) Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14):e49–e57

    Article  CAS  PubMed  Google Scholar 

  4. Bousmalis K, Trigeorgis G, Silberman N, Krishnan D, Erhan D (2016) Domain separation networks. Advances in neural information processing systems 29

  5. Chen C, Chen Z, Jiang B, Jin X (2019) Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. Proc AAAI Conf Artif Intell 33:3296–3303

    Google Scholar 

  6. Chen C, Dou Q, Chen H, Qin J, Heng PA (2019) Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation. Proc AAAI Conf Artif Intell 33:865–872

    Google Scholar 

  7. Chen C, Dou Q, Chen H, Qin J, Heng PA (2020) Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation. IEEE Trans Med Imaging 39(7):2494–2505

    Article  PubMed  Google Scholar 

  8. Chen X, Lian C, Wang L, Deng H, Kuang T, Fung SH, Gateno J, Shen D, Xia JJ, Yap PT (2021) Diverse data augmentation for learning image segmentation with cross-modality annotations. Med Image Anal 71:102060

    Article  PubMed  PubMed Central  Google Scholar 

  9. Chen X, Kuang T, Deng H, Fung SH, Gateno J, Xia JJ, Yap PT (2022) Dual adversarial attention mechanism for unsupervised domain adaptive medical image segmentation. IEEE Transactions on Medical Imaging

  10. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3213–3223

  11. Csurka G, Volpi R, Chidlovskii B (2021) Unsupervised domain adaptation for semantic image segmentation: a comprehensive survey. arXiv:2112.03241

  12. Cui H, Yuwen C, Jiang L, Xia Y, Zhang Y (2021) Bidirectional cross-modality unsupervised domain adaptation using generative adversarial networks for cardiac image segmentation. Comput Biol Med 136:104726

    Article  PubMed  Google Scholar 

  13. Cui Z, Li C, Du Z, Chen N, Wei G, Chen R, Yang L, Shen D, Wang W (2021) Structure-driven unsupervised domain adaptation for cross-modality cardiac segmentation. IEEE Trans Med Imaging 40(12):3604–3616

    Article  PubMed  Google Scholar 

  14. Dou Q, Ouyang C, Chen C, Chen H, Glocker B, Zhuang X, Heng PA (2019) PnP-adaNet: Plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation. IEEE Access 7:99065–99076

    Article  Google Scholar 

  15. Dou Q, Liu Q, Heng PA, Glocker B (2020) Unpaired multi-modal segmentation via knowledge distillation. IEEE Trans Med Imaging 39(7):2415–2425

    Article  PubMed  Google Scholar 

  16. Ge Y, Xue Z, Cao T, Liao S (2019) Unpaired whole-body MR to CT synthesis with correlation coefficient constrained adversarial learning. In: Medical imaging 2019: Image Processing, SPIE, vol 10949. pp 28–35

  17. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Advances in neural information processing systems 27

  18. Han X, Qi L, Yu Q, Zhou Z, Zheng Y, Shi Y, Gao Y (2021) Deep symmetric adaptation network for cross-modality medical image segmentation. IEEE Trans Med Imaging 41(1):121–132

    Article  PubMed  Google Scholar 

  19. Hoffman J, Tzeng E, Park T, Zhu JY, Isola P, Saenko K, Efros A, Darrell T (2018) Cycada: Cycle-consistent adversarial domain adaptation. In: International conference on machine learning, Pmlr. pp 1989–1998

  20. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1125–1134

  21. Jafari M, Francis S, Garibaldi JM, Chen X (2022) LMISA: A lightweight multi-modality image segmentation network via domain adaptation using gradient magnitude and shape constraint. Medical Image Analysis p 102536

  22. Kim T, Cha M, Kim H, Lee JK, Kim J (2017) Learning to discover cross-domain relations with generative adversarial networks. In: International conference on machine learning, PMLR. pp 1857–1865

  23. Kuroki S, Charoenphakdee N, Bao H, Honda J, Sato I, Sugiyama M (2019) Unsupervised domain adaptation based on source-guided discrepancy. Proc AAAI Conf Artif Intell 33:4122–4129

    Google Scholar 

  24. Lee AH, Hiramatsu N (2011) Role of n-3 series polyunsaturated fatty acids in cardiovascular disease prevention. Nutr Diet Suppl 3:93

  25. Li K, Yu L, Wang S, Heng PA (2020) Towards cross-modality medical image segmentation with online mutual knowledge distillation. Proc AAAI Conf Artif Intell 34:775–783

    Google Scholar 

  26. Liao X, Qian Y, Chen Y, Xiong X, Wang Q, Heng PA (2020) Mmtlnet: Multi-modality transfer learning network with adversarial training for 3D whole heart segmentation. Comput Med Imaging Graph 85:101785

    Article  PubMed  Google Scholar 

  27. Long M, Cao Y, Wang J, Jordan M (2015) Learning transferable features with deep adaptation networks. In: International conference on machine learning, PMLR. pp 97–105

  28. Long M, Zhu H, Wang J, Jordan MI (2016) Unsupervised domain adaptation with residual transfer networks. Advances in neural information processing systems 29

  29. Lu C, Zheng S, Gupta G (2022) Unsupervised domain adaptation for cardiac segmentation: Towards structure mutual information maximization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 2588–2597

  30. Van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11)

  31. Mancini M, Porzi L, Bulo SR, Caputo B, Ricci E (2018) Boosting domain adaptation by discovering latent domains. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3771–3780

  32. Mollet NR, Dymarkowski S, Bogaert J (2003) MRI and CT revealing carcinoid heart disease. Eur Radiol 13(6):L14–L18

    Article  PubMed  Google Scholar 

  33. Park T, Liu MY, Wang TC, Zhu JY (2019) Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 2337–2346

  34. Pei C, Wu F, Huang L, Zhuang X (2021) Disentangle domain features for cross-modality cardiac image segmentation. Med Image Anal 71:102078

    Article  PubMed  Google Scholar 

  35. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention, Springer. pp 234–241

  36. Shen D, Wu G, Suk HI (2017) Deep learning in medical image analysis. Annu Rev Biomed Eng 19:221

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Sun B, Saenko K (2016) Deep coral: Correlation alignment for deep domain adaptation. In: European conference on computer vision, Springer. pp 443–450

  38. Tomar D, Lortkipanidze M, Vray G, Bozorgtabar B, Thiran JP (2021) Self-attentive spatial adaptive normalization for cross-modality domain adaptation. IEEE Trans Med Imaging 40(10):2926–2938

    Article  PubMed  Google Scholar 

  39. Tsai YH, Hung WC, Schulter S, Sohn K, Yang MH, Chandraker M (2018) Learning to adapt structured output space for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7472–7481

  40. Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7167–7176

  41. Vesal S, Gu M, Kosti R, Maier A, Ravikumar N (2021) Adapt everywhere: unsupervised adaptation of point-clouds and entropy minimization for multi-modal cardiac image segmentation. IEEE Trans Med Imaging 40(7):1838–1851

    Article  PubMed  Google Scholar 

  42. Vs V, Gupta V, Oza P, Sindagi VA, Patel VM (2021) MeGA-CDA: Memory guided attention for category-aware unsupervised domain adaptive object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 4516–4526

  43. Vu TH, Jain H, Bucher M, Cord M, Pérez P (2019) Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 2517–2526

  44. Wang M, Deng W (2018) Deep visual domain adaptation: A survey. Neurocomputing 312:135–153

    Article  Google Scholar 

  45. Wang Q, Du Y, Fan H, Ma C (2022) Towards collaborative appearance and semantic adaptation for medical image segmentation. Neurocomputing 491:633–643

  46. Wang R, Zheng G (2022) CyCMIS: Cycle-consistent cross-domain medical image segmentation via diverse image augmentation. Med Image Anal 76:102328

  47. Wang S, Yu L, Li K, Yang X, Fu CW, Heng PA (2019) Boundary and entropy-driven adversarial learning for fundus image segmentation. In: International conference on medical image computing and computer-assisted intervention, Springer. pp 102–110

  48. Wilson G, Cook DJ (2020) A survey of unsupervised deep domain adaptation. ACM Trans Intell Syst Technol (TIST) 11(5):1–46

    Article  CAS  Google Scholar 

  49. Wu F, Zhuang X (2020) CF distance: A new domain discrepancy metric and application to explicit domain adaptation for cross-modality cardiac image segmentation. IEEE Trans Med Imaging 39(12):4274–4285

    Article  PubMed  Google Scholar 

  50. Wu F, Zhuang X (2021) Unsupervised domain adaptation with variational approximation for cardiac segmentation. IEEE Trans Med Imaging 40(12):3555–3567

    Article  PubMed  Google Scholar 

  51. Yi Z, Zhang H, Tan P, Gong M (2017) Dualgan: Unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE international conference on computer vision. pp 2849–2857

  52. Zhou Z, Qi L, Yang X, Ni D, Shi Y (2022) Generalizable cross-modality medical image segmentation via style augmentation and dual normalization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 20856–20865

  53. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp 2223–2232

  54. Zhuang X (2018) Multivariate mixture model for myocardial segmentation combining multi-source images. IEEE Trans Pattern Anal Mach Intell 41(12):2933–2946

    Article  PubMed  Google Scholar 

  55. Zhuang X, Shen J (2016) Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med Image Anal 31:77–87 t

  56. Zhuang X, Li L, Payer C, Štern D, Urschler M, Heinrich MP, Oster J, Wang C, Smedby Ö, Bian C et al (2019) Evaluation of algorithms for multi-modality whole heart segmentation: an open-access grand challenge. Med Image Anal 58:101537

    Article  PubMed  PubMed Central  Google Scholar 

  57. Zhuang X, Xu J, Luo X, Chen C, Ouyang C, Rueckert D, Campello VM, Lekadir K, Vesal S, RaviKumar N, et al. (2020) Cardiac segmentation on late gadolinium enhancement MRI: a benchmark study from multi-sequence cardiac MR segmentation challenge. arXiv:2006.12434

  58. Zou Y, Yu Z, Kumar B, Wang J (2018) Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Proceedings of the European conference on computer vision (ECCV). pp 289–305

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China (Grant No. 61976126), Shandong Nature Science Foundation of China (No. ZR2019MF003).

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, Shandong, China

    Dapeng Li, Yanjun Peng, Jindong Sun & Yanfei Guo

  2. Shandong Province Key Laboratory of Wisdom Mining Information Technology, Shandong University of Science and Technology, Qingdao, 266590, Shandong, China

    Yanjun Peng

Authors
  1. Dapeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanjun Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jindong Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yanfei Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Dapeng Li: Conceptualization, Methodology, Software, Writing the original draft. Yanjun Peng: Data curation, Supervision. Jindong Sun: Investigation, Formal analysis, Software. Yanfei Guo: Visualization, Writing-review and editing.

Corresponding author

Correspondence to Yanjun Peng.

Ethics declarations

Conflicts of interest

The authors declare there are no conflicts of interest regarding the publication of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, D., Peng, Y., Sun, J. et al. Unsupervised deep consistency learning adaptation network for cardiac cross-modality structural segmentation. Med Biol Eng Comput 61, 2713–2732 (2023). https://doi.org/10.1007/s11517-023-02833-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02833-y

Keywords

Search

Navigation