CyCMIS: Cycle-consistent Cross-domain Medical Image Segmentation via diverse image augmentation

doi:10.1016/j.media.2021.102328

Medical Image Analysis

Volume 76, February 2022, 102328

https://doi.org/10.1016/j.media.2021.102328 Get rights and content

Highlights

•
Propose an end-to-end unsupervised cross-domain image segmentation framework integrating diverse image translation with semantic image segmentation into one network.
•
Propose the diverse inter-domain semantic consistency loss and the intra-domain semantic consistency loss to regularize the cross-modality segmentation process.
•
Extensive validation on two public datasets.

Abstract

Domain shift, a phenomenon when there exists distribution discrepancy between training dataset (source domain) and test dataset (target domain), is very common in practical applications and may cause significant performance degradation, which hinders the effective deployment of deep learning models to clinical settings. Adaptation algorithms to improve the model generalizability from source domain to target domain has significant practical value. In this paper, we investigate unsupervised domain adaptation (UDA) technique to train a cross-domain segmentation method which is robust to domain shift, and which does not require any annotations on the test domain. To this end, we propose Cycle-consistent Cross-domain Medical Image Segmentation, referred as CyCMIS, integrating online diverse image translation via disentangled representation learning and semantic consistency regularization into one network. Different from learning one-to-one mapping, our method characterizes the complex relationship between domains as many-to-many mapping. A novel diverse inter-domain semantic consistency loss is then proposed to regularize the cross-domain segmentation process. We additionally introduce an intra-domain semantic consistency loss to encourage the segmentation consistency between the original input and the image after cross-cycle reconstruction. We conduct comprehensive experiments on two publicly available datasets to evaluate the effectiveness of the proposed method. Results demonstrate the efficacy of the present approach.

Graphical abstract

Introduction

The past few years witnessed remarkable progress in medical image analysis due to the increasing availability of data and the rapid development of deep learning techniques (Litjens, Kooi, Bejnordi, Setio, Ciompi, Ghafoorian, Van Der Laak, Van Ginneken, Sánchez, 2017, Bernard, Lalande, Zotti, Cervenansky, Yang, Heng, Cetin, Lekadir, Camara, Ballester, et al., 2018, Zhuang, Li, Payer, Štern, Urschler, Heinrich, Oster, Wang, Smedby, Bian, et al., 2019, Wu, Tang, Li, Firmin, Yang, 2021). Deep learning-based methods, such as convolutional neural networks (CNNs), are excellent at learning from large amounts of data (Yang, Zhang, Firmin, Li, 2021, Yang, Ye, Xia, 2022), but can be poor at generalizing learned knowledge to new test datasets that differ from the training dataset (Choudhary, Tong, Zhu, Wang, 2020, Guan, Liu, 2021). Such distribution discrepancy between the training dataset (source domain) and the test dataset (target domain), referred as domain shift, occurs frequently in medical imaging due to factors such as changing imaging modalities (e.g. computed tomography (CT), magnetic resonance imaging (MRI), Ultrasound, etc.), protocols (e.g. MRI protocols like PDw, T1w, T2w, etc.), scanning parameters (e.g. MRI pulse sequence parameters like repetition time, echo time, inversion time, flip angle, etc.), subject cohorts, or machines from different vendors and clinical sites. Domain shift is very common in practical applications and may cause significant performance degradation, which hinders the effective deployment of deep learning models to clinical settings. For semantic segmentation, which is a prerequisite for many clinical applications including disease diagnosis, surgical planning and computer assisted interventions, manually annotating data with pixel-level annotations for each test domain is not a feasible solution. Adaptation algorithms to improve the model generalizability from source domain to target domain has significant practical value. In this work, we investigate unsupervised domain adaptation (UDA) technique to train a cross-domain segmentation method which is robust to domain shift, and which does not require any annotations on the test domain.

Given respectively two sets of unpaired data in two different domains, majority of existing unsupervised cross-domain segmentation methods are based on either feature alignment (Dou, Ouyang, Chen, Chen, Glocker, Zhuang, Heng, 2019, Wu, Zhuang, 2020) or image translation (Chartsias, Joyce, Dharmakumar, Tsaftaris, 2017, Chen, Ouyang, Tarroni, Schlemper, Qiu, Bai, Rueckert, 2019, Chen, Lian, Wang, Deng, Kuang, Fung, Gateno, Yap, Xia, Shen, 2020, Chen, Lian, Wang, Deng, Kuang, Fung, Gateno, Shen, Xia, Yap, 2021). Typically, image translation-based methods consist of two components: a cross-domain image synthesis component where adversarial learning with cycle-consistent reconstruction is leveraged to solve the problem of data without correspondence, and an image segmentation component. The identified limitations of existing methods include: (1) most of them do not preserve semantic information in the process of image translation such that any error generated in the cross-domain image synthesis stage will be passed to the image segmentation stage; and (2) the cross-domain image synthesis stage cannot benefit from the high-level semantic information obtained from the image segmentation stage.

In this paper, we propose Cycle-consistent Cross-domain Medical Image Segmentation, referred as CyCMIS, integrating diverse image translation via disentangled representation learning and semantic consistency regularization into one network. Instead of learning one-to-one mapping as in CycleGAN (Zhu et al., 2017), our method characterizes the complex relationship between domains as many-to-many mapping (Yang et al., 2019b), where images are embedded onto two spaces: a domain-invariant content space and a domain-specific attribute space. By enforcing diversity seeking regularization (Yang, Hong, Jang, Zhao, Lee, 2019, Mao, Lee, Tseng, Ma, Yang, 2019), disentangled content features can be combined with different attribute vectors to produce diverse images with identical content, which can be regarded as a way of online diverse image augmentation. Based on this, we further propose a novel diverse inter-domain semantic consistency loss to regularize the cross-domain segmentation. We additionally introduce an intra-domain semantic consistency loss to encourage the segmentation consistency between the original input and the image after cross-cycle reconstruction. Our contributions are summarized as follows:

1.
We propose an end-to-end unsupervised cross-domain image segmentation framework integrating diverse image translation with semantic image segmentation into one network such that the two parts can benefit from each other, i.e., better image translation will improve cross-domain image segmentation and conversely, better image segmentation will regularize cross-domain image translation.
2.
We introduce two consistency losses, i.e., the diverse inter-domain semantic consistency loss and the intra-domain semantic consistency loss, to further regularize the cross-domain segmentation process.
3.
We demonstrate on two public datasets that the proposed end-to-end network, which takes both content features and diverse appearance information into account, producing better segmentation than state-of-the-art (SOTA) cross-domain segmentation methods.

Section snippets

Related work

Unsupervised domain adaptation has been applied to a large number of medical image analysis tasks (Choudhary, Tong, Zhu, Wang, 2020, Guan, Liu, 2021). The existing methods can be largely categorized into two groups: shallow learning-based methods and deep learning-based methods. Below we will give a review of related work.

Method

Let $x \in X$ and $y \in Y$ be images from two domains, and $m_{x} \in M_{X}$ and $m_{y} \in M_{Y}$ be corresponding labels to $x$ and $y$ , respectively. Note that $x$ and $y$ are not necessarily paired, and we have no access to $M_{Y}$ in the training phase. Our goal is to design a network to segment unlabeled images in the target domain $Y$ by making use of $X$ and $M_{X}$ in the source domain. Fig. 1 shows an overview of the proposed CyCMIS framework which consists of two modules: a diverse image translation (DIT) module and a domain-specific

Experiments and results

In this section, we present the experimental results of the proposed CyCMIS framework. Two publicly available image datasets were used in our study: cardiac MR dataset from the Multi-Sequence Cardiac MR Segmentation (MS-CMRSeg) challenge (Zhuang, 2018) and CT-MR dataset from the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge (Zhuang et al., 2019). The MS-CMRSeg challenge dataset contains 45 paired bSSFP CMR and LGE CMR images with ground truth annotations while the MM-WHS challenge

Discussions

Developing UDA methods for cross-domain medical image segmentation is challenging. The goal of the present study is to develop and validate an accurate method that can mitigate such a challenge. In this paper, we presented a cycle-consistent cross-domain medical image segmentation method called CyCMIS, integrating online diverse image translation via disentangled representation learning and semantic consistency regularization into one network. We conducted extensive validation studies to

Conclusion

In this paper, we proposed an end-to-end unsupervised cross-domain medical image segmentation method, taking advantage of diverse image translation via disentangled representation learning and consistency regularization into one network. We characterized the complex relationship between domains as many-to-many mapping and introduced a novel diverse inter-domain semantic consistency loss to regularize the cross-domain segmentation process. We additionally introduced an intra-domain semantic

CRediT authorship contribution statement

Runze Wang: Methodology, Software, Validation, Writing – original draft. Guoyan Zheng: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Supervision, Funding acquisition.

Declaration of Competing Interest

None.

Acknowledgments

This study was partially supported by Shanghai Municipal Science and Technology Commission via Project 20511105205 and by the Natural Science Foundation of China via project U20A20199.

References (44)

J. Cai et al.
Towards cross-modal organ translation and segmentation: a cycle-and shape-consistent generative adversarial network
Med Image Anal
(2019)
A. Chartsias et al.
Disentangle, align and fuse for multimodal and semi-supervised image segmentation
IEEE Trans Med Imaging
(2020)
A. Choudhary et al.
Advancing medical imaging informatics by deep learning-based domain adaptation
Yearb Med Inform
(2020)
J. Patravali et al.
2D-3D Fully Convolutional Neural Networks for Cardiac Mr Segmentation
In: International Workshop on Statistical Atlases and Computational Models of the Heart
(2017)
W. Yan et al.
Edge-guided output adaptor: highly efficient adaptation module for cross-vendor medical image segmentation
IEEE Signal Process Lett
(2019)
G. Zeng et al.
Icmsc: intra-and cross-modality semantic consistency for unsupervised domain adaptation on hip joint bone segmentation
arXiv preprint arXiv:2012.12570
(2020)
H. Zhao et al.
Pyramid Scene Parsing Network
Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2881–2890
(2017)
O. Bernard et al.
Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?
IEEE Trans Med Imaging
(2018)
A. Chartsias et al.
Adversarial Image Synthesis for Unpaired Multi-modal Cardiac Data
In: International workshop on simulation and synthesis in medical imaging
(2017)
A. Chartsias et al.
Disentangled representation learning in cardiac image analysis
Med Image Anal
(2019)

C. Chen et al.

Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation

IEEE Trans Med Imaging

(2020)

C. Chen et al.

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

(2019)

C. Chen et al.

Unsupervised Multi-modal Style Transfer for Cardiac Mr Segmentation

In: International Workshop on Statistical Atlases and Computational Models of the Heart

(2019)

X. Chen et al.

Anatomy-regularized representation learning for cross-modality medical image segmentation

IEEE Trans Med Imaging

(2020)

X. Chen et al.

Diverse data augmentation for learning image segmentation with cross-modality annotations

Med Image Anal

(2021)

V. Cheplygina et al.

Transfer learning for multicenter classification of chronic obstructive pulmonary disease

IEEE J Biomed Health Inform

(2017)

Q. Dou et al.

Pnp-adanet: plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation

IEEE Access

(2019)

Y. Ganin et al.

Domain-adversarial training of neural networks

The journal of machine learning research

(2016)

H. Guan et al.

Domain adaptation for medical image analysis: a survey

arXiv preprint arXiv:2102.09508

(2021)

K. He et al.

Deep Residual Learning for Image Recognition

In: Proceedings of the IEEE conference on computer vision and pattern recognition pp. 770–778

(2016)

T. Heimann et al.

Learning without Labeling: Domain Adaptation for Ultrasound Transducer Localization

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

(2013)

Y. Hiasa et al.

Cross-modality Image Synthesis from Unpaired Data Using Cyclegan

In: International workshop on simulation and synthesis in medical imaging

(2018)

Cited by (28)

Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives
2024, Computers in Biology and Medicine
Deep learning has demonstrated remarkable performance across various tasks in medical imaging. However, these approaches primarily focus on supervised learning, assuming that the training and testing data are drawn from the same distribution. Unfortunately, this assumption may not always hold true in practice. To address these issues, unsupervised domain adaptation (UDA) techniques have been developed to transfer knowledge from a labeled domain to a related but unlabeled domain. In recent years, significant advancements have been made in UDA, resulting in a wide range of methodologies, including feature alignment, image translation, self-supervision, and disentangled representation methods, among others. In this paper, we provide a comprehensive literature review of recent deep UDA approaches in medical imaging from a technical perspective. Specifically, we categorize current UDA research in medical imaging into six groups and further divide them into finer subcategories based on the different tasks they perform. We also discuss the respective datasets used in the studies to assess the divergence between the different domains. Finally, we discuss emerging areas and provide insights and discussions on future research directions to conclude this survey.
Cycle consistent twin energy-based models for image-to-image translation
2024, Medical Image Analysis
Domain shift refers to change of distributional characteristics between the training (source) and the testing (target) datasets of a learning task, leading to performance drop. For tasks involving medical images, domain shift may be caused because of several factors such as change in underlying imaging modalities, measuring devices and staining mechanisms. Recent approaches address this issue via generative models based on the principles of adversarial learning albeit they suffer from issues such as difficulty in training and lack of diversity. Motivated by the aforementioned observations, we adapt an alternative class of deep generative models called the Energy-Based Models (EBMs) for the task of unpaired image-to-image translation of medical images. Specifically, we propose a novel method called the Cycle Consistent Twin EBMs (CCT-EBM) which employs a pair of EBMs in the latent space of an Auto-Encoder trained on the source data. While one of the EBMs translates the source to the target domain the other does vice-versa along with a novel consistency loss, ensuring translation symmetry and coupling between the domains. We theoretically analyze the proposed method and show that our design leads to better translation between the domains with reduced langevin mixing steps. We demonstrate the efficacy of our method through detailed quantitative and qualitative experiments on image segmentation tasks on three different datasets vis-a-vis state-of-the-art methods.
EDRL: Entropy-guided disentangled representation learning for unsupervised domain adaptation in semantic segmentation
2023, Computer Methods and Programs in Biomedicine
Deep learning-based approaches are excellent at learning from large amounts of data, but can be poor at generalizing the learned knowledge to testing datasets with domain shift, i.e., when there exists distribution discrepancy between the training dataset (source domain) and the testing dataset (target domain). In this paper, we investigate unsupervised domain adaptation (UDA) techniques to train a cross-domain segmentation method which is robust to domain shift, eliminating the requirement of any annotations on the target domain.
To this end, we propose an Entropy-guided Disentangled Representation Learning, referred as EDRL, for UDA in semantic segmentation. Concretely, we synergistically integrate image alignment via disentangled representation learning with feature alignment via entropy-based adversarial learning into one network, which is trained end-to-end. We additionally introduce a dynamic feature selection mechanism via soft gating, which helps to further enhance the task-specific feature alignment. We validate the proposed method on two publicly available datasets: the CT-MR dataset and the multi-sequence cardiac MR (MS-CMR) dataset.
On both datasets, our method achieved better results than the state-of-the-art (SOTA) methods. Specifically, on the CT-MR dataset, our method achieved an average DSC of 84.8% when taking CT as the source domain and MR as the target domain, and an average DSC of 84.0% when taking MR as the source domain and CT as the target domain.
Results from comprehensive experiments demonstrate the efficacy of the proposed EDRL model for cross-domain medical image segmentation.
Multi-modality cardiac image computing: A survey
2023, Medical Image Analysis
Multi-modality cardiac imaging plays a key role in the management of patients with cardiovascular diseases. It allows a combination of complementary anatomical, morphological and functional information, increases diagnosis accuracy, and improves the efficacy of cardiovascular interventions and clinical outcomes. Fully-automated processing and quantitative analysis of multi-modality cardiac images could have a direct impact on clinical research and evidence-based patient management. However, these require overcoming significant challenges including inter-modality misalignment and finding optimal methods to integrate information from different modalities.
This paper aims to provide a comprehensive review of multi-modality imaging in cardiology, the computing methods, the validation strategies, the related clinical workflows and future perspectives. For the computing methodologies, we have a favored focus on the three tasks, i.e., registration, fusion and segmentation, which generally involve multi-modality imaging data, either combining information from different modalities or transferring information across modalities. The review highlights that multi-modality cardiac imaging data has the potential of wide applicability in the clinic, such as trans-aortic valve implantation guidance, myocardial viability assessment, and catheter ablation therapy and its patient selection. Nevertheless, many challenges remain unsolved, such as missing modality, modality selection, combination of imaging and non-imaging data, and uniform analysis and representation of different modalities. There is also work to do in defining how the well-developed techniques fit in clinical workflows and how much additional and relevant information they introduce. These problems are likely to continue to be an active field of research and the questions to be answered in the future.
Unpaired, unsupervised domain adaptation assumes your domains are already similar
2023, Medical Image Analysis
Unsupervised domain adaptation is a popular method in medical image analysis, but it can be tricky to make it work: without labels to link the domains, domains must be matched using feature distributions. If there is no additional information, this often leaves a choice between multiple possibilities to map the data that may be equally likely but not equally correct. In this paper we explore the fundamental problems that may arise in unsupervised domain adaptation, and discuss conditions that might still make it work. Focusing on medical image analysis, we argue that images from different domains may have similar class balance, similar intensities, similar spatial structure, or similar textures. We demonstrate how these implicit conditions can affect domain adaptation performance in experiments with synthetic data, MNIST digits, and medical images. We observe that practical success of unsupervised domain adaptation relies on existing similarities in the data, and is anything but guaranteed in the general case. Understanding these implicit assumptions is a key step in identifying potential problems in domain adaptation and improving the reliability of the results.
MIC-Net: A deep network for cross-site segmentation of COVID-19 infection in the fog-assisted IoMT
2023, Information Sciences
Citation Excerpt :
The aim is to improve the classification performance across different domains. Moreover, the work [42] presented a cycle-consistent framework for efficient semantic segmentation from cross domains in the medical field, which integrates consistency regularization and online diverse image translation into a single framework to promote modeling complicated relationships between domains such as many-to-many mappings. In [48], the authors presented a wholly automated machine-agnostic approach for segmenting and quantifying COVID-19 lesions from multisource CT scans.
The automatic segmentation of COVID-19 pneumonia from a computerized tomography (CT) scan has become a major interest for scholars in developing a powerful diagnostic framework in the Internet of Medical Things (IoMT). Federated deep learning (FDL) is considered a promising approach for efficient and cooperative training from multi-institutional image data. However, the nonindependent and identically distributed (Non-IID) data from health care remain a remarkable challenge, limiting the applicability of FDL in the real world. The variability in features incurred by different scanning protocols, scanners, or acquisition parameters produces the learning drift phenomena during the training, which impairs both the training speed and segmentation performance of the model. This paper proposes a novel FDL approach for reliable and efficient multi-institutional COVID-19 segmentation, called MIC-Net. MIC-Net consists of three main building modules: the down-sampler, context enrichment (CE) module, and up-sampler. The down-sampler was designed to effectively learn both local and global representations from input CT scans by combining the advantages of lightweight convolutional and attention modules. The contextual enrichment (CE) module is introduced to enable the network to capture the contextual representation that can be later exploited to enrich the semantic knowledge of the up-sampler through skip connections. To further tackle the inter-site heterogeneity within the model, the approach uses an adaptive and switchable normalization (ASN) to adaptively choose the best normalization strategy according to the underlying data. A novel federated periodic selection protocol (FED-PCS) is proposed to fairly select the training participants according to their resource state, data quality, and loss of a local model. The results of an experimental evaluation of MIC-Net on three publicly available data sets show its robust performance, with an average dice score of 88.90% and an average surface dice of 87.53%.

View all citing articles on Scopus

¹: Authors contribute equally to the paper.

View full text

CyCMIS: Cycle-consistent Cross-domain Medical Image Segmentation via diverse image augmentation

Highlights

Abstract

Graphical abstract

Introduction

Section snippets

Related work

Method

Experiments and results

Discussions

Conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgments

Med Image Anal

IEEE Trans Med Imaging

Yearb Med Inform

IEEE Signal Process Lett

arXiv preprint arXiv:2012.12570

Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?

IEEE Trans Med Imaging

Adversarial Image Synthesis for Unpaired Multi-modal Cardiac Data

In: International workshop on simulation and synthesis in medical imaging

Disentangled representation learning in cardiac image analysis

Med Image Anal

Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation

IEEE Trans Med Imaging

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Unsupervised Multi-modal Style Transfer for Cardiac Mr Segmentation

In: International Workshop on Statistical Atlases and Computational Models of the Heart

Anatomy-regularized representation learning for cross-modality medical image segmentation

IEEE Trans Med Imaging

Diverse data augmentation for learning image segmentation with cross-modality annotations

Med Image Anal

Transfer learning for multicenter classification of chronic obstructive pulmonary disease

IEEE J Biomed Health Inform

Pnp-adanet: plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation

IEEE Access

Domain-adversarial training of neural networks

The journal of machine learning research

Domain adaptation for medical image analysis: a survey

arXiv preprint arXiv:2102.09508

Deep Residual Learning for Image Recognition

In: Proceedings of the IEEE conference on computer vision and pattern recognition pp. 770–778

Learning without Labeling: Domain Adaptation for Ultrasound Transducer Localization

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Cross-modality Image Synthesis from Unpaired Data Using Cyclegan

In: International workshop on simulation and synthesis in medical imaging