Tripartite-GAN: Synthesizing liver contrast-enhanced MRI to improve tumor detection
Graphical abstract
Introduction
The magnetic resonance examination of liver cancer relies heavily on contrast agent (CA) injection. Specifically, as shown in Fig. 1, in the non-contrast enhanced magnetic resonance imaging (NCEMRI) obtained without CA injection, the area of hemangioma (a benign tumor) and hepatocellular carcinoma (HCC, a malignant tumor) could barely find their difference to distinguish. On the contrary, in contrast-enhanced MRI (CEMRI) obtained by the CA injection, the area of hemangioma is gradual central filling and bright at the edge and the area of HCC is entirely or mostly bright through the whole tumor. It is no doubt that CA injection gives the two kinds of tumors their diagnosis specificity, which provides an accurate and easy way to diagnose hemangioma and HCC.
However, gadolinium-based CA brings inevitable shortcomings, which suffers from high-risk, time-consuming, and expensive (Idée et al., 2006). The high-risk is due to the gadolinium-based CA injection, which may induce nephrogenic systemic fibrosis (Marckmann et al., 2006), especially for patients with compromised kidney function. The time-consuming comes from the MRI process itself and the waiting-time after CA injection. The expensive mainly comes from CA, in the USA alone, conservatively, if each dose of CA is $60, the direct material expense alone equates to roughly $1.2 billion in 2016 (Statistics from IQ-AI Limited Company, USA). It will be significant for clinical diagnosis if CEMRI can be successfully synthesized without CA injection. Therefore, this work focuses on the end-to-end method of synthesizing liver CEMRI from liver NCEMRI for tumor detection.
There is currently no reported synthesis of liver CEMRI for tumor detection because of three unique challenges: 1) The difficulty in discriminating the tumor features extracted in NCEMRI. That is to say. It is easy to confuse the features of hemangioma and HCC when extracting the features because of the low discrimination of hemangioma and HCC in NCEMRI. Therefore, the synthesis network is required to pay more attention to the detailed feature for improving the feature representation. 2) The difficulty in learning the highly nonlinear mapping between multi-class NCEMRI and multi-class CEMRI. That is to say. Each anatomy can be seen as a class. Different from the synthesis of single-class medical images (e.g., brain MRI (Yang et al., 2018), lesion area patches (Frid-Adar et al., 2018)), liver MRI has multi-class anatomy (i.e., liver, spleen, spine, and so on). Therefore, multi-class liver MRI has the risk of causing misclassification of the anatomy. The synthesis network is required to explicitly capture global dependencies of multi-class feature representations regardless of locations. 3) The difficulty in alleviating the blurring problem of synthetic CEMRI. The blurring of synthetic image is a problem that GAN always needs to alleviate (Korkinof et al., 2018), and the problem becomes more serious due to the complex anatomy of multi-class liver MRI. That is to say. For our task of synthesizing liver CEMRI to improve tumor detection, we must ensure not only the high quality of CEMRI synthesis but also the clarity of the tumor area. Therefore, an effective loss function is necessary for our task due to the loss function determines the ability to learn the highly nonlinear mapping between the source image and the target image.
In this paper, we propose a novel Tripartite Generative Adversarial Network (Tripartite-GAN) as a non-invasive, time-saving, and inexpensive clinical tool to synthesize liver CEMRI without CA injection for tumor detection. Specifically, for the first time, the Tripartite-GAN combines three associated-network (an attention-aware generator, a convolutional neural network-based (CNN-based) discriminator, and a region-based convolutional neural network-based (R-CNN-based) detector), which simultaneously achieves CEMRI synthesis and tumor detection in an end-to-end framework. Firstly, in order to overcome the aforementioned challenges of 1) and 2), the newly designed attention-aware generator expands the receptive field via hybrid convolution, integrates local features with their global dependencies via dual attention module (DAM), and improves the convergence of loss via residual learning. This is capable of effectively extracting the diagnosis-specific features of two types of tumor and accurately learning the highly nonlinear mapping between multi-class NCEMRI and multi-class CEMRI. Secondly, in order to overcome the aforementioned challenge of 3) for achieving high-quality CEMRI synthesis, which equivalent to real CEMRI. The CNN-based discriminator is trained to discriminate the real or fake of synthetic CEMRI, and then promotes the generator to synthesize highly authentic CEMRI via adversarial-strategy. Thirdly, the R-CNN-based detector is combined to the generator via back-propagation for the first time, which achieves that CEMRI synthesis and tumor detection promote each other in an end-to-end framework. Moreover, the attention maps obtained from the generator newly added into the detector improve the performance of tumor detection. The contributions of this study are mainly in four aspects:
- 1.
For the first time, synthesizing CEMRI without CA injection for liver tumor detection is achieved, which provides a safe, time-saving, and inexpensive clinical tool to synthesize CEMRI without CA injection.
- 2.
The newly proposed Tripartite-GAN successfully combined the regular two-participant GAN and the detector via back-propagation for the first time, which achieves that CEMRI synthesis and tumor detection promote each other in an end-to-end framework.
- 3.
The newly designed attention-aware generator is powerful in feature extraction with the help of hybrid convolution, residual learning, and DAM. Specifically, the hybrid convolution enlarges the receptive field efficiently, the residual learning benefits the convergence to facilitate the training of the generator, and the DAM enhances feature representation learning of tumor specificity and context learning of multi-class liver MRI.
- 4.
Attention maps from the generator newly added into the detector in the manner of residual connection improve VGG-16 based convolution operation to extract tumor information better, which improves the performance of tumor detection.
Section snippets
Related work on tumor diagnosis in liver MRI
Studies have shown that in liver tumor diagnosis, MRI is useful and more sensitive than other modalities in the tumor diagnosis and characterization of hemangioma and HCC with the help of CA injection (Digumarthy, Sahani, Saini, 2005, Low, 2007, Halavaara, Breuer, Ayuso, Balzer, Bellin, Blomqvist, Carter, Grazioli, Hammerstingl, Huppertz, et al., 2006). However, the dependence of existing magnetic resonance examination on CA injection also brings inevitable shortcoming, which suffers from
An overview of Tripartite-GAN
For effective CEMRI synthesis and tumor detection, our Tripartite-GAN executes the competition between three participants: the newly designed attention-aware generator (Section 3.2), the CNN-based discriminator (Section 3.3), and the R-CNN-based detector (Section 3.4). Fig. 2 shows the design of our newly proposed Tripartite-GAN. Specifically, the attention-aware generator is a hybrid convolution network to synthesize tumor-specific CEMRI, which facilitates tumor detection. The generator is
Materials and implementation
The experimental datasets we used totaling 265 subjects (75 subjects of hemangioma, 138 subjects of HCC, and 52 subjects of health), and each subject has corresponding NCEMRI and CEMRI (after gadolinium CA injection) collected after standard clinical liver MRI examinations. And all subjects are provided after approval by the McGill University Health Centre. The corresponding axial T1 FS Pre-Contrast MRI [4mm; 512 × 512px] and axial T1 FS Delay MRI [4mm; 512 × 512px] are selected for our
Conclusions and discussion
Our proposed Tripartite-GAN successfully synthesized liver CEMRI without CA injection on a dataset of 265 subjects. The synthesized liver CEMRI has the equivalent value of real CEMRI in clinical diagnosis and then used to accurate tumor detection. All of the results demonstrate that Tripartite-GAN can aid in the clinical diagnosis as a safe, time-saving, and inexpensive tool to synthesize and detect CEMRI without CA injection. The effective application of the hybrid convolution, residual
Declaration of Competing Interest
None.
Acknowledgments
This work was funded by the China Scholarship Council (No. 201808370212), the National Natural Science Foundation of China (61971271), the Taishan Scholars Project of Shandong Province (Tsqn20161023) and the Primary Research and Development Plan of Shandong Province (No. 2018GGX101018, No. 2019QYTPY020).
References (39)
- et al.
Cross-modality synthesis from CT to PET using FCN and GAN networks for improved automated lesion detection
Eng. Appl. Artif. Intell.
(2019) - et al.
GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification
Neurocomputing
(2018) Abdominal MRI advances in the detection of liver tumours and characterisation
Lancet Oncol.
(2007)- et al.
Diffeomorphic demons: efficient non-parametric image registration
NeuroImage
(2009) - et al.
Automatic spondylolisthesis grading from MRIs across modalities using faster adversarial recognition network
Med. Image Anal.
(2019) - et al.
A region based convolutional network for tumor detection and classification in breast mammography
Deep Learning and Data Labeling for Medical Applications
(2016) - Beers, A., Brown, J., Chang, K., Campbell, J. P., Ostmo, S., Chiang, M. F., Kalpathy-Cramer, J., 2018. High-resolution...
- et al.
Triple generative adversarial nets
Advances in Neural Information Processing Systems
(2017) - et al.
End-to-end adversarial retinal image synthesis
IEEE Trans. Med. Imaging
(2017) - et al.
MRI in detection of hepatocellular carcinoma (HCC)
Cancer Imaging
(2005)
Automatic brain tumor detection and segmentation using u-net based fully convolutional networks
Annual Conference on Medical Image Understanding and Analysis
Generating synthetic CTs from magnetic resonance images using generative adversarial networks
Med. Phys.
Dual attention network for scene segmentation
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Fast R-CNN
Proceedings of the IEEE International conference on Computer Vision
Generative adversarial nets
Advances in Neural Information Processing Systems
Liver tumor characterization: comparison between liver-specific gadoxetic acid disodium-enhanced MRI and biphasic CT-A multicenter trial
J. Comput. Assist. Tomogr.
Deep residual learning for image recognition
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Relation networks for object detection
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Squeeze-and-excitation networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Cited by (67)
Deep learning based synthesis of MRI, CT and PET: Review and analysis
2024, Medical Image AnalysisSpatiotemporal knowledge teacher–student reinforcement learning to detect liver tumors without contrast agents
2023, Medical Image AnalysisHeuristic multi-modal integration framework for liver tumor detection from multi-modal non-enhanced MRIs
2023, Expert Systems with ApplicationsNCCT-CECT image synthesizers and their application to pulmonary vessel segmentation
2023, Computer Methods and Programs in BiomedicineData synthesis and adversarial networks: A review and meta-analysis in cancer imaging
2023, Medical Image Analysis