Tripartite-GAN: Synthesizing liver contrast-enhanced MRI to improve tumor detection

https://doi.org/10.1016/j.media.2020.101667Get rights and content

Highlights

  • A first method is proposed to promote tumor detection by CEMRI synthesis without CA injection.

  • A novel Tripartite-GAN successfully combined regular two-participant GAN and the detector via back-propagation.

  • The innovative attention-aware generator enhances feature extraction via hybrid convolution, residual learning, and DAM.

  • Attention maps from the generator added into the detector improves tumor detection.

Abstract

Contrast-enhanced magnetic resonance imaging (CEMRI) is crucial for the diagnosis of patients with liver tumors, especially for the detection of benign tumors and malignant tumors. However, it suffers from high-risk, time-consuming, and expensive in current clinical diagnosis due to the use of the gadolinium-based contrast agent (CA) injection. If the CEMRI can be synthesized without CA injection, there is no doubt that it will greatly optimize the diagnosis. In this study, we propose a Tripartite Generative Adversarial Network (Tripartite-GAN) as a non-invasive, time-saving, and inexpensive clinical tool by synthesizing CEMRI to detect tumors without CA injection. Specifically, our innovative Tripartite-GAN combines three associated-networks (an attention-aware generator, a convolutional neural network-based discriminator, and a region-based convolutional neural network-based detector) for the first time, which achieves CEMRI synthesis and tumor detection promoting each other in an end-to-end framework. The generator facilitates detector for accurate tumor detection via synthesizing tumor-specific CEMRI. The detector promotes the generator for accurate CEMRI synthesis via the back-propagation. In order to synthesize CEMRI of equivalent clinical value to real CEMRI, the attention-aware generator expands the receptive field via hybrid convolution, and enhances feature representation and context learning of multi-class liver MRI via dual attention mechanism, and improves the performance of convergence of loss via residual learning. Moreover, the attention maps obtained from the generator newly added into the detector improve the performance of tumor detection. The discriminator promotes the generator to synthesize high-quality CEMRI via the adversarial learning strategy. This framework is tested on a large corpus of axial T1 FS Pre-Contrast MRI and axial T1 FS Delay MRI of 265 subjects. Experimental results and quantitative evaluation demonstrate that the Tripartite-GAN achieves high-quality CEMRI synthesis that peak signal-to-noise rate of 28.8 and accurate tumor detection that accuracy of 89.4%, which reveals that Tripartite-GAN can aid in the clinical diagnosis of liver tumors.

Introduction

The magnetic resonance examination of liver cancer relies heavily on contrast agent (CA) injection. Specifically, as shown in Fig. 1, in the non-contrast enhanced magnetic resonance imaging (NCEMRI) obtained without CA injection, the area of hemangioma (a benign tumor) and hepatocellular carcinoma (HCC, a malignant tumor) could barely find their difference to distinguish. On the contrary, in contrast-enhanced MRI (CEMRI) obtained by the CA injection, the area of hemangioma is gradual central filling and bright at the edge and the area of HCC is entirely or mostly bright through the whole tumor. It is no doubt that CA injection gives the two kinds of tumors their diagnosis specificity, which provides an accurate and easy way to diagnose hemangioma and HCC.

However, gadolinium-based CA brings inevitable shortcomings, which suffers from high-risk, time-consuming, and expensive (Idée et al., 2006). The high-risk is due to the gadolinium-based CA injection, which may induce nephrogenic systemic fibrosis (Marckmann et al., 2006), especially for patients with compromised kidney function. The time-consuming comes from the MRI process itself and the waiting-time after CA injection. The expensive mainly comes from CA, in the USA alone, conservatively, if each dose of CA is $60, the direct material expense alone equates to roughly $1.2 billion in 2016 (Statistics from IQ-AI Limited Company, USA). It will be significant for clinical diagnosis if CEMRI can be successfully synthesized without CA injection. Therefore, this work focuses on the end-to-end method of synthesizing liver CEMRI from liver NCEMRI for tumor detection.

There is currently no reported synthesis of liver CEMRI for tumor detection because of three unique challenges: 1) The difficulty in discriminating the tumor features extracted in NCEMRI. That is to say. It is easy to confuse the features of hemangioma and HCC when extracting the features because of the low discrimination of hemangioma and HCC in NCEMRI. Therefore, the synthesis network is required to pay more attention to the detailed feature for improving the feature representation. 2) The difficulty in learning the highly nonlinear mapping between multi-class NCEMRI and multi-class CEMRI. That is to say. Each anatomy can be seen as a class. Different from the synthesis of single-class medical images (e.g., brain MRI (Yang et al., 2018), lesion area patches (Frid-Adar et al., 2018)), liver MRI has multi-class anatomy (i.e., liver, spleen, spine, and so on). Therefore, multi-class liver MRI has the risk of causing misclassification of the anatomy. The synthesis network is required to explicitly capture global dependencies of multi-class feature representations regardless of locations. 3) The difficulty in alleviating the blurring problem of synthetic CEMRI. The blurring of synthetic image is a problem that GAN always needs to alleviate (Korkinof et al., 2018), and the problem becomes more serious due to the complex anatomy of multi-class liver MRI. That is to say. For our task of synthesizing liver CEMRI to improve tumor detection, we must ensure not only the high quality of CEMRI synthesis but also the clarity of the tumor area. Therefore, an effective loss function is necessary for our task due to the loss function determines the ability to learn the highly nonlinear mapping between the source image and the target image.

In this paper, we propose a novel Tripartite Generative Adversarial Network (Tripartite-GAN) as a non-invasive, time-saving, and inexpensive clinical tool to synthesize liver CEMRI without CA injection for tumor detection. Specifically, for the first time, the Tripartite-GAN combines three associated-network (an attention-aware generator, a convolutional neural network-based (CNN-based) discriminator, and a region-based convolutional neural network-based (R-CNN-based) detector), which simultaneously achieves CEMRI synthesis and tumor detection in an end-to-end framework. Firstly, in order to overcome the aforementioned challenges of 1) and 2), the newly designed attention-aware generator expands the receptive field via hybrid convolution, integrates local features with their global dependencies via dual attention module (DAM), and improves the convergence of loss via residual learning. This is capable of effectively extracting the diagnosis-specific features of two types of tumor and accurately learning the highly nonlinear mapping between multi-class NCEMRI and multi-class CEMRI. Secondly, in order to overcome the aforementioned challenge of 3) for achieving high-quality CEMRI synthesis, which equivalent to real CEMRI. The CNN-based discriminator is trained to discriminate the real or fake of synthetic CEMRI, and then promotes the generator to synthesize highly authentic CEMRI via adversarial-strategy. Thirdly, the R-CNN-based detector is combined to the generator via back-propagation for the first time, which achieves that CEMRI synthesis and tumor detection promote each other in an end-to-end framework. Moreover, the attention maps obtained from the generator newly added into the detector improve the performance of tumor detection. The contributions of this study are mainly in four aspects:

  • 1.

    For the first time, synthesizing CEMRI without CA injection for liver tumor detection is achieved, which provides a safe, time-saving, and inexpensive clinical tool to synthesize CEMRI without CA injection.

  • 2.

    The newly proposed Tripartite-GAN successfully combined the regular two-participant GAN and the detector via back-propagation for the first time, which achieves that CEMRI synthesis and tumor detection promote each other in an end-to-end framework.

  • 3.

    The newly designed attention-aware generator is powerful in feature extraction with the help of hybrid convolution, residual learning, and DAM. Specifically, the hybrid convolution enlarges the receptive field efficiently, the residual learning benefits the convergence to facilitate the training of the generator, and the DAM enhances feature representation learning of tumor specificity and context learning of multi-class liver MRI.

  • 4.

    Attention maps from the generator newly added into the detector in the manner of residual connection improve VGG-16 based convolution operation to extract tumor information better, which improves the performance of tumor detection.

Section snippets

Related work on tumor diagnosis in liver MRI

Studies have shown that in liver tumor diagnosis, MRI is useful and more sensitive than other modalities in the tumor diagnosis and characterization of hemangioma and HCC with the help of CA injection (Digumarthy, Sahani, Saini, 2005, Low, 2007, Halavaara, Breuer, Ayuso, Balzer, Bellin, Blomqvist, Carter, Grazioli, Hammerstingl, Huppertz, et al., 2006). However, the dependence of existing magnetic resonance examination on CA injection also brings inevitable shortcoming, which suffers from

An overview of Tripartite-GAN

For effective CEMRI synthesis and tumor detection, our Tripartite-GAN executes the competition between three participants: the newly designed attention-aware generator (Section 3.2), the CNN-based discriminator (Section 3.3), and the R-CNN-based detector (Section 3.4). Fig. 2 shows the design of our newly proposed Tripartite-GAN. Specifically, the attention-aware generator is a hybrid convolution network to synthesize tumor-specific CEMRI, which facilitates tumor detection. The generator is

Materials and implementation

The experimental datasets we used totaling 265 subjects (75 subjects of hemangioma, 138 subjects of HCC, and 52 subjects of health), and each subject has corresponding NCEMRI and CEMRI (after gadolinium CA injection) collected after standard clinical liver MRI examinations. And all subjects are provided after approval by the McGill University Health Centre. The corresponding axial T1 FS Pre-Contrast MRI [4mm; 512  ×  512px] and axial T1 FS Delay MRI [4mm; 512  ×  512px] are selected for our

Conclusions and discussion

Our proposed Tripartite-GAN successfully synthesized liver CEMRI without CA injection on a dataset of 265 subjects. The synthesized liver CEMRI has the equivalent value of real CEMRI in clinical diagnosis and then used to accurate tumor detection. All of the results demonstrate that Tripartite-GAN can aid in the clinical diagnosis as a safe, time-saving, and inexpensive tool to synthesize and detect CEMRI without CA injection. The effective application of the hybrid convolution, residual

Declaration of Competing Interest

None.

Acknowledgments

This work was funded by the China Scholarship Council (No. 201808370212), the National Natural Science Foundation of China (61971271), the Taishan Scholars Project of Shandong Province (Tsqn20161023) and the Primary Research and Development Plan of Shandong Province (No. 2018GGX101018, No. 2019QYTPY020).

References (39)

  • H. Dong et al.

    Automatic brain tumor detection and segmentation using u-net based fully convolutional networks

    Annual Conference on Medical Image Understanding and Analysis

    (2017)
  • H. Emami et al.

    Generating synthetic CTs from magnetic resonance images using generative adversarial networks

    Med. Phys.

    (2018)
  • J. Fu et al.

    Dual attention network for scene segmentation

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2019)
  • R. Girshick

    Fast R-CNN

    Proceedings of the IEEE International conference on Computer Vision

    (2015)
  • I. Goodfellow et al.

    Generative adversarial nets

    Advances in Neural Information Processing Systems

    (2014)
  • J. Halavaara et al.

    Liver tumor characterization: comparison between liver-specific gadoxetic acid disodium-enhanced MRI and biphasic CT-A multicenter trial

    J. Comput. Assist. Tomogr.

    (2006)
  • K. He et al.

    Deep residual learning for image recognition

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2016)
  • H. Hu et al.

    Relation networks for object detection

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2018)
  • J. Hu et al.

    Squeeze-and-excitation networks

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2018)
  • Cited by (67)

    View all citing articles on Scopus
    View full text