Skip to main content
Springer Nature Link
Log in

Synthesizing Multi-Contrast MR Images Via Novel 3D Conditional Variational Auto-Encoding GAN

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

As two different modalities of medical images, Magnetic Resonance (MR) and Computer Tomography (CT), provide mutually-complementary information to doctors in clinical applications. However, to obtain both images sometimes is cost-consuming and unavailable, particularly for special populations. For example, patients with metal implants are not suitable for MR scanning. Also, it is probably infeasible to acquire multi-contrast MR images during once clinical scanning. In this context, to synthesize needed MR images for patients whose CT images are available becomes valuable. To this end, we present a novel generative network, called CAE-ACGAN, which incorporates the advantages of Variational Auto-Encoder (VAE) and Generative Adversarial Network (GAN) with an auxiliary discriminative classifier network. We apply this network to synthesize multi-contrast MR images from single CT and conduct experiments on brain datasets. Our main contributions can be summarized as follows: 1)We alleviate the problems of images blurriness and mode collapse by integrating the advantages of VAE and GAN; 2) We solve the complicated cross-domain, multi-contrast MR synthesis task using the proposed network; 3) The technique of random-extraction-patches is used to lower the limit of insufficient training data, enabling to obtain promising results even with limited available data; 4) By comparing with other typical networks, we are able to yield nearer-real, higher-quality synthetic MR images, demonstrating the effectiveness and stability of our proposed network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The dataset used to support the findings of this study are available from the corresponding author upon request.

References

  1. Villeirs GM, Vaerenbergh KV, Vakaet L, Bral S, Claus F, Neve WJD (2005) Interobserver delineation variation using CT versus combined CT + MRI in intensity-modulated radiotherapy for prostate cancer. Strahlenther Onkol 181(7):424–430

    Article  Google Scholar 

  2. Vinod SK, Jameson MG, Min M, Holloway LC (2016) Uncertainties in volume delineation in radiation oncology: a systematic review and recommendations for future studies. Radiother Oncol 121(2):169–179

    Article  Google Scholar 

  3. Han X (2017) MR-based synthetic CT generation using a deep convolutional neural network method. Med Phys 44(4):1408–1419

    Article  Google Scholar 

  4. Wolterink JM, Dinkla AM, Savenije MHF, Seevinck PR, van den Berg CAT, Išgum I (2017) Deep MR to CT synthesis using unpaired data, SASHIMI 2017: Simulation and Synthesis in Medical Imaging, vol.10557 of Lecture Notes in Computer Science, pp. 14–23

  5. Nie D, Trullo R, Lian J, Wang L, Petitjean C, Ruan S, Wang Q, Shen D (2018) Medical image synthesis with deep convlutional adversarial networks. IEEE Trans Biomed Eng 65(12):2720–2730

    Article  Google Scholar 

  6. Xu K; Cao JW; Xia KJ; Yang, H, et.al. (2019) Multichannel residual conditional GAN-leveraged abdominal pseudo-CT generation via Dixon MR images. IEEE Access 7:163823–163830

  7. Qian P, Chen Y, Kuo JW, Zhang YD, Jiang Y et al (2019) mDixon-based synthetic CT generation for PET attenuation correction on abdomen and pelvis jointly using transfer fuzzy clustering and active learning-based classification. IEEE Trans Med Imaging 39(4):819–832

    Article  Google Scholar 

  8. Qian P, Xu K, Wang T, Zheng QK, Yang H, Baydoun A, Zhu J, Traughber B, Muzic RF Jr (2020) Estimating CT from MR abdominal images using novel generative adversarial networks. J Grid Comput 18(2):211–226

    Article  Google Scholar 

  9. Yang H, Xia K, Anqi B, Qian P, Khosravi MR (2019) Abdomen MRI synthesis based on conditional GAN. 2019 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, pp. 1021–1025

  10. Iglesias JE, Konukoglu E, Zikic D, Glocker B, Leemput KV, Fischl B (2013) Is synthesizing MRI contrast useful for inter-modality analysis? Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013: 16th International Conference, Nagoya, Japan, September 22–26, Proceedings, Part I (pp.631–638)

  11. Tulder G, Bruijne M (2015) Why does synthesized data improve multi-sequence classification? Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, vol.9349 of Lecture Notes in Computer Science, pp.531–538

  12. Calimeri F, Marzullo A, Stamile C, Terracina G (2017) Biomedical data augmentation using generative adversarial neural networks. Artificial Neural Networks and Machine Learning – ICANN 2017, vol.10614 of Lecture Notes in Computer Science, pp. 626–634

  13. Chartsias A, Joyce T, Dharmakumar R, Tsaftaris S (2017) Adversarial Image synthesis for unpaired multi-modal cardiac data. SASHIMI 2017: Simulation and Synthesis in Medical Imaging, vol.10557 of Lecture Notes in Computer Science 10557, pp. 3–13

  14. Zhu J, Park T, Isola P, Efros A (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks, 2017 IEEE International Conference on Computer Vision (ICCV), Venice

  15. Dong N, Lei Y, Tian S, Wang T, Patel P, Curran WJ, Jani AB, Liu T, Yang X (2019) Synthetic MRI-aided multi-organ segmentation on male pelvic CT using cycle consistent deep attention network. Radiother Oncol 141:192–199

    Article  Google Scholar 

  16. Lei Y, Wang T, Tian S, Dong X, Jani AB, Schuster D, Curran WJ, Patel P, Liu T, Yang X (2019) Male pelvic multi-organ segmentation aided by CBCT-based synthetic MRI. Phys Med Biol 65(3)

  17. Jiang J, Hu YC, Tyagi N, Zhang P, Rimner A, Mageras GS, Deasy JO, Veeraraghavan H (2018) Tumor-aware, adversarial domain adaptation from ct to mri for lung cancer segmentation. Med Image Comput Comput Assist Interv 11071:777–785

    Google Scholar 

  18. Duchateau N, Sermesant M, Delingette H, Ayache N (2018) Model-based generation of large databases of cardiac images: synthesis of pathological cine MR sequences from real healthy cases. IEEE Trans Med Imaging 37:755–766

    Article  Google Scholar 

  19. Dar S, Yurt M, Karacan L, Erdem A, Erdem E, Cukur T (2018) Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Trans Med Imaging 38:2375–2388

    Article  Google Scholar 

  20. Zhang Z, Yang L, Zheng Y (2018) Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City

  21. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems, 2:2672–2680

  22. Han C, Hayashi H, Rundo L, Araki R (2018) GAN-based synthetic brain MR image generation. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). Washington

  23. Kwon G, Han C, Kim DS (2019) Generation of 3D Brain MRI using auto-encoding generative adversarial networks. the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2019), arXiv:1908.02498 [eess.IV]

  24. Rosca M, Lakshminarayanan B, Warde-Farley D, Mohamed S (2017) Variational approaches for auto-encoding generative adversarial networks. arXiv:1706.04987 [stat.ML]

  25. Mirza M, Osindero S (2014) Conditional generative adversarial nets, arXiv:1411.1784 [cs. LG]

  26. Nguyen H V, Zhou K, Vemulapalli R (2015) Cross-domain synthesis of medical images using efficient location-sensitive deep network. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer International Publishing, 9349:677–684

  27. Dar SU, Yurt M, Karacan L, Erdem A, Erdem CT (2019) Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Trans Med Imaging 38(10):2375–2388

    Article  Google Scholar 

  28. Kingma DP, Welling M (2013) Auto-encoding Variational Bayes. arXiv:1312.6114

  29. Nguyen H, Zhou K, Vemulapalli R (2015) Cross-domain synthesis of medical images using efficient location-sensitive deep network. Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015, vol.9349 of Lecture Notes in Computer Science, pp. 677–684

  30. Bi L, Kim J, Kumar A, Feng D, Fulham M (2017) Synthesis of positron emission tomography (PET) images via multi-channel generative adversarial networks (GANs). RAMBO 2017, CMMI 2017, SWITCH 2017: Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment, vol. 10555 of Lecture Notes in Computer Science, pp. 43–51

  31. Hu Y, Gibson E, Lee L, Xie W, Barratt D, Vercauteren T, Noble J (2017) Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks. RAMBO 2017, CMMI 2017, SWITCH 2017: molecular imaging, reconstruction and analysis of moving body organs, and stroke imaging and treatment, vol.10555 of Lecture Notes in Computer Science, pp. 105–115

  32. Ben-Cohen A, Klang E, Raskin S, Amitai M, Greenspan H (2017) Virtual PET images from CT data using deep convolutional networks: initial results. SASHIMI 2017: Simulation and Synthesis in Medical Imaging, vol.10557 of Lecture Notes in Computer Science, pp. 49–57

  33. Zhao H, Li H, Stroh S, Cheng L (2018) Synthesizing retinal and neuronal images with generative adversarial nets. Med Image Anal 49:14–26

    Article  Google Scholar 

  34. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks, arXiv:1611.07004 [cs.CV]

  35. Zhu J-Y, Zhang R, Pathak D, Darrell T, Efros A A, Wang O, Shechtman E (2017) Toward multimodal image-to-image translation, In Neural Information Processing Systems, 2017 (NIPS 2017)

  36. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein GAN, arXiv:1701.07875 [stat.ML]

  37. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville A (2017) Improved training of Wasserstein GANs, arXiv:1704.00028 [cs.LG]

  38. Kingma DP, Rezende DJ, Mohamed S, Welling M (2014) Semi-supervised learning with deep generative models. Advances in Neural Information Processing Systems 27 (NIPS 2014), arXiv:1406.5298 [cs.LG]

  39. Sohn K, Lee H, Yan X (2015) Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems 28 (NIPS 2015)

  40. Larsen A B L, Sønderby S K, Winther O (2015) Autoencoding beyond pixels using a learned similarity metric. arXiv preprint arXiv:1512.09300

  41. Bao J, Chen D, Wen F, Li H, Hua G (2017) CVAE-GAN: fine-grained image generation through asymmetric training. 2017 IEEE International Conference on Computer Vision (ICCV), 1:2764–2773

  42. Odena A, Olah C, Shlens J. (2016) Conditional image synthesis with auxiliary classifier GANs. arXiv:1610.09585 [stat.ML]

  43. Johnson PB, Padgett KR, Chen KL, Dogan N (2016) Evaluation of the tool “Reg refine” for user-guided deformable image registration. J Appl Clin Med Phys 17:158–170

    Article  Google Scholar 

  44. Muzic RF, Cornelius S (2001) COMKAT: compartment model kinetic analysis tool. J Nucl Med 42:636–645

    Google Scholar 

  45. Fang Y-HD, Asthana P, Salinas C, Huang H-M, Muzic RF (2010) Integrated software environment based on COMKAT for analyzing tracer pharmacokinetics with molecular imaging. J Nucl Med 51:77–84

    Article  Google Scholar 

  46. Nie D, Trullo R, Lian J, Wang L, Petitjean C, Ruan S, Wang Q, Shen D (2018) Medical image synthesis with deep convlutional adversarial networks. IEEE Trans Biomed Eng 65.12(2018):2720–2730

    Article  Google Scholar 

  47. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

Download references

Funding

This work was partly supported by the National Natural Science Foundation of China under grants 61772241 and 61702225, by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under grant 18KJB520048, by the Science and Technology Demonstration Project of Social Development of Wuxi under grant WX18IVJN002, and by JiangSu 333 Expert Engineering under grant BRA2019189.

Author information

Authors and Affiliations

  1. School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, 214122, China

    Huan Yang, Yizhang Jiang & Pengjiang Qian

  2. Jiangsu Key Laboratory of Media Design and Software Technology, Jiangnan University, Wuxi, 214122, China

    Huan Yang, Xianling Lu, Yizhang Jiang & Pengjiang Qian

  3. School of Internet of Things Engineering, Jiangnan University, Wuxi, 214122, China

    Huan Yang & Xianling Lu

  4. School of Architecture Building and Civil Engineering, Loughborough University, Loughborough, LE11 3TU, UK

    Shui-Hua Wang

  5. School of Educational Science, Nanjing Normal University, Nanjing, 210096, China

    Zhihai Lu

  6. Wuxi IoT Innovation Center Co., Ltd, Wuxi, 214000, China

    Jian Yao

Authors
  1. Huan Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Xianling Lu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Shui-Hua Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Zhihai Lu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Jian Yao
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Yizhang Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Pengjiang Qian
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Xianling Lu or Pengjiang Qian.

Ethics declarations

Conflict of interest

We declare that there are no conflicts of interest.

Code availability

The code used to implement the findings of this study are available from the corresponding author upon request.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, H., Lu, X., Wang, SH. et al. Synthesizing Multi-Contrast MR Images Via Novel 3D Conditional Variational Auto-Encoding GAN. Mobile Netw Appl 26, 415–424 (2021). https://doi.org/10.1007/s11036-020-01678-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-020-01678-1

Keywords