Skip to main content
SpringerLink
Log in

A novel structure preserving generative adversarial network for CT to MR modality translation of spine

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a structure preserving generative adversarial network (S-P GAN) to solve the problem of small structure information loss during the modality translation from computed tomography (CT) to magnetic resonance (MR). A novel generator of the S-P GAN is designed to encode features from CT, where an original CT information branch and its corresponding high-frequency information branch form a dual-branch structure. The small details are highlighted by using a filter in the high-frequency information branch, which offers a complement to the integrity of structural information in CT. Meanwhile, a mixed attention mechanism is introduced to better fuse the dual-branch features for decoding features to MR, where small structure features get more attention in channel and space. Additionally, a new joint loss function is proposed to guide the adversarial training of S-P GAN, which contains structural consistency constrain, pixel translation constrain, and adversarial constrain, so that global similarity and local detail consistency are obtained at the same time. Experimental results show that the results of the S-P GAN are superior to the state-of-the-art models in mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM). In real clinical situations, the proposed method has shown good performance in the diagnosis of lumbar disk herniation, the new and old degree of compressibility fracture, and the Modic change of cartilage end plate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The datasets generated during and/or analyzed during the current study are not available due to proprietary reasons.

References

  1. Wang L, Wang H, Huang Y, Yan B, Chang Z, Liu Z, Zhao M, Cui L, Song J, Li F (2022) Trends in the application of deep learning networks in medical image analysis: Evolution between 2012 and 2020. Eur J Radiol 146:110069

    Article  PubMed  Google Scholar 

  2. Al-Riyami K, Vöö S, Gnanasegaran G, Pressney I, Meir A, Casey A, Molloy S, Allibone J, Bomanji J (2019) The role of bone spect/ct in patients with persistent or recurrent lumbar pain following lumbar spine stabilization surgery. Eur J Nucl Med Mol Imaging 46:989–998

    Article  PubMed  Google Scholar 

  3. Khurana B, Prevedello LM, Bono CM, Lin E, McCormack ST, Jimale H, Harris MB, Sodickson AD (2018) CT for thoracic and lumbar spine fractures: can CT findings accurately predict posterior ligament complex injury? Eur Spine J 27:3007–3015

    Article  PubMed  Google Scholar 

  4. Das A, Bhattacharya M (2011) Affine-based registration of CT and MR modality images of human brain using multiresolution approaches: comparative study on genetic algorithm and particle swarm optimization. Neural Comput Appl 20:223–237

    Article  Google Scholar 

  5. Tavolaro C, Ghaffar S, Zhou H, Nguyen QT, Bellabarba C, Bransford RJ (2019) Is routine MRI of the spine necessary in trauma patients with ankylosing spinal disorders or is a CT scan sufficient? Spine J 19(8):1331–1339

    Article  PubMed  Google Scholar 

  6. Zhou Q, Ye S, Wen M, Huang Z, Ding M, Zhang X (2022) Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer. Neural Comput Appl 34(24):21741–21761

    Article  Google Scholar 

  7. Maksymowych WP (2019) The role of imaging in the diagnosis and management of axial spondyloarthritis. Nat Rev Rheumatol 15(11):657–672

    Article  CAS  PubMed  Google Scholar 

  8. Guerrini L, Mazzocchi S, Giomi A, Milli M, Carpi R (2020) An operational approach to the execution of MR examinations in patients with Cied. Radiol Med Torino 125:1311–1321

    Article  PubMed  Google Scholar 

  9. Patel DM, Weinberg BD, Hoch MJ (2020) Ct myelography: clinical indications and imaging findings. Radiographics 40(2):470–484

    Article  PubMed  Google Scholar 

  10. Celard P, Iglesias E, Sorribes-Fdez J, Romero R, Vieira AS, Borrajo L (2023) A survey on deep learning applied to medical images: from simple artificial neural networks to generative models. Neural Comput Appl 35(3):2291–2323

    Article  CAS  PubMed  Google Scholar 

  11. Nallamothu PT, Bharadiya JP (2023) Artificial intelligence in orthopedics: a concise review. Asian J Orthop Res 9(1):17–27

    Google Scholar 

  12. Kaji S, Kida S (2019) Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging. Radiol Phys Technol 12:235–248

    Article  PubMed  Google Scholar 

  13. Singh NK, Raza K (2021) Medical image generation using generative adversarial networks: a review. Health Inform A Comput Perspect Healthc, 77–96

  14. Rubin J, Abulnaga SM (2019) CT-to-MR conditional generative adversarial networks for ischemic stroke lesion segmentation. In: 2019 IEEE International conference on healthcare informatics (ICHI), pp 1–7

  15. Jin C-B, Kim H, Liu M, Han IH, Lee JI, Lee JH, Joo S, Park E, Ahn YS, Cui X (2019) Dc2anet: generating lumbar spine MR images from CT scan data based on semi-supervised learning. Appl Sci 9(12):2521

    Article  Google Scholar 

  16. Kalantar R, Messiou C, Winfield JM, Renn A, Latifoltojar A, Downey K, Sohaib A, Lalondrelle S, Koh D-M, Blackledge MD (2021) Ct-based pelvic t1-weighted MR image synthesis using unet, unet++ and cycle-consistent generative adversarial network (cycle-gan). Front Oncol 11:665807

    Article  PubMed  PubMed Central  Google Scholar 

  17. Yi X, Walia E, Babyn P (2019) Generative adversarial network in medical imaging: A review. Med Image Anal 58:101552

    Article  PubMed  Google Scholar 

  18. Choudhary A, Tong L, Zhu Y, Wang MD (2020) Advancing medical imaging informatics by deep learning-based domain adaptation. Yearb Med Inform 29(01):129–138

    Article  PubMed  PubMed Central  Google Scholar 

  19. Lim S, Shin M, Paik J (2022) Point cloud generation using deep adversarial local features for augmented and mixed reality contents. IEEE Trans Consum Electron 68(1):69–76

    Article  Google Scholar 

  20. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  21. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  22. Wang T, Lei Y, Fu Y, Wynne JF, Curran WJ, Liu T, Yang X (2021) A review on medical imaging synthesis using deep learning and its clinical applications. J Appl Clin Med Phys 22(1):11–36

    Article  PubMed  Google Scholar 

  23. Zhou T, Li Q, Lu H, Cheng Q, Zhang X (2023) Gan review: models and medical image fusion applications. Inform Fusion 91:134–148

    Article  Google Scholar 

  24. Abu-Srhan A, Almallahi I, Abushariah MA, Mahafza W, Al-Kadi OS (2021) Paired-unpaired unsupervised attention guided Gan with transfer learning for bidirectional brain MR-CT synthesis. Comput Biol Med 136:104763

    Article  PubMed  Google Scholar 

  25. Matsuo H, Nishio M, Nogami M, Zeng F, Kurimoto T, Kaushik S, Wiesinger F, Kono AK, Murakami T (2022) Unsupervised-learning-based method for chest MRI-CT transformation using structure constrained unsupervised generative attention networks. Sci Rep 12(1):11090

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  26. Liu Y, Chen A, Shi H, Huang S, Zheng W, Liu Z, Zhang Q, Yang X (2021) Ct synthesis from MRI using multi-cycle Gan for head-and-neck radiation therapy. Comput Med Imaging Graph 91:101953

    Article  PubMed  Google Scholar 

  27. Armanious K, Jiang C, Fischer M, Küstner T, Hepp T, Nikolaou K, Gatidis S, Yang B (2020) Medgan: medical image translation using gans. Comput Med Imaging Graph 79:101684

    Article  PubMed  Google Scholar 

  28. Yan S, Wang C, Chen W, Lyu J (2022) Swin transformer-based Gan for multi-modal medical image translation. Front Oncol 12:942511

    Article  PubMed  PubMed Central  Google Scholar 

  29. Liu S, Zhu C, Xu F, Jia X, Shi Z, Jin M (2022) Bci: Breast cancer immunohistochemical image generation through pyramid pix2pix. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1815–1824

  30. Ranjan A, Lalwani D, Misra R (2022) Gan for synthesizing CT from t2-weighted MRI data towards MR-guided radiation treatment. Magn Reson Mater Phys Biol Med 35(3):449–457

    Article  CAS  Google Scholar 

  31. Xie J (2021) Multi-task medical image-to-images translation using transformer for chest x-ray radiography. In: 2021 2nd International conference on artificial intelligence and computer engineering (ICAICE), pp 708–715

  32. Chen Y, Lin Y, Xu X, Ding J, Li C, Zeng Y, Xie W, Huang J (2023) Multi-domain medical image translation generation for lung image classification based on generative adversarial networks. Comput Methods Programs Biomed 229:107200

    Article  PubMed  Google Scholar 

  33. Wu C, Zhang H, Chen J, Gao Z, Zhang P, Muhammad K, Del Ser J (2022) Vessel–Gan: angiographic reconstructions from myocardial CT perfusion with explainable generative adversarial networks. Future Gener Comput Syst 130:128–139

    Article  Google Scholar 

  34. Yedla RR, Dubey SR (2021) On the performance of convolutional neural networks under high and low frequency information. In: Computer vision and Image processing: 5th international conference, CVIP 2020, Prayagraj, India, December 4-6, 2020, Revised Selected Papers, Part III 5, pp 214–224. Springer

  35. Cong R, Zhang Y, Yang N, Li H, Zhang X, Li R, Chen Z, Zhao Y, Kwong S (2022) Boundary guided semantic learning for real-time covid-19 lung infection segmentation system. IEEE Trans Consum Electron 68(4):376–386

    Article  Google Scholar 

  36. Chetia R, Boruah S, Sahu P (2021) Quantum image edge detection using improved Sobel mask based on NEQR. Quantum Inf Process 20:1–25

    Article  ADS  MathSciNet  Google Scholar 

  37. Woo S, Park J, Lee JY, Kweon IS (2018) CBAM: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  38. Xu R, Zhou Z, Zhang W, Yu Y (2017) Face transfer with generative adversarial network. arXiv preprint arXiv:1710.06090

  39. Hou X, Shen L, Sun K, Qiu G (2017) Deep feature consistent variational autoencoder. In: 2017 IEEE Winter conference on applications of computer vision (WACV), pp 1133–1141

  40. Gao X, Fang Y (2011) A note on the generalized degrees of freedom under the l1 loss function. J Stat Plan Inference 141(2):677–686

    Article  Google Scholar 

  41. Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S (2017) Least squares generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2794–2802

  42. Fedorov A, Beichel R, Kalpathy-Cramer J, Finet J, Fillion-Robin J-C, Pujol S, Bauer C, Jennings D, Fennessy F, Sonka M et al (2012) 3d slicer as an image computing platform for the quantitative imaging network. Magn Reson Imaging 30(9):1323–1341

    Article  PubMed  PubMed Central  Google Scholar 

  43. Zheng C, Cham TJ, Cai J (2021) The spatially-correlative loss for various image translation tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 16407–16417

  44. Park T, Efros AA, Zhang R, Zhu JY (2020) Contrastive learning for unpaired image-to-image translation. In: Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, 23–28, 2020, Proceedings, Part IX 16, pp 319–345. Springer

  45. Ang SP, Phung SL, Field M, Schira MM (2022) An improved deep learning framework for MR-to-CT image synthesis with a new hybrid objective function. In: 2022 IEEE 19th International symposium on biomedical imaging (ISBI), pp 1–5

  46. Cheng B, Liu Z, Peng Y, Lin Y (2023) General image-to-image translation with one-shot image guidance. arXiv preprint arXiv:2307.14352

  47. Torbunov D, Huang Y, Yu H, Huang J, Yoo S, Lin M, Viren B, Ren Y (2023) UVCGAN: UNET vision transformer cycle-consistent Gan for unpaired image-to-image translation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 702–712

  48. D’Aprile P, Tarantino A, D’Aprile P, Tarantino A (2021) MRI in postoperative spine. MRI of Degenerative Disease of the Spine: A Case-Based Atlas, 19–25

  49. Schwaiger BJ, Schneider C, Kronthaler S, Gassert FT, Böhm C, Pfeiffer D, Baum T, Kirschke JS, Karampinos DC, Makowski MR et al (2021) Ct-like images based on t1 spoiled gradient-echo and ultra-short echo time MRI sequences for the assessment of vertebral fractures and degenerative bone changes of the spine. Eur Radiol 31:4680–4689

    Article  PubMed  PubMed Central  Google Scholar 

  50. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62,273,163, the Key R &D Project of Shandong Province under Grant No. 2022CXGC010503, and the Youth Foundation of Shandong Province under Grant No. ZR202102230323.

Author information

Author notes

  1. Guangxin Dai and Junxiao Su have contributed equally to this work.

Authors and Affiliations

  1. School of Electrical Engineering, University of Jinan, Jinan, 250000, Shandong, China

    Guangxin Dai, Menghua Zhang & Weijie Huang

  2. Department of Orthopedics, Qilu Hospital of Shandong University, Jinan, 250012, Shandong, China

    Junxiao Su & Xinyu Liu

Authors
  1. Guangxin Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junxiao Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Menghua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weijie Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xinyu Liu or Weijie Huang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dai, G., Su, J., Zhang, M. et al. A novel structure preserving generative adversarial network for CT to MR modality translation of spine. Neural Comput & Applic 36, 4101–4114 (2024). https://doi.org/10.1007/s00521-023-09254-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09254-w

Keywords

Search

Navigation