Skip to main content
SpringerLink
Log in

Fast and accurate super-resolution of MR images based on lightweight generative adversarial network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Single image super-resolution reconstruction (SISR) can effectively and economically improve the spatial resolution of magnetic resonance (MR) images, and it helps more accurate early clinical diagnosis and subsequent analysis. To increase the imaging speed and reduce the patient’s pain and motion artifacts, many studies have moved from only considering the quality of the reconstructed image to proposing some lightweight models. However, the model’s lightweight will limit its performance, and high-resolution MR images are often reconstructed with a single target (LOSS). In this work, we propose a lightweight generative adversarial network to alleviate this problem. The network mainly contains generators and discriminators. The generator uses a global cascade module to extract image features, and multi-scale up sampling of high-frequency and low-frequency features of different depths. As the cascaded modules lead to similar features, a consistent spatial attention module is used to weigh them and share the up-sampling module to reduce network parameters. The discriminator judges the authenticity of the input MR image, and it constructs two losses with the pre-trained VGG network to assist the generator training and provide diversified standards for the generation of MR images. In addition, we use knowledge transfer to train the network to explore the toplimit of network performance. Qualitative and quantitative experiments on the FASTMRI dataset show that the MR images generated by the designed multiple targets (loss) have better visual effects in detail. The proposed network has advantages in running time and parameter memory and achieved the highest precision results compared with state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Agustsson E, Timofte R (2017) NTIRE 2017 challenge on single image super-resolution: dataset and study. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1122–1131

  2. Ahn N, Kang B, Sohn K-a (2018) Fast, accurate, and, lightweight super-resolution with cascading residual network. ArXiv abs/180308664 n. pag

  3. Bosse S, Maniry D, Müller K, Wiegand T, Samek W (2018) Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans Image Process 27:206–219

    Article  MathSciNet  MATH  Google Scholar 

  4. Carmi E, Liu S, Alon N, Fiat A, Fiat D (2006) Resolution enhancement in MRI. Magn Reson Imaging 24(2):133–154

    Article  Google Scholar 

  5. Chen Y, Xie Y, Zhou Z, Shi F, Christodoulou AG, Li D (2018) Brain MRI super resolution using 3D deep densely connected neural networks. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp 739–742

  6. Chen Y, Shi F, Christodoulou A, Zhou Z, Xie Y, Li D (2018) Efficient and accurate MRI super-resolution using a generative adversarial network and 3D multi-level densely connected network. ArXiv abs/1803.01417: n. pag

  7. Chu X, Zhang B, Ma H, Xu R, Li J, Li Q (2021) Fast, accurate and lightweight super-resolution with neural architecture search. 2020 25th International Conference on Pattern Recognition (ICPR):59–64

  8. Dai T, Cai J, Zhang Y, Xia S, Zhang L (2019) Second-order attention network for single image super-resolution. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 11057–11066

  9. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38:295–307

    Article  Google Scholar 

  10. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. ECCV https://doi.org/10.48550/arXiv.1608.00367

  11. Farsiu S, Robinson M, Elad M, Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13:1327–1344

    Article  Google Scholar 

  12. Fitzgibbon, AW, Pollefeys M, Van Gool L, Zisserman A (2006) European conference on computer vision (ECCV). ECCV 2006

  13. Giannakidis A, Oktay O, Keegan J, Spadotto V, Firmin DN (2017) Super-resolution reconstruction of late gadolinium enhancement cardiovascular magnetic resonance images using a residual convolutional neural network. In: 25th scientific meeting of the International Society for Magnetic Resonance in medicine (ISMRM 2017)

  14. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. Adv Neural Inf Proces Syst 3:139–144. https://doi.org/10.1145/3422622

    Article  Google Scholar 

  15. Greenspan H, Oz G, Kiryati N, Peled S (2002) MRI inter-slice reconstruction using super-resolution. Magn Reson Imaging 20:437–446

    Article  MATH  Google Scholar 

  16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  17. Hui Z, Gao X, Yang Y, Wang X (2019) Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM international conference on multimedia, n. pag

  18. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. ICML

  19. Jolicoeur-Martineau A (2019) The relativistic discriminator: a key element missing from standard GAN. ArXiv abs/180700734 n. pag

  20. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1646–1654

  21. Kim S, Hong J-H, Kang I, Kwak N (2019) Semantic sentence matching with densely-connected recurrent and co-attentive information. ArXiv abs/180511360 n. pag

  22. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017) Deep Laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 5835–5843

  23. Ledig C, Theis L, Huszár F, Caballero J, Aitken AP, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 105–114

  24. Li X, Orchard M (2001) New edge-directed interpolation. IEEE Trans Image Process 10(10):1521–1527

    Article  Google Scholar 

  25. Li Y, Iwamoto Y, Lin L, Xu R, Tong R, Chen Y-W (2021) VolumeNet: a lightweight parallel network for super-resolution of MR and CT volumetric data. IEEE Trans Image Process 30:4840–4854

    Article  MathSciNet  Google Scholar 

  26. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1132–1140. https://doi.org/10.48550/arXiv.1707.02921

  27. Liu J, Tang J, Wu G (2020) Residual feature distillation network for lightweight image super-resolution. ECCV Workshops

  28. Lyu Q, Shan H, Wang G (2020) MRI super-resolution with ensemble learning and complementary priors. IEEE Trans Comput Imaging 6:615–624

    Article  Google Scholar 

  29. Manjón J, Coupé P, Buades A, Collins D, Robles M (2010) MRI Superresolution using self-similarity and image priors. Int J Biomed Imaging, n. pag

  30. McDonagh SG, Hou B, Alansary A, Oktay O, Kamnitsas K, Rutherford MA, Hajnal JV, Kainz B (2017) Context-sensitive super-resolution for fast fetal magnetic resonance imaging. CMMI/RAMBO/SWITCH@MICCAI

  31. Mittal A, Soundararajan R, Bovik A (2013) Making a “completely blind” image quality analyzer. IEEE Signal Process Lett 20:209–212

    Article  Google Scholar 

  32. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22:1345–1359

    Article  Google Scholar 

  33. Park SC, Park M, Kang M (2003) Super-resolution image reconstruction: a technical overview. IEEE Signal Process Mag 20:21–36

    Article  Google Scholar 

  34. Ramzi Z, Ciuciu P, Starck J (2020) Benchmarking MRI reconstruction neural networks on large public datasets. Appl Sci 10:1816

    Article  Google Scholar 

  35. Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1874–1883

  36. Shi J, Li Z, Ying S, Wang C, Liu Q, Zhang Q, Yan P (2019) MR image super-resolution via wide residual networks with fixed skip connection. IEEE J Biomed Health Inform 23:1129–1140

    Article  Google Scholar 

  37. Shilling RZ, Robbie TQ, Bailloeul T, Mewes K, Mersereau R, Brummer M (2009) A super-resolution framework for 3-D high-resolution and high-contrast imaging using 2-D multislice MRI. IEEE Trans Med Imaging 28:633–644

    Article  Google Scholar 

  38. Song D, Wang Y, Chen H, Xu C, Xu C, Tao D (2021) AdderSR: towards energy efficient image super-resolution. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR):15643–15652

  39. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. 2017 IEEE conference on computer vision and pattern recognition (CVPR):2790–2798

  40. Tai Y, Yang J, Liu X, Xu C (2017) MemNet: a persistent memory network for image restoration. 2017 IEEE international conference on computer vision (ICCV):4549–4557

  41. Tanno R, Worrall DE, Ghosh A, Kaden E, Sotiropoulos SN, Criminisi A, Alexander DC (2017) Bayesian image quality transfer with CNNs: exploring uncertainty in dMRI super-resolution. MICCAI

  42. Tian C, Zhuge R, Wu Z, Xu Y, Zuo W, Chen C, Lin C (2020) Lightweight image super-resolution with enhanced CNN. Knowl Based Syst 205:106235

    Article  Google Scholar 

  43. Timofte R, De Smet V, Van Gool L (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. ACCV

  44. Tsai R, Huang T. (1984) Multiframe image restoration and registration. Adv Comput Vis Image Process 317–339

  45. Wang Z, Bovik A, Sheikh H, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612

    Article  Google Scholar 

  46. Wang Z, Liu D, Yang J, Han W, Huang T (2015) Deep networks for image super-resolution with sparse prior. In: 2015 IEEE international conference on computer vision (ICCV), pp 370–378

    Chapter  Google Scholar 

  47. Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Loy CC, Qiao Y, Tang X (2018) ESRGAN: enhanced super-resolution generative adversarial networks. ArXiv abs/180900219: n. pag

  48. Woo S, Park J, Lee J-Y, Kweon I-S (2018) CBAM: convolutional block attention module. ECCV

  49. Xia Y, Ravikumar N, Greenwood J, Neubauer S, Petersen S, Frangi AF (2021) Super-resolution of cardiac MR cine imaging using conditional GANs and unsupervised transfer learning. Med Image Anal 71:102037

    Article  Google Scholar 

  50. Xue X, Wang Y, Li J, Jiao Z, Ren Z, Gao X (2020) Progressive sub-band residual-learning network for MR image super resolution. IEEE J Biomed Health Inform 24:377–386

    Article  Google Scholar 

  51. Yang J, Wright J, Huang T, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19:2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  52. Yuan Y, Liu S, Zhang J, Zhang Y, Dong C, Lin L (2018) Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW) (2018): 814–81409

  53. Yue L, Shen H, Li J, Yuan Q, Zhang H, Zhang L (2016) Image super-resolution: The techniques, applications, and future. Signal Process 128:389–408

    Article  Google Scholar 

  54. Zbontar J, Knoll F, Sriram A, Muckley M, Bruno M, Defazio A, Parente M, Geras K, Katsnelson J, Chandarana H, Zhang Z, Drozdzal M, Romero A, Rabbat MG, Vincent P, Pinkerton J, Wang D, Yakubova N, Owens E, Zitnick CL, Recht M, Sodickson D, Lui Y (2018) fastMRI: an open dataset and benchmarks for accelerated MRI. ArXiv abs/181108839 (2018): n. pag

  55. Zeng K, Zheng H, Cai C, Yang Y, Zhang K, Chen Z (2018) Simultaneous single- and multi-contrast super-resolution for brain MRI images based on a convolutional neural network. Comput Biol Med 99:133–141

    Article  Google Scholar 

  56. Zhang C, Ma Y. (2012) Ensemble machine learning: methods and applications. https://link.springer.com/book/10.1007%2F978-1-4419-9326-7

  57. Zhang Y, Tian Y, Kong Y, Zhong B, Fu YR (2018) Residual dense network for image super-resolution. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 2472–2481

  58. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu YR (2018) Image super-resolution using very deep Residual Channel attention networks. ECCV

  59. Zhang W, Liu Y, Dong C, Qiao Y (2019) RankSRGAN: generative adversarial networks with ranker for image super-resolution. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp 3096–3105

  60. Zhao C, Carass A, Dewey BE, Prince JL (2018) Self super-resolution for magnetic resonance images using deep networks. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp 365–368

  61. Zhao X, Zhang Y, Zhang T, Zou X (2019) Channel splitting network for single MR image super-resolution. IEEE Trans Image Process 28:5649–5662

    Article  MathSciNet  MATH  Google Scholar 

  62. Zhao X, Zhang H, Zhou Y, Bian W, Zhang T, Zou X (2019) Gibbs-ringing artifact suppression with knowledge transfer from natural images to MR images. Multimed Tools Appl:1–23

  63. Zhao X, Hu X, Liao Y, He T, Zhang T, Zou X, Tian J (2020) Accurate MR image super-resolution via lightweight lateral inhibition network. Comput Vis Image Underst 201:103075

    Article  Google Scholar 

  64. Zhu X, Cheng D, Zhang Z, Lin S, Dai J (2019) An empirical study of spatial attention mechanisms in deep networks. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 6687–6696

Download references

Acknowledgements

We wish to express our gratitude to the anonymous reviewers for their insightful comments.

Funding

The work is supported by Beijing Outstanding Talents Training Fund Youth Top Individual Project,

Premium Funding Project for Academic Human Resources Development in Beijing Union University under.

grant BPHR2020EZ01.

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of information service engineering, Beijing Union University, Beijing, China

    Hangyu Li

  2. Institute of Fundamental and Interdisciplinary Sciences, Beijing Union University, Beijing, 100101, China

    Zuxing Xuan

  3. Anhui Province Key Laboratory of Special Heavy Load Robot, Anhui University of Technology, Ma’anshan, 243032, China

    Jianpin Zhou

  4. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China

    Xiyuan Hu

  5. School of Information Engineering, Beijing Forestry University, Beijing, China

    Bo Yang

Authors
  1. Hangyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zuxing Xuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianpin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiyuan Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zuxing Xuan.

Ethics declarations

The authors declare that they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Xuan, Z., Zhou, J. et al. Fast and accurate super-resolution of MR images based on lightweight generative adversarial network. Multimed Tools Appl 82, 2465–2487 (2023). https://doi.org/10.1007/s11042-022-13326-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13326-9

Keywords

Search

Navigation