Skip to main content
Springer Nature Link
Log in

Multi-scale generative adversarial network for image super-resolution

  • Foundations
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

In recent years, deep convolutional neural networks (CNNs) have been widely employed in image super-resolution. Thanks to the power of deep CNNs, the reconstruction performance is largely improved. However, the high-frequency information and details in the low-resolution image still can hardly be reconstructed. To deal with the above problems, we propose a multi-scale generative adversarial network in this paper. The multi-scale Pyramid module inside the generator could extract the features containing high-frequency information, and then the high-resolution image with the results of the bicubic interpolations is reconstructed. The discriminator in our model is used to identify the authenticity of the input image after refactoring. Our final loss function includes an adversarial loss and the mean square error (L2) reconstruction loss. In order to further improve the efficiency of training, the generator is pre-trained with the L2 loss, so as to improve the efficiency of the discriminator optimization. Compared with the algorithms based solely on normal plain convolutional networks, the proposed algorithm performs better in two indexes PSNR and SSIM of the super-resolution task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The datasets analyzed during the current study are available in the BSD100, BSD500, SET5 and SET14 repositories, https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/, http://people.rennes.inria.fr/Aline.Roumy/results/SR_BMVC12.html, https://github.com/jbhuang0604/SelfExSR.

References

  • Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel ML (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Bowden R, Collomosse J, Mikolajczyk K (eds) Proceedings of the british machine vision conference. BMVA Press, pp 135.1–135.10. https://doi.org/10.5244/c.26.135

    Chapter  Google Scholar 

  • Bulat A, Yang J, Tzimiropoulos G (2018) To learn image super-resolution, use a gan to learn how to do image degradation first. Springer, Cham, pp 185–200

    Google Scholar 

  • Chen L, Li Q, Jiang J (2020) Noisy practical facial super-resolution method via deformable constrained model with small dataset. Multimed Tools Appl 79(3):2577–2600. https://doi.org/10.1007/s11042-019-08277-7

    Article  Google Scholar 

  • Chu X, Zhang B, Ma H, Xu R, Li J, Li Q (2019) Fast, accurate and lightweight super-resolution with neural architecture search. In: 2020 25th International conference on pattern recognition (ICPR), pp 59–64. https://doi.org/10.1109/ICPR48806.2021.9413080

  • Dong C, Loy CC, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer vision – ECCV 2014, ECCV 2014. Lecture notes in computer science, vol 8692. Springer, Cham. https://doi.org/10.1007/978-3-319-10593-2_13

    Chapter  Google Scholar 

  • Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer vision – ECCV 2016, ECCV 2016. Lecture notes in computer science, vol 9906. Springer, Cham. https://doi.org/10.1007/978-3-319-46475-6_25

    Chapter  Google Scholar 

  • Fattal R (2007) Image upsampling via imposed edge statistics. ACM Trans Graph 26(3):95–es. https://doi.org/10.1145/1276377.1276496

    Article  Google Scholar 

  • Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Bengio Y (2014) Generative adversarial nets. Adv Neural Inform Process Syst. https://doi.org/10.3156/jsoft.29.5_177_2

    Article  Google Scholar 

  • Gunturk BK, Batur AU, Altunbasak Y, Hayes MH, Mersereau RM (2003) Eigenface-domain super-resolution for face recognition. IEEE Trans Image Process A Publ IEEE Signal Process Soc 12(5):597–606

    Article  Google Scholar 

  • Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. IEEE/CVF conference on computer vision & pattern recognition. https://doi.org/10.1109/cvpr.2018.00179

  • Hu J, Shen L, Albanie S, Sun G, Wu E (2017) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/cvpr.2018.00745

    Article  Google Scholar 

  • Huang H, Wu N (2011) Fast facial image super-resolution via local linear transformations for resource-limited applications. IEEE Trans Circuits Syst Video Technol 21(10):1363–1377. https://doi.org/10.1109/tcsvt.2011.2163461

    Article  Google Scholar 

  • Jo Y, Yang S, Kim SJ (2020) Investigating loss functions for extreme super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 424–425. https://doi.org/10.1109/CVPR50498.2020.00220

  • Keys RG (2003) Cubic convolution interpolation for digital image processing. IEEE Trans Acoust Speech Signal Process. https://doi.org/10.1109/tassp.1981.1163711

    Article  MathSciNet  MATH  Google Scholar 

  • Kim J, Kwon Lee J, Mu Lee K (2016) Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654. https://doi.org/10.1109/cvpr.2016.182

  • Kim J, Kwon Lee J, Mu Lee K (2016) Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645. https://doi.org/10.1109/cvpr.2016.181

  • Lai WS, Huang JB, Ahuja N, Yang MH (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. IEEE conference on computer vision & pattern recognition, pp 5835–5843. IEEE Computer Society. https://doi.org/10.1109/CVPR.2017.618

  • Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A et al (2016) Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), 2017, pp 105–114. https://doi.org/10.1109/cvpr.2017.19

  • Lee Y, Jun J, Hong Y, Jeon M (2019) Practical license plate recognition in unconstrained surveillance systems with adversarial super-resolution. In: Proceedings of the 14th international joint conference on computer vision, imaging and computer graphics theory and applications, vol 5, VISAPP, ISBN 978-989-758-354-4; ISSN 2184-4321, pp 68–76. https://doi.org/10.5220/0007378300680076

  • Li J, Fang F, Mei K, Zhang G (2018) Multi-scale residual network for image super-resolution. In Proceedings of the European conference on computer vision (ECCV), pp 517–532. https://doi.org/10.1007/978-3-030-01237-3_32

  • Lim, B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1132–1140. https://doi.org/10.1109/CVPRW.2017.151

  • Liu L, Wang S, Wan L (2019) Component semantic prior guided generative adversarial network for face super-resolution. IEEE Access. https://doi.org/10.1109/access.2019.2921859

    Article  Google Scholar 

  • Lugmayr A, Danelljan M, Timofte R (2020) Unsupervised learning for real-world super-resolution. 2019 IEEE/CVF International conference on computer vision workshop (ICCVW). IEEE. https://doi.org/10.1109/iccvw.2019.00423

  • Martin D, Fowlkes C, Tal D, Malik J (2002) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. IEEE international conference on computer vision. IEEE. https://doi.org/10.1109/iccv.2001.937655

  • Pang Y, Cao J, Wang J, Han J (2019) Jcs-net: joint classification and super-resolution network for small-scale pedestrian detection in surveillance images. IEEE Trans Inform Forens Secur 14:3322. https://doi.org/10.1109/TIFS.2019.2916592

    Article  Google Scholar 

  • Peled S, Yeshurun Y (2015) Superresolution in mri: application to human white matter fiber tract visualization by diffusion tensor imaging. Mag Resonan Med Off J Soc Mag Reson Med 45(1):29–35

    Article  Google Scholar 

  • Son S, Lee KM (2021) SRWarp: generalized image super-resolution under arbitrary transformation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7782–7791. https://doi.org/10.1109/CVPR46437.2021.00769

  • Sun N, Li H (2019) Super resolution reconstruction of images based on interpolation and full convolutional neural network and application in medical fields. IEEE Access 7:186470. https://doi.org/10.1109/access.2019.2960828

    Article  Google Scholar 

  • Wang Z, Liu D, Yang J, Han W, Huang T (2016) Deep networks for image super-resolution with sparse prior. 2015 IEEE international conference on computer vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2015.50

  • Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C et al (2018) Esrgan: enhanced super-resolution generative adversarial networks. In: Leal-Taixé L, Roth S (eds) Computer vision – ECCV 2018 workshops: Munich, Germany, September 8–14, 2018, proceedings part V. Springer International Publishing, Cham

    Google Scholar 

  • Wang L, Dong X, Wang Y, Ying X, Lin Z, An W, Guo Y (2021) Exploring sparsity in image super-resolution for efficient inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4917–4926). https://doi.org/10.1109/CVPR46437.2021.00488

  • Wu B, Duan H, Liu Z, Sun G (2017) Srpgan: perceptual generative adversarial network for single image super resolution. In: SRPGAN: Perceptual generative adversarial network for single image super resolution. arXiv:abs/1712.05927

  • Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873. https://doi.org/10.1109/tip.2010.2050625

    Article  MathSciNet  MATH  Google Scholar 

  • Yuan Y, Liu S, Zhang J, Zhang Y, Dong C, Lin L (2018) Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW). IEEE. https://doi.org/10.1109/cvprw.2018.00113

  • Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: Boissonnat J-D, Chenin P, Cohen A, Gout C, Lyche T, Mazure M-L, Schumaker L (eds) Curves and surfaces. Springer, Heidelberg

    Google Scholar 

  • Zhang L, Zhang H, Shen H, Li P (2010) A super-resolution reconstruction algorithm for surveillance images. Signal Process 90(3):848–859. https://doi.org/10.1016/j.sigpro.2009.09.002

    Article  MATH  Google Scholar 

  • Zhao T, Ren W, Zhang C, Ren D, Hu Q (2018) Unsupervised degradation learning for single image super-resolution. CoRR. arxiv:1812.04240

Download references

Funding

This work was supported in part by the Major Project of Natural Science Research of the Jiangsu Higher Education Institutions of China under Grant 18KJA520012, and in part by the Xuzhou Science and Technology Plan Project under Grant KC19197.

Author information

Authors and Affiliations

  1. Xuzhou University of Technology, Xuzhou, 221000, China

    Jiang Daihong, Dai Lei & Dai Yueming

  2. China University of Mining and Technology, Xuzhou, 221000, China

    Zhang Sai

Authors
  1. Jiang Daihong
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Zhang Sai
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Dai Lei
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Dai Yueming
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

JDH: Conceptualization, Methodology, Writing—Original Draft. ZS: Visualization, Writing—Review & Editing. DL: Software, Data Curation. DYM: Resources, Validation.

Corresponding author

Correspondence to Jiang Daihong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest. The authors have no relevant financial or non-financial interest to disclose.

Additional information

Communicated by Jiang Daihong.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Daihong, J., Sai, Z., Lei, D. et al. Multi-scale generative adversarial network for image super-resolution. Soft Comput 26, 3631–3641 (2022). https://doi.org/10.1007/s00500-022-06822-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-022-06822-5

Keywords