Skip to main content
SpringerLink
Log in

Back-projection-based progressive growing generative adversarial network for single image super-resolution

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Recent advanced deep learning studies have shown the positive role of feedback mechanism in image super-resolution task. However, current feedback mechanism only calculates residual errors of images with the same resolution without considering the useful features that may be carried by different resolution features. In this paper, to explore the potential of feedback mechanism, we design a new network structure (progressive up- and downsampling back-projection units) to construct a generative adversarial network for single image super-resolution and use progressive growing methodologies to train it. Unlike previous feedback structure, we use progressively increasing scale factor to build up- and down-projection units, which aims to learn fruitful features across scales. This method allows us to get more meaningful information from early feature maps. Additionally, we train our network progressively; in the process of training, we start from single layer network structure and add new layers as the training goes on. By this mean, the training process can be greatly accelerated and stabilized. Experiments on benchmark dataset with the state-of-the-art methods show that our network achieves 0.01 dB, 0.11 dB, 0.13 dB and 0.4 dB better PSNR results than that of RDN+, MDSR, D-DBPN and EDSR on 8\(\times \) enlargement, respectively, and also achieves favorable performance against the state-of-the-art methods on 2\(\times \) and 4\(\times \) enlargement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceeding s of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

  2. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for imagerecognition. arXiv preprint arXiv:1512.03385 (2015)

  3. Denton, E.L., Chintala, S., Fergus, R., et al.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, pp. 1486–1494 (2015)

  4. Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., Webb, R.: Learning from simuilated and unsupervised images through adversarial training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

  5. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

  6. Ulg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: flownet. Evolution of optical flow estimation with deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  7. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer (2016)

  8. Dong, C., Loy, C.C., He, K., Tang, X.: Image super resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  9. Haris, M., Widyanto, M.R., Nobuhara, H.: Inception learning super-resolution. Appl. Opt. 56(22), 6043–6048 (2017)

    Article  Google Scholar 

  10. Kim, J., KwonLee, J., MuLee, K.: Accurate image super resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  11. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep Laplacian pyramid networks for fast and accurate superresolution. In: CVPR (2017)

  12. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: CVPR (2017)

  13. Shi, W., Caballero, J., Ledig, C., Zhuang, X., Bai, W., Bhatia, K., de Marvao, A.M.S.M., Dawes, T., ORegan, D., Rueckert, D.: Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: MICCAI (2013)

  14. Zou, W.W., Yuen, P.C.: Very low resolution face recognition problem. In: TIP (2012)

  15. Zhang, L., Wu, X.: An edge-guided image interpolation algorithm via directional filtering and data fusion. In: TIP (2006)

  16. Zhang, K., Gao, X., Tao, D., Li, X.: Single image super resolution with non-local means and steering kernel regression. In: TIP (2012)

  17. Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Asian Conference on Computer Vision, pp. 111–126. Springer (2014)

  18. Timofte, R., De, V., Gool, L.V.: Anchored neighborhood regression for fast example-based super-resolution. In: ICCV (2013)

  19. Peleg, T., Elad, M.: A statistical prediction model based on sparse representations for single image super-resolution. In: TIP (2014)

  20. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: ICCV (2017)

  21. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: CVPR (2018)

  22. Zhang Y., Li K.: Image super-resolution using very deep residual channel attention network. In: European Conference on Computer Vision (2018)

  23. Li, J., Fang, F.: Multi-scale residual network for image super-resolution. In: European Conference on Computer Vision, pp. 527–542 (2018)

  24. Ledig, C., Theis, L., Huszar, F., et al.: Photo-realistic single image super-resolution using a generative adversarial network (2016). arXiv:1609.04802v5

  25. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution (2015). arXiv:1511.04491v2

  26. Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M.: Enhanced deep residual networks for single image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)

  27. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: CVPRW (2017)

  28. Zhang, Y., Tian, Y., Kong, Y., et al.: Residual dense network for image super-resolution (2018)

  29. Fellemanand, D.J., VanEssen, D.C.: Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1(1), 1–47 (1991)

    Article  Google Scholar 

  30. Kravitz, D.J., Saleem, K.S., Baker, C.I., Ungerleider, L.G., Mishkin, M.: The ventral visual pathway: an expanded neural framework for the processing of object quality. Trends Cognit. Sci. 17(1), 26–49 (2013)

    Article  Google Scholar 

  31. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution (2018). arXiv:1803.02735

  32. Karras, T., Aila, T., Laine, S., et al.: Progressive growing of GANs for improved quality, stability, and variation (2017). arXiv:1710.10196

  33. Li, D., Wang, Z.: Face video super-resolution with identity guided generative adversarial networks. In: Chinese Conference on Computer Vision (CCCV), pp. 357–369 (2017)

  34. Wang, X., Yu, K., Wu, S., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: European Conference on Computer Vision (ECCV), pp. 63–79 (2018)

  35. Zhang, D., Jie, S., Gang, H., et al.: Sharp and real image super-resolution using generative adversarial network. In: International Conference on Neural Information Processing, pp. 217–226 (2017)

  36. Irani, M., Peleg, S.: Improving resolution by image registration. CVGIP Gr. Models Image Process. 53(3), 231–239 (1991)

    Article  Google Scholar 

  37. Li Z., Yang J.: Feedback Network for Image Super-Resolution. arXiv:1903.09814 (2019)

  38. Gulrajani, I., Ahmed, F., Arjovsky, M., et al.: Improved training of wasserstein GANs (2017). arXiv:1704.00028

  39. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers surpassing human-level performance on imagenet classification In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

  40. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.-H., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M., et al.: Ntire 2017 challenge on single image super-resolution: methods and results. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1110–1121. IEEE (2017)

  41. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  42. Bevilacqua, M., Roumy, A., Guillemot, C., Morel, M.L.A.: Low-complexity single-image super-resolution based on no nnegative neighbor embedding. In: British Machine Vision Conference (BMVC) (2012)

  43. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International Conference on Curves and Surfaces, pp. 711–730 (2010)

  44. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)

    Article  Google Scholar 

  45. Huang, J.B., Singh, A., Ahuja, N.: Single image super resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197-5206 (2015)

  46. Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimed. Tools Appl. 76, 1–28 (2016)

    Google Scholar 

  47. Dong, C., Loy, C.C., Tang, X.: Accelerating the super resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407. Springer (2016)

  48. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: a persistent memory network for image restoration. In: ICCV (2017)

  49. Irani, M., Peleg, S.: Motion analysis for image enhancement: resolution, occlusion, and transparency. J. Vis. Commun. Image Represent. 4(4), 324–335 (1993)

    Article  Google Scholar 

  50. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported partially by National Key Research and Development Program (ID 2018AAA003203)

Author information

Authors and Affiliations

  1. University of Electronic Science and Technology of China, Chengdu, China

    Tingsong Ma & Wenhong Tian

Authors
  1. Tingsong Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenhong Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenhong Tian.

Ethics declarations

Conflict of interest

Author Tingsong Ma declares that he has no conflict of interest. Author Wenhong Tian declares that he has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, T., Tian, W. Back-projection-based progressive growing generative adversarial network for single image super-resolution. Vis Comput 37, 925–938 (2021). https://doi.org/10.1007/s00371-020-01843-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01843-3

Keywords

Search

Navigation