Skip to main content
SpringerLink
Log in

Image super-resolution reconstruction based on generative adversarial network model with feedback and attention mechanisms

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Despite the rapid development of single-image super-resolution (SISR) methods of generative adversarial networks (GAN), which can reconstruct visually realistic images, the problem of high discrepancy between the recovered details or textures and the ground truth persists. To address this issue, an SISR reconstruction GAN based on a feedback and attention mechanism (FBSRGAN) is proposed. Specifically, we select a network with a feedback mechanism as the generator, which can gradually create high-resolution images through the feedback connection. The attention mechanism is combined with the feedback block to adaptively select useful feature information and effectively process the feedback stream and enhance the image output quality. We use the relative average least squares GAN loss to reduce the instability of the optimization generator process to guide GAN to obtain more realistic results. The results show that compared with the ESRGAN method, when the amplification factor is 4, PSNR and SSIM of the proposed method increase by 0.386 and 0.0141, respectively, and PI decreases by 0.284, while the number of parameters is only 18.5% of that of ESRGAN, when tested on the Set5 dataset. Compared with existing GAN-based SR methods, FBSRGAN achieves superior performance in terms of both perceptual ability and image distortion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Agustsson E, Timofte R (2017). NTIRE 2017 challenge on single image super-resolution: dataset and study. in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1122–1131

  2. Ali A, Zhu Y, Zakarya M (2021) A data aggregation based approach to exploit dynamic spatio-temporal correlations for citywide crowd flows prediction in fog computing. Multimed Tools Appl. https://doi.org/10.1007/s11042-020-10486-4

  3. Bevilacqua M, Roumy A, Guillemot C, et al (2012). Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In British machine vision conference (BMVC), pp. 1-10

  4. Blau Y, Mechrez R, Timofte R, et al (2018). The 2018 PIRM challenge on perceptual image super-resolution. In proc. Int. Conf. European conference on computer vision (ECCV), pp. 334-355

  5. Bulat A, Yang J, Tzimiropoulos G, et al (2018). To learn image super-resolution, use a GAN to learn how to do image degradation first. In proc. Int. Conf. The European conference on computer vision (ECCV), pp. 185-200.

  6. Cao C, Liu X, Yang Y, et al (2015). Look and think twice: capturing top-down visual attention with feedback convolutional neural networks. In proc. Int. Conf. International conference on computer vision (ICCV), pp. 2956–2964.

  7. Dai S, Han M, Xu W, Wu Y, Katsaggelos AK (2009) Softcuts: a soft edge smoothness prior for color image super-resolution. IEEE Trans Image Process 18(5):969–981

    Article  MathSciNet  Google Scholar 

  8. Dong, C., Loy CC, et al (2014) Learning a Deep Convolutional Network for Image Super-Resolution. in Proc. Int. Conf. European conference on computer vision (ECCV), pp. 184–199

  9. Han W, Chang S, Liu D, et al (2018). Image super-resolution via dual-state recurrent networks. In proc. Int. Conf. Computer vision and pattern recognition (CVPR) pp.1654-1663

  10. Haris M, Shakhnarovich G, Ukita N, et al (2018). Deep back-projection networks for super-resolution. In proc. Int. Conf. Computer vision and pattern recognition (CVPR), pp. 1664-1673

  11. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. Computer Vision and Pattern Recognition, pp:7132–7141

  12. Huang J, Singh A, Ahuja N, et al (2015). Single image super-resolution from transformed self-exemplars. In proc. Int. Conf. Computer vision and pattern recognition (CVPR), pp. 5197-5206

  13. Izonin I, Tkachenko R, Peleshko D , et al. Learning-based image super-resolution using weight coefficients of synaptic connections[C]// 2015 Xth international scientific and technical conference "computer sciences and information technologies" (CSIT). IEEE, 2015.

  14. Jianbing, Shen, Xin, et al (2019) Visual object tracking by hierarchical attention Siamese network. IEEE transactions on cybernetics, pp 3068–3080.

  15. Jolicoeur-Martineau A. (2018). The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734

  16. Kim J, Lee JK, Lee KM (2016). Deeply-recursive convolutional network for image super-resolution. in Proc. Int. Conf. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1637-1645.

  17. Kim J, Lee JK, and Lee KM (2016) Accurate image superresolution using very deep convolutional networks. In CVPR, 2016. 1, 2, 5, 7, 8.

  18. Kim SY, Oh J, and Kim M (2019). Deep sritm:Joint learning of super-resolution and inverse tone-mapping for 4k uhd hdr applications. IEEE Conf. Comput. Vis. Pattern Recognit., pp.3116–3125

  19. Lai Q , Wang W , Sun H , et al(2019) Video saliency prediction using spatiotemporal residual attentive networks. IEEE transactions on image processing, PP(99):1-1

  20. Ledig, C., Theis, L., Huszar, F., et al (2016). Photo-realistic single image super-resolution using a generative adversarial network. In proc. Int. Conf. Computer vision and pattern recognition (CVPR), pp. 105-114.

  21. Li Z, Yang J, Liu Z, et al (2019). Feedback network for image super-resolution. In proc. Int. Conf. Computer vision and pattern recognition (CVPR), pp. 3867-3876.

  22. Lim B, Son S, Kim H, et al (2017). Enhanced deep residual networks for single image super-resolution. In proc. Int. Conf. Computer vision and pattern recognition (CVPR), pp. 1132-1140

  23. Ma C, Jiang Z, Rao Y, Lu J, and Zhou J (2020) Deepface super-resolution with iterative collaboration between attentive recovery and landmark estimation. IEEE Conf. Comput. Vis. PatternRecognit., pp. 5569–5578.

  24. Ma C, Rao Y , Cheng Y , et al (2020) Structure-preserving super resolution with gradient guidance. In proc. Int. Conf. Computer vision and Pattern Recognition. pp. 7769–7778

  25. Martin D, Fowlkes C, Tal D, et al (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In proc. Int. Conf. International conference on computer vision (ICCV), pp. 416–423

  26. Nan F, Zeng QL, Xing YN et al (2020) Single image super-resolution reconstruction based on the ResNeXt network. Multimed Tools Appl 79:34459–34470

    Article  Google Scholar 

  27. Peleshko D , Rak T , Peleshko M , et al (2016) Two-frames image superresolution based on the aggregate divergence matrix. 2016 IEEE first international conference on Data Stream Mining & Processing (DSMP). IEEE computer society, pp. 235-238

  28. Rashkevych Y, Peleshko D , Vynokurova O, et al (2017) Single-frame image super-resolution based on singular square matrix operator. 2017 IEEE first Ukraine conference on electrical and computer engineering (UKRCON). IEEE Computer Society, pp 944–948

  29. Sajjadi, M S., Scholkopf, B., Hirsch, M., et al (2017). EnhanceNet: single image super-resolution through automated texture synthesis. In proc. Int. Conf. International conference on computer vision (ICCV), pp. 4501–4510.

  30. Soh JW, Park GY, Jo J, et al (2019). Natural and realistic single image super-resolution with explicit natural manifold discrimination. in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), pp. 8122–8131.

  31. Tai, Y., Yang, J., Liu, X., et al (2017). Image super-resolution via deep recursive residual network. in Proc Int. Conf. Computer Vision and Pattern Recognition (CVPR), PP. 2790–2798.

  32. Timofte R, Agustsson E, Gool L, et al (2017). NTIRE 2017 challenge on single image super-resolution methods and results. in Proc. Int. Conf. computer vision and pattern recognition (CVPR), pp. 1110–1121

  33. Tong T, Li G, Liu X, et al (2017). Image super-resolution using dense skip connections. In proc. Int. Conf. International conference on computer vision (ICCV), pp. 4799–4807

  34. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (n.d.) Residual attention network for image classification. Computer Vision and Pattern Recognition, pp.6450–6458

  35. Wang, X., Yu, K., Dong, C., et al (2018). Recovering realistic texture in image super-resolution by deep spatial feature transform. In proc. Int. Conf. 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 606-615.

  36. Wang X, Yu K, Wu S, et al (2018). ESRGAN: enhanced super-resolution generative adversarial networks. In proc. Int. Conf. European conference on computer vision (ECCV), pp. 63-79

  37. Wang Z, Bovik AC, Sheikh HR et al (2004) Image quality assessment: from error visibility to structural similarity. Transactions on Image Processing 13(4):600–612

    Article  Google Scholar 

  38. Yao X, Wu Q, Zhang P, Bao F (2019) Adaptive rational fractal interpolation function for image super-resolution via local fractal analysis. Image Vis Comput 82:39–49

    Article  Google Scholar 

  39. Zhang Yulun, Li K, Li K, et al (2018) Image Super-Resolution Using Very Deep Residual Channel Attention Networks[J]. 2018ECCV.

  40. Yulun Z, Li K, Li K et al (2018) Image super-resolution using very deep Residual Channel attention networks. European Conference on Computer Vision, pp:294–310

  41. Zeyde R, Elad M, Protter M, et al (2010). On single image scale-up using sparse-representations. In proc. Int. Conf international conference on curves and surfaces, pp. 711–730

  42. Zhang R, Isola P, Efros AA, et al (2018). The unreasonable effectiveness of deep features as a perceptual metric. in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), pp. 586–595.

Download references

Author information

Authors and Affiliations

  1. School of Information Science & Engineering, Xinjiang University, Urumqi, 830046, China

    Yongqiang Wang

  2. School of Software, Xinjiang University, Urumqi, 830046, China

    Xue Li, Fangzhe Nan, Feng Liu, Hua Li, Haitao Wang & Yurong Qian

Authors
  1. Yongqiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xue Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fangzhe Nan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Feng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hua Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Haitao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yurong Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yurong Qian.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Li, X., Nan, F. et al. Image super-resolution reconstruction based on generative adversarial network model with feedback and attention mechanisms. Multimed Tools Appl 81, 6633–6652 (2022). https://doi.org/10.1007/s11042-021-11679-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11679-1

Keywords

Search

Navigation