Skip to main content
SpringerLink
Log in

Boundary equilibrium SR: effective loss functions for single image super-resolution

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Recently, single image super-resolution (SISR) has made great progress due to the rapid development of deep convolutional neural networks (CNN), and the application of Generative Adversarial Networks (GAN ) has made super-resolution networks even more effective. However, GAN-based methods have many problems such as lengthy and unstable convergence. To solve these problems, this paper presents a mechanism that employs boundary equilibrium in the image super-resolution network to balance the convergence of the generator and the discriminator and improve the visual quality of the generated synthetic images. Furthermore, current methods often use perceptual loss based on the VGG network. However, experiments show that the visual quality improvement brought by this perceptual loss is very limited, so we propose an improved perceptual loss based on Learned Perceptual Image Patch Similarity (LPIPS) to acquire better human visual effects rather than adopting the traditional perceptual loss based on VGG. The experimental results clearly show that using our proposed method can considerably improve the performance of image super-resolution and obtain clearer details than state-of-the-arts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: dataset and study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 126–135

  2. Bevilacqua M, Roumy A, Guillemot C et al (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the british machine vision conference. BMVA Press, pp 135.1–135.10. https://doi.org/10.5244/C.26.135https://doi.org/10.5244/C.26.135

  3. Chowdhuri D, SKK S, Babu MR et al (2012) Very low resolution face recognition in parallel environment. Int J Comput Sci Inform Technol 3(3):4408–4410

    Google Scholar 

  4. Dai T, Cai J, Zhang Y et al (2019) Second-order attention network for single image super-resolution. In: Conference on computer vision and pattern recognition, pp 11,057–11,066

  5. Dong C, Loy CC, He K et al (2014) Learning a deep convolutional network for image super-resolution. In: European conference on computer vision. Springer, pp 184–199

  6. Dong C, Loy CC, He K et al (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  7. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: ECCV, pp 391–407

  8. Goodfellow IJ, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial nets. In: Proceedings of the 27th international conference on neural information processing systems - vol 2 MIT Press, Cambridge, MA, USA, NIPS’14, pp 2672–2680

  9. Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 5197–5206

  10. Jo Y, Yang S, Kim SJ (2020) Investigating loss functions for extreme super-resolution. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops, pp 1705–1712

  11. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: ECCV, pp 694–711

  12. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  13. Ledig C, Theis L, Huszár F et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Conference on computer vision and pattern recognition, pp 105–114

  14. Li Y, Xiao N, Ouyang W (2018) BEGAN: boundary equilibrium generative adversarial networks. IEEE Access 6:11,342– 11,348

    Article  Google Scholar 

  15. Lim B, Son S, Kim H et al (2017) Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1132–1140

  16. Liu J, Zhang W, Tang Y et al (2020) Residual feature aggregation network for image super-resolution. In: Conference on computer vision and pattern recognition, pp 2356–2365

  17. Martin D, Fowlkes C, Tal D et al (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: IEEE international conference on computer vision, IEEE, pp 416–423

  18. Nasrollahi K, Moeslund TB (2014) Super-resolution: a comprehensive survey. Mach Vis Appl 25(6):1423–1468

    Article  Google Scholar 

  19. Niu B, Wen W, Ren W et al (2020) Single image super-resolution via a holistic attention network. In: ECCV, pp 191–207

  20. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:151106434

  21. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention, pp 234–241

  22. Shi W, Caballero J, Huszár F et al (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883

  23. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  24. Wang X, Yu K, Wu S et al (2019) ESRGAN: enhanced super-resolution generative adversarial networks. In: ECCV 2018 workshops, pp 63–79

  25. Xu J, Chae Y, Stenger B et al (2018) Dense bynet: residual dense network for image super resolution. In: 2018 25th IEEE international conference on image processing (ICIP). IEEE, pp 71-75

  26. Zeyde R, Elad M, Protter M (2012) On single image scale-up using sparse-representations. In: Curves and surfaces, Berlin, Heidelberg, pp 711–730

  27. Zhang J, Jia K, Jia J et al (2018a) An improved approach to infer protein-protein interaction based on a hierarchical vector space model. Bmc Bioinformatics 19(1):161

    Article  Google Scholar 

  28. Zhang R, Isola P, Efros AA et al (2018b) The unreasonable effectiveness of deep features as a perceptual metric. Proc IEEE Comput Society Conf Comput Vis Pattern Recognit, (1):586–595

  29. Zhang Y, Li K, Li K et al (2018c) Image super-resolution using very deep residual channel attention networks. In: ECCV, pp 294–310

Download references

Acknowledgements

This work has been supported by the Natural Science Foundation of Shandong Province (No. ZR2019MF011), and the Postdoctoral Science Foundation of China (No. 2017M622210). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V used for this research.

Author information

Authors and Affiliations

  1. School of Mechanical Electrical & Information Engineering, Shandong University, Weihai, 264200, China

    Zhechen Zhang, Shuo Chen, Fei Yang & Pan Jingchang

  2. Department of Educational Technology, Ocean University of China, Qingdao, 266000, China

    Weigang Lu

  3. School of Computer Science and Technology, Shandong University, Qingdao, 266000, China

    Fei Yang

Authors
  1. Zhechen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weigang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pan Jingchang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fei Yang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Lu, W., Chen, S. et al. Boundary equilibrium SR: effective loss functions for single image super-resolution. Appl Intell 53, 17128–17138 (2023). https://doi.org/10.1007/s10489-022-04162-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04162-3

Keywords

Search

Navigation