Skip to main content
SpringerLink
Log in

A novel perceptual loss function for single image super-resolution

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, various deep neural networks have been proposed to improve the performance in the single image super-resolution (SISR) task. The commonly used per-pixel MSE loss function captures less perceptual difference and tends to make the super-resolved images overly smooth, while the perceptual loss function defined on image features extracted from one or two layers of a pretrained network yields more visually pleasing results. We propose a new perceptual loss function via combining features from multiple levels, which incorporates the discrepancy between the reconstruction and the ground truth in different structures. In addition, some variants of the proposed perceptual loss are explored. Extensive quantitative and qualitative comparisons with the state-of-the-art methods demonstrate that our loss function can drive the same network to produce better results when used alone or combined with other loss functions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: dataset and study. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW) , pp 126–135

  2. Bevilacqua M, Roumy A, Guillemot C, Morel M-LA (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: British machine vision conference

  3. Blau Y, Mechrez R, Timofte R, Michaeli T, Zelnik-Manor L (2018) The pirm challenge on perceptual super resolution. (https://www.pirm2018.org/PIRM-SR.html)

  4. Blau Y, Michaeli T (2018) The Perception-Distortion tradeoff. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 6628–6237

  5. Bosse S, Maniry D, Müller KR, Wiegand T, Samek W (2018) Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans Image Process 27(1):206–219

    Article  MathSciNet  MATH  Google Scholar 

  6. Bruna J, Sprechmann P, LeCun Y (2016) Super-resolution with deep convolutional sufficient statistics. In: International conference on learning representations (ICLR)

  7. Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European conference on computer vision (ECCV), pp 184–199

  8. Dong C, Loy CC, Tang X (2016) Accelerating the superresolution convolutional neural network. In: European conference on computer vision (ECCV), pp 391–407

  9. Greenspan H (2009) Super-resolution in medical imaging. The Comput J 52 (1):43–63

    Article  Google Scholar 

  10. Hochstein S, Ahissar M (2002) View from the top Hierarchies and reverse hierarchies in the visual system. Neuron 36(5):791–804

    Article  Google Scholar 

  11. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision (ECCV), pp 694–711

  12. Kim J, Lee JK, Lee KM (2016) Accurate image superresolution using very deep convolutional networks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1646–1654

  13. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1637–1645

  14. Kingma D, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR)

  15. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems (NIPS), pp 1097–1105

  16. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017) Deep laplacian pyramid networks for fast and accurate superresolution. In: IEEE conference on computer vision and pattern recognition (CVPR) , pp 624–632

  17. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4681–4690

  18. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 136–144

  19. Mahendran A, Vedaldi A (2016) Visualizing deep convolutional neural networks using natural pre-images. Int J Comput Vis, pp 1–23

  20. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: IEEE international conference on computer vision (ICCV), pp 416–423

  21. Park S-J, Son H, Cho S, Hong K-S, Lee S (2018) SRFeat: single image super-resolution with feature discrimination. In: European conference on computer vision (ECCV), pp 439–455

  22. Sajjadi MSM, Scholkopf B, Hirsch M (2017) EnhanceNet: single image super-resolution through automated texture synthesis. In: International conference on computer vision (ICCV), pp 4491–4500

  23. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations (ICLR)

  24. Tan W, Yan B, Bare B (2018) Feature super-resolution: make machine see more clearly. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 3994–4002

  25. Tong T, Li G, Liu X, Gao Q (2017) Image super-resolution using dense skip connections. In: International conference on computer vision (ICCV), pp 4799–4807

  26. Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Loy CC (2018) ESRGAN: enhanced super-resolution generative adversarial networks. In: European conference on computer vision workshops (ECCVW), pp 63–79

  27. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13 (4):600–612

    Article  Google Scholar 

  28. Yang C-Y, Ma C, Yang M-H (2014) Single-image super-resolution: a benchmark. In: European conference on computer vision (ECCV), pp 372–386

  29. Yosinski J, Clune J, Nguyen A, Fuchs T, Lipson H (2015) Understanding neural networks through deep visualization. In: International conference on machine learning - deep learning workshop

  30. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: Curves and surfaces, pp 711–730

  31. Zhang L, Zhang H, Shen H, Li P (2010) A super-resolution reconstruction algorithm for surveillance images. Signal Process 90(3):848–859

    Article  MATH  Google Scholar 

  32. Zhang R, Isola1 P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 586–595

  33. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 6848–6856

  34. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2472–2481

  35. Zhao H, Gallo O, Frosio I, Kautz J (2017) Loss functions for image restoration with neural networks. In: IEEE transactions on computational imaging, vol 3, pp 47–57

Download references

Acknowledgements

Thanks to the National Natural Science Foundation of China (No. NSFC-61471067, 81671651), the National Great Science Specific Project (No. 2015ZX03002008), the Beijing Municipal Natural Science Foundation (No. 1472024), Beijing University of Posts and Telecommunications (No. 2013XD-04, 2015XD-02), the Beijing Key Laboratory of Work Safety and Intelligent Monitoring Foundation.

Author information

Authors and Affiliations

  1. School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Qiong Wu, Chunxiao Fan, Yong Li & Jiahao Hu

  2. Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China

    Yang Li

Authors
  1. Qiong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chunxiao Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiahao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunxiao Fan.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Q., Fan, C., Li, Y. et al. A novel perceptual loss function for single image super-resolution. Multimed Tools Appl 79, 21265–21278 (2020). https://doi.org/10.1007/s11042-020-08878-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08878-7

Keywords

Search

Navigation