Skip to main content
SpringerLink
Log in

Fine-grained scale space learning for single image super-resolution

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Recent deep convolutional neural networks have achieved great reconstruction accuracy for single image super-resolution (SISR). Most of them, however, need to train a specific set of parameters for a single scaling factor or a particular group of scaling factors. This means, multiple sets of model parameters have to be used for different scaling factors, each of which can be already very large. In this paper, we study a new problem of fine-grained scale space learning of SISR, which uses one set of parameters while achieving varying scales. Specifically, we aim to use an arbitrary base SISR \(\times 2\) model to realize high-quality SISR for a continuous-integer spectrum of scaling factors, e.g., \(2 \sim 8\). To this end, first for the base scaling factor 2, we propose low-resolution reconstruction, blind kernel estimation and recursive error compensation, to generate three loss functions, helping to boost the training quality of the base model. Then, we cascade the boosted SISR \(\times \)2 model and extend the low-resolution reconstruction to incorporate multiple LR loss functions covering \(\{3,4,\ldots ,2^n\}\) scales. By this way, the SISR \(\times \)2 model can be effectively tuned to work well for continuous-integer scaling factors, with exactly the same set of parameters. Extensive experiments verify the capability of our approach to enable state-of-the-art methods to realize fine-grained scale space learning of SISR, with higher accuracy and much less parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Note that to apply the proposed framework on ZSSR, we reconstruct scale \(2^n\) by reusing the ZSSR model with scale 2, denoted as ZSSR-2 and obtain the results of other scales by downsampling. This scheme has a similar cascade manner as ours and significantly decreases the parameter size.

References

  1. Agustsson, E., Timofte, R., Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: CVPRW, pp. 1122–1131 (2017)

  2. Chen, H., Wang, Y., Xu, C., Shi, B., Xu, C., Tian, Q., Xu, C.: Addernet: Do we really need multiplications in deep learning? (2021)

  3. Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function. In: CVPR, pp. 8628–8638 (2021)

  4. Chen, Y., Shi, F., Christodoulou, A.G., Xie, Y., Zhou, Z., Li, D.: Efficient and accurate mri super-resolution using a generative adversarial network and 3d multi-level densely connected network. In: MICCAI (2018)

  5. Dong, C., Chen, C.L., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE TPAMI 38(2), 295–307 (2016)

    Article  Google Scholar 

  6. Dong, C., Chen, C.L., Tang, X.: Accelerating the super-resolution convolutional neural network. In: ECCV, pp. 391–407 (2016)

  7. Feng, W., Tian, F.P., Zhang, Q., Zhang, N., Sun, J.: Fine-grained change detection of misaligned scenes with varied illuminations. In: ICCV (2015)

  8. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: ICCV, pp. 349–356 (2009)

  9. Hamid Rahim, S., Alan Conrad, B., Gustavo, D.V.: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE TIP 14(12), 2117–2128 (2005)

    Google Scholar 

  10. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: CVPR (2018)

  11. Hu, X., Mu, H., Zhang, X., Wang, Z., Tan, T., Sun, J.: Meta-sr: A magnification-arbitrary network for super-resolution. In: CVPR (2019)

  12. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: CVPR, pp. 5197–5206 (2015)

  13. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: CVPR (2018)

  14. Irani, M., Peleg, S.: Motion analysis for image enhancement: Resolution, occlusion, and transparency. J. Vis. Commun. Image Represent. 4(4), 324–335 (1993)

    Article  Google Scholar 

  15. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp. 1646–1654 (2016)

  16. Kong, X., Zhao, H., Qiao, Y., Dong, C.: Classsr: A general framework to accelerate super-resolution networks by data characteristic. In: CVPR, pp. 12,016–12,025 (2021)

  17. Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: CVPR, pp. 5835–5843 (2017)

  18. Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE TPAMI PP(99), 1 (2017)

  19. Li, J., Fang, F., Li, J., Mei, K., Zhang, G.: Mdcn: Multi-scale dense cross network for image super-resolution. TCSVT 31(7), 2547–2561 (2021)

    Google Scholar 

  20. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: CVPRW, pp. 1132–1140 (2017)

  21. Liu, Y., Wan, L., Fan, L.: Fine-grained scale space learning for single image super-resolution. In: CGI (2022)

  22. Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: ICCV, pp. 2272–2279 (2010)

  23. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, vol. 2, pp. 416–423 (2001)

  24. Pablo, A., Michael, M., Charless, F., Jitendra, M.: Contour detection and hierarchical image segmentation. IEEE TPAMI 33(5), 898–916 (2011)

    Article  Google Scholar 

  25. Pan, J., Liu, Y., Sun, D., Ren, J., Cheng, M.M., Yang, J., Tang, J.: Image formation model guided deep image super-resolution. In: AAAI (2020)

  26. Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp. 1874–1883 (2016)

  27. Shocher, A., Cohen, N., Irani, M.:“zero-shot”super-resolution using deep internal learning. In: CVPR (2017)

  28. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: CVPR, pp. 2790–2798 (2017)

  29. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: ICCV, pp. 4549–4557 (2017)

  30. Thornton, M.W., Atkinson, P.M., Holland, D.A.: Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping. Int. J. Remote Sens. 27(3), 473–491 (2006)

    Article  Google Scholar 

  31. Vedaldi, A., Lenc, K.: Matconvnet:convolutional neural networks for matlab. In: ACM MM, pp. 689–692 (2015)

  32. Wang, L., Dong, X., Wang, Y., Ying, X., Lin, Z., An, W., Guo, Y.: Exploring sparsity in image super-resolution for efficient inference. In: CVPR (2021)

  33. Wang, L., Wang, Y., Lin, Z., Yang, J., An, W., Guo, Y.: Learning for scale-arbitrary super-resolution from scale-specific networks. In: arXiv (2020)

  34. Wang, L., Wang, Y., Lin, Z., Yang, J., An, W., Guo, Y.: Learning for scale-arbitrary super-resolution from scale-specific networks. arXiv preprint arXiv:2004.03791 (2020)

  35. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE TIP 13(4), 600–612 (2004)

    Google Scholar 

  36. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE TIP 19(11), 2861–2873 (2010)

    MathSciNet  MATH  Google Scholar 

  37. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International Conference on Curves and Surfaces, pp. 711–730 (2012)

  38. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for -ple degradations. In: CVPR (2017)

  39. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp. 286–301 (2018)

  40. Zou, W.W.W., Yuen, P.C.: Very low resolution face recognition problem. IEEE TIP 21(1), 327–40 (2012)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The work is supported by the research fund for The Tianjin Key Lab for Advanced Signal Processing, Civil Aviation University of China (2019ASP-TJ01) and the National Natural Science Foundation of China (Grant No. 61902275).

Author information

Authors and Affiliations

  1. College of Intelligence and Computing, Tianjin University, Tianjin, China

    Ye Liu, Liang Wan, Fan Lyu & Wei Feng

Authors
  1. Ye Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liang Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fan Lyu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liang Wan.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Wan, L., Lyu, F. et al. Fine-grained scale space learning for single image super-resolution. Vis Comput 38, 3377–3389 (2022). https://doi.org/10.1007/s00371-022-02551-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02551-w

Keywords

Search

Navigation