Skip to main content
SpringerLink
Log in

Multi-scale fractal residual network for image super-resolution

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Recent studies have shown that the use of deep convolutional neural networks (CNNs) can improve the performance of single image super-resolution reconstruction (SISR) methods. However, the existing CNN-based SISR model ignores the multi-scale features and shallow and deep features of the image, resulting in relatively low image reconstruction performance. To address these issues, this paper proposes a new multi-scale fractal residual network (MSFRN) for image super-resolution. On the basis of residual learning, a multi-scale fractal residual block (MSFRB) is designed. This block uses convolution kernels of different sizes to extract image multi-scale features and uses multiple paths to extract and fuse image features of different depths. Then, the shallow features extracted at the shallow feature extraction stage and the local features output by all MSFRBs are used to perform global hierarchical feature fusion. Finally, through sub-pixel convolution, the fused global features are used to reconstruct high-resolution images from low-resolution images. The experimental results on the five standard benchmark datasets show that MSFRN improved subjective visual effects and objective image quality evaluation indicators, and is superior to other state-of-the-art SISR methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Zhang H, Yang Z, Zhang L, Shen H (2013) Super-resolution reconstruction for multi-angle remote sensing images considering resolution differences. Remote Sens 6(1):637–657

    Article  Google Scholar 

  2. Zou WWW, Yuen PC (2012) Very low resolution face recognition problem. IEEE Trans Image Process 21(1):327–340

    Article  MathSciNet  Google Scholar 

  3. Shi W, Caballero J, Ledig C, Zhuang X, Bai W, Bhatia K, De Marvao AMSM, Dawes T, O’Regan D, Rueckert D (2013) Cardiac image super-resolution with global correspondence using multi-atlas PatchMatch. In: 16th International Conference on Medical Image Computing and Computer Assisted Intervention, pp. 9-16

  4. Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10(10):1521–1527

    Article  Google Scholar 

  5. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238

    Article  Google Scholar 

  6. Tai Y-W, Liu S, Brown MS, Lin S (2010) Super resolution using edge prior and single image detail synthesis. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2400–2407

  7. Zhang K, Gao X, Member S, IEEE, Tao D (2012) Single image super-resolution with non-local means and steering kernel regression. IEEE Trans Image Process 21(11):4544–4556

    Article  MathSciNet  Google Scholar 

  8. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  9. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1646-1654

  10. Kim J, Lee JK, Lee KM (2016) deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1637-1645

  11. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: proceedings-30th IEEE Conference on Computer Vision and Pattern Recognition, pp. 5835-5843

  12. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: Proceedings-30th IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790-2798

  13. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings-30th IEEE Conference on Computer Vision and Pattern Recognition, pp. 105-114

  14. N. Ahn, B. Kang, K.-A. Sohn (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 252-268

  15. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1132-1140

  16. Cao F, Liu H (2019) Single image super-resolution via multi-scale residual channel attention network. Neurocomputing 358:424–436

    Article  Google Scholar 

  17. Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1874-1883

  18. Jiang K, Wang Z, Yi P, Jiang J (2020) Hierarchical dense recursive network for image super-resolution. Pattern Recogn 107:107475

    Article  Google Scholar 

  19. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings-30th IEEE Conference on Computer Vision and Pattern Recognition, pp. 2261-2269

  20. Larsson G, Maire M, Shakhnarovich G (2017) FractalNet: ultra-deep neural networks without residuals. In: 5th international conference on Learning Representations

  21. Zhou Y, Dong J, Yang Y (2019) Deep fractal residual network for fast and accurate single image super resolution. Neurocomputing

  22. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 770–778

  23. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: 28th Annual Conference on Neural Information Processing Systems 2014, pp. 2672–2680

  24. Cao F, Chen B (2019) New architecture of deep recursive convolution networks for super-resolution. Knowl-Based Syst 178:98–110

    Article  Google Scholar 

  25. Soh JW, Cho NI (2020) Lightweight single image super-resolution with multi-scale spatial attention networks. IEEE Access 8:35383–35391

    Article  Google Scholar 

  26. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-ResNet and the impact of residual connections on learning. In: 31st AAAI Conference on Artificial Intelligence, pp. 4278-4284

  27. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2472–2481

  28. Qin J, Huang Y, Wen W (2020) Multi-scale feature fusion residual network for single image super-resolution. Neurocomputing 379:334–342

    Article  Google Scholar 

  29. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–9

  30. Yu W, Yang K, Yao H, Sun X, Xu P (2017) Exploiting the complementary strengths of multi-layer CNN features for image. Neurocomputing 237:235–241

    Article  Google Scholar 

  31. Zhao H, Gallo O, Frosio I, Kautz J (2017) Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging 3(1):47–57

    Article  Google Scholar 

  32. Timofte R, Agustsson E, Gool LV, Yang M-H, Zhang L, Lim B et al (2017) NTIRE 2017 challenge on single image super-resolution: methods and results. In: 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1110-1121

  33. Bevilacqua M, Roumy A, Guillemot C, Morel M-LA (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding, In: Proceedings of British Machine Vision Conference, pp. 1-10

  34. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: 7th International Conference on Curves and Surfaces, pp. 711-730

  35. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the IEEE International Conference on Computer Vision,pp. 416–423

  36. Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 5197–5206

  37. Matsui Y, Ito K, Aramaki Y, Fujimoto A, Ogawa T, Yamasaki T, Aizawa K (2017) Sketch-based manga retrieval using manga109 dataset. Multimed Tools Appl 76(20):21811–21838

    Article  Google Scholar 

  38. Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: ICLR 2015 - conference track proceedings

  39. A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer (2017) Automatic differentiation in pytorch, Advances in Neural Information Processing Systems Autodiff Workshop

  40. Keys RG (1981) Cubic convolution interpolation for digital image processing. IEEE Transactions on Acoustics, Speech, and Signal Processing ASSP 29(6):1153–1160

    Article  MathSciNet  Google Scholar 

  41. Timofte R, de Smet V, van Gool L (2015) A+: adjusted anchored neighborhood regression for fast super-resolution. In: 12th Asian Conference on Computer Vision, pp. 111-126

  42. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: 14th European Conference on Computer Vision(ECCV), pp. 391-407

  43. Li J, Fang F, Mei K, Zhang G (2018) Multi-scale residual network for image super-resolution. In: 15th European Conference on Computer Vision, pp. 527-542

Download references

Acknowledgments

This work was supported by the Science and Technology Support of Tianjin Key Research and the Development Plan Project (Grant no.18YFZCGX00930). The authors also acknowledge the anonymous reviewers for their helpful comments on the manuscript.

Author information

Authors and Affiliations

  1. School of Electronics and Information Engineering, Tiangong University, Tianjin, 300387, China

    Xinxin Feng, Xianguo Li & Jianxiong Li

  2. Tianjin Key Laboratory of Optoelectronic Detection Technology and System, Tianjin, 300387, China

    Xinxin Feng, Xianguo Li & Jianxiong Li

Authors
  1. Xinxin Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xianguo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianxiong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xianguo Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feng, X., Li, X. & Li, J. Multi-scale fractal residual network for image super-resolution. Appl Intell 51, 1845–1856 (2021). https://doi.org/10.1007/s10489-020-01909-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01909-8

Keywords

Search

Navigation