Skip to main content
SpringerLink
Log in

An image super-resolution network based on multi-scale convolution fusion

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, we propose a multi-scale convolution adaptive fusion super-resolution reconstruction network. Firstly, the input is passed through three convolution kernels of different sizes, and then the results are added and fused. Then, after pooling and full connection, the output results of the convolution layer with different sizes are weighted and added by the Softmax weighting mechanism to get the fusion feature map. Since the weights of the different branches with different convolution kernel sizes can be adaptively changed with the input information, the SR reconstruction is effectively improved. The detailed comparative experiments on the public datasets show that the SR reconstruction effect of our model is better than that of some state-of-the-art networks in objective criteria PSNR, SSIM, and subjective visual effect.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Harris, J.L.: Diffraction and resolving power. J. Opt. Soc. Am. (1917–1983), 54(7), 931–933 (1964)

  2. Goodman, J.W.: Introduction to Fourier Optics. Roberts and Company Publishers (2005)

  3. Tsai, R.Y., Huang, T.S.: Multi-frame image restoration and registration. Adv. Comput. Vis. Image Process. 1(2), 317–339 (1984)

    Google Scholar 

  4. Dai, S., Han, M., Xu, W., et al.: Soft edge smoothness prior for alpha channel super resolution. CVPR. 7, 1–8 (2007)

    Google Scholar 

  5. Aly, H.A., Dubois, E.: Image up-sampling using total-variation regularization with a new observation model. IEEE Trans. Image Process. 14(10), 1647–1659 (2005)

    Article  Google Scholar 

  6. Shan, Q., Li, Z., Jia, J., et al.: Fast image/video upsampling. ACM Trans. Gr. (TOG) 27(5), 1–7 (2008)

    Article  Google Scholar 

  7. He, H., Siu, W.C.: Single image super-resolution using Gaussian process regression//CVPR. IEEE 2011, 449–456 (2011)

    Google Scholar 

  8. Zhang, K., Gao, X., Tao, D., et al.: Single image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 4544–4556 (2012)

  9. Chan, T.M., Zhang, J., Pu, J., et al.: Neighbor embedding based super-resolution algorithm through edge detection and feature selection. Pattern Recognit. Lett. 30(5), 494–502 (2009)

    Article  Google Scholar 

  10. Yang, X., Liu, L., Zhu, C., et al.: An improved anchor neighborhood regression SR method based on low-rank constraint. Vis. Comput. 1–14 (2020)

  11. Kuanar, S., Mahapatra, D., Bilas, M., et al.: Multi-path dilated convolution network for haze and glow removal in nighttime images. Vis. Comput. 3, 1–14 (2021)

    Google Scholar 

  12. Kuanar, S., Conly, C., Rao, K.R.: Deep learning based HEVC in-loop filtering for decoder quality enhancement//2018 Picture Coding Symposium (PCS). IEEE, pp. 164–168 (2018)

  13. Dong, C., Loy, C.C. He, K., et al.: Learning a deep convolutional network for image super-resolution//European Conference on Computer Vision. Springer, Cham, pp. 184–199 (2014)

  14. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  15. Lim, B., Son, S., Kim, H., et al.: Enhanced deep residual networks for single image super-resolution//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  16. Zhang, Y., Li, K., Li, K., et al.: Image super-resolution using very deep residual channel attention networks//Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018)

  17. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for single image super-resolution. arXiv preprint arXiv:1904.05677 (2019)

  18. Hu, Y., Gao, X., Li, J., et al.: Single image super-resolution via cascaded multi-scale cross network. arXiv preprint arXiv:1802.08808 (2018)

  19. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3147–3155 (2017)

  20. Yang, X., Li, X., Li, Z., et al.: Image super-resolution based on deep neural network of multiple attention mechanism. J. Vis. Commun. Image Represent 75, 103019 (2021)

  21. Zhang, Y., Tian, Y., Kong, Y., et al.: Residual dense network for image super-resolution//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)

  22. Hui, Z., Gao, X., Yang, Y., et al.: Lightweight image super-resolution with information multi-distillation network//Proceedings of the 27th ACM International Conference on Multimedia, pp. 2024–2032 (2019)

  23. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 723–731 (2018)

  24. Liu, J., Tang, J., Wu, G.: Residual feature distillation network for lightweight image super-resolution. arXiv preprint arXiv:2009.11551 (2020)

  25. Yang, X., Guo, Y., Li, Z., et al.: Image super-resolution network based on a multi-branch attention mechanism. Signal Image Video Process. 1–9 (2021)

  26. Ledig, C., Theis, L., Huszár, F., et al.: Photo-realistic single image super-resolution using a generative adversarial network//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

  27. Ma, C., Rao, Y., Cheng, Y., Chen, C., Lu, J., Zhou, J.: Structure-Preserving Super Resolution with Gradient Guidance. arXiv preprint arXiv :2003.13081 (2020)

  28. Blau, Y., Mechrez, R., Timofte, R., et al.: The 2018 pirm challenge on perceptual image super-resolution//Proceedings of the European Conference on Computer Vision (ECCV), pp. 334–355 (2018)

  29. Zhang, R, Isola, P., Efros, A.A., et al.: The unreasonable effectiveness of deep features as a perceptual metric//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

  30. Li, X., Wang, W., Hu, X., et al.: Selective kernel networks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 510–519 (2019)

  31. Timofte, R., Agustsson, E., Van Gool, L., et al.: Ntire 2017 challenge on single image super-resolution: Methods and results//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017)

  32. Bevilacqua, M., Roumy, A., Guillemot, C., et al.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding 135-1 (2012)

  33. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations//International Conference on Curves and Surfaces. Springer, Berlin, Heidelberg, pp. 711–730 (2010)

  34. Martin, D., Fowlkes, C., Tal, D., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics//Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2, pp. 416–423. IEEE (2001)

  35. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

  36. Fujimoto, A., Ogawa, T., Yamamoto, K., et al.: Manga109 dataset and creation of metadata//Proceedings of the 1st International Workshop on Comics Analysis, Processing and Understanding, pp. 1–5 (2016)

  37. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network//European Conference on Computer Vision, pp. 391–407. Springer, Cham (2016)

  38. Lai, W.S. Huang, J.B. Ahuja, N., et al.: Deep Laplacian pyramid networks for fast and accurate super-resolution//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 624–632 (2017)

  39. Tai, Y., Yang, J., Liu, X., et al.: Memnet: a persistent memory network for image restoration//Proceedings of the IEEE International Conference on Computer Vision, pp. 4539–4547 (2017)

  40. Li, Z., Yang, J., Liu, Z., et al.: Feedback network for image super-resolution//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3867–3876 (2019)

  41. Lai, W.S., Huang, J.B., Ahuja, N., et al.: Fast and accurate image super-resolution with deep Laplacian pyramid networks. IEEE Trans. Pattern Anal. Mach. Intell. 41(11), 2599–2613 (2018)

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the National Natural Science Foundation of China (61573182), and by the Fundamental Research Funds for the Central Universities (NS2020025).

Author information

Authors and Affiliations

  1. College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Xin Yang, Yitian Zhu, Yingqing Guo & Dake Zhou

Authors
  1. Xin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yitian Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yingqing Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dake Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Yang.

Ethics declarations

Conflict of interest

Xin Yang declares that he has no conflict of interest. Yitian Zhu declares that he has no conflict of interest. Yingqing Guo declares that he has no conflict of interest. Dake Zhou declares that he has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, X., Zhu, Y., Guo, Y. et al. An image super-resolution network based on multi-scale convolution fusion. Vis Comput 38, 4307–4317 (2022). https://doi.org/10.1007/s00371-021-02297-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02297-x

Keywords

Search

Navigation