Skip to main content
Log in

Separable feature complementary network with branch-wise and multi-scale spatial attention for lightweight image super-resolution

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Recent advances in single image super-resolution (SISR) have shown promising results, but networks with optimal performance tend to have heavy computation, making them unsuitable for edge devices. How to achieve better results with fewer parameters is still a problem that requires further research. To overcome this issue, we propose a separable feature complementary network using branch-wise attention and multi-scale spatial attention (SFCN-BMSA). The network contains a feature complementary module, which utilizes a limited number of small-sized convolution kernels to combine long-range features from different positions on the feature map and utilizes them to enhance image reconstruction. In addition, we design a feature fusion module with branch-wise attention, which can fuse the features of different branches according to the importance of each branch. Finally, we also design a multi-scale spatial attention module, which utilizes three dilated convolutions with the size of 5\(\times \)5 to calculate attention from different spatial scales and combines them to obtain more refined attention while utilizing a larger receptive field. Experiments show that the proposed neural network achieves better reconstruction results with lower parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availability

The data and materials used in this study are not publicly available. Due to data confidentiality and proprietary nature, the data and materials will not be provided.

References

  1. Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Signal Process 128, 389–408 (2016)

    Article  Google Scholar 

  2. Malczewski, K., Stasiński, R.: Super resolution for multimedia, image, and video processing applications. In: Recent Advances in Multimedia Signal Processing and Communications, pp. 171–208. Springer, Berlin, Heidelberg (2009)

  3. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Machine Vis Appl 25, 1423–1468 (2014)

    Article  Google Scholar 

  4. Wang, Z., Chen, J., Hoi, S.C.: Deep learning for image super-resolution: a survey. IEEE Trans. Patt. Anal. Mach. Intell. 43(10), 3365–3387 (2020)

    Article  Google Scholar 

  5. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans Patt Anal Machine Intelligence 38(2), 295–307 (2015)

    Article  Google Scholar 

  6. Dong, C., Loy, C. C., Tang, X.: Accelerating the super-resolution convolutional neural network, In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, Proceedings, Part II 14. pp. 391-407 (2016)

  7. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolu-tional neural network, In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1874-1883 (2016)

  8. Kim, J., Lee, J. K., Lee, K. M.: Accurate image super-resolution using very deep convolutional networks, In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1646–1654 (2016)

  9. Liu, J., Tang, J., Wu, G.: Residual feature distillation network for lightweight image super-resolution. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28,Proceedings, Part III 16. pp. 41–55 (2020)

  10. Dou, H., Zhang, W., Liang, X.: Image Super Resolution Recon-struction Algorithm Based on Deep Learning, In: 2019 IEEE 2nd International Conference on Information Communication and Signal Processing (ICICSP). pp. 306–310 (2019)

  11. Han, W., Chang, S., Liu, D., Yu, M., Witbrock, M., Huang, T. S.: Image su-per-resolution via dual-state recurrent networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1654-1663 (2018)

  12. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks, In: Proceedings of the European conference on computer vision (ECCV). pp. 286–301 (2018)

  13. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual net-works for single image super-resolution, In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 136–144 (2017)

  14. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial net-work, In: Proceedings of the IEEE conference on computer vision and pattern recogni-tion. pp. 4681–4690 (2017)

  15. Cao, F., Li, K.: A new method for image super-resolution with multi-channel constraints. Knowl-Based Syst 146, 118–128 (2018)

    Article  Google Scholar 

  16. Li, K., Cao, F.: Super-resolution using neighbourhood regression with local structure prior. Signal Process.: Image Commun. 72, 58–68 (2019)

    Google Scholar 

  17. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Guo, B.: Swin trans-former: hierarchical vision transformer using shifted windows, In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)

  18. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: image restoration using swin transformer, In: Proceedings of the IEEE/CVF international conference on computer vision pp. 1833-1844 (2021)

  19. Conde, M.V., Choi, U.J., Burchi, M., Timofte, R., Swin2SR: Swinv2 transformer for compressed image super-resolution and restoration. In Computer Vision-ECCV,: Workshops: Tel Aviv, Israel, October 23–27, 2022. Proceedings, Part II 2023, 669–687 (2022)

  20. Zhang, Y., Li, K., Li, K., Zhong, B., Fu, Y.: Residual non-local attention networks for image restoration. arXiv preprint arXiv:1903.10082 (2019)

  21. Mei, Y., Fan, Y., Zhou, Y.: Image super-resolution with non-local sparse attention. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. pp. 3517–3526 (2021)

  22. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applica-tions. arXiv preprint arXiv:1704.04861 (2017)

  23. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolu-tional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1874–1883 (2016)

  24. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image su-per-resolution with information multi-distillation network, In: Proceedings of the 27th ACM international conference on multimedia. pp. 2024–2032 (2019)

  25. Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)

  26. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L.: Ntire: challenge on single image super-resolution: methods and results. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 114–125 (2017)

  27. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)

  28. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations, In: Curves and Surfaces: 7th International Conference, Avignon, France, June 24–30, 2010. Revised Selected Papers, Vol 7, 711–730 (2012)

  29. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics In: Proceedings Eighth IEEE international conference on computer vision, ICCV, pp. 416-423 (2001)

  30. Huang, J. B., Singh, A., Ahuja, N.: Single image super-resolution from trans-formed self-exemplars, In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5197–520 (2015)

  31. Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multim. Tools Appl. 76, 1811–21838 (2017)

    Article  Google Scholar 

  32. Kingma, D. P., Ba, J.: Adam: a method for stochastic optimization. arXiv pre-print arXiv:1412.6980 (2014)

  33. Lai, W. S., Huang, J. B., Ahuja, N., Yang, M. H.: Deep Laplacian pyramid net-works for fast and accurate super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 624–632 (2017)

  34. Ahn, N., Kang, B., Sohn, K. A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European conference on computer vision (ECCV). pp. 252–268 (2018)

  35. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 723–731 (2018)

  36. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: Proceedings of the IEEE international conference on computer vi-sion. pp. 4539–4547 (2017)

  37. Muqeet, A., Hwang, J., Yang, S., Kang, J., Kim, Y., Bae, S.H.: Multi-attention based ultra lightweight image super-resolution. In: Computer Vision-ECCV 2020 Work-shops: Glasgow, UK, August 23–28, 2020. Proceedings, Part III, (16), 103–118 (2020)

  38. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image su-per-resolution with information multi-distillation network. In: Proceedings of the 27th ACM international conference on multimedia. pp. 2024–2032 (2019)

  39. Cheng, G., Matsune, A., Du, H., Liu, X., Zhan, S.: Exploring more diverse network architectures for single image super-resolution. Knowl-Based Syst. 235, 107648 (2022)

    Article  Google Scholar 

  40. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: image restoration using swin transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1833–1844 (2021)

Download references

Acknowledgements

This work was supported by the Provincial Key Laboratory Performance Subsidy Project (22567612 H), the Natural Science Foundation of Hebei Province (F2019203195), and the National Natural Science Foundation of China (62106214).

Funding

This work was supported by the Provincial Key Laboratory Performance Subsidy Project (22567612 H), the Natural Science Foundation of Hebei Province (F2019203195), and the National Natural Science Foundation of China (62106214).

Author information

Authors and Affiliations

Authors

Contributions

QMH wrote the main manuscript text. WMZ, YQL, and HBL provided guidance and equipment support. All authors reviewed the manuscript.

Corresponding author

Correspondence to Qiming Han.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, W., Han, Q., Li, Y. et al. Separable feature complementary network with branch-wise and multi-scale spatial attention for lightweight image super-resolution. SIViP 18, 1715–1724 (2024). https://doi.org/10.1007/s11760-023-02880-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02880-w

Keywords

Navigation