Skip to main content
SpringerLink
Log in

Multi-feature self-attention super-resolution network

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In recent years, single-image super-resolution (SISR) methods based on the attention mechanism have been widely explored and achieved remarkable performances. However, most existing networks only explore channel correlations or the spatial long-distance dependences in a single scale while ignoring the mutual guidance of multi-scale information, resulting in the loss of high-frequency information in the reconstructed image. To address this issue, we propose a multi-feature self-attention super-resolution network (MFSN) to embed multi-scale encoding information into the attention mechanism. Specifically, the network consists of a shallow feature extraction subnetwork, a multi-feature alignment subnetwork (MFAN) and a reconstruction subnetwork. The MFAN is composed of an adjacent feature alignment residual block (AFAB) and a dense backward fusion block (DBFB), where AFAB explores multi-scale encoding information using the low-resolution space statistics with larger receptive fields to weight and align the original-scale feature map, so as to extract more discriminative high-frequency features adaptively. Meanwhile, the contrast-aware channel attention module adopts contrast pooling that is more suitable for low-level computer vision tasks to realize the adaptive selection of channel feature in the AFAB. Structurally, the DBFB adopts the backward fusion mechanism to fuse the output of each AFAB, from the deep layer to the shallow layer to make full use of the hierarchical features. Experimental results demonstrate the superiority of our MFSN network in terms of both quantitative metrics and visual quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Greenspan, H.: Super-resolution in medical imaging. Comput. J. 52(1), 43–63 (2009)

    Article  Google Scholar 

  2. Zhang, D., Shao, J., Li, X., Shen, H.T.: Remote sensing image super-resolution via mixed high-order attention network. IEEE Trans. Geosci. Remote Sens. 59(6), 5183–5196 (2020)

    Article  Google Scholar 

  3. Swaminathan, A., Wu, M., Liu, K.R.: Digital image forensics via intrinsic fingerprints. IEEE Trans. Inf. Forensics Secur. 3(1), 101–117 (2008)

    Article  Google Scholar 

  4. Ward, C.M., Harguess, J., Crabb, B., Parameswaran, S.: Image quality assessment for determining efficacy and limitations of super-resolution convolutional neural network (SRCNN). In: Applications of Digital Image Processing XL, vol. 10396, pp. 19–30 (2017)

  5. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  6. Niu, B., Wen, W., Ren, W., Zhang, X., Yang, L., Wang, S., Zhang, K., Cao, X., Shen, H.: Single image super-resolution via a holistic attention network. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pp. 191–207 (2020)

  7. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., : Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

  8. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  9. Sajjadi, M.S., Scholkopf, B., Hirsch, M.: Enhancenet: single image super-resolution through automated texture synthesis. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4491–4500 (2017)

  10. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3147–3155 (2017)

  11. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: a persistent memory network for image restoration. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4539–4547 (2017)

  12. Tian, C., Zhang, Y., Zuo, W., Lin, C.-W., Zhang, D., Yuan, Y.: A heterogeneous group CNN for image super-resolution. IEEE Trans. Neural Netw. Learn. Syst. (2022). https://doi.org/10.1109/TNNLS.2022.3210433

    Article  Google Scholar 

  13. Tian, C., Zhang, X., Lin, J.C.-W., Zuo, W., Zhang, Y., Lin, C.-W.: Generative adversarial networks for image super-resolution: a survey. arXiv preprint arXiv:2204.13620 (2022)

  14. Tian, C., Zheng, M., Zuo, W., Zhang, B., Zhang, Y., Zhang, D.: Multi-stage image denoising with the wavelet transform. Pattern Recogn. 134, 109050 (2023)

    Article  Google Scholar 

  15. Zhang, Q., Xiao, J., Tian, C., Chun-Wei Lin, J., Zhang, S.: A robust deformed convolutional neural network (CNN) for image denoising. CAAI Trans. Intell. Technol. 8, 331–342 (2022)

    Article  Google Scholar 

  16. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  17. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018)

  18. Dai, T., Cai, J., Zhang, Y., Xia, S.-T., Zhang, L.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11065–11074 (2019)

  19. Niu, B., Wen, W., Ren, W., Zhang, X., Yang, L., Wang, S., Zhang, K., Cao, X., Shen, H.: Single image super-resolution via a holistic attention network. In: Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pp. 191–207 (2020)

  20. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: SwinIR: image restoration using swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1833–1844 (2021)

  21. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H.: Restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022)

  22. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part IV 13, pp. 184–199 (2014)

  23. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part II 14, pp. 391–407 (2016)

  24. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  25. Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., Wu, W.: Feedback network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3867–3876 (2019)

  26. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)

  27. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1664–1673 (2018)

  28. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

  29. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

  30. Li, J., Fang, F., Mei, K., Zhang, G.: Multi-scale residual network for image super-resolution. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 517–532 (2018)

  31. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep Laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 624–632 (2017)

  32. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 723–731 (2018)

  33. Ahn, N., Kang, B., Sohn, K.-A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 252–268 (2018)

  34. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 2024–2032 (2019)

  35. Li, B., Wang, B., Liu, J., Qi, Z., Shi, Y.: s-LWSR: super lightweight super-resolution network. IEEE Trans. Image Process. 29, 8368–8380 (2020)

    Article  Google Scholar 

  36. Luo, X., Xie, Y., Zhang, Y., Qu, Y., Li, C., Fu, Y.: Latticenet: towards lightweight image super-resolution with lattice block. In: Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, pp. 272–289 (2020)

  37. Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G.: Residual feature aggregation network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2359–2368 (2020)

  38. Zhang, Y., Wei, D., Qin, C., Wang, H., Pfister, H., Fu, Y.: Context reasoning attention network for image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4278–4287 (2021)

  39. Mei, Y., Fan, Y., Zhou, Y.: Image super-resolution with non-local sparse attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3517–3526 (2021)

  40. Wang, L., Dong, X., Wang, Y., Ying, X., Lin, Z., An, W., Guo, Y.: Exploring sparsity in image super-resolution for efficient inference. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4917–4926 (2021)

  41. Wang, D., Tang, H., Pan, J., Tang, J.: Learning a tree-structured channel-wise refinement network for efficient image deraining. In: 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6 (2021). https://doi.org/10.1109/ICME51207.2021.9428187

  42. Tang, H., Yuan, C., Li, Z., Tang, J.: Learning attention-guided pyramidal features for few-shot fine-grained recognition. Pattern Recogn. 130, 108792 (2022)

    Article  Google Scholar 

  43. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  44. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  45. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  46. Liu, J.-J., Hou, Q., Cheng, M.-M., Wang, C., Feng, J.: Improving convolutional networks with self-calibrated convolutions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10096–10105 (2020)

  47. Mei, Y., Fan, Y., Zhou, Y.: Image super-resolution with non-local sparse attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3517–3526 (2021)

  48. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.-H., Zhang, L.: Ntire 2017 challenge on single image super-resolution: methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017)

  49. Bevilacqua, M., Roumy, A., Guillemot, C., Morel, M.-l.A.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the British Machine Vision Conference (2012)

  50. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces: 7th International Conference, Avignon, France, June 24–30, 2010, Revised Selected Papers 7, pp. 711–730 (2012)

  51. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 416–423 (2001)

  52. Huang, J.-B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

  53. Dodgson, N.A.: Quadratic interpolation for image resampling. IEEE Trans. Image Process. 6(9), 1322–1326 (1997)

    Article  Google Scholar 

  54. Wang, Z., Bovik, A.C., Sheikh, H.R.: Structural similarity based image quality assessment. In: Digital Video Image Quality and Perceptual Coding, pp. 225–242 (2017)

  55. Muqeet, A., Hwang, J., Yang, S., Kang, J., Kim, Y., Bae, S.-H.: Multi-attention based ultra lightweight image super-resolution. In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pp. 103–118 (2020)

  56. Zhao, H., Kong, X., He, J., Qiao, Y., Dong, C.: Efficient image super-resolution using pixel attention. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 56–72 (2020)

  57. Esmaeilzehi, A., Ahmad, M.O., Swamy, M.: SRNSSI: a deep light-weight network for single image super resolution using spatial and spectral information. IEEE Trans. Comput. Imaging 7, 409–421 (2021)

    Article  Google Scholar 

  58. Sun, L., Pan, J., Tang, J.: Shufflemixer: an efficient convnet for image super-resolution. arXiv preprint arXiv:2205.15175 (2022)

  59. Zhang, M., Wang, H., Zhang, Z., Chen, Z., Shen, J.: Lightweight multi-scale asymmetric attention network for image super-resolution. Micromachines 13(1), 54 (2022)

    Article  Google Scholar 

  60. Zhu, F., Zhao, Q.: Efficient single image super-resolution via hybrid residual feature learning with compact back-projection network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 2453–2460 (2019)

Download references

Funding

This work was supported by the National Key Research and Development Program of China (Grant No. 2022ZD0160400) and the National Natural Science Foundation of China (Grant Nos. 62071323 and 62176178). We gratefully acknowledge the support from Shanghai Artificial Intelligence Laboratory.

Author information

Authors and Affiliations

  1. School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China

    Aiping Yang, Zihao Wei, Jinbin Wang, Jiale Cao, Zhong Ji & Yanwei Pang

  2. Shanghai Artificial Intelligence Laboratory, Shanghai, China

    Aiping Yang

Authors
  1. Aiping Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zihao Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinbin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiale Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhong Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yanwei Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinbin Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, A., Wei, Z., Wang, J. et al. Multi-feature self-attention super-resolution network. Vis Comput 40, 3473–3486 (2024). https://doi.org/10.1007/s00371-023-03046-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03046-y

Keywords

Search

Navigation