Skip to main content
Log in

Single image super-resolution via a ternary attention network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Recently, the deep convolutional neural network (CNN) has been widely explored in single image super-resolution (SISR) and achieves excellent performance. However, most of the existing CNN-based SISR methods mainly focus on wider or deeper architecture design, ignoring the internal dependencies between the features of different layers in the network and the intrinsic statistical properties of the feature maps. It hinders the maximization of the network representation power. We propose a ternary attention mechanism network (TAN) for effective feature extraction and feature correlation learning to address this issue. Specifically, we introduced a layer attention mechanism(LAM) to fully use the features generated by each layer of the network. Furthermore, we present a spatial attention mechanism(SAM) that uses the internal statistical characteristics of the features to enhance themself. Finally, we design a new channel attention mechanism(CAM) to ensure the feature diversity in channel dimensions. Extensive experiments show that our TAN achieves better both quantitative metrics and visual quality compared with state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

Some or all data, models, or code generated or used during the study are available from the corresponding author by request.

References

  1. Zou WWW, Yuen PC (2011) Very low resolution face recognition problem. IEEE Trans Image Process 21(1):327–340

    Article  MathSciNet  MATH  Google Scholar 

  2. Zhang L, Zhang H, Shen H, Li P (2010) A super-resolution reconstruction algorithm for surveillance images. Signal Process 90(3):848–859

    Article  MATH  Google Scholar 

  3. Rasti P, Uiboupin T, Escalera S, Anbarjafari G (2016) Convolutional neural network super resolution for face recognition in surveillance monitoring. In: International conference on articulated motion and deformable objects. Springer, pp 175–184

  4. Shi W, Caballero J, Ledig C, Zhuang X, Bai W, Bhatia K, de Marvao AMSM, Dawes T, O’Regan D, Rueckert D (2013) Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 9–16

  5. Isaac JS, Kulkarni R (2015) Super resolution techniques for medical image processing. In: International conference on technologies for sustainable development (ICTSD). IEEE, pp 1–6

  6. Huang Y, Shao L, Frangi AF (2017) Simultaneous super-resolution and cross-modality synthesis of 3d medical images using weakly-supervised joint convolutional sparse coding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6070–6079

  7. Zhang L, Xiaolin W u (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238

    Article  Google Scholar 

  8. Dong W, Zhang L, Shi G, Xiaolin W (2011) Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Trans Image Process 20(7):1838–1857

    Article  MathSciNet  MATH  Google Scholar 

  9. Zhang K, Gao X, Tao D, Li X (2012) Single image super-resolution with non-local means and steering kernel regression. IEEE Trans Image Process 21(11):4544–4556

    Article  MathSciNet  MATH  Google Scholar 

  10. Wang S, Zhang L, Liang Y, Pan Q (2012) Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2216–2223

  11. Zhang Y, Li K, Li K, Wang L, Zhong B, Yun F (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp 286–301

  12. Dong C, Loy CCh, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European conference on computer vision. Springer, pp 184–199

  13. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  14. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 624–632

  15. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 136–144

  16. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3147–3155

  17. Tai Y, Yang J, Liu X, Chunyan Xu. (2017) Memnet: a persistent memory network for image restoration. In: Proceedings of the IEEE international conference on computer vision, pp 4539–4547

  18. Zhang Y, Tian Y, Kong Y u, Zhong B, Yun F (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481

  19. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2018) Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Trans Pattern Anal Mach Intell 41(11):2599–2613

    Article  Google Scholar 

  20. Sajjadi MSM, Scholkopf B, Hirsch M (2017) Enhancenet: single image super-resolution through automated texture synthesis. In: Proceedings of the ieee international conference on computer vision, pp 4491–4500

  21. Zhang K, Zuo W, Zhang L (2018) Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3262–3271

  22. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1664–1673

  23. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645

  24. Dai T, Cai J, Zhang Y, Xia Shu-Tao, Zhang L (2019) Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11065–11074

  25. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  26. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks arXiv:1406.2661

  27. Zhang Z, Wang Z, Lin Z, Qi H (2019) Image super-resolution by neural texture transfer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7982–7991

  28. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

  29. Jie H u, Li S, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7132–7141

  30. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  31. Woo S, Park J, Lee Joon-Young, In So Kweon. (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  32. Yanting H u, Li J, Huang Y, Gao X (2019) Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Trans Circuits Syst Video Technol 30(11):3911–3927

    Google Scholar 

  33. Zontak M, Irani M (2011) Internal statistics of a single natural image. In: CVPR 2011. IEEE, pp 977–984

  34. Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 349–356

  35. Buades A, Coll B, Morel J-M (2005) A non-local algorithm for image denoising. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05). IEEE, vol 2, pp 60–65

  36. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  37. Sze V, Chen Y-H, Yang T-J, Emer JS (2017) Efficient processing of deep neural networks: a tutorial and survey. Proc IEEE 105(12):2295–2329

    Article  Google Scholar 

  38. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  39. Huang S, Sun J, Yang Y, Fang Y, Lin P, Que Y (2018) Robust single-image super-resolution based on adaptive edge-preserving smoothing regularization. IEEE Trans Image Process 27(6):2650–2663

    Article  MathSciNet  MATH  Google Scholar 

  40. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  41. Kingma DP, Adam JB (2014) A method for stochastic optimization. arXiv:1412.6980

  42. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European conference on computer vision. Springer, pp 391–407

  43. Li Z, Yang J, Liu Z, Yang X, Jeon G, Wei W (2019) Feedback network for image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3867–3876

  44. Niu B, Wen W, Ren W, Ren X, Yang L, Wang S, Zhang K, Cao X, Shen H (2020) Single image super-resolution via a holistic attention network. In: European conference on computer vision. Springer, pp 191–207

  45. Peleg T, Elad M (2014) A statistical prediction model based on sparse representations for single image super-resolution. IEEE Trans Image Process 23(6):2569–2582

    Article  MathSciNet  MATH  Google Scholar 

  46. Zhang K, Zuo W, Gu S, Zhang L (2017) Learning deep CNN denoiser prior for image restoration. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2808–2817

Download references

Acknowledgements

This work is supported by National Key Research and Development Program of China (grant number: 2021YFF0306401).

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Lianping Yang or Xin Wang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, L., Tang, J., Niu, B. et al. Single image super-resolution via a ternary attention network. Appl Intell 53, 13067–13081 (2023). https://doi.org/10.1007/s10489-022-04129-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04129-4

Keywords

Navigation