Skip to main content
SpringerLink
Log in

Lightweight adaptive enhanced attention network for image super-resolution

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, convolutional neural networks have obtained significant success in super-resolution (SR) with remarkable performance. However, as the depth of the network increases, the model parameters and computational overhead become complex. In this paper, we propose a lightweight adaptive enhanced attention network named LAEAN by cascading a series of shared adaptive enhanced attention modules (AEAM). Specifically, the AEAM is implemented by adding a novel up-down sampling module (UDSM) in the position and channel attention modules (PCAM) to adaptively capture wider and richer contextual information. The UDSM is designed to aggregate representative features while enabling the PCAM to learn more discriminative features. Furthermore, we propose a weighted fusion module (WFM) to flexibly combine informative features from all AEAMs for further boosting reconstruction performance. The experimental results demonstrate that our LAEAN is efficient and lightweight with only \(\sim\)1.5M training parameters, and outperforms most state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Ahn N, Kang B, Sohn KA (2018) Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network

    Book  Google Scholar 

  2. Ali A, Zhu Y, Chen Q, Yu J, Cai H (2019) Leveraging Spatio-Temporal Patterns for Predicting Citywide Traffic Crowd Flows Using Deep Hybrid Neural Networks. In: International Conference on Parallel and Distributed Systems (ICPADS). pp 125–132

  3. Ali A, Zhu Y, Zakarya M (2021) A data aggregation based approach to exploit dynamic spatio-temporal correlations for citywide crowd flows prediction in fog computing. Multimed Tools Appl

  4. Awan N, Ali A, Khan F, Zakarya M, Alturki R, Kundi M, Alshehri MD, Haleem M (2021) Modeling dynamic spatio-temporal correlations for urban traffic flows prediction. IEEE Access 9:26502–26511

    Article  Google Scholar 

  5. Bevilacqua M, Roumy A, Guillemot C, Morel MA (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 23rd British Machine Vision Conference (BMVC). pp 1–10

  6. Cheng X, Li X, Yang J (2019) Triple-attention mixed-link network for single-image super-resolution. Appl Sci 9(15):2992

    Article  Google Scholar 

  7. Chu X, Zhang B, Ma H, Xu R, Li J, Li Q (2019) Fast, accurate and lightweight super-resolution with neural architecture search. In: International Conference on Pattern Recognition (ICPR). pp 59–64

  8. Dong C, Loy CC, He K, Tang X (2014) Learning a Deep Convolutional Network for Image Super-Resolution. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) European Conference on Computer Vision, vol. 8692. Cham. Springer International Publishing. Series Title: Lecture Notes in Computer Science, pp 184–199

  9. Dong C, Loy CC, Tang X (2016) Accelerating the Super-Resolution Convolutional Neural Network. In: Leibe B, Matas J, Sebe N, Welling M (eds) European Conference on Computer Vision. pp 391–407

  10. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2020) Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp 3146–3154

  11. Guo J, Ma X, Sansom A, McGuire M, Kalaani A, Chen Q, Tang S, Yang Q, Fu S (2020) Spanet: Spatial Pyramid Attention Network for enhanced image recognition. In: IEEE International Conference on Multimedia and Expo (ICME). pp 1–6

  12. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 1664–1673

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 770–778

  14. Huang J (2020) Image super-resolution reconstruction based on generative adversarial network model with double discriminators. Multimed Tools Appl 79(39–40):29639–29662

    Article  Google Scholar 

  15. Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 5179–5206

  16. Hu J, Shen L, Albanie S, Sun G, Wu E (2017) Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 7132–7141

  17. Kim J-H, Choi J-H, Cheon M, Lee J-S (2018) RAM: Residual attention module for single image super-resolution. arXiv:1811.12043

  18. Kim J-H, Choi J-H, Cheon M, Lee J-S (2020) MAMNet: Multi-path adaptive modulation network for image super-resolution. Neurocomputing 402:38–49

  19. Kim J, Lee JK, Lee KM (2016) Accurate Image Super-Resolution Using Very Deep Convolutional Networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 1646–1654

  20. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 1637–1645

  21. Lai W, Huang J, Ahuja N, Yang M-H (2017) Deep Laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 5835–5843

  22. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J,  Zehan, Wang (2016) Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 4681–4690

  23. Li Z (2019) Image super-resolution using attention based DenseNet with residual deconvolution. arXiv:1907.05282

  24. Li J, Faming Fang, Kangfu Mei, and Guixu Zhang (2018) Multi-scale residual network for image super-resolution. In: Proceedings of the European Conference on Computer Vision (ECCV). pp 517–532

  25. Li Z, Li Q, Wei W, Yang J, Li Z, Yang X (2020) Deep recursive up-down sampling networks for single image super-resolution. Neurocomputing 398:377–388

    Article  Google Scholar 

  26. Liao X, Li K, Zhu X, Ray Liu KJ (2020) Robust detection of image operator chain with two-stream convolutional neural network. IEEE J Sel Top Sign Proces 14(5):955–968

    Article  Google Scholar 

  27. Liao X, Yin J, Chen M, Qin Z (2020) Adaptive payload distribution in multiple images steganography based on image texture features. IEEE Trans Dependable Secure Comput 1–1

  28. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced Deep Residual Networks for Single Image Super-Resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp 136–144

  29. Lin F,  Fookes C, Chandran V, Sridharan S (2007) Super-resolved faces for improved face recognition from surveillance video advances in biometrics. In: International Conference on Advances in Biometrics. pp 1–10

  30. Lin Z, Feng M, Dos Santos CN, Yu M, Xiang B, Zhou B, Bengio Y (2017) A structured self-attentive sentence embedding. arXiv:1703.03130

  31. Lin G, Shen C, Van Dan Hengel A, Reid I (2016) Efficient piecewise training of deep structured models for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 3194–3203

  32. Liu J, Zhang W, Tang Y, Tang J, Wu G (2020) Residual feature aggregation network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp 2359–2368

  33. Long C, Zhang H, Xiao J, Nie L, Chua TS (2017) SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 5659–5667

  34. Lyn J, Yan S (2020) Non-local second-order attention network for single image super resolution. In: International Cross-Domain Conference for Machine Learning and Knowledge Extraction. pp 267–279

  35. Ma H, Chu X, Zhang B, Wan S, Zhang B (2019) A matrix-in-matrix neural network for image super resolution. arXiv:1903.07949

  36. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: IEEE International Conference on Computer Vision (CVPR). pp 416–423

  37. Mao X, Shen C, Yang Y (2016) Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv:1606.08921

  38. Park J, Woo S, Lee J-Y, Kweon IS (2018) BAM: Bottleneck attention module. arXiv:1807.06514 (2018)

  39. Qin J, Xie Z, Shi Y, Wen W (2019) Difficulty-aware image super resolution via deep adaptive dual-network. In: IEEE International Conference on Multimedia and Expo (ICME). pp 586–591

  40. Sajjadi MSM, Scholkopf B, Hirsch M (2017) EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp 4491–4500 

  41. Shen T, Zhou T, Long G, Jiang J, Pan S, Zhang C (2017) DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp 5446–5455

  42. Shi W, Caballero J, Huszar F, Totz J, Wang Z (2016) Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 1874–1883

  43. Shi W, Caballero J,  Ledig C,  Zhuang X,  Rueckert D (2013) Cardiac Image Super-Resolution with Global Correspondence Using Multi-Atlas PatchMatch. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp 9–16

  44. Sinha A, Dolz J (2020) Multi-scale self-guided attention for medical image segmentation. IEEE J of Biomed and Health Inform PP(99):121–130

  45. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 3147–3155

  46. Tai Y, Yang J, Liu X, Xu C (2017) MemNet: A persistent memory network for image restoration. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp 4539–4547

  47. Tian C, Xu Y, Zuo W, Lin C-W, Zhang D (2021) Asymmetric CNN for image super-resolution. arXiv:2103.13634

  48. Timofte R, Agustsson E, Van Gool L, Yang M-H, Zhang L (2017) NTIRE 2017 Challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp 114–125

  49. Timofte R, De Smet V, Van Gool L (2014) A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution. In: Asian Conference on Computer Vision. pp 111–126

  50. Timofte R, De V, Van Gool L (2014) Anchored neighborhood regression for fast example-based super-resolution. In: Anchored Neighborhood Regression for Fast Example-Based Super-Resolution. pp 1920–1927

  51. Wang L, Wang Y, Liang Z, Lin Z, Yang J, An W, Guo Y (2019) Learning Parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp 12250–12259

  52. Wang C, Li Z, Shi J (2019) Lightweight image super-resolution with adaptive weighted learning network. arXiv:1904.02358 (2019)

  53. Woo S, Park J, Lee J-Y, Kweon IS (2018) CBAM: Convolutional block attention module. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds) Proceedings of the European Conference on Computer Vision (ECCV). pp 3–19

  54. Zeyde R, Elad M, Protter M (2012) On single image scale-up using sparse-representations. In: Boissonnat J-D, Chenin P, Cohen A, Gout C, Lyche T, Mazure M-L,  Schumaker L (eds) International Conference on Curves and Surfaces. pp 711–730

  55. Zhang H, Xiao J, Jin Z (2021) Multi-scale image super-resolution via a single extendable deep network. IEEE J Sel Top Sign Proces 15(2):253–263

    Article  MathSciNet  Google Scholar 

  56. Zhang L, Xiaolin W (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238

    Article  Google Scholar 

  57. Zhang K, Gao X, Tao D, Li X (2012) Single Image Super-Resolution With Non-Local Means and Steering Kernel Regression. IEEE Trans Image Process 21(11):4544–4556

    Article  MathSciNet  MATH  Google Scholar 

  58. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV). pp 286–301

  59. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 2472–2481

Download references

Funding

This work was supported by the National Natural Science Foundation of China (No. 51979085, 61903124), Guangdong Water Resources Science and Technology Innovation Project (No. 2020-04).

Author information

Authors and Affiliations

  1. College of Computer and Information, Hohai University, Nanjing, 211100, China

    Li Wang, Lizhong Xu, Jie Shen & Fengcheng Huang

  2. School of Energy and Power Engineering, Nangjing Institute of Technology, Nangjing, 211167, China

    Jianqiang Shi

Authors
  1. Li Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lizhong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianqiang Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fengcheng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lizhong Xu.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, L., Xu, L., Shi, J. et al. Lightweight adaptive enhanced attention network for image super-resolution. Multimed Tools Appl 81, 6513–6537 (2022). https://doi.org/10.1007/s11042-021-11444-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11444-4

Keywords

Search

Navigation