Skip to main content
SpringerLink
Log in

Lightweight multi-scale aggregated residual attention networks for image super-resolution

  • 1193: Intelligent Processing of Multimedia Signals
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recently, single image super-resolution (SISR) based on convolutional neural networks (CNNs) has represented great progress. However, due to the huge number of parameters, these models cannot work well in many real-world applications, most of which fail to exploit the multi-scale features and the hierarchical features for lightweight and accurate image SR. In this paper, a lightweight multi-scale aggregated residual attention network (MARAN) is proposed by exploring multi-scale contextual information and multi-level features. The network consists of shallow feature extraction, recursively stacked multiple multi-scale aggregated residual attention groups (MARAGs), multi-level feature fusion block (MLFFB), and reconstruction part. Specifically, the MARAGs produce the hierarchical multi-scale deep features, the MLFFB effectively fuses the hierarchical features with multi-scale aggregated residual attention. Each MARAG is composed of cascaded multi-scale aggregated residual attention blocks (MARABs) and each MARAB contains a multi-scale aggregated unit and a dual-attention unit. The multi-scale aggregated unit expands group convolution with cross-path connection. The dual-attention unit can adaptively modulate region-based information and channel-wise features. Qualitative and quantitative experiments on four benchmark datasets demonstrate that the proposed MARAN achieves better performance against state-of-the-art methods with fewer parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Ahn N, Kang B, Sohn K (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In: The European conference on computer vision (ECCV), Munich, Germany, pp 252– 268

  2. Aitken A, Ledig C, Theis L, Caballero J, Wang Z, Shi W (2017) Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize, arXiv:1707.02937

  3. Anagun Y, Isik S, Seke E (2019) SRLIbrary: comparing different loss functions for super-resolution over various convolutional architectures. J Vis Commun Image Represent 61:178–187

    Article  Google Scholar 

  4. Arbel’aez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916

    Article  Google Scholar 

  5. Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel ML (2012) Low-complexity single image super-resolution based on nonnegative neighbor embedding. In: British machine vision conference (BMVC), BMVA press surrey, UK, pp 1–10

  6. Brownlee J (2016) Overfitting and underfitting with machine learning algorithms. Machine Learning Mastery 21

  7. Cao F, Chen B (2019) New architecture of deep recursive convolution networks for super-resolution. Knowl-Based Syst 178:98–110

    Article  Google Scholar 

  8. Chen L, Zhang H, Xiao J, Nie L, Shao J, Liu W, Chua T-S (2017) Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In: IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, USA, pp 5659–5667

  9. Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: The European conference on computer vision (ECCV), pp 184–199

  10. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: The European conference on computer vision (ECCV), The Netherlands, pp 391–407

  11. Gao S, Cheng MM, Zhao K, Zhang XY, Yang MH, Torr PH (2019) Res2net: A new multi-scale backbone architecture. IEEE Trans Pattern Anal Mach Intell 1–1

  12. He K, Zhang X, Ren S. h., Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, USA, pp 770–778

  13. Hou Q, Cheng M-M, Hu X, Borji A, Tu Z, Torr P (2017) Deeply supervised salient object detection with short connections. In: IEEE conference on computer vision and pattern recognition (CVPR). Honolulu, USA, pp 5300–5309

  14. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv:1704.04861

  15. Hu Y, Li J, Huang Y, Gao X (2019) Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology

  16. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, USA, pp 7132–7141

  17. Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: The IEEE conference on computer vision and pattern recognition (CVPR), Boston, USA, pp 5197–5206

  18. Huang J-J, Siu W-C, Liu T-R (2015) Fast image interpolation via random forests. IEEE Trans Image Process 24(10):3232–3245

    Article  MathSciNet  Google Scholar 

  19. Hui Z, Wang X, Gao X (2018) Fast and accurate single image super-resolution via information distillation network. In: IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, USA, pp 723–731

  20. Jiang K, Wang Z, Yi P, Jiang J (2018) A progressively enhanced network for video satellite imagery super-resolution. IEEE Signal Process Lett 25 (11):1630–1634

    Article  Google Scholar 

  21. Jing P, Guan W, Bai X, Guo H, Su Y (2020) Single image super-resolution via low-rank tensor representation and hierarchical dictionary learning. Multimed Tools Appl 1–19

  22. Kim J, Kwon Lee J, Lee K. M. u. (2016) Deeply-recursive convolutional network for image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, USA, pp 1637–1645

  23. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, USA, pp 1646–1654

  24. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: International conference for learning representations (ICLR), San Diego, USA, pp 1–13

  25. Lai WS, Huang JB, Ahuja N, Yang MH (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, USA, pp 624–632

  26. Ledig C, Theis L, Husz’ar F, Caballero J (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, USA, pp 4681–4690

  27. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW). Honolulu, USA, pp 136–144

  28. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, USA, pp 136–144

  29. Lin F, Fookes C, Chandran V, Sridharan S (2007) Super-resolved faces for improved face recognition from surveillance video. In: The international conference on biometrics. Springer, Berlin, pp 1–10

  30. Lu P, Barazzetti L, Chandran V, Gavaghan K, Weber S, Gerber N, Reyes M (2018) Highly accurate facial nerve segmentation refinement from CBCT/CT imaging using a super-resolution classification approach. IEEE Trans Biomed Eng 65(1):178–188

    Article  Google Scholar 

  31. Pang S, Chen Z, Yin F (2021) Convolutional neural network based sub-pixel line-edged angle detection with applications in measurement. IEEE Sensors J 21(7):9314–9322

    Article  Google Scholar 

  32. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in PyTorch. Advances in Neural Information Processing Systems Autodiff Workshop 1–4

  33. Qiao J, Song H, Zhang K, Zhang X, Liu Q (2019) Image super-resolution using conditional generative adversarial network. IET Image Process 13 (14):2673–2679

    Article  Google Scholar 

  34. Romano Y, Protter M, Elad M (2014) Single image interpolation via adaptive nonlocal sparsity-based modeling. IEEE Trans Image Process 23 (7):3085–3098

    Article  MathSciNet  Google Scholar 

  35. Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, USA, pp 1874– 1883

  36. Shi W, Caballero J, Ledig C, Zhuang X, Bai W, Bhatia K, de Marvao AMSM, Dawes T, ORegan D, Rueckert D (2013) Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: The international conference on medical image computing and computer-assisted intervention. Berlin, Heidelberg, pp 9–16

  37. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, USA, pp 3147–3155

  38. Tai Y, Yang J, Liu X, Xu C (2017) Memnet: A persistent memory network for image restoration. In: IEEE international conference on computer vision (ICCV), Venice, Italy, pp 4539–4547

  39. Tang Z, Li S, Luo L, Fu M, Peng H, Zhou Q (2019) Image super-resolution via simplified dense network with non-degenerate layers. IEEE Access 24775–24787

  40. Timofte R, Agustsson E, Van Gool L, Yang M. -H., Zhang L (2017) Ntire 2017 challenge on single image super-resolution: methods and results. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), Honolulu, USA, pp 114–125

  41. Wang Z, Chen J, Hoi SC (2020) Deep learning for image super-resolution: a survey. IEEE Trans Pattern Anal Mach Intell 1–1

  42. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, USA, pp 6450–6458

  43. Xue S, Qiu W, Liu F, Jin X (2020) Wavelet-based residual attention network for image super-resolution. Neurocomputing 382:116–126

    Article  Google Scholar 

  44. Yang W, Wang W, Zhang X, Sun S, Liao Q (2019) Lightweight feature fusion network for single image super-resolution. IEEE Signal Process Lett 26(4):538–542

    Article  Google Scholar 

  45. Yang A, Yang B, Ji Z, Pang Y, Shao L (2020) Lightweight group convolutional network for single image super-resolution. Inf Sci 516:220–233

    Article  Google Scholar 

  46. Yang X, Zhang Y, Guo Y, Zhou D (2021) An image super-resolution deep learning network based on multi-level feature extraction module. Multimed Tools Appl 80(5):7063–7075

    Article  Google Scholar 

  47. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse representations. In: International conference on curves and surfaces, Heidelberg, Berlin, pp 711–730

  48. Zhang K, Gao X, Tao D, Li X (2012) Single image super-resolution with non-local means and steering kernel regression. Trans Image Process 21 (11):4544–4556

    Article  MathSciNet  Google Scholar 

  49. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: European conference on computer vision (ECCV), Munich, Germany, pp 286–301

  50. Zhang Y, Li K, Li K, Zhong B, Fu Y (2019) Residual non-local attention networks for image restoration, arXiv:1903.10082

  51. Zhang W, Liu Y, Dong C, Qiao Y (2019) Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In: IEEE international conference on computer vision (ICCV), Seoul, Korea, pp 3096–3105

  52. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, USA, pp 2472–2481

  53. Zhang S, Yuan Q, Li J, Sun J, Zhang X (2020) Scene-adaptive remote sensing image super-resolution using a multiscale attention network. IEEE Transactions on Geoscience and Remote Sensing

  54. Zhao L, Bai H, Liang J, Zeng B, Wang A, Zhao Y (2019) Simultaneous color-depth super-resolution with conditional generative adversarial networks. Pattern Recogn 88:356–369

    Article  Google Scholar 

  55. Zhou L, Wang Z, Luo Y, Xiong Z (2019) Separability and compactness network for image recognition and super-resolution. IEEE Trans Neural Netw Learn Syst 30(11):3275–3286

    Article  Google Scholar 

  56. Zhu L, Zhan S, Zhang HJN (2019) Stacked U-shape networks with channel-wise attention for image super-resolution. Neurocomputing 345:58–66

    Article  Google Scholar 

  57. Zou WWW, Yuen PC (2012) Very low resolution face recognition problem. IEEE Trans Image Process 21(1):327–340

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work was supported by Natural Science Foundations of China (Nos.61771091, 61871066), National High Technology Research and Development Program (863 Program) of China (No. 2015AA016306), Natural Science Foundation of Liaoning Province of China (No. 20170540159), and Fundamental Research Fund for the Central Universities of China (No.DUT17LAB04).

Author information

Authors and Affiliations

  1. School of Information and Communication Engineering, Dalian University of Technology, Dalian, 116023, China

    Shurong Pang, Zhe Chen & Fuliang Yin

Authors
  1. Shurong Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhe Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fuliang Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhe Chen.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pang, S., Chen, Z. & Yin, F. Lightweight multi-scale aggregated residual attention networks for image super-resolution. Multimed Tools Appl 81, 4797–4819 (2022). https://doi.org/10.1007/s11042-021-11138-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11138-x

Keywords

Search

Navigation