Skip to main content
SpringerLink
Log in

Deformable and residual convolutional network for image super-resolution

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Recent research on image super-resolution (SR) has greatly progressed with the development of convolutional neural networks (CNNs). However, the fixed geometric structures of standard convolution filters largely limit the learning capacity of CNNs for image SR. To effectively address this problem, we propose a deformable and residual convolutional network (DefRCN) for image SR. Specifically, a deformable residual convolution block (DRCB) is developed to augment spatial sampling locations and enhance the transformation modelling capability of CNNs. In addition, we optimize the residual convolution block to reduce the model redundancy and alleviate the vanishing-gradient in backpropagation. In addition, the proposed upsample block allows the network to directly process low-resolution images, which reduces the computational resource cost. Extensive experiments on benchmark datasets verify that the proposed method achieves a high quantitative and qualitative performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Jia K, Wang X, Tang X (2013) Image transformation based on learning dictionaries across image spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(2):367–380

    Article  Google Scholar 

  2. Yang J, Wright J, Huang TS et al (2010) Image super-resolution via sparse representation. IEEE transactions on image processing 19(11):2861–2873

    Article  MathSciNet  Google Scholar 

  3. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: International conference on curves and surfaces. Springer, Berlin, pp 711–730

  4. Timofte R, De Smet V, Van Gool L (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: Asian conference on computer vision. Springer, Cham, pp 111–126

  5. Yang CY, Yang MH (2013) Fast direct super-resolution by simple functions. In: Proceedings of the IEEE international conference on computer vision, pp 561–568

  6. Schulter S, Leistner C, Bischof H (2015) Fast and accurate image upscaling with super-resolution forests. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3791–3799

  7. Dong C, Loy CC, He K et al (2016) Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(2):295–307

    Article  Google Scholar 

  8. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  9. Kim J, Lee JK, Lee KM (2016) Deeply recursive convolutional network for image super-resolution. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645

  10. Lai WS, Huang JB, Ahuja N et al (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 624–632

  11. Tai Y, Yang J, Liu X et al (2017) Memnet: a persistent memory network for image restoration. In: Proceedings of the IEEE international conference on computer vision, pp 4539–4547

  12. Zhang Y, Tian Y, Kong Y et al (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481

  13. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1664–1673

  14. Zhang Y, Li K, Li K et al (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp 286–301

  15. Gu J, Lu H, Zuo W et al (2019) Blind super-resolution with iterative kernel correction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1604–1613

  16. Soh JW, Park GY, Jo J et al (2019) Natural and realistic single image super-resolution with explicit natural manifold discrimination. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8122–8131

  17. Lim B, Son S, Kim H et al (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 136–144

  18. Liu H, Cao F, Wen C et al (2020) Lightweight multi-scale residual networks with attention for image super-resolution. Knowledge-Based Systems 203:106103

    Article  Google Scholar 

  19. Liu H, Cao F (2020) Improved dual-scale residual network for image super-resolution. Neural Netw 132:84–95

    Article  Google Scholar 

  20. Cao F, Yao K, Liang J (2020) Deconvolutional neural network for image super-resolution. Neural Netw 132:394–404

    Article  Google Scholar 

  21. Cao F, Liu H (2019) Single image super-resolution via multi-scale residual channel attention network. Neurocomputing 358:424–436

    Article  Google Scholar 

  22. Cao F, Chen B (2019) New architecture of deep recursive convolution networks for superresolution. Knowledge-Based Systems 178:98–110

    Article  Google Scholar 

  23. Ledig C, Theis L, Huszr F et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  24. Wang X, Yu K, Wu S et al (2018) Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV), pp 63–79

  25. Shocher A, Cohen N, Irani M (2018) Zero-shot super-resolution using deep internal learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3118–3126

  26. Bulat A, Yang J, Tzimiropoulos G (2018) To learn image super-resolution, use a gan to learn how to do image degradation first. In: Proceedings of the European conference on computer vision (ECCV), pp 185–200

  27. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3147–3155

  28. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  29. Tong T, Li G, Liu X et al (2017) Image super-resolution using dense skip connections. In: Proceedings of the IEEE international conference on computer vision, pp 4799–4807

  30. Jaderberg M, Simonyan K, Zisserman A (2015) Spatial transformer networks. In: Advances in neural information processing systems, pp 2017–2025

  31. Jeon Y, Kim J (2017) Active convolution: learning the shape of convolution for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 4201–4209

  32. Dai J, Qi H, Xiong Y et al (2017) Deformable convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 764–773

  33. Xu Z, Xu X, Wang L, et al. (2017) Deformable convnet with aspect ratio constrained nms for object detection in remote sensing imagery. Remote Sensing 9(12):1312

    Article  Google Scholar 

  34. Ren Y, Zhu C, Xiao S (2018) Deformable faster r-cnn with aggregating multi-layer features for partially occluded object detection in optical remote sensing images. Remote Sens 10(9):1470

    Article  Google Scholar 

  35. Ouyang W, Wang X, Zeng X et al (2015) Deepid-net: deformable deep convolutional neural networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2403–2412

  36. Zhang M, Li X, Xu M et al (2018) RBC semantic segmentation for sickle cell disease based on deformable U-Net. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 695–702

  37. Sun M, Zhang G, Dang H et al (2019) Accurate gastric cancer segmentation in digital pathology images using deformable convolution and multi-scale embedding networks. IEEE Access 7:75530–75541

    Article  Google Scholar 

  38. Jin Q, Meng Z, Pham TD et al (2019) DUNet: a deformable network for retinal vessel segmentation. Knowledge-Based Systems 178:149–162

    Article  Google Scholar 

  39. Zhu J, Fang L, Ghamisi P (2018) Deformable convolutional neural networks for hyperspectral image classification. IEEE Geosci Remote Sens Lett 15(8):1254–1258

    Article  Google Scholar 

  40. Timofte R, Agustsson E, Van Gool L et al (2017) Ntire 2017 challenge on single image super-resolution: methods and results. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 114–125

  41. Bevilacqua M, Roumy A, Guillemot C et al (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: BMVC

  42. Martin D, Fowlkes C, Tal D et al (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings eighth IEEE international conference on computer vision, pp 416–423

  43. Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5197–5206

  44. Wang Z, Bovik AC, Sheikh HR et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4):600–612

    Article  Google Scholar 

  45. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  46. Paszke A, Gross S, Chintala S et al (2017) Automatic differentiation in pytorch

Download references

Acknowledgements

This work was supported by the Scientific Research Project of Tianjin Municipal Education Commission [grant number 2019KJ105].The authors also acknowledge the anonymous reviewers for their helpful comments on the manuscript.

Author information

Authors and Affiliations

  1. School of Computer and Information Engineering, Tianjin Chengjian University, Tianjin, China

    Yan Zhang, Yemei Sun & Shudong Liu

Authors
  1. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yemei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shudong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yemei Sun.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Sun, Y. & Liu, S. Deformable and residual convolutional network for image super-resolution. Appl Intell 52, 295–304 (2022). https://doi.org/10.1007/s10489-021-02246-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02246-0

Keywords

Search

Navigation