Skip to main content
Springer Nature Link
Log in

Image interpolation using convolutional neural networks with deep recursive residual learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recent developments of image super-resolution often utilize the deep convolutional neural network (CNN) and residual learning to relate the observed low-resolution pixels and unknown high-resolution pixels. However, image interpolation assumes that the observed image was directly down-sampled without low-pass filtering, such that the aliased down-sampled low-resolution image exhibits jags and chaos that cannot be easily modeled by conventional residual learning in super-resolution. In this paper, we propose a new framework to exploit the residual dense network using hierarchical levels of recursive residual learning and densely connected convolutional layers for image interpolation. The proposed deep recursive network iteratively reconstructs hierarchical levels of image details for aliased and discontinuous residual of interpolated pixels. Experimental results on popular Set16, Set18, and Urban12 image datasets show that the proposed method outperforms state-of-the-art image interpolation methods using local and nonlocal autoregressive models, random forests and deep CNN, in terms of PSNR (0.27–1.57 dB gain), SSIM and subjective evaluations. More importantly, model parameters of the proposed method are significantly less than that of existing deep CNN for image interpolation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Bai X, Zhang T, Wang C, ABD EL-LATIF AA, Niu X (2013) A fully automatic player detection method based on one-class SVM. IEICE Trans Inf Syst E96.D(2):387–391

    Article  Google Scholar 

  2. Colonnese S, Rinauro S, Scarano G (2013) Bayesian image interpolation using Markov random fields driven by visually relevant image features. Signal Process Image Commun 28(8):967–983

    Article  Google Scholar 

  3. Dong W, Zhang L, Lukac R, Shi G (2013) Sparse representation based image interpolation with nonlocal autoregressive modeling. IEEE Trans Image Process 22(4):1382–1394

    Article  MathSciNet  MATH  Google Scholar 

  4. Dong C, Deng Y, Loy CC, Tang X (2015) Compression artifacts reduction by a deep convolutional network. 2015 International Conference on Computer Vision (ICCV), pages 576–584

  5. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. Proceedings of European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, pp 1–16

  6. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  7. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, pp 770–778

  8. Huang JJ, Siu W-C, Liu TR (2015) Fast image interpolation via random forests. IEEE Trans Image Process 24(10):3232–3245

    Article  MathSciNet  MATH  Google Scholar 

  9. Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5197–5206

  10. Huang G, Liu Z, Maaten LVD (2017) Densely connected convolutional networks. CVPR

  11. Hung K-W, Siu W-C (2012) Robust soft-decision interpolation using weighted least squares. IEEE Trans Image Process 21(3):1061–1069

    Article  MathSciNet  MATH  Google Scholar 

  12. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T Caffe: convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM international conference on Multimedia (MM '14). ACM, New York, NY, USA, 675–678

  13. Kim J, Lee JK, Lee KM (2015) Accurate image super-resolution using very deep convolutional networks. CVPR 1646–1654

  14. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. CVPR

  15. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25 (NIPS 2012), Lake Tahoe, Nevada, pp 1–9

  16. Lakshman H, Lim W-Q, Schwarz H, Marpe D, Kutyniok G, Wiegand T (2015) Image interpolation using shearlet based iterative refinement. Signal Process Image Commun 36:83–94

    Article  Google Scholar 

  17. Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10(10):1521–1527

    Article  Google Scholar 

  18. Li M, Liu J, Ren J, Guo Z (2015) Adaptive general scale interpolation based on weighted autoregressive models. IEEE Trans Circuit Syst Video Technol 25(2):200–211

    Article  Google Scholar 

  19. Liu X, Zhao D, Xiong R, Ma S, Gao W, Sun H (2011) Image Interpolation via Regularized Local Linear Regression. IEEE Trans Image Process 20(12):3455–3469

    Article  MathSciNet  MATH  Google Scholar 

  20. Liu D, Wang Z, Wen B, Yang J, Han W, Huang TS (2016) Robust single image super-resolution via deep networks with sparse prior. IEEE Trans Image Process 25(7):3194–3207

    Article  MathSciNet  MATH  Google Scholar 

  21. Mallat S, Yu G (2010) Super-resolution with sparse mixing estimators. IEEE Trans Image Process 19(11):2889–2900

    Article  MathSciNet  MATH  Google Scholar 

  22. Nemirovsky S, Porat M (2009) On texture and image interpolation using Markov models. Signal Process Image Commun 24(3):139–157

    Article  Google Scholar 

  23. Romano Y, Protter M, Elad M (2014) Single image interpolation via adaptive nonlocal sparsity-based modeling. IEEE Trans Image Process 23(7):3085–3098

    Article  MathSciNet  MATH  Google Scholar 

  24. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. CVPR 2790–2798

  25. Tai Y, Yang J, Liu X (2017) MemNet: a persistent memory network for image restoration. ICCV

  26. Tong T, Li G, Liu X (2017) Image super-resolution using dense skip connections. ICCV 4809–4817

  27. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  28. Yan X, Wang S, Li L, Abd El-Latif AA, Wei Z, Niu X (2013) A new assessment measure of shadow image quality based on error diffusion techniques. J Inf Hiding Multimedia Signal Process 4(2)

  29. Yang W, Liu J, Xia S (2017) Variation learning guided convolutional network for image interpolation. ICIP 1652–1656

  30. Yang W, Liu J, Li M, Guo Z Isophote-constrained autoregressive model with adaptive window extension for image interpolation. IEEE Trans Circuit Syst Video Technol. Accepted for publication

  31. Zhang X, Wu X (2008) Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation. IEEE Trans Image Process 17(6):887–896

    Article  MathSciNet  Google Scholar 

  32. Zhang Y, Tian Y, Kong Y (2018) Residual dense network for image super-resolution. CVPR

  33. Zhu S, He Z, Liu S, Zeng B MMSE-directed linear image interpolation based on non-local geometric similarity. IEEE Signal Process Lett. Accepted for publication

Download references

Acknowledgements

The authors wish to acknowledge the financial support from: (i) Natural Science Foundation China (NSFC) under the Grant No. 61602312, 61620106008; and (ii) Shenzhen Commission for Scientific Research & Innovations under the Grant No. JCYJ20160226191842793.

Author information

Authors and Affiliations

  1. College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China

    Kwok-Wai Hung, Kun Wang & Jianmin Jiang

Authors
  1. Kwok-Wai Hung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianmin Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kwok-Wai Hung.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hung, KW., Wang, K. & Jiang, J. Image interpolation using convolutional neural networks with deep recursive residual learning. Multimed Tools Appl 78, 22813–22831 (2019). https://doi.org/10.1007/s11042-019-7633-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7633-1

Keywords