Skip to main content
Springer Nature Link
Log in

Efficient image super-resolution integration

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The super-resolution (SR) problem is challenging due to the diversity of image types with little shared properties as well as the speed required by online applications, e.g., target identification. In this paper, we explore the merits and demerits of recent deep learning-based and conventional patch-based SR methods and show that they can be integrated in a complementary manner, while balancing the reconstruction quality and time cost. Motivated by this, we further propose an integration framework to take the results from FSRCNN and A+ methods as inputs and directly learn a pixel-wise mapping between the inputs and the reconstructed results using the Gaussian conditional random fields. The learned pixel-wise integration mapping is flexible to accommodate different upscaling factors. Experimental results show that the proposed framework can achieve superior SR performance compared with the state of the arts while being efficient.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. http://www.gris.tu-darmstadt.de/research/visinf/software/index.en.htm.

References

  1. Chang, H., Yeung, D., Xiong, Y.: Super-resolution through neighbor embedding. In: CVPR (2004)

  2. Chorowski, J.K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition. In: NIPS (2015)

  3. Dai, D., Timofte, R., Van Gool, L.: Jointly optimized regressors for image super-resolution. In: EG (2015)

  4. Dai, S., Han, M., Xu, W., Wu, Y., Gong, Y.: Soft edge smoothness prior for alpha channel super resolution. In: CVPR (2007)

  5. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: ECCV (2014)

  6. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE TPAMI 38, 295–307 (2016)

    Article  Google Scholar 

  7. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: ECCV (2016)

  8. Duchon, C.: Lanczos filtering in one and two dimensions. J. Appl. Meteorol. 18, 1016–1022 (1979)

    Article  Google Scholar 

  9. Efrat, N., Glasner, D., Apartsin, A., Nadler, B., Levin, A.: Accurate blur models vs. image priors in single image super-resolution. In: ICCV (2013)

  10. Farsiu, S., Robinson, D., Elad, M., Milanfar, P.: Advances and challenges in super-resolution. Int. J. Imaging Syst. Technol. 14, 47–57 (2004)

    Article  Google Scholar 

  11. Fattal, R.: Image upsampling via imposed edge statistics. In: ACM TOG (2007)

  12. Freedman, G., Fattal, R.: Image and video upscaling from local self-examples. ACM TOG 30, 12 (2011)

    Article  Google Scholar 

  13. Freeman, W., Jones, T., Pasztor, E.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 56–65 (2002)

    Article  Google Scholar 

  14. Freeman, W., Liu, C.: Markov random fields for super-resolution and texture synthesis. Adv. Markov Random Fields Vis. Image Process. 1, 3 (2011)

    Google Scholar 

  15. He, X., Zemel, R.S., Carreira-Perpiñán, M.Á.: Multiscale conditional random fields for image labeling. In: CVPR (2004)

  16. Huang, J., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: CVPR (2015)

  17. Jancsary, J., Nowozin, S., Rother, C.: Loss-specific training of non-parametric image restoration models: a new state of the art. In: ECCV (2012)

  18. Jancsary, J., Nowozin, S., Sharp, T., Rother, C.: Regression tree fields an efficient, non-parametric approach to image labeling problems. In: CVPR (2012)

  19. Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging 2, 109–122 (2016)

    Article  MathSciNet  Google Scholar 

  20. Khler, T., Huang, X., Schebesch, F., Aichert, A., Maier, A., Hornegger, J.: Robust multiframe super-resolution employing iteratively re-weighted minimization. IEEE Trans. Comput. Imaging 2, 42–58 (2016)

    Article  MathSciNet  Google Scholar 

  21. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: CVPR (2016)

  22. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)

  23. Kim, K., Franz, M., Schölkopf, B.: Kernel hebbian algorithm for single-frame super-resolution. In: ECCV Workshop on Statistical Learning in Computer Vision (2004)

  24. Kim, K.I., Kwon, Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE TPAMI 32, 1127–1133 (2010)

    Article  Google Scholar 

  25. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001)

  26. Michaeli, T., Irani, M.: Nonparametric blind super-resolution. In: ICCV (2013)

  27. Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. 20, 21–36 (2003)

    Article  Google Scholar 

  28. Roweis, S., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)

    Article  Google Scholar 

  29. Salvador, J., Perez-Pellitero, E.: Naive bayes super-resolution forest. In: ICCV (2015)

  30. Schmidt, U., Rother, C., Nowozin, S., Jancsary, J., Roth, S.: Discriminative non-blind deblurring. In: CVPR (2013)

  31. Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: CVPR (2015)

  32. Shen, X., Tao, X., Gao, H., Zhou, C., Jia, J.: Deep automatic portrait matting. In: ECCV (2016)

  33. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv:1409.1556

  34. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: CVPR (2017)

  35. Tappen, M., Liu, C., Adelson, E., Freeman, W.: Learning gaussian conditional random fields for low-level vision. In: CVPR (2007)

  36. Thévenaz, P., Blu, T., Unser, M.: Image interpolation and resampling. Handb. Med. Imaging Process. Anal. 1, 393–420 (2000)

    Article  Google Scholar 

  37. Timofte, R., De, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: ICCV (2013)

  38. Timofte, R., De Smet, V., Van Gool, L.: A+: Adjusted anchored neighborhood regression for fast super-resolution. In: ACCV (2014)

  39. Wang, Q., Tang, X., Shum, H.: Patch based blind image super resolution. In: ICCV (2005)

  40. Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: ICCV (2015)

  41. Xie, Z., Xu, K., Liu, L., Xiong, Y.: 3d shape segmentation and labeling via extreme learning machine. In: CGF (2014)

  42. Xu, K., Shi, Y., Zheng, L., Zhang, J., Liu, M., Huang, H., Su, H., Cohen-Or, D., Chen, B.: 3d attention-driven depth acquisition for object identification. ACM TOG 35, 238 (2016)

    Google Scholar 

  43. Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution via sparse representation. IEEE TIP 19, 2861–2873 (2010)

    MathSciNet  MATH  Google Scholar 

  44. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces (2010)

  45. Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM TOMS 23, 550–560 (1997)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We thank the anonymous reviewers for the insightful and constructive comments. This work is in part supported by an SRG grant from City University of Hong Kong (Ref. 7004889), and by NSFC grant from National Natural Science Foundation of China (Ref. 91748104, 61632006, 61425002, 61702194).

Funding

This study was funded by an SRG grant from City University of Hong Kong (Ref. 7004889), and by NSFC grant from National Natural Science Foundation of China (Ref. 91748104, 61632006, 61425002, 61702194).

Author information

Authors and Affiliations

  1. Dalian University of Technology, Dalian, China

    Ke Xu, Xin Wang, Xin Yang, Qiang Zhang, Baocai Yin & Xiaopeng Wei

  2. South China University of Technology, Guangzhou, China

    Shengfeng He

  3. City University of Hong Kong, Kowloon, Hong Kong

    Ke Xu & Rynson W. H. Lau

Authors
  1. Ke Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Xin Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xin Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Shengfeng He
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Qiang Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Baocai Yin
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Xiaopeng Wei
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Rynson W. H. Lau
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Xin Yang or Shengfeng He.

Ethics declarations

Conflict of interest

Ke Xu, XinWang, Xin Yang, Shengfeng He, Qiang Zhang, Baocai Yin, Xiaopeng Wei and Rynson W.H. Lau declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, K., Wang, X., Yang, X. et al. Efficient image super-resolution integration. Vis Comput 34, 1065–1076 (2018). https://doi.org/10.1007/s00371-018-1554-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-018-1554-2

Keywords