Skip to main content
SpringerLink
Log in

Single-image super-resolution via patch-based and group-based local smoothness modeling

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Local smoothness and nonlocal self-similarity of natural images are two main priors in the image restoration (IR) problem. Many IR methods have widely used patch-based modeling. Recently, the concept of grouping-based technique, the nonlocal patches with similar structures, has been introduced as the basic unit of sparse representation. In the group-based methods, the nonlocal self-similarity and the local sparsity properties are combined in a unified framework using the sparsity-based techniques. In this paper, a new model is proposed which utilizes both the patch and the group as the basic units of image modeling, called patch-based and group-based local smoothness modeling (PGLSM). More, precisely, in the proposed PGLSM scheme, the local smoothness in the patch-based unit is exploited by an isotropic total variation method and the local smoothness in the group-based unit is exploited by group-based sparse representation method. In this way, a novel technique for high-fidelity single-image super-resolution (SISR) via PGLSM is proposed, called SR-PGLSM. By adding nonlocal means (NLM) as the complementary regularization term to PGLSM, another technique for SISR is modeled, called SR_PGLSM_NLM. In order to efficiently solve the above variational problems, the split Bergman iterative technique has been leveraged. Extensive experimental results validate the effectiveness and robustness of the proposed methods. Our proposed schemes can recover more fine structures and achieve better results than the competing methods with the scaling factor of 2 and 3 and for noisy images both subjectively and objectively in most cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. 20(3), 21–36 (2003)

    Google Scholar 

  2. Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Signal Process. 128, 389–408 (2016)

    Google Scholar 

  3. Kwok, W., Sun, H.: Multi-directional interpolation for spatial error concealment. IEEE Trans. Consum. Electron. 39, 455–460 (1993)

    Google Scholar 

  4. Zhang, X., Wu, X.: Image interpolation by adaptive 2D autoregressive modeling and soft-decision estimation. IEEE Trans. Image Process. 17(6), 887–896 (2008)

    MathSciNet  Google Scholar 

  5. Protter, M., Elad, M., Takeda, H., Millanfar, P.: Generalizing the non-local means to super-resolution reconstruction. IEEE Trans. Image Process. 18(1), 36–51 (2009)

    MathSciNet  MATH  Google Scholar 

  6. Candocia, F.M., Principe, J.C.: Super-resolution of images based on local correlations. IEEE Trans. Neural Netw. Learn. Syst. 10(2), 372–380 (1999)

    Google Scholar 

  7. Stark, H., Oskoui, P.: High-resolution image recovery from image plane arrays, using convex projections. J. Opt. Soc. Am. 6(11), 1715–1726 (1989)

    Google Scholar 

  8. Tao, D., Cheng, J., Lin, X., Yu, J.: Local structure preserving discriminative projections for RGB-D sensor-based scene classification. Inf. Sci. 320, 383–394 (2015)

    MathSciNet  Google Scholar 

  9. Takeda, H., Farsiu, S., Milanfar, P.: Kernel regression for image processing and reconstruction. IEEE Trans. Image Process. 16(2), 349–366 (2007)

    MathSciNet  Google Scholar 

  10. Fattal, R.: Image upsampling via imposed edge statistics. ACM Trans. Graph. 26(3), 95 (2007)

    Google Scholar 

  11. Wang, L., Xiang, S., Meng, G., Wu, H., Pan, C.: Edge-directed single-image super-resolution via adaptive gradient magnitude self interpolation. IEEE Trans. Circuits Syst. Video Technol. 23(8), 1289–1299 (2013)

    Google Scholar 

  12. Song, Q., Xiong, R., Liu, D., Xiong, Z.H., Wu, F., Gao, W.: Fast image super-resolution via local adaptive gradient field sharpening transform. IEEE Trans. Image Process. 27(4), 1966–1980 (2018)

    MathSciNet  MATH  Google Scholar 

  13. Marquina, A., Osher, S.J.: Image super-resolution by TV regularization and Bregman iteration. J. Sci. Comput. 37(3), 367–382 (2008)

    MathSciNet  MATH  Google Scholar 

  14. Yang, X., Zhang, J., Liu, Y., et al.: Super-resolution image reconstruction using fractional-order total variation and adaptive regularization parameters. Vis. Comput. (2018). https://doi.org/10.1007/s00371-018-1570-2

    Article  Google Scholar 

  15. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: 2005 IEEE Conference on Computer Vision and Pattern Recognition, pp. 60–65 (2005)

  16. Chang, H., Yeung, D.-Y., Xiong, Y.: Super-resolution through neighbor embedding. In: CVPR, pp. 275–282 (2004)

  17. Yang, C.-Y., Yang, M.-H.: Fast direct super-resolution by simple functions. In: ICCV, pp. 561–568 (2013)

  18. Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1920–1927 (2013)

  19. Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: 2014 Asian Conference on Computer Vision and Pattern Recognition, pp. 111–126 (2014)

  20. Zhang, K., Tao, D., Gao, X., Li, X., Xiong, Z.: Learning multiple linear mappings for efficient single image super-resolution. IEEE Trans. Image Process. 24(3), 846–861 (2015)

    MathSciNet  MATH  Google Scholar 

  21. Zhang, Y., Zhang, Y., Zhang, J., Dai, Q.: CCR: clustering and collaborative representation for fast single image super-resolution. IEEE Trans. Multimed. 18(3), 405–417 (2016)

    Google Scholar 

  22. Zhang, K., Wang, B., Zuo, W., Zhang, H., Zhang, L.: Joint learning of multiple regressors for single image super-resolution. IEEE Trans. Signal Process. Lett. 23(1), 102–106 (2016)

    Google Scholar 

  23. Romano, Y., Isidoro, J., Milanfar, P.: RAISR: rapid and accurate image super resolution. IEEE Trans. Comput. Imaging 3(1), 110–125 (2017)

    MathSciNet  Google Scholar 

  24. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision (ECCV) 2014: Computer Vision, pp. 184–199 (2014)

  25. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: CVPR (2018)

  26. Kim, J, Kwon Lee, J., Lee, K.M.: Accurate image super resolution using very deep convolutional networks. In: CVPR (2016)

  27. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super resolution using a generative adversarial network. In: CVPR (2017)

  28. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: CVPRW (2017)

  29. Zhang, Y., Yapeng, T., Kong, Y., Zhong, B., Yun, F.: Residual dense network for image super-resolution. In: CVPR (2018)

  30. Zhang, Y., Li, K., Li, K., Wang, L., Zhong. B., Fu, Y.: Image superresolution using very deep residual channel attention networks. In: ECCV (2018)

  31. Zhang, Y., Li, K., et al.: Residual non-local attention networks for image restoration. In: ICLR (2019)

  32. Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution as sparse representation of raw image patches. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  33. Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    MathSciNet  MATH  Google Scholar 

  34. Yang, J., Wang, Z., Lin, Z., Cohen, S., Huang, T.: Coupled dictionary training for image super-resolution. IEEE Trans. Image Process. 21(8), 3467–3478 (2012)

    MathSciNet  MATH  Google Scholar 

  35. Li, J., Wu, J., Deng, H., Liu, J.: A self-learning image super-resolution method via sparse representation and non-local similarity. Neurocomputing 184, 196–206 (2016)

    Google Scholar 

  36. Dong, W., Zhang, L., Shi, G., Wu, X.: Image deblurring and superresolution by adaptive sparse domain selection and adaptive regularization. IEEE Trans. Image Process. 20(7), 1838–1857 (2011)

    MathSciNet  MATH  Google Scholar 

  37. Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22(4), 1620–1630 (2013)

    MathSciNet  MATH  Google Scholar 

  38. Peleg, T., Elad, M.: A statistical prediction model based on sparse representations for single image super-resolution. IEEE Trans. Image Process. 23(6), 2569–2582 (2014)

    MathSciNet  MATH  Google Scholar 

  39. Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-Laplacian priors. In: Proceeding of NIPS, vol. 22, pp. 1–9 (2009)

  40. Hughes, N.P., Tarassenko, L.: Novel signal shape descriptors through wavelet transforms and dimensionality reduction. In: 2003 Optical Science and Technology, pp. 763–773 (2003)

  41. Tropp, J.A.: Algorithms for simultaneous sparse approximation. Part II: convex relaxation. Signal Process. 86(3), 589–602 (2006)

    MATH  Google Scholar 

  42. Liu, J., Yang, W., Zhang, X., Guo, Z.: Retrieval compensated group structured sparsity for image super-resolution. IEEE Trans. Multimed 19(2), 302–316 (2017)

    Google Scholar 

  43. Soltani-Farani, A., Rabiee, H.R., Hosseini, S.A.: Spatial-aware dictionary learning for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 53(1), 527–541 (2014)

    Google Scholar 

  44. Rakotomamonjy, A.: Surveying and comparing simultaneous sparse approximation (or group-lasso) algorithms. Signal Process. 91(7), 1505–1526 (2011)

    MATH  Google Scholar 

  45. Eslahi, N., Aghagolzadeh, A., Andargoli, S.M.H.: Image/video compressive sensing recovery using joint adaptive sparsity measure. Neurocomputing 200, 88–109 (2016)

    Google Scholar 

  46. Zhang, J., Zhao, D., Gao, W.: Group-based sparse representation for image restoration. IEEE Trans. Image Process. 23(8), 3336–3351 (2014)

    MathSciNet  MATH  Google Scholar 

  47. Cai, J.-F., Candès, E.J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20, 1956–1982 (2010)

    MathSciNet  MATH  Google Scholar 

  48. Goldstein, T., Osher, S.: The split Bregman algorithm for L1 regularized problems. SIAM J. Image. Sci. 2(2), 323–343 (2009)

    MATH  Google Scholar 

  49. Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process. 18(11), 2419–2434 (2009)

    MathSciNet  MATH  Google Scholar 

  50. Zhang, Z., Xu, Y., Yang, J., Li, X., Zhang, D.: A survey of sparse representation: algorithms and applications. IEEE. Open Access 3, 490–530 (2015)

    Google Scholar 

  51. Aharon, M., Elad, M., Bruckstein, A.: K-svd: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)

    MATH  Google Scholar 

  52. Elad, M., Yavneh, I.: A plurality of sparse representation is better than the sparsest one alone. IEEE Trans. Inf. Theory 55(10), 4701–4714 (2009)

    MathSciNet  MATH  Google Scholar 

  53. Dong, W., Shi, G., Wu, X., Zhang, L.: A learning-based method for compressive image recovery. J. Vis. Commun. Image Represent. 24(7), 1055–1063 (2013)

    Google Scholar 

  54. Zhang, X., Burger, M., Bresson, X., Osher, S.: Bregmanized nonlocal regularization for deconvolution and sparse reconstruction. SIAM J. Image. Sci. 3(3), 253–276 (2010)

    MathSciNet  MATH  Google Scholar 

  55. Moreau, J.J.: Proximitéet dualité dans un espace hilbertien. Bull. Soc. Math. France 93, 273–299 (1965)

    MathSciNet  MATH  Google Scholar 

  56. Dong, W., Shi, G., Li, X.: Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans. Image Process. 22(2), 700–711 (2013)

    MathSciNet  MATH  Google Scholar 

  57. Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Image. Vis. 20(1–2), 89–97 (2004)

    MathSciNet  MATH  Google Scholar 

  58. Bioucas-Dias, J., Figueiredo, M.: A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16(12), 2992–3004 (2007)

    MathSciNet  Google Scholar 

  59. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 23rd British Machine Vision Conference (BMVC) (2012)

  60. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.-D., Chenin, P., Cohen, A., Gout, C., Lyche, T., Mazure, M.-L., Schumaker, L. (eds.) Curves and Surfaces, pp. 711–730. Springer, Berlin (2012)

    Google Scholar 

  61. Huang, J.B., Singh, A., Ahuja, N.: Single image superresolution using transformed self-exemplars. In: CVPR (2015)

  62. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001)

  63. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error measurement to structural similarity. IEEE Trans. Image Process. 3(4), 600–612 (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical and Computer Engineering, Babol Noshirvani University of Technology, Babol, Iran

    Elhameh Mikaeli & Ali Aghagolzadeh

  2. Faculty of Electrical and Computer Engineering, Sahand University of Technology, Tabriz, Iran

    Masoumeh Azghani

Authors
  1. Elhameh Mikaeli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Aghagolzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Masoumeh Azghani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ali Aghagolzadeh.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mikaeli, E., Aghagolzadeh, A. & Azghani, M. Single-image super-resolution via patch-based and group-based local smoothness modeling. Vis Comput 36, 1573–1589 (2020). https://doi.org/10.1007/s00371-019-01756-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01756-w

Keywords

Search

Navigation