Skip to main content
SpringerLink
Log in

Gradient-based low rank method and its application in image inpainting

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Conventional inpainting methods generally apply textures that are most similar to the areas around the missing region or use large image database. Recently, low rank property of data shows that the non-convex optimization decreases measurements. In this paper, we propose a new image prior, which implies the low rank prior knowledge of image gradients. The proposed detail-preserving image inpainting algorithm adopts the low rank regularization to gradient similarity minimization, termed gradient-based low rank approximation (Grad-LR), namely that we employ the low rank constraints in the horizontal and vertical gradients of the image and then reconstruct the desired image using the adaptive iterative singular-value thresholding of both derivatives. In the method, by incorporating the spatially adaptive iterative singular-value thresholding (SAIST) to optimize our gradient scheme, the deterministic annealing iterates the procedure efficiently. As a result, the strength of the algorithm is obvious when filling large missing region. Experimental results consistently demonstrate that the proposed algorithm works well for both structural and texture images and outperforms other techniques, in terms of both objective and subjective performance measures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. The code is available in http://www.csee.wvu.edu/~xinl/demo/saist.html

  2. The PSNR is defined as: PSNR = 20log10255/RMSE, where RMSE is the root mean error estimated between the ground truth and the reconstructed image

  3. Available at http://www.csee.wvu.edu/~xinl/code/patch_recon.zip.

References

  1. Algazi VR, Ford GE, Potharlanka R (1991) Directional interpolation of images based on visual properties and rank order filtering. In: Acoustics, Speech, and Signal Processing. ICASSP-91., 1991 International conference on, pp 3005–3008. IEEE

  2. Allebach J, Wong PW (1996) Edge-directed interpolation. In: Image processing. Proceedings, international conference on 3:707–710. IEEE

  3. Arias P, Caselles V, Sapiro G (2009) A variational framework for non-local image inpainting. In: Energy minimization methods in computer vision and pattern recognition. Springer, Berlin, pp 345–358

  4. Arias P, Facciolo G, Caselles V, Sapiro G (2011) A variational framework for exemplar-based image inpainting. Int J Comput Vis 93(3):319–347

    Article  MathSciNet  MATH  Google Scholar 

  5. Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on computer graphics and interactive techniques, pp 417–424

  6. Bertozzi AL, Esedoglu S, Gillette A (2007) Inpainting of binary images using the Cahn-Hilliard equation. IEEE Trans Image Process 16(1):285–291 IEEE

    Article  MathSciNet  MATH  Google Scholar 

  7. Bornemann F, März T (2007) Fast image inpainting based on coherence transport. J Math Imaging Vision 28(3):259–278

    Article  MathSciNet  Google Scholar 

  8. Brandtberg T, McGraw JB, Warner TA et al (2003) Image restoration based on multi-scale relationships of image structures. IEEE Trans Geosci Remote Sens 41(1):102–110 IEEE

    Article  Google Scholar 

  9. Burger M, He L, Schönlieb CB (2009) Cahn-Hilliard inpainting and a generalization for grayvalue images. SIAM J Imag Sci 2(4):1129–1167

    Article  MathSciNet  MATH  Google Scholar 

  10. Cai JF, Chan RH, Shen L, Shen Z (2009) Simultaneously inpainting in image and transformed domains. Numerische Mathematik 112(4):509–533

    Article  MathSciNet  MATH  Google Scholar 

  11. Cai JF, Candès EJ, Shen Z (2010) A singular value thresholding algorithm for matrix completion. SIAM J Optim 20(4):1956–1982

    Article  MathSciNet  MATH  Google Scholar 

  12. Carrato S, Ramponi G, Marsi S (1996) A simple edge-sensitive image interpolation filter. Proc. IEEE Int. Conf. Image Processing 3:711–714 IEEE

    Article  Google Scholar 

  13. Caselles V (2011) Exemplar-based image inpainting and applications. SIAM News 44(10):1–3

    Google Scholar 

  14. Chan TF, Shen J (2005) Variational image inpainting. Comm Pure Appl Math 58(2005):579–619

    Article  MathSciNet  MATH  Google Scholar 

  15. Cho TS, Joshi N, Zitnick CL, Kang SB, Szeliski R, Freeman WT (2010) A content-aware image prior. In: Computer Vision and Pattern Recognition (CVPR), 2010 I.E. Conference on, pp 169–176. IEEE

  16. Cho TS, Zitnick CL, Joshi N, Kang SB, Szeliski R, Freeman WT (2012) Image restoration by matching gradient distributions. IEEE Trans Pattern Anal Mach Intell 34(4):683–694

    Article  Google Scholar 

  17. Coombs T, Schmidt L (2000) An empirical analysis of image restoration: Texaco’s racism crisis. J Public Relat Res 12(2):163–178

    Article  Google Scholar 

  18. Dabov K et al (2007) Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans Image Process 16(8):2080–2095

    Article  MathSciNet  Google Scholar 

  19. Dong W, Shi G, Li X (2013) Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans Image Process 22(2):700–711 IEEE

    Article  MathSciNet  MATH  Google Scholar 

  20. Elad M, Starck JL, Querre P, Donoho DL (2005) Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl Comput Harmon Anal 19(3):340–358

    Article  MathSciNet  MATH  Google Scholar 

  21. Facciolo G, Arias P, Caselles V, Sapiro G (2009) Exemplar-based interpolation of sparsely sampled images. In: Energy minimization methods in computer vision and pattern recognition. Springer, Berlin, p 331–344

  22. Fadili MJ, Starck JL, Murtagh F (2009) Inpainting and zooming using sparse representations. Comput J 52(1):64–79

    Article  Google Scholar 

  23. Fergus R, Singh B, Hertzmann A, Roweis ST, Freeman WT (2006) Removing camera shake from a single photograph. ACM Trans Graph (TOG) 25(3):787–794

    Article  MATH  Google Scholar 

  24. Gepshtein S, Keller Y (2013) Image completion by diffusion maps and spectral relaxation. IEEE Trans Image Process 22(8):2983–2994 IEEE

    Article  MathSciNet  MATH  Google Scholar 

  25. Guleryuz OG (2006) Non-linear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-part II: adaptive algorithms. IEEE Trans Image Process 15(3):555–571 IEEE

    Article  Google Scholar 

  26. Jensen K, Anastassiou D (1995) Subpixel edge localization and the interpolation of still images. IEEE Trans Image Process 4(3):285–295 IEEE

    Article  Google Scholar 

  27. Ji H, Huang S, Shen Z, Xu Y (2011) Robust video restoration by joint sparse and low rank matrix approximation. SIAM J Imag Sci 4(4):1122–1142

    Article  MathSciNet  MATH  Google Scholar 

  28. Kindermann S, Osher S, Jones PW (2005) Deblurring and denoising of images by nonlocal functionals. Multiscale Model Simul 4(4):1091–1115

    Article  MathSciNet  MATH  Google Scholar 

  29. Li X (2008) Patch-based image interpolation: algorithms and applications. In: Proceedings local nonlocal approximation in image processing, pp 1–6. http://www.eurasip.org/Proceedings/Ext/LNLA2008/papers/cr1019.pdf

  30. Li X (2011) Image recovery via hybrid sparse representations: A deterministic annealing approach. IEEE J Sel Top Sign Proces. 5(5):953–962 IEEE

    Article  Google Scholar 

  31. Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10(10):1521–1527 IEEE

    Article  Google Scholar 

  32. Liu Y, Caselles V (2013) Exemplar-based image inpainting using multiscale graph cuts. IEEE Trans Image Process 22(5):1699–1711 IEEE

    Article  MathSciNet  MATH  Google Scholar 

  33. Liu Q, Wang S, Ying L, Peng X, Zhu Y, Liang D (2013) Adaptive dictionary learning in sparse gradient domain for image recovery. IEEE Trans Image Process 22(12):4652–4663 IEEE

    Article  MathSciNet  MATH  Google Scholar 

  34. Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A (2009) Non-local sparse models for image restoration. In Computer vision, 2009 I.E. 12th international conference on, pp 2272–2279. IEEE

  35. Morse BS, Schwartzwald D (1998) Isophote-based interpolation. In: Image processing. ICIP 98. Proceedings. 1998 international conference on, pp 227–231. IEEE

  36. Ogawa T, Haseyama M (2011) Missing image data reconstruction based on adaptive inverse projection via sparse representation. IEEE Trans Multimedia 13(5):974–992 IEEE

    Article  Google Scholar 

  37. Ogawa T, Haseyama M (2013) Missing texture reconstruction method based on error reduction algorithm using Fourier transform magnitude estimation scheme. IEEE Trans Image Process 22(3):1252–1257 IEEE

    Article  MathSciNet  MATH  Google Scholar 

  38. Papafitsoros K, Schoenlieb CB, Sengul B (2013) Combined first and second order total variation inpainting using split Bregman. Image Processing On Line 3:112–136

    Article  Google Scholar 

  39. Paul G, Cardinale J, Sbalzarini IF (2013) Coupling image restoration and segmentation: a generalized linear model/Bregman perspective. Int J Comput Vis 104(1):69–93

    Article  MathSciNet  MATH  Google Scholar 

  40. Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639

    Article  Google Scholar 

  41. Ratakonda K, Ahuja N (1998) POCS based adaptive image magnification. In: Image processing. ICIP 98. Proceedings. 1998 international conference on, pp 203–207. IEEE

  42. Roth S, Black MJ (2009) Fields of experts. Int J Comput Vis 82(2):205–229

    Article  Google Scholar 

  43. Schönlieb CB, Bertozzi A (2011) Unconditionally stable schemes for higher order inpainting. Commun Math Sci 9(2):413–457

    Article  MathSciNet  MATH  Google Scholar 

  44. Sun J, Xu Z, Shum HY (2011) Gradient profile prior and its applications in image super-resolution and enhancement. IEEE Trans Image Process 20(6):1529–1542

    Article  MathSciNet  MATH  Google Scholar 

  45. Takeda H, Farsiu S, Milanfar P (2007) Kernel regression for image processing and reconstruction. IEEE Trans Image Process 16(2):349–366 IEEE

    Article  MathSciNet  Google Scholar 

  46. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. Image Processing, IEEE Transactions 13(4):600–612 IEEE

    Article  Google Scholar 

  47. Wang S, Zhang L, Liang Y (2012) Nonlocal spectral prior model for low-level vision. In: Computer vision-ACCV 2012. Springer, Berlin, pp 231–244

  48. Wang M, Yan B, Ngan KN (2013) An efficient framework for image/video inpainting. Signal Process Image Commun 28(7):753–762

    Article  Google Scholar 

  49. Wexler Y, Shechtman E, Irani M (2007) Space-time completion of video. IEEE Trans Pattern Anal Mach Intell 29(3):463–476

    Article  Google Scholar 

  50. Xu Z, Sun J (2010) Image inpainting by patch propagation using patch sparsity. IEEE Trans Image Process 19(5):1153–1165. IEEE

  51. Zhang X, Wu X (2008) Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation. IEEE Trans Image Process 17(6):887–896. IEEE

  52. Zuo W, Zhang L, Song C, Zhang D (2013) Texture enhanced image denoising via gradient histogram preservation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 1203–1210

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported in part by the National Natural Science Foundation of China under 61661031, 61362001, 61503176, Jiangxi advanced projects for post-doctoral research funds (2014KY02), the international postdoctoral exchange fellowship program, the international scientific and technological cooperation projects of Jiangxi Province (No.20141BDH80001).

Author information

Authors and Affiliations

  1. Department of Electronic Information Engineering, Nanchang University, Nanchang, 330031, China

    Hongyang Lu, Qiegen Liu, Minghui Zhang, Yuhao Wang & Xiaohua Deng

  2. Institute of Space Science and Technology, Nanchang University, Nanchang, 330031, China

    Hongyang Lu & Xiaohua Deng

Authors
  1. Hongyang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiegen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Minghui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuhao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaohua Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiegen Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, H., Liu, Q., Zhang, M. et al. Gradient-based low rank method and its application in image inpainting. Multimed Tools Appl 77, 5969–5993 (2018). https://doi.org/10.1007/s11042-017-4509-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4509-0

Keywords

Search

Navigation