Skip to main content
SpringerLink
Log in

Image inpainting algorithm based on tensor decomposition and weighted nuclear norm

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

For a damaged image, recovering an image with missing entire rows or columns is a challenging problem arising in many real applications, such as digital image inpainting. For this kind of information missing situation, the diffusion-based inpainting methods are tend to produce blur, the exemplar-based methods are prone to error filling and the neural network-based methods are highly dependent on data. Many existing approaches formulate this problem as a general low-rank matrix approximate one which cannot handle this special structural missing very well. In this paper, we propose a novel image inpainting algorithm named nonlocal low-rank tensor completion (NLLRTC) based on the nonlocal self-similarity prior and the low-rank prior. By using the nonlocal self-similarity of image patches, we directly stack these patches into a three-dimensional similar tensor instead of pulling them into column vectors, then the similar tensor can be completed by tensor ring (TR) decomposition. By leveraging the alternating direction method under the augmented Lagrangian multiplier framework, the optimization results can be obtained. Moreover, a weighted nuclear norm is added to the tensor completion model to achieve better inpainting performance, which we call weighted nonlocal low-rank tensor completion (WNLLRTC) algorithm. Our empirical studies show encouraging results on both quantitative assessment and visual interpretation of our proposed methods in comparison to some state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Barnes C, Zhang FL (2017) A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media 3(01):5–22

    Article  Google Scholar 

  2. Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp 417–424

  3. Cai J, Candés EJ, Shen Z (2008) A singular value thresholding algorithm for matrix completion. Siam Journal on Optimization 20(4):1956–1982

    Article  MathSciNet  MATH  Google Scholar 

  4. Candès EJ, Recht B (2009) Exact matrix completion via convex optimization. Found Comput Math 9(6):717

    Article  MathSciNet  MATH  Google Scholar 

  5. Cao X, Ren W, Zuo W, Guo X, Foroosh H (2015) Scene text deblurring using text-specific multiscale dictionaries. IEEE Trans Image Process 24 (4):1302–1314

    Article  MathSciNet  MATH  Google Scholar 

  6. Criminisi A, Perez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process 13(9):1200–1212

    Article  Google Scholar 

  7. Daisy M, Buyssens P, Tschumperlé D, Lézoray O (2014) A smarter exemplar-based inpainting algorithm using local and global heuristics for more geometric coherence. In: 2014 IEEE international conference on image processing (ICIP), IEEE, pp 4622–4626

  8. Dong W, Shi G, Li X (2012) Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans Image Process 22(2):700–711

    Article  MathSciNet  MATH  Google Scholar 

  9. Efros AA, Leung TK (2002) Texture synthesis by non-parametric sampling. In: Proceedings of the Seventh IEEE international conference on computer vision

  10. Elad M, Starck JL, Querre P, Donoho DL (2005) Simultaneous cartoon and texture image inpainting using morphological component analysis (mca). Appl Comput Harmon Anal 19(3):340–358

    Article  MathSciNet  MATH  Google Scholar 

  11. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  12. Gu S, Xie Q, Meng D, Zuo W, Feng X, Zhang L (2017) Weighted nuclear norm minimization and its applications to low level vision. Int J Comput Vis 121(2):183–208

    Article  MATH  Google Scholar 

  13. Guo Q, Gao S, Zhang X, Yin Y, Zhang C (2017) Patch-based image inpainting via two-stage low rank approximation. IEEE Trans Vis Comput Graph 24(6):2023–2036

    Article  Google Scholar 

  14. He L, Wang Y (2014) Iterative support detection-based split bregman method for wavelet frame-based image inpainting. IEEE Trans Image Process 23 (12):5470–5485

    Article  MathSciNet  MATH  Google Scholar 

  15. Hillar CJ, Lim L (2013) Most tensor problems are np-hard. Journal of the ACM (JACM) 60(6):1–39

    Article  MathSciNet  MATH  Google Scholar 

  16. Hu SM, Zhang FL, Wang M, Martin RR, Wang J (2013) Patchnet: A patch-based image representation for interactive library-driven image editing. ACM Transactions on Graphics (TOG) 32(6):1–12

    Google Scholar 

  17. Hu Y, Zhang D, Ye J, Li X, He X (2012) Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Trans Pattern Anal Mach Intell 35(9):2117–2130

    Article  Google Scholar 

  18. Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500

    Article  MathSciNet  MATH  Google Scholar 

  19. Komodakis N (2006) Image completion using global optimization. In: 2006 IEEE computer society conference on computer vision and pattern recognition

  20. Le Meur O, Gautier J, Guillemot C (2011) Examplar-based inpainting based on local geometry. In: 2011 18Th IEEE international conference on image processing, IEEE, pp 3401–3404

  21. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  22. Li F, Zeng T (2014) A universal variational framework for sparsity-based image inpainting. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 23(10):4242

    Article  MathSciNet  MATH  Google Scholar 

  23. Li W, Zhao L, Lin Z, Xu D, Lu D (2015) Non-local image inpainting using low-rank matrix completion. In: Computer graphics forum, vol 34, Wiley Online Library, pp 111–122

  24. Lin Z, Chen M, Ma Y (2010) The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv preprint arXiv:1009.5055

  25. Liu J, Musialski P, Wonka P, Ye J (2012) Tensor completion for estimating missing values in visual data. IEEE Trans Pattern Anal Mach Intell 35 (1):208–220

    Article  Google Scholar 

  26. Luo SJ, Sun YT, Shen IC, Chen BY, Chuang YY (2015) Geometrically consistent stereoscopic image editing using patch-based synthesis. IEEE Transactions on Visualization & Computer Graphics 21(1):56–67

    Article  Google Scholar 

  27. Masci J, Meier U, Cireṡan D., Schmidhuber J (2011) Stacked convolutional auto-encoders for hierarchical feature extraction. In: International conference on artificial neural networks, Springer, pp 52–59

  28. Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759

  29. Oseledets IV (2011) Tensor-train decomposition. SIAM J Sci Comput 33(5):2295–2317

    Article  MathSciNet  MATH  Google Scholar 

  30. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544

  31. Perona P, Malik J (2002) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639

    Article  Google Scholar 

  32. Ren W, Cao X, Pan J, Guo X, Zuo W, Yang MH (2016) Image deblurring via enhanced low-rank prior. IEEE Trans Image Process 25 (7):3426–3437

    Article  MathSciNet  MATH  Google Scholar 

  33. Steck H (2010) Training and testing of recommender systems on data missing not at random. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 713–722

  34. Toh KC, Yun S (2010) An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pacific Journal of Optimization 6(615-640):15

    MathSciNet  MATH  Google Scholar 

  35. Tschumperlé D (2006) Fast anisotropic smoothing of multi-valued images using curvature-preserving pde’s. Int J Comput Vis 68(1):65–82

    Article  MATH  Google Scholar 

  36. Wen Z, Yin W, Zhang Y (2012) Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm. Mathematical Programming Computation 4(4):333–361

    Article  MathSciNet  MATH  Google Scholar 

  37. Xu Z, Sun J (2010) Image inpainting by patch propagation using patch sparsity. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 19(5):1153

    Article  MathSciNet  MATH  Google Scholar 

  38. Xu Z, Sun J (2010) Image inpainting by patch propagation using patch sparsity. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 19(5):1153

    Article  MathSciNet  MATH  Google Scholar 

  39. Ye X, Yang J, Sun X, Li K, Hou C, Wang Y (2015) Foreground–background separation from video clips via motion-assisted matrix restoration. IEEE Trans Circuits Syst Video Technol 25(11):1721–1734

    Article  Google Scholar 

  40. Yeh R, Chen C, Lim TY, Hasegawa-Johnson M, Do MN (2016) Semantic image inpainting with perceptual and contextual losses. arXiv preprint arXiv:1607.07539 2(3)

  41. Yuan L, Li C, Mandic D, Cao J, Zhao Q (2019) Tensor ring decomposition with rank minimization on latent space: an efficient approach for tensor completion. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9151–9158

  42. Zhang Z, Ganesh A, Liang X, Ma Y (2012) Tilt: Transform invariant low-rank textures. Int J Comput Vis 99(1):1–24

    Article  MathSciNet  MATH  Google Scholar 

  43. Zhang FL, Wang J, Shechtman E, Zhou ZY, Shi JX, Hu SM (2017) Plenopatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization & Computer Graphics 23(5):1561–1573

    Article  Google Scholar 

  44. Zhang J, Zhao D, Gao W (2014) Group-based sparse representation for image restoration. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 23(8):3336

    Article  MathSciNet  MATH  Google Scholar 

  45. Zhao Q, Zhou G, Xie S, Zhang L, Cichocki A (2016) Tensor ring decomposition. arXiv preprint arXiv:1606.05535

  46. Zhou M, Chen H, Paisley J, Ren L, Li L, Xing Z, Dunson D, Sapiro G, Carin L (2012) Nonparametric bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 21(1):130–44

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The research has been supported in part by the National Natural Science Foundation of China (12071263, 61671276, 11971269), the Natural Science Foundation of Shandong Province of China (ZR2019MF045), and the National Science Fund for Distinguished Young Scholars (61625102).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong Jianzhu University, Jinan, 250101, China

    Xuya Liu

  2. Qingdao Hi-image Technologies Co., Ltd., Qingdao, 266000, China

    Caiyan Hao

  3. Guangdong Technion-Israel Institute of Technology, Shantou, 515063, China

    Zezhao Su

  4. Shandong Chengshi Electronic Technology Limited Company, Jinan, 250031, China

    Zerong Qi

  5. School of Mathematics, Shandong University, Jinan, 250100, China

    Shujun Fu

  6. Department of Intervention Medicine, The Second Hospital of Shandong University, Jinan, 250100, China

    Yuliang Li

  7. Department of Radiology, Peking University Third Hospital, Beijing, 100089, China

    Hongbin Han

  8. Beijing Key Laboratory of Magnetic Resonance Imaging Equipment and Technique, Beijing, 100089, China

    Hongbin Han

Authors
  1. Xuya Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Caiyan Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zezhao Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zerong Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shujun Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yuliang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hongbin Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shujun Fu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, X., Hao, C., Su, Z. et al. Image inpainting algorithm based on tensor decomposition and weighted nuclear norm. Multimed Tools Appl 82, 3433–3458 (2023). https://doi.org/10.1007/s11042-022-12635-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12635-3

Keywords

Search

Navigation