Skip to main content
Springer Nature Link
Log in

Tensor based approach for inpainting of video containing sparse text

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Videos received from certain sources may contain irrelevant contents which might reduce the amount of information conveyed by it. This paper proposes an effective tensor based video inpainting approach to improve the quality of these videos by removing and replacing unwanted contents with relevant information. The proposed method employs reweighted tensor decomposition technique to identify and discard the inappropriate sparse components of the video data. These sparse components are substituted with proper contents by utilising spatio-temporal consistency through reweighted tensor completion. The replacement is carried out in such a way that the resulting video possesses superior temporal consistency and visual credibility. The proposed method is applied to sparse text removal of videos having dynamic content in various extents and is found that our method outperforms its counterparts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Arizona State University, Yuv video sequences. http://trace.eas.asu.edu/yuv/

  2. Baburaj M, George SN (2017) Twist tensor total variation regularized-reweighted nuclear norm based tensor completion for video missing area recovery, Information Sciences

  3. Bornard R, Lecan E, Laborelli L, Chenot JH (2002) Missing data correction in still images and image sequences. In: Proceedings of the tenth ACM international conference on multimedia. ACM, pp 355–361

  4. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3(1):1–122

    Article  Google Scholar 

  5. Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell 23(11):1222–1239

    Article  Google Scholar 

  6. Candes EJ, Wakin MB, Boyd SP (2008) Enhancing sparsity by reweighted 1 minimization. J Fourier Anal Appl 14(5):877–905

    Article  MathSciNet  Google Scholar 

  7. Carroll JD, Chang JJ (1970) Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition. Psychometrika 35 (3):283–319

    Article  Google Scholar 

  8. Choi JK, Dong B, Zhang X (2017) An edge driven wavelet frame model for image restoration. arXiv:170107158

  9. Cichocki A, Mandic D, De Lathauwer L, Zhou G, Zhao Q, Caiafa C, Phan HA (2015) Tensor decompositions for signal processing applications: from two-way to multiway component analysis. IEEE Signal Proc Mag 32(2):145–163

    Article  Google Scholar 

  10. Comon P (2014) Tensors: a brief introduction. IEEE Signal Proc Mag 31(3):44–53

    Article  Google Scholar 

  11. Criminisi A, Perez P, Toyama K (2003) Object removal by exemplar-based inpainting. In: 2003. Proceedings. 2003 IEEE computer society conference on computer vision and pattern recognition. IEEE, vol 2, pp II–II

  12. De Lathauwer L, De Moor B, Vandewalle J (2000) A multilinear singular value decomposition. SIAM J Matrix Anal Appl 21(4):1253–1278

    Article  MathSciNet  Google Scholar 

  13. De Lathauwer L (2009) A survey of tensor methods. In: 2009. ISCAS 2009. IEEE international symposium on circuits and systems. IEEE, pp 2773–2776

  14. Demanet L, Song B, Chan T (2003) Image inpainting by correspondence maps: a deterministic approach. Appl Comput Math 1100(217-50):99

    Google Scholar 

  15. Ding T, Sznaier M, Camps OI (2007) A rank minimization approach to video inpainting. In: 2007. ICCV 2007. IEEE 11th international conference on computer vision. IEEE, pp 1–8

  16. Drori I, Cohen-Or D, Yeshurun H (2003) Fragment-based image completion. In: ACM transactions on graphics (TOG). ACM, vol 22, pp 303–312

  17. Ebdelli M, Le Meur O, Guillemot C (2015) Video inpainting with short-term windows: application to object removal and error concealment. IEEE Trans Image Process 24(10):3034–3047

    Article  MathSciNet  Google Scholar 

  18. Efros AA, Leung TK (1999) Texture synthesis by non-parametric sampling. In: 1999. The proceedings of the seventh IEEE international conference on computer vision. IEEE, vol 2, pp 1033–1038

  19. Goldfarb D, Qin Z (2014) Robust low-rank tensor recovery: models and algorithms. SIAM J Matrix Anal Appl 35(1):225–253

    Article  MathSciNet  Google Scholar 

  20. Granados M, Tompkin J, Kim K, Grau O, Kautz J, Theobalt C (2012) How not to be seen—object removal from videos of crowded scenes. In: Computer graphics forum, Wiley online library, vol 31, pp 219–228

  21. Grasedyck L, Kressner D, Tobler C (2013) A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen 36(1):53–78

    Article  MathSciNet  Google Scholar 

  22. Guo Q, Zhang C, Zhang Y, Liu H (2016) An efficient svd-based method for image denoising. IEEE Trans Circ Syst Video Technol 26(5):868–880

    Article  Google Scholar 

  23. He K, Sun J (2012) Statistics of patch offsets for image completion. In: Computer vision–ECCV 2012. Springer, pp 16–29

  24. He W, Zhang H, Zhang L, Shen H (2016) Total-variation-regularized low-rank matrix factorization for hyperspectral image restoration. IEEE Trans Geosci Remote Sens 54(1):178–188

    Article  Google Scholar 

  25. Hu W, Tao D, Zhang W, Xie y, Yang Y (2016) The twist tensor nuclear norm for video completion.. In: IEEE transactions on neural networks and learning systems

  26. Ji H, Huang S, Shen Z, Xu Y (2011) Robust video restoration by joint sparse and low rank matrix approximation. SIAM J Imaging Sci 4(4):1122–1142

    Article  MathSciNet  Google Scholar 

  27. Jia J, Tang CK (2003) Image repairing: Robust image synthesis by adaptive nd tensor voting. In: 2003. Proceedings. 2003 IEEE computer society conference on computer vision and pattern recognition. IEEE, vol 1, pp I–I

  28. Jia J, Tai YW, Wu TP, Tang CK (2006) Video repairing under variable illumination using cyclic motions. IEEE Trans Pattern Anal Mach Intell 28(5):832–839

    Article  Google Scholar 

  29. Kilmer ME, Braman K, Hao N, Hoover RC (2013) Third-order tensors as operators on matrices: a theoretical and computational framework with applications in imaging. SIAM J Matrix Anal Appl 34(1):148–172

    Article  MathSciNet  Google Scholar 

  30. Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500

    Article  MathSciNet  Google Scholar 

  31. Komodakis N, Tziritas G (2007) Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans Image Process 16(11):2649–2661

    Article  MathSciNet  Google Scholar 

  32. Kressner D, Steinlechner M, Vandereycken B (2014) Low-rank tensor completion by riemannian optimization. BIT Numer Math 54(2):447–468

    Article  MathSciNet  Google Scholar 

  33. Krishnamurthy A, Singh A (2013) Low-rank matrix and tensor completion via adaptive sampling. In: Advances in neural information processing systems, pp 836–844

  34. Ling CH, Lin CW, Su CW, Chen YS, Liao HYM (2011) Virtual contour guided video object inpainting using posture mapping and retrieval. IEEE Trans Multimed 13(2):292–302

    Article  Google Scholar 

  35. Liu Y, Caselles V (2013) Exemplar-based image inpainting using multiscale graph cuts. IEEE Trans Image Process 22(5):1699–1711

    Article  MathSciNet  Google Scholar 

  36. Lu C, Feng J, Chen Y, Liu W, Lin Z, Yan S (2016) Tensor robust principal component analysis: Exact recovery of corrupted low-rank tensors via convex optimization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5249–5257

  37. Martin CD, Shafer R, LaRue B (2013) An order-p tensor factorization with applications in imaging. SIAM J Sci Comput 35(1):A474–A490

    Article  MathSciNet  Google Scholar 

  38. Newson A, Almansa A, Fradet M, Gousseau Y, Pérez P (2014) Video inpainting of complex scenes. SIAM J Imaging Sci 7(4):1993–2019

    Article  MathSciNet  Google Scholar 

  39. Patwardhan KA, Sapiro G, Bertalmio M (2005) Video inpainting of occluding and occluded objects. In: 2005. ICIP 2005. IEEE international conference on image processing, vol 2. IEEE, pp II–69

  40. Patwardhan KA, Sapiro G, Bertalmío M (2007) Video inpainting under constrained camera motion. IEEE Trans Image Process 16(2):545–553

    Article  MathSciNet  Google Scholar 

  41. Peng Y, Suo J, Dai Q, Xu W (2014) Reweighted low-rank matrix recovery and its application in image restoration. IEEE Trans Cybern 44(12):2418–2430

    Article  Google Scholar 

  42. Pritch Y, Kav-Venaki E, Peleg S (2009) Shift-map image editing. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 151–158

  43. Semerci O, Hao N, Kilmer ME, Miller EL (2014) Tensor-based formulation and nuclear norm regularization for multienergy computed tomography. IEEE Trans Image Process 23(4):1678–1693

    Article  MathSciNet  Google Scholar 

  44. Umeda Y, Arakawa K (2012) Removal of film scratches using exemplar-based inpainting with directional median filter. In: International symposium on communications and information technologies (ISCIT), 2012. IEEE, pp 6–11

  45. Venkatesh MV, Cheung ScS, Zhao J (2009) Efficient object-based video inpainting. Pattern Recogn Lett 30(2):168–179

    Article  Google Scholar 

  46. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  47. Wexler Y, Shechtman E, Irani M (2004) Space-time video completion. In: 2004. CVPR 2004. Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition. IEEE, vol 1, pp I–I

  48. Xu Y, Hao R, Yin W, Su Z (2013) Parallel matrix factorization for low-rank tensor completion. arXiv:13121254

  49. Yoo S, Park RH (2009) Red-eye detection and correction using inpainting in digital photographs. IEEE Trans Consum Electron 55(3):1006–1014

  50. Zhang Z, Aeron S (2017) Exact tensor completion using t-svd. IEEE Trans Signal Process 65(6):1511–1526

    Article  MathSciNet  Google Scholar 

  51. Zhang Z, Ely G, Aeron S, Hao N, Kilmer M (2014) Novel methods for multilinear data completion and de-noising based on tensor-svd. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3842–3849

  52. Zhou X, Yang C, Zhao H, Yu W (2015) Low-rank modeling and its applications in image analysis. ACM Comput Surv (CSUR) 47(2):36

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics, Communication Engineering, National Institute of Technology Calicut, NIT Campus Post Office, Calicut, Kerala, India

    Baburaj M & Sudhish N. George

Authors
  1. Baburaj M
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Sudhish N. George
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Baburaj M.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

M, B., George, S.N. Tensor based approach for inpainting of video containing sparse text. Multimed Tools Appl 78, 1805–1829 (2019). https://doi.org/10.1007/s11042-018-6251-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6251-7

Keywords