Skip to main content
Log in

Low-rank image completion with entropy features

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a novel method to complete the images or textures with the property of low rank. Our method leverages saliency detection with two entropy features to estimate initial corrupted regions. Then an iterative optimization model for low-rank and sparse errors recovery is designed to complete the corrupted images. Our iterative model can improve the initial corrupted regions and generate accurate and continuous corrupted regions via fully connected CRFs. By introducing a F-norm term in our model to absorb small noise, we can generate completed images which are more precise and have lower rank. Experiments indicate that our method introduces less local distortions than example-based methods for images with regular structures. It is also superior to the previous low-rank image completion method especially when the images contain low-rank corrupted regions. Furthermore, we show that the entropy features benefit the existing saliency detection methods too.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ballester, C., Caselles, V., Verdera, J., Bertalmio, M., Sapiro, G.: A variational model for filling-in gray level and color images. In: IEEE International Conference on Computer Vision, pp. 10–16 (2001)

  2. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 341–352 (2009)

    Article  Google Scholar 

  3. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417–424 (2000)

  4. Bertalmio, M., Vese, L., Sapiro, G., Osher, S.: Simultaneous structure and texture image inpainting. IEEE Trans. Image Process. 12(8), 882–889 (2003)

    Article  Google Scholar 

  5. Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9(6), 717–772 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  6. Criminisi, A., Pérez, P., Toyama, K.: Object removal by exemplar based inpainting. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 721–728 (2003)

  7. Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31(4), 13–15 (2012)

    Article  Google Scholar 

  8. Drori, I., Cohen-Or, D., Yeshurun, H.: Fragment-based image completion. ACM Trans. Graph. 22(3), 303–312 (2003)

    Article  Google Scholar 

  9. Efros, A., Leung, T. K.: Texture synthesis by non-parametric sampling. In: The Proceedings of the 7th IEEE International Conference on Computer Vision (1999)

  10. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 341–346 (2001)

  11. He, K., Sun, J.: Image completion approaches using the statistics of similar patches. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2423–2435 (2014)

    Article  Google Scholar 

  12. Huang, J.B., Kang, S.B., Ahuja, N., Kopf, J.: Image completion using planar structure guidance. ACM Trans. Graph. 33(4), 1–10 (2014)

    Google Scholar 

  13. Krähenbühl, P., Koltun, V.: Efficient inference in fully connected CRFs with gaussian edge potentials. In: Computer Vision and Pattern Recognition, pp. 109–117 (2012)

  14. Liang, X., Ren, X., Zhang, Z., Ma, Y.: Repairing sparse low-rank texture. Eur. Conf. Comput. Vis. 7576(1), 482–495 (2012)

    Google Scholar 

  15. Pritch, Y., Kav-Venaki, E., Peleg, S.: Shift-map image editing. In: 2009 IEEE 12th International Conference on Computer Vision (2009)

  16. Qin, Y., Lu, H.C., Xu, Y.Z., Wang, H.: Saliency detection via cellular automata. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)

  17. Ting, H., Chen, S., Liu, J., Tang, X.: Image inpainting by global structure and texture propagation. In: Proceedings of the 15th international conference on Multimedia, pp. 517–520 (2007)

  18. Wexler, Y., Shechtman, E., Irani, M.: Spacetime completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007)

    Article  Google Scholar 

  19. Yang, C., Yang, M.S., Zhang, L.H., Lu, H.C.: Saliency detection via graph-based manifold ranking. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3166–3173 (2009)

  20. Zhou, Z., Li, X., Wright, J., Candes, E., Ma, Y.: Stable principal component pursuit. In: IEEE International Symposium on Information Theory Proceedings, 2010 41(3), 1518–1522 (2010)

Download references

Acknowledgments

We would like to thank all the reviewers for their valuable comments and feedback. Junjie Cao is supported by the NSFC (Nos. 61363048, 61262050) and the Fundamental Research Funds for the Central Universities (No. DUT16QY02). Xiuping Liu is supported by the NSFC (No. 61370143). Weiming Wang is supported by Fundamental Research Funds for the Central Universities (No. DUT16RC(3)061). Jun Wang is supported by NSFC (No. 61402224), Fundamental Research Funds for the Central Universities (Nos. NE2014402, NE2016004), Natural Science Foundation of Jiangsu Province (BK2014833), NUAA Fundamental Research Funds (NS2015053) and Jiangsu Specially-Appointed Professorship.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xiuping Liu.

Appendix: LADM Algorithm

Appendix: LADM Algorithm

We solve the optimization model via the LADM algorithm. We first construct the augmented Lagrangian function as follows:

$$\begin{aligned} L_{\mu }(A,S,Z,Y)= & {} \Vert L\Vert _{*}+\lambda \Vert S\Vert _{1}+\frac{\beta }{2} \Vert Z\Vert _{F}^{2}\nonumber \\&+\,\langle Y, P_{\bar{\varOmega }}(L+S+F)-P_{\bar{\varOmega }}(D)\rangle \nonumber \\&+\,\frac{\mu }{2}\Vert P_{\bar{\varOmega }}(L+S+F)-P_{\bar{\varOmega }}(D)\Vert _{F}^{2}.\nonumber \\ \end{aligned}$$
(11)

Solving Eq. (11) is equal to solving the following LADM iteration problem:

$$\begin{aligned} (L_{k+1},S_{k+1},Z_{k+1})= & {} \arg \min _{L,S,Z} L_{\mu _{k}}(L,S,Z,Y_{k}),\nonumber \\ Y_{k+1}= & {} Y_{k}+\mu _{k}\cdot P_{\bar{\varOmega }}(L+S+Z-D),\nonumber \\ \mu _{k+1}= & {} \rho \cdot \mu _{k}, \end{aligned}$$
(12)

Here, \(\rho >1\) is a constant value and the value can be tuned. Here we use the fixed value 1.5 for the whole iterative process.

We now focus on efficiently solving the above iterative scheme. In general, it is computationally expensive to minimize over all the variables L, S and Z simultaneously [14]. The solutions can be achieved by an alternating minimizing strategy:

$$\begin{aligned} L_{k+1}= & {} \arg \min _{L} L_{\mu _{k}}(L,S_{k},Z_{k},Y_{k}) , \end{aligned}$$
(13)
$$\begin{aligned} S_{k+1}= & {} \arg \min _{S} L_{\mu _{k}}(L_{k},S,Z_{k},Y_{k}) , \end{aligned}$$
(14)
$$\begin{aligned} Z_{k+1}= & {} \arg \min _{Z} L_{\mu _{k}}(L_{k},S_{k},Z,Y_{k}) , \end{aligned}$$
(15)

So the optimization can be converted to some subproblems (13), (14) and (15). Those subproblems can be easily solved, and the solutions are close to the solutions of the original problem. In turns, we solve (13), (14) and (15) as follows:

$$\begin{aligned} L_{k+1}= & {} \arg \min _{L} \Vert L\Vert _{*}+\frac{\mu _{k}}{2}\left\| L+S_{k}+Z_{k}-D+\frac{1}{\mu _{k}}Y_{k}\right\| _{F}^{2}\nonumber \\= & {} \mathcal {S}_{\frac{1}{\mu _{k}}}(L-A_{k}), \end{aligned}$$
(16)

and

$$\begin{aligned} S_{k+1}= & {} \arg \min _{S} \lambda \Vert S\Vert _{1}+\frac{\mu _{k}}{2}\left\| L_{k}+S+Z_{k}-D+\frac{1}{\mu _{k}}Y_{k}\right\| _{F}^{2}\nonumber \\= & {} \mathcal {T}_{\frac{\lambda }{\mu _{k}}}(S-B_{k}), \end{aligned}$$
(17)

and

$$\begin{aligned} Z_{k+1}= & {} \arg \min _{Z} \frac{\beta }{2}\Vert Z\Vert _{F}^{2}\nonumber \\&+\,\frac{\mu _{k}}{2}\left\| L_{k}+S_{k}+Z-D+\frac{1}{\mu _{k}}Y_{k}\right\| _{F}^{2}\nonumber \\= & {} \frac{\mu _{k}}{\beta +\mu _{k}}(P_{\bar{\varOmega }}(D-L_{k}-S_{k})+Y_{k}). \end{aligned}$$
(18)

\(\mathcal {T}_{\eta }(x)\) is a soft-thresholding operator defined as:

$$\begin{aligned} \mathcal {T}_{\eta }(x)= sgn(x)\cdot \max (x-\eta ) \end{aligned}$$
(19)

and \(\mathcal {S}_{\eta }(X)\) is defined as:

$$\begin{aligned} \mathcal {S}_{\eta }(X)= UT_{\eta }(\varSigma )V^{T}, \end{aligned}$$
(20)

in which \(U\varSigma V^{T}\) is the SVD of X and \(\mathcal {T}_{\eta }(\cdot )\) is used for each pixel of matrix \(\varSigma \). The \(A_{k}\) and \(B_{k}\) in the above equations are, respectively, presented as:

$$\begin{aligned} A_{k}= & {} P_{\bar{\varOmega }}(L_{k}+S_{k}+Z_{k}-D)-\frac{1}{\mu _{k}} Y_{k}, \end{aligned}$$
(21)
$$\begin{aligned} B_{k}= & {} P_{\bar{\varOmega }}(L_{k}+S_{k}+Z_{k}-D)-\frac{\lambda }{\mu _{k}} Y_{k}. \end{aligned}$$
(22)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, J., Zhou, J., Liu, X. et al. Low-rank image completion with entropy features. Machine Vision and Applications 28, 129–139 (2017). https://doi.org/10.1007/s00138-016-0811-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-016-0811-5

Keywords

Navigation