Skip to main content
SpringerLink
Log in

Structure-preserving image completion with multi-level dynamic patches

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, we present a novel structure-preserving image completion approach equipped with dynamic patches. We formulate the image completion problem into an energy minimization framework that accounts for coherence within the hole and global coherence simultaneously. The completion of the hole is achieved through iterative optimizations combined with a multi-scale solution. In order to avoid abnormal structure and disordered texture, we utilize a dynamic patch system to achieve efficient structure restoration. Our dynamic patch system functions in both horizontal and vertical directions of the image pyramid. In the horizontal direction, we conduct a parallel search for multi-size patches in each pyramid level and design a competitive mechanism to select the most suitable patch. In the vertical direction, we use large patches in higher pyramid level to maximize the structure restoration and use small patches in lower pyramid level to reduce computational workload. We test our approach on massive images with complex structure and texture. The results are visually pleasing and preserve nice structure. Apart from effective structure preservation, our approach outperforms previous state-of-the-art methods in time consumption.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ashikhmin, M.: Synthesizing natural textures. In: Proceedings of the 2001 Symposium on Interactive 3D graphics, pp. 217–226. ACM (2001)

  2. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10(8), 1200–1211 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  3. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24:1–24:11 (2009)

    Article  Google Scholar 

  4. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2002)

  5. Bugeau, A., Bertalmio, M., Caselles, V., Sapiro, G.: A comprehensive framework for image inpainting. IEEE Trans. Image Process. 19(10), 2634–2645 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  6. Burt, P.J., Adelson, E.H.: A multiresolution spline with application to image mosaics. ACM Trans. Graph. 2(4), 217–236 (1983)

    Article  Google Scholar 

  7. Cai, N., Su, Z., Lin, Z., Wang, H., Yang, Z., Ling, B.W.K.: Blind inpainting using the fully convolutional neural network. In: The Visual Computer pp. 1–13 (2015)

  8. Chan, T.F., Kang, S.H., Shen, J.: Euler’s elastica and curvature based inpainting. SIAM J. Appl. Math. 63(2), 491–506 (2010)

    MathSciNet  MATH  Google Scholar 

  9. Chen, T., Cheng, M.M., Tan, P., Shamir, A., Hu, S.M.: Sketch2photo: internet image montage. ACM Trans. Graph. 28(5), 124 (2009)

    Google Scholar 

  10. Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 721–728. IEEE (2003)

  11. Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31(4), 82:1–82:10 (2012)

    Article  Google Scholar 

  12. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 341–346. ACM (2001)

  13. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1033–1038. IEEE (1999)

  14. Ford Jr., L.R., Fulkerson, D.R.: Flows in Networks. Princeton University Press, Princeton (2015)

    MATH  Google Scholar 

  15. Gracias, N., Mahoor, M., Negahdaripour, S., Gleason, A.: Fast image blending using watersheds and graph cuts. Image Vis. Comput. 27(5), 597–607 (2009)

    Article  Google Scholar 

  16. Hao, C., Chen, Y., Wu, W., Wu, E.: Image completion with perspective constraint based on a single image. Sci China Inf Sci 58(9), 1–12 (2015)

    Article  Google Scholar 

  17. Hua, M., Wang, W.: Effective structure restoration for image completion using internet resources. Vis. Comput 31(6–8), 1113–1122 (2015)

    Article  Google Scholar 

  18. Ignácio, U.A., Jung, C.R.: Block-based image inpainting in the wavelet domain. Vis. Comput. 23(9–11), 733–741 (2007)

    Article  Google Scholar 

  19. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 107:1–107:14 (2017)

    Google Scholar 

  20. Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.: Graphcut textures: image and video synthesis using graph cuts. ACM Trans. Graph. 22(3), 277–286 (2003). doi:10.1145/882262.882264

    Article  Google Scholar 

  21. Levin, A., Zomet, A., Weiss, Y.: Learning how to inpaint from global image statistics. In: Proceedings of IEEE International Conference on Computer Vision, vol. 1, pp. 305–312 (2003)

  22. Liang, L., Liu, C., Xu, Y.Q., Guo, B., Shum, H.Y.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. 20(3), 127–150 (2001)

    Article  Google Scholar 

  23. Liu, B., Li, P., Sheng, B., Wu, E.: Image completion with dynamic patches. In: Proceedings of the Computer Graphics International Conference, p. 3. ACM (2017)

  24. Nill, N.B., Bouzas, B.: Objective image quality measure derived from digital image power spectra. Opt. Eng. 31(4), 813–825 (1992)

    Article  Google Scholar 

  25. Qin, X., Shen, J., Mao, X., Li, X., Jia, Y.: Robust match fusion using optimization. IEEE Trans. Cybern. 45(8), 1549–1560 (2015)

  26. Qin, X., Shen, J., Mao, X., Li, X., Jia, Y.: Structured-patch optimization for dense correspondence. IEEE Trans. Multimed. 17(3), 295–306 (2015)

    Article  Google Scholar 

  27. Shen, J., Jin, X., Zhou, C., Wang, C.C.: Gradient based image completion by solving the Poisson equation. Comput. Graph. 31(1), 119–126 (2007)

    Article  Google Scholar 

  28. Shen, J., Zhao, Y., Yan, S., Li, X., et al.: Exposure fusion using boosting Laplacian pyramid. IEEE Trans. Cybern. 44(9), 1579–1590 (2014)

    Article  Google Scholar 

  29. Sunkavalli, K., Johnson, M.K., Matusik, W., Pfister, H.: Multi-scale image harmonization. ACM Trans. Graph. 29(4), 125:1–125:10 (2010). doi:10.1145/1778765.1778862

    Article  Google Scholar 

  30. Wei, L.Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 479–488. ACM Press/Addison-Wesley Publishing Co. , New York (2000)

  31. Wexler, Y., Shechtman, E., Irani, M.: Space–time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007)

    Article  Google Scholar 

  32. Whitaker, R.T.: A level-set approach to image blending. IEEE Trans. Image Process. 9(11), 1849–1861 (2000)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors would like to thank all reviewers for their helpful suggestions and constructive comments. The work was supported by the National Natural Science Foundation of China (No. 61632003), and a Grant from the Research Grants Council of Hong Kong (No. 28200215).

Author information

Authors and Affiliations

  1. Department of Mathematics and Information Technology, The Education University of Hong Kong, Hong Kong, Hong Kong

    Bowen Liu

  2. Faculty of Information Technology, Macau University of Science and Technology, Macao, Macao

    Ping Li

  3. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  4. School of Computer Science and Engineering, South China University of Technology, Guangzhou, China

    Yongwei Nie

  5. University of Macau and SKLCS-IOSCAS, Macao, Macao

    Enhua Wu

Authors
  1. Bowen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bin Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yongwei Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Enhua Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ping Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, B., Li, P., Sheng, B. et al. Structure-preserving image completion with multi-level dynamic patches. Vis Comput 35, 85–98 (2019). https://doi.org/10.1007/s00371-017-1454-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-017-1454-x

Keywords

Search

Navigation