Abstract
Automatic image completion can provide convenient editing of consumer images. Most image completion methods find the best patch locally and then copy it to the hole region for texture inpainting. Since the best patch is fixed in size, it is hard to adapt these methods efficiently either to various patterns or to content synthesis. Meanwhile, salient image structures can be estimated and propagated to guide the texture inpainting process for more plausible results. This paper presents a novel image completion method using structure guided texture inpainting. We do not require any interactions to achieve automatic two-stage image completion. In the structure completion stage, the salient structures around the hole region are detected, and then the missing structure curves are completed with Euler spiral. The textures along the structure curves are completed through belief propagation. In the texture inpainting stage, we propose multi-scale patches and global optimization to inpaint the remaining texture in the hole regions guided by the completed structures. First, with defined patch sizes, the hole region is divided into lattice patches, making it possible for multiple patch sizes to render multiscale descriptions of this image. A multi-scale graph is then built for the hole region and formulated as a posterior probability model. Second, using a simulated annealing based Markov chain Monte Carlo method, an inference algorithm is designed to find a global optimization solution for the posterior probability model. The experiments show that our method can automatically complete the hole region and preserve plausible structure shapes of existing ones in various scenarios. The texture inpainting results are more convincing with guidance from the completed structures, and our method can guarantee and accelerate convergence of the global optimization.
Similar content being viewed by others
References
Ting H, Chen S F, Liu J Z, et al. Image inpainting by global structure and texture propagation. In: Proceedings of the 15th international Conference on Multimedia. New York: ACM, 2007. 517–520
Chen X W, Zhou B, Xu F, et al. Automatic image completion with structure propagation and texture synthesis. Int J Softw Eng Knowl Eng, 2010, 20: 1097–1117
Chen X W, Xu F. Automatic image inpainting by heuristic texture and structure completion. In: Proceedings of the 16th International Multimedia Modeling Conference, Chongqing, 2010. 110–119
Sun J, Yuan L, Jia J Y, et al. Image completion with structure propagation. ACM Trans Graph, 2005, 24: 861–868
Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 2007. 417–424
Chan T F, Kang S H, Shen J H. Euler’s elastica and curvature-based inpainting. SIAM J Appl Math, 2003, 63: 564–592
Bornemann D F, Marz T. Fast image inpainting based on coherence transport. J Math Imaging Vis, 2007, 28: 259–278
Criminisi A, Pérez P, Toyama K. Object removal by exemplar-based inpainting. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Madison, 2003. II:721–II:728
Criminisi A, Pérez P, Toyama K. Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process, 2004, 13: 1200–1212
Drori I, Cohen-Or D, Yeshurun H. Fragment-based image completion. ACM Trans Graph, 2003, 22: 303–312
Komodakis N, Tziritas G. Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans Image Process, 2007, 16: 2649–2661
Cao F, Gousseau Y, Masnou S, et al. Geometrically guided exemplar-based inpainting. SIAM J Imaging Sci, 2011, 4: 1143–1179
Cho T S, Avidan S, Freeman W T. The patch transform. IEEE Trans Pattern Anal Mach Intell, 2010, 32: 1489–1501
Wexler Y, Shechtman E, Irani M. Space-time completion of video. IEEE Trans Pattern Anal Mach Intell, 2007, 29: 463–476
Pritch Y, Kav-Venaki E, Peleg S. Shift-map image editing. In: Proceedings of International Conference on Computer Vision (ICCV), Kyoto, 2009. 151–158
Bertalmio M, Vese L, Sapiro G, et al. Simultaneous structure and texture image inpainting. IEEE Trans Image Process, 2003, 12: 882–889
Jia J Y, Tang C K. Image repairing: robust image synthesis by adaptive ND tensor voting. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Madison, 2003. I: 643–I:650
Pavic D, Schönefeld V, Kobbelt L. Interactive image completion with perspective correction. Visual Comput, 2006, 22: 671–681
Hays J, Efros A A. Scene completion using millions of photographs. ACM Trans Graph, 2007, 26: 4
Barnes C, Shechtman W, Finkelstein A, et al. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graphics, 2009, 28: 24
Arbelaez P, Maire M, Fowlkes C, et al. Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell, 2011, 33: 898–916
Mordohai P. A perceptual organization approach for figure completion, binocular and multiple-view stereo and machine learning using tensor voting. Dissertation for the Doctoral Degree. University of Southern California, 2005
Kimia B, Frankel I, Popescu A M. Euler spiral for shape completion. Int J Comput Vis, 2003, 54: 159–182
Muja M, Lowe D G. Fast approximate nearest neighbors with automatic algorithm configuration. In: Proceedings of International Conference on Computer Vision Theory and Applications, Lisboa, 2009. 331–340
Broadway J Y, Yedidia J S, Freeman WT, et al. Generalized belief propagation. In: Proceedings of Neural Information Processing Systems Conference, Denver, 2000. 689–695
Metropolis N, Rosenbluth A W, Rosenbluth M N, et al. Equation of state calculations by fast computing machines. J Chem Phys, 1953, 21: 1087–1092
Tu Z W, Zhu S C. Image segmentation by data-driven Markov chain Monte Carlo. IEEE Trans Pattern Anal Mach Intell, 2002, 24: 657–673
Huang T J, Tian Y H, Li J, et al. Salient region detection and segmentation for general object recognition and image understanding. Sci China Inf Sci, 2011, 54: 2461–2470
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Chen, X., Zhou, B., Guo, Y. et al. Structure guided texture inpainting through multi-scale patches and global optimization for image completion. Sci. China Inf. Sci. 57, 1–16 (2014). https://doi.org/10.1007/s11432-012-4772-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11432-012-4772-7