A color-gradient patch sparsity based image inpainting algorithm with structure coherence and neighborhood consistency
Introduction
The recovery of lost or corrupted parts of the image data, which is called image inpainting or image completion, is an important research subject of computer vision and image processing. Image inpainting has been widely studied and applied in various fields including video inpainting [1], art conservation and digital restoration [2], etc. Nowadays, most image inpainting methods can be roughly divided into two categories, the diffusion-based inpainting methods and the exemplar-based inpainting methods.
The diffusion-based methods are the fundamental approaches built on partial differential equation (PDE) or total variation (TV), where the missing area is filled gradually via diffusing the image information from source area into missing area. The first inpainting algorithm proposed by Bertalmio et al. [3] is a diffusion-based approach, where the missing area was filled by diffusing the information along the isophote direction. Since then, the diffusion-based algorithms have achieved remarkable performances. Bertalmio et al. [4] introduced the Navier–Stokes equation in fluid dynamics into image inpainting task [4]. Chan and Shen [5] brought in the TV model to recover the damaged area. As the TV model cannot hold the connectivity, later they proposed an image inpainting model based upon the curvature-driven diffusion method [6]. Some other PDE models are also applied to the task of image inpainting [7], [8], [9], [10], [11]. The diffusing-based algorithms have achieved excellent results for filling the non-textured or small damaged area. However, the diffusion-based approaches implicitly assume the content of the missing region is smooth or nontextured, they incline to bring in smooth effect in textured area or large damaged region.
The exemplar-based algorithms implement the inpainting task at a patch level and they have got plausible results for inpainting relatively larger missing region. The idea derived from the texture synthesis technique proposed by Efors and Leung [12], where the texture was synthesized by sampling the best match patch from the source region and later it was extended by Harrison [13] for image inpainting. As natural images consist of various structures and textures, the texture synthesis technique cannot restore the damaged region with complicated textures and relatively larger missing structures. Criminisi et al. [14], [15] proposed an outstanding exemplar-based inpainting algorithm for repairing larger missing region with composite structures and textures. In the algorithm, priority function was defined to decide the filling order that encourages to fill structure part, and sum of squared difference (SSD) was applied to search the most similar patch in the source region for the filling missing region. The algorithm first formulated the procedures of exemplar-based inpainting algorithms and afterwards many approaches follow this route.
Many approaches improve the performance of Criminisi's algorithm from two sides, the priority function and the patch inpainting. Regarding priority function, Cheng et al. [16] redefined the priority function. Sun et al. [17] repaired the structure part preferentially along the manually added auxiliary line. Wu and Ruan [18] used a cross-isophote diffusion item to decide the filling order. Structure sparsity was defined in Xu and Sun [19] to determine the filling order and later it was combined with a modified confidence term to compute priority in Hesabi and MahdaviAmiri [20]. Jemi Florinabel et al. [21] developed a method using the DCT coefficients of a patch located at fill-front to decide the filling order. Le Meur et al. [22] investigated on using structure tensors to determine the filling order. Wang et al. [23] applied the D–S evidence theory to compute priority. Zhang and Lin [24] designed a priority scheme based upon color distribution.
As for the patch inpainting, local and non-local methods exist in the literature. The former methods searched one best patch from the source region to fill the target patch, such as the works in [14], [15], [16], [17], [18], [20], [21], [22], [23], [24]. Though it is simple, it is easy to result in the block effect and the seam effect. The latter methods use a linear combination of several candidate patches to fill the missing region. For example, Wong and Orchard [25] applied the nonlocal image information from multiple patches to fill the target area. Bin et al. [26] considered image inpainting as sequential incomplete signal recovery under the assumption that every image patch could be sparsely represented over a redundant dictionary. Wohlberg [27] filled the missing region with a sparse linear combination of example blocks extracted from the target image or the external training image set. Xu and Sun [19] sparsely represented the missing region using multiple candidate patches with local patch consistency constraint in color space. In addition, except for using SSD to search patches in most algorithms, Jemi Florinabel et al. [21] considered the edge information and Le Meur et al. [22] applied structure tensors to search candidate patches.
Although the exemplar-based algorithms with non-local methods lessen the block effect and the seam effect, there are still some defects that limit their applications. This paper proposes a color-gradient patch sparsity (CGPS) based inpainting algorithm with structure coherence and neighborhood consistency, where the CGPS is reflected in color-gradient structure sparsity (CGSS) and patch sparse representation. Specifically, to maintain structure coherence, the CGSS is designed based on a weighted color-gradient distance (WCGD) to determine the filling order of all patches located at fill-front. For maintaining neighborhood consistency, the WCGD is applied to search multiple candidate patches and an optimization equation with local patch consistency constraints in color and gradient spaces is constructed to obtain sparse linear combination coefficients of candidate patches. Furthermore, to reduce computational complexity, candidate patches are searched in a limited region that is adaptively decided by the CGSS value of target patch. Compared with existing patch sparsity based inpainting approaches that only use color information, the proposed algorithm combines both color and gradient information, which ensures a better maintenance of structure coherence, texture clarity and neighborhood consistency. Moreover, through limiting the search region size via the CGSS, the inpainting efficiency can be significantly improved. Experimental results on scratch removal, text removal, block removal and object removal demonstrate the advantages of the proposed approach.
The remainder of this paper is organized as follows. Section 2 presents the proposed image inpainting algorithm, including filling order determination, candidate patches search and patch sparse representation. Section 3 presents the experimental results on a variety of images, which demonstrates the superior performance over five existing algorithms. Conclusions are made in Section 4.
Section snippets
Inpainting based on color-gradient patch sparsity
This section expounds the proposed CGPS based image inpainting algorithm with structure coherence and neighborhood consistency. The algorithm procedure is shown in Fig. 1. Given the degraded image I and missing region , the fill-front is determined and confidence is initialized as for and for . In Step 2, the CGSS is defined based on the WCGD to measure the confidence of a patch located at the structure region and further applied to determine the filling order.
Experiments
In this section, extensive experiments are conducted to verify the performance of our proposed image inpainting algorithm. Firstly, the effects of the weight coefficient , the number M of candidate patches for sparse representation and amplification factor λ on inpainting performance are analyzed. Then the performance of our algorithm is compared with Telea's [7], Barnes's [28], Zhang's [24], Shen's [26] and Xu's [19] performance on block, scratch, text and object removal. In our algorithm,
Conclusions
This paper proposes a CGPS based image inpainting algorithm, which utilizes color and gradient information in the inpainting process, to maintain structure coherence and neighborhood consistency for block, scratch, text and object removal. The novelty of this work includes four aspects: (1) the CGSS is applied to obtain a robust filling order to maintain structure coherence; (2) the WCGD is used to search candidate patches; (3) local search is adopted to enhance algorithm efficiency; and (4)
Acknowledgments
This work is supported in part by the National Natural Science Foundation of China (Grant no. 61373180), the Fundamental Research Funds for the Central Universities (Grant nos. SWJTU09CX039, SWJTU10CX09), and “the 2014 Doctoral Innovation Funds of Southwest Jiaotong University” and “the Fundamental Research Funds for the Central Universities”.
References (28)
- et al.
Nonlocal video denoising, simplification and inpainting using discrete regularization on graphs
Signal Process.
(2010) - et al.
Crack detection and inpainting for virtual restoration of paintingsthe case of the Ghent altarpiece
Signal Process.
(2013) - et al.
Nontexture inpainting by curvature-driven diffusions
J. Vis. Commun. Image Represent.
(2001) - et al.
Combined frequency and spatial domain-based patch propagation for image completion
Comput. Graph.
(2011) - M. Bertalmio, G. Sapiro, V. Caselles, C. Ballester, Image inpainting, in: Proceedings of ACM International Conference...
- M. Bertalmio, A.L. Bertozzi, G. Sapiro, Navier–stokes, fluid dynamics, and image and video inpainting, in: Proceedings...
- et al.
Mathematical models for local nontexture inpaintings
SIAM J. Appl. Math.
(2001) An image inpainting technique based on the fast marching method
J. Graph. Tools
(2004)- et al.
Fast dual minimization of the vectorial total variation norm and applications to color image processing
Inverse Probl. Imaging
(2008) - et al.
Orientation-matching minimization for image denoising and inpainting
Int. J. Comput. Vis.
(2011)
Nonlocal Mumford–Shah regularizers for color image restoration
IEEE Trans. Image Process.
A primal–dual method for total variation-based wavelet domain inpainting
IEEE Trans. Image Process.
Cited by (12)
Sparse recovery of missing image samples using a convex similarity index
2018, Signal ProcessingCitation Excerpt :The missing sample recovery problem arises in many applications in the literature of signal processing [1]. It is also known as inpainting in the context of audio and image processing [2–6]. Sometimes the problem is known as signal recovery from incomplete data [7].
A global and local feature weighted method for ancient murals inpainting
2020, International Journal of Machine Learning and CyberneticsNon-local SVD denoising of MRI based on sparse representations
2020, Sensors (Switzerland)A comprehensive review on state-of-the-art image inpainting techniques
2020, Scalable ComputingStructured sparse image inpainting algorithm based on offset difference
2019, Guangdianzi Jiguang/Journal of Optoelectronics Laser