Multi-scale gradient attention guidance and adaptive style fusion for image inpainting

Abstract

Image inpainting aims to fill in the missing regions of damaged images with plausible content. Existing inpainting methods tend to produce ambiguous artifacts and implausible structures. To address the above issues, our method aims to fully utilize the information of known regions to provide style and structural guidance for missing regions. Specifically, the Adaptive Style Fusion (ASF) module reduces artifacts by transferring visual style features from known regions to missing regions. The Gradient Attention Guidance (GAG) module generates accurate structures by aggregating semantic information along gradient boundary regions. In addition, the Multi-scale Attentional Feature Extraction (MAFE) module extracts global contextual information and enhances the representation of image features. The sufficient experimental results on the three datasets demonstrate that our proposed method has superior performance in terms of visual plausibility and structural consistency compared to state-of-the-art inpainting methods.

Introduction

Image inpainting aims to generate visually credible contents and structures for damaged images, which is widely applied in image processing tasks, such as image restoration [1], face editing [2], and object removal [3], [4]. Although breakthrough progress has been made in image inpainting, it still faces great challenges in generating a consistent visual style and clear textural details.

Existing image inpainting methods can be divided into two categories including traditional methods and learning-based methods. Traditional inpainting methods typically use the local texture similarity to perform pixel interpolation, and that include diffusion-based [5], [6], [7] and patch-based [8], [9], [10], [11] methods. However, the traditional methods lack a high-level understanding for image content and structure, and it is difficult to generate meaningful semantic content. The learning-based methods use convolutional neural networks to generate high-level semantic features to fill in missing content. However, they always produce distorted structures and blurred textures in the face of large missing regions or complex contexts.

With the deepening study of learning-based inpainting methods, the researchers found that the regular convolution cannot distinguish between missing foreground pixels and known background pixels, which will lead to the drift of the mean and variance of features in the normalization process. To solve the problem, Partial convolution (Pconv) [12] was proposed instead of regular convolution, where only background pixels are convolved. To address the limitations of partial convolution, Deepfillv2 [13] applied gated convolution and channel soft masks to treat different background pixels equally to improve inpainting performance. These methods still cannot intrinsically solve the mean and variance shift problems during normalization. Regional Normalization (RN) [14] can normalize missing foreground region and background region respectively, which replace instance normalization (IN) [15] in convolutional networks. However, since the missing foreground and background regions are processed independently, the semantic information of the background cannot guide the missing regions construction. As a result, there are no visual style characteristics transferred from the background to foreground region, and the edge connection transition between two regions is not smooth.

Recently, structural information has been increasingly studied to guide image inpainting and has play an important role in generating reasonably realistic images of missing content, including edges [16], [17], salient map [18], [19], and gradient maps [20]. The key of assisting is to make full use of relatively accurate and complete structures to guide the network to generate missing content. Edge Connect (EC) [16] proposed a two-stage model, including an edge generator and an image generator, with the relatively complete edge map predicted in the first stage serving as a condition for the second stage. Foreground-aware inpainting [18] embedded a salient map to guide the generator to reconstruct plausible content. Compared with edge and salient maps, the gradient map contains richer texture information. Zhang [20] designed a Gradient Augmented Inpainting Network (GAIN) used an image gradient map instead of an edge map and proposed a dual-branch structure for completing the image and gradient map separately. However, the above methods simply concatenate image features and structural information to guide the network during network inference. The network will weaken and forget structural information with the network deepen and structure sparsity increasing, which is difficult to fully understand structural information.

In the process of network reasoning and inpainting, the missing regions and the known regions lack the necessary visual style connection. The Adaptive Style Fusion Residual Block (ASF ResBlock) transmits statistical information internally and performs style transfer in the feature space, effectively combining the content of the former with the style of the latter, thereby reducing artifacts significantly. Furthermore, the Gradient Attention Guidance (GAG) module employs an attention mechanism to encourage information sharing between image feature maps and gradient feature maps in the network to generate accurate semantic structures. Finally, the Multi-scale Attentional Feature Extraction (MAFE) adopts the structure of a spatial pyramid to capture and enhance the deep features from different receptive fields of GAG modules. Therefore, the network alleviates artifacts and inaccurate structures under the interaction of the above modules. The main contributions of this paper are summarized as follows:

• The ASF module introduces the perspective of style transfer, which can explicitly describe style characteristics from known region and transfer to the missing region. It can effectively reduce artifacts and color differences through adaptive fusion between known and missing regions.

• We propose a multi-scale gradient guidance framework, which applies gradient attention to guide the network generates trustworthy visual structures and textures on multi-scale layers of decoder.

• Experiments on three public datasets, Paris Street View [21], CelebA-HQ [22], and Places2 [23], demonstrate qualitatively and quantitatively the significant performance of the proposed method.