Image Inpainting for Large and Irregular Mask Based on Partial Convolution and Cross Semantic Attention

Abstract

Image inpainting is inferring missing pixels in images using known content and surrounding regions. To achieve this task, existing methods have utilized convolutional neural networks and adversarial generative networks. However, these approaches produce blurry and unrealistic completion results when the region of the missing pixels is large and irregular. This is because there is insufficient information to fill the missing region, and the networks fail to utilize known pixel information. Therefore, this study proposes a coarse-to-fine strategy using a pre-trained partial convolution-based encoder-decoder network. Our two-stage image inpainting strategy consists of a coarse inpainting network using partial convolution and a fine inpainting network using cross-attention layers based on an Unsupervised Cross-space Generative Adversarial Network (UCTGAN). In the first stage, the coarse network completes the missing region to feed the clue for inpainting into the following stage. Then, in the second stage, the fine network projects instance images and the coarse clue image into a low-dimensional manifold space and combines low-dimensional spatial features across semantic attention layers. Finally, the generator reconstructs a reconstructed image conditioned on the coarse completion. In the experiments, we qualitatively and quantitatively evaluated generated images in terms of image quality and similarity. As a result, our framework can generate precise completion results for a large and irregular missing region better than the previous methods, such as the methods based on partial convolution or the original UTCGAN.