Image inpainting based on deep learning: A review

Highlights

• Classify image inpainting methods based on deep learning from a new perspective.

• Summarizes the current research status in the field of image inpainting.

• Select some representative image inpainting methods for comparison and analysis.

• The research direction and development trend of image inpainting are prospected.

Abstract

Image inpainting aims to restore the pixel features of damaged parts in incomplete image and plays a key role in many computer vision tasks. Image inpainting technology based on deep learning is a major current research hotspot. To deeply understand related methods and technologies, this article combs and summarizes the latest research status in this field. Firstly, we summarize inpainting methods of different types of neural network structure based on deep learning, then analyze and study important technical improvement mechanisms. In addition, various algorithms are comprehensively reviewed from the aspects of model network structure and restoration methods. And we select some representative image inpainting methods for comparison and analysis. Finally, the current problems of image inpainting are summarized, and the future development trend and research direction are prospected.

Introduction

Image inpainting is a technology that aims to restore the damaged part of pixel features in the incomplete image, and then reconstruct and generate high-quality and deep semantic approximation to the original image. In recent years, the implementation of artificial intelligence scientific research and deep learning related technologies has achieved vigorous development along with the substantial increase in computer computing power, which has brought important promotion and improvement to science technology and the quality of human life. Image inpainting technology based on deep learning plays an important role in many computer vision applications [1] (such as target removal in image editing technology, old photo restoration, defective cultural relics and font restoration, facial restoration, etc.) and has become a major research hotspot in computer vision.

In traditional image inpainting technology, the related methods are mostly machine learning algorithms based on statistical probability. Marcelo Bertalmio et al. [2] proposed a Markov Random Field (MRF) image inpainting algorithm on the basis of structure migration mapping statistics and multi-directional features for large-scale damaged image restoration. The inpainting algorithm is mainly used for target removal, which can better maintain the continuity of repaired image structure and the consistency between adjacent pixels. Shen et al. [3] proposed an improved sparse representation inpainting algorithm in the light of similar matching blocks, which achieved good restoration effects in the inpainting of color damaged images with multiple damaged shapes in a small area. Tsai et al. [4] proposed a matrix completion method with automatic rank estimation based on low-rank decomposition is used to extract restored high-quality images from images with different degrees of low sampling rate. Considering the above, Bertalmio et al. [5] introduced conjugate gradient method based on riemannian manifold to optimize matrix completion and combined convolution neural network to preprocess sample images. The method of block processing is adopted to further save operation space and improve the quality of restored images. These methods have made improvements on traditional machine learning algorithms to promote image inpainting effects in different ways. However, compared with the deep learning image inpainting technology, the restored images generated by the traditional methods when processing large image data in damaged areas often lack semantic consistency and texture structure coherence.

Around 2014, with the rise of deep learning, image inpainting technology has been deeply applied in the field of computer vision. Many researchers have continuously carried out in-depth research on the problem of high-quality image inpainting at the level of generating semantic understanding [6], [7], [8], and then a large number of classical image inpainting methods based on deep learning have emerged [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], Some scholars have also summarized the work in this field. Recent work [1], [19] summarized the image inpainting technology based on deep learning. Omar Elharrouss et al. [1] divided the image inpainting model methods proposed in some classic papers into three categories from a global perspective, namely, sequence-based methods, CNN-based methods and GAN-based methods. Qiang et al. [19] summarized the main image inpainting methods based on deep learning in recent years and classified the existing methods into three network structure types of image inpainting methods based on convolutional autoencoder network, generative adversarial network and recurrent neural network according to the inpainting network structure.

In the past few years, deep learning has made great breakthroughs in the field of image inpainting. A hybrid network model based on the combination of autoencoder and Generative Adversarial Network (GAN) [20], [21], [22], [23], an improved autoencoder based on attention mechanism [24], [25], [26], [27], [28], [29], and improved shared codec network layer based on coarse-to-fine network [30], [31], [32], [33], [34], [35] have emerged, which gradually repair damaged images at the semantic level. Based on the above work, this paper makes a more comprehensive and detailed summary of image inpainting related network models based on deep learning in recent years, aiming to provide a more comprehensive and in-depth learning perspective for subsequent research in related fields.