Image Inpainting Based on Edge Features and Attention Mechanism
Pages 64 - 71
Abstract
Image inpainting as a kind important application in our life and entertainment, it also is a popular task of computer vision. The latest deep learning-based approaches have shown promising results for the challenging task of inpainting damaged regions of an image. However, there are still structural differences between the restored images and the ground truth images. Aiming at this problem, we propose a model of image inpainting called ECF-Net, ECF-Net incorporates edge information into the process of image inpainting to help damaged images to obtain more structures similar to the ground truth images, which to guide the generation of the feature of the damaged area. At the same time, we introduce the knowledge consistency attention mechanism in ECF-Net, which can obtain more reasonable semantics to eliminate blurs for image inpainting. Extensive experiments on various datasets such as CelebA-HQ, Places2 and the Paris StreetView clearly demonstrate that our method gets a better performance in vision.
References
[1]
Bertalmio M, Sapiro G, Caselles V, Image Inpainting. Proceedings of Annual Conference on Computer Graphics & Interactive Techniques, 2000:417–424.
[2]
Shen J, Chan T F. Mathematical Models for Local Nontexture Inpaintings. SIAM Journal on Applied Mathematics, 2002, 62(3): 1019-1043
[3]
RUDIN LI, OSHER S, FATEMI E. Nonlinear Total Variation Based Noise Removal Algorithms. Physica D: Nonlinear Phenomena, 1992,60(1): 259.
[4]
Criminisi A, Pérez P, Toyama K. Region Filling and Object Removal by Exemplar-based Image Inpainting. Image Processing, IEEE Transactions on, 2004, 13(9): 1200-1212.
[5]
Barnes C, Dan B G, Shechtman E, The PatchMatch randomized matching algorithm for image manipulation. Communications of the ACM, 2011.
[6]
Bertalmio M, Vese L, Guillermo S T, Simultaneous structure and texture image inpainting. IEEE Transactions on Image Processing, 2003, 12(8).
[7]
GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, Generative adversarial nets. Proceedings of the 26th International Conference on Neural Information Processing Systems. Goslar: Euro graphics Association Press, 2014: 2672-2680.
[8]
Pathak D, Krhenbhl P, Donahue J, Context encoders: feature learning by inpainting. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 2536-2544.
[9]
Iizuka S, Simo-Serra E, Ishikawa H. Globally and locally consistent image completion. ACM Transactions on Graphics, 2017, 36(4)
[10]
Nazeri K, Ng E, Joseph T, EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning. ICCV Workshops. 2019.
[11]
Jingyuan Li, Fengxinag He, Lefei Zhang, Bo Du, and Dacheng Tao. Progressive reconstruction of visual structure for image inpainting. In Proc. ICCV, pages 6721–6729,2019.
[12]
Ren Y, X Yu, Zhang R, StructureFlow: Image Inpainting via Structure-aware Appearance Flow. IEEE, 2019.
[13]
Yu J, Lin Z, Yang J, Generative image inpainting with contextual attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 5505-5514.
[14]
Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. In Proc. ICCV, pages 4170–4179, 2019.
[15]
Li J, Wang N, Zhang L, Recurrent Feature Reasoning for Image Inpainting. IEEE, 2020.
[16]
LIU G, REDA F A, SHIH K J, Image inpainting for irregular holes using partial convolutions. Computer Vision- ECCV 2018. Heidelberg: Springer,2018: 85-100.
[17]
Szegedy C, Ioffe S, Vanhoucke V, Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. 2016.
[18]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei AEfros. Image-to-image translation with conditional adversarial networks. In Proc. CVPR, 2017.
[19]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[20]
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. ICCV, pages 3730–3738, 2015.
[21]
Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei Efros. What makes paris look like paris? ACM TOG, 31(4):101, 2012.
[22]
Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 40(6):1452–1464, 2018.
[23]
Manoj Kumar Singh, "Image Reconstruction and Edge Detection based upon Neural Approximation Characteristics," Journal of Image and Graphics, 2013, 1(1):12-16.
Recommendations
Image inpainting based on deep learning: A review
AbstractImage inpainting is an important research direction in the study of computer vision, and is widely used in image editing and photo inpainting etc. Traditional image inpainting algorithms are often difficult to deal with large-scale image deletion,...
Highlights- Reviewing the image inpainting on deep learning of the past 15 years.
- Summarizing the prominent deep learning methods especially in feature fusion.
- Pointing out the problem and difficulty in image inpainting task.
- Providing ...
Edge-Guided Image Inpainting with Transformer
Advances in Visual ComputingAbstractImage inpainting aims to complete missing regions by extracting the features of the image through the information of the known region. Traditional image inpainting approaches like patch-based and diffusion-based methods are robust for simple ...
Image Retrieval Using Digital Image Inpainting Techniques
Image retrieval is an inverse problem in digital image processing. In this paper, the authors deal with restoration of image using digitally image inpainting methods. In this inpainting technique, one can extract a missing an important part or can ...
Comments
Information & Contributors
Information
Published In
January 2022
391 pages
ISBN:9781450395465
DOI:10.1145/3512388
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 March 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICIGP 2022
ICIGP 2022: 2022 the 5th International Conference on Image and Graphics Processing
January 7 - 9, 2022
Beijing, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 54Total Downloads
- Downloads (Last 12 months)8
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format