Huan Zheng
Meng Wang
ABSTRACT
Deep learning based inpainting methods have obtained promising performance for image restoration, however current image inpainting methods still tend to produce unreasonable structures and blurry textures when processing the damaged images with heavy corruptions. In this paper, we propose a new image inpainting method termed Global Context Modeling Network (GCM-Net). By capturing the global contextual information, GCM-Net can potentially improve the performance of recovering the missing region in the damaged images with irregular masks. To be specific, we first use four convolution layers to extract the shadow features. Then, we design a progressive multi-scale fusion block termed PMSFB to extract and fuse the multi-scale features for obtaining local features. Besides, a dense context extraction (DCE) module is also designed to aggregate the local features extracted by PMSFBs. To improve the information flow, a channel attention guided residual learning module is deployed in both the DCE and PMSFB, which can reweight the learned residual features and refine the extracted information. To capture more global contextual information and enhance the representation ability, a coordinate context attention (CCA) based module is also presented. Finally, the extracted features with rich information are decoded as the image inpainting result. Extensive results on the Paris Street View, Places2 and CelebA-HQ datasets demonstrate that our method can better recover the structures and textures, and deliver significant improvements, compared with some related inpainting methods.
- Naoufal Amrani, Joan Serra-Sagristà, Pascal Peter, and Joachim Weickert. 2017. Diffusion-based inpainting for coding remote-sensing data. IEEE Geoscience and Remote Sensing Letters 14, 8 (2017), 1203--1207.Google Scholar
Cross Ref
- Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (2009), 24. Google ScholarDigital Library
- Ugo V Boscain, Roman Chertovskih, Jean-Paul Gauthier, Dario Prandi, and Alexey Remizov. 2018. Highly corrupted image inpainting through hypoelliptic diffusion. Journal of Mathematical Imaging and Vision 60, 8 (2018), 1231--1245. Google ScholarDigital Library
- Pierre Buyssens, Olivier Le Meur, Maxime Daisy, David Tschumperlé, and Olivier Lézoray. 2016. Depth-guided disocclusion inpainting of synthesized RGB-D images. IEEE Transactions on Image Processing 26, 2 (2016), 525--538. Google ScholarDigital Library
- Tony F Chan and Jianhong Shen. 2001. Nontexture inpainting by curvaturedriven diffusions. Journal of visual communication and image representation 12, 4 (2001), 436--449. Google ScholarDigital Library
- Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on image processing 13, 9 (2004), 1200--1212. Google ScholarDigital Library
- Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei Efros. 2012. What makes paris look like paris? ACM Transactions on Graphics 31, 4 (2012). Google ScholarDigital Library
- Xinjian Gao, Zhao Zhang, Tingting Mu, Xudong Zhang, Chaoran Cui, and Meng Wang. 2020. Self-attention driven adversarial similarity learning network. Pattern Recognition 105 (2020), 107331.Google ScholarCross Ref
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139--144. Google ScholarDigital Library
- Zongyu Guo, Zhibo Chen, Tao Yu, Jiale Chen, and Sen Liu. 2019. Progressive image inpainting with full-resolution residual network. In Proceedings of the 27th ACM International Conference on Multimedia. 2496--2504. Google ScholarDigital Library
- Kaiming He and Jian Sun. 2012. Statistics of patch offsets for image completion. In European conference on computer vision. Springer, 16--29.Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.Google ScholarCross Ref
- Xin Hong, Pengfei Xiong, Renhe Ji, and Haoqiang Fan. 2019. Deep fusion network for image completion. In Proceedings of the 27th ACM International Conference on Multimedia. 2033--2042. Google ScholarDigital Library
- Yibing Song Wei Huang Hongyu Liu, Bin Jiang and Chao Yang. 2020. Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations. In Proceedings of the European Conference on Computer Vision.Google Scholar
- Alain Hore and Djemel Ziou. 2010. Image quality metrics: PSNR vs. SSIM. In 2010 20th international conference on pattern recognition. IEEE, 2366--2369. Google ScholarDigital Library
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4700--4708.Google ScholarCross Ref
- Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision. 1501--1510.Google ScholarCross Ref
- Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG) 36, 4 (2017), 1--14. Google ScholarDigital Library
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Jingyuan Li, Fengxiang He, Lefei Zhang, Bo Du, and Dacheng Tao. 2019. Progressive reconstruction of visual structure for image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5962--5971.Google ScholarCross Ref
- Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao. 2020. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7760--7768.Google ScholarCross Ref
- Guilin Liu, FitsumAReda, Kevin J Shih, Ting-ChunWang, AndrewTao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV). 85--100.Google ScholarDigital Library
- Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. 2019. Coherent semantic attention for image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4170--4179.Google ScholarCross Ref
- Or Lotan and Michal Irani. 2016. Needle-match: Reliable patch matching under high uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 439--448.Google ScholarCross Ref
- Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi. 2019. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212 (2019).Google Scholar
- Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).Google Scholar
- Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2536--2544.Google ScholarCross Ref
- Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. 2019. Structureflow: Image inpainting via structure-aware appearance flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 181--190.Google ScholarCross Ref
- Jianhong Shen and Tony F Chan. 2002. Mathematical models for local nontexture inpaintings. SIAM J. Appl. Math. 62, 3 (2002), 1019--1043.Google ScholarDigital Library
- Ling Shen, Richang Hong, Haoran Zhang, Hanwang Zhang, and Meng Wang. 2019. Single-shot semantic image inpainting with densely connected generative networks. In Proceedings of the 27th ACM International Conference on Multimedia. 1861--1869. Google ScholarDigital Library
- Yong-Goo Shin, Min-Cheol Sagong, Yoon-Jae Yeo, Seung-Wook Kim, and Sung- Jea Ko. 2020. Pepsi++: Fast and lightweight network for image inpainting. IEEE transactions on neural networks and learning systems 32, 1 (2020), 252--265.Google ScholarCross Ref
- Linsen Song, Jie Cao, Lingxiao Song, Yibo Hu, and Ran He. 2019. Geometry-aware face completion and editing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2506--2513.Google ScholarDigital Library
- Ning Wang, Jingyuan Li, Lefei Zhang, and Bo Du. 2019. MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting.. In IJCAI. 3748--3754. Google ScholarDigital Library
- Yanyan Wei, Zhao Zhang, Yang Wang, Mingliang Xu, Yi Yang, Shuicheng Yan, and Meng Wang. 2021. DerainCycleGAN: Rain attentive CycleGAN for single image deraining and rainmaking. IEEE Transactions on Image Processing 30 (2021), 4788--4801.Google ScholarCross Ref
- Yanyan Wei, Zhao Zhang, Haijun Zhang, Richang Hong, and Meng Wang. 2019. A Coarse-to-Fine Multi-stream Hybrid Deraining Network for Single Image Deraining. In 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 628--637.Google ScholarCross Ref
- Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding. 2019. Image inpainting with learnable bidirectional attention maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8858--8867.Google ScholarCross Ref
- Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo. 2019. Foreground-aware image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5840--5848.Google ScholarCross Ref
- Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 (2015).Google Scholar
- Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. 2017. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6721--6729.Google ScholarCross Ref
- Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. 2017. Semantic image inpainting with deep generative models. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5485--5493.Google ScholarCross Ref
- Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5505--5514.Google ScholarCross Ref
- Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2019. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4471--4480.Google ScholarCross Ref
- Tao Yu, Zongyu Guo, Xin Jin, ShilinWu, Zhibo Chen,Weiping Li, Zhizheng Zhang, and Sen Liu. 2020. Region normalization for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12733--12740.Google ScholarCross Ref
- Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. 2019. Learning pyramid-context encoder network for high-quality image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1486--1494.Google ScholarCross Ref
- Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Selfattention generative adversarial networks. In International conference on machine learning. PMLR, 7354--7363.Google Scholar
- Yulun Zhang, Kunpeng Li, Kai Li, LichenWang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV). 286--301.Google ScholarDigital Library
- Zhao Zhang, Zemin Tang, Yang Wang, Zheng Zhang, Choujun Zhan, Zhengjun Zha, and Meng Wang. 2021. Dense Residual Network: Enhancing global dense feature flow for character recognition. Neural Networks 139 (2021), 77--85.Google ScholarCross Ref
- Zhao Zhang, Yanyan Wei, Haijun Zhang, Yi Yang, Shuicheng Yan, and Meng Wang. 2021. Data-Driven Single Image Deraining: A Comprehensive Review and New Perspectives. TechRxiv preprint (2021).Google Scholar
- Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. 2019. Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1438--1447.Google ScholarCross Ref
- Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence 40, 6 (2017), 1452--1464.Google Scholar
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223--2232.Google ScholarCross Ref
Index Terms
- GCM-Net: Towards Effective Global Context Modeling for Image Inpainting
Recommendations
Image Retrieval Using Digital Image Inpainting Techniques
Image retrieval is an inverse problem in digital image processing. In this paper, the authors deal with restoration of image using digitally image inpainting methods. In this inpainting technique, one can extract a missing an important part or can ...
An efficient framework for image/video inpainting
Image inpainting has been widely applied to many applications, such as restoring corrupted old photos, erasing video logos, concealing errors in a digital video processing system, and so on. However, traditional geometric inpainting methods suffer low ...
A new structure tensor based image inpainting algorithm
A new structure tensor based image inpainting algorithm STIA is proposed for solving the deficiencies of the classical Criminisi method, such as the error repair accumulation, high time complexity caused by the unreasonable design of the patch priority, ...
Comments