Skip to main content
SpringerLink
Log in

An Improved Method for Semantic Image Inpainting with GANs: Progressive Inpainting

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Semantic image inpainting is getting more and more attention due to its increasing usage. Existing methods make inference based on either local data or external information. Generating Adversarial Networks, as a research focus in recent years, has been proven to be useful in inpainting work. One of the most representative is the deep-generative-model-based approach, which use undamaged images for training and repair the corrupted image with the trained networks. However, this method is too dependent on the training process, easily resulting in the completed image blurry in details. In this paper, we propose an improved method named progressive inpainting. With the trained networks, we use back-propagation to find the most appropriate input distribution and use the generator to repair the corrupted image. Instead of repairing the image in one step, we take a pyramid strategy from a low-resolution image to higher one, with the purpose of getting a clear completed image and reducing the reliance on the training process. The advantage of progressive inpainting is that we can predict the general distribution of the corrupted image and then gradually refine the details. Experiment results on two datasets show that our method successfully reconstructs the image and outperforms most existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Afonso MV, Bioucas-Dias JM, Figueiredo MA (2011) An augmented lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE Trans Image Process 20(3):681–695

    Article  MathSciNet  MATH  Google Scholar 

  2. Shen J, Chan TF (2002) Mathematical models for local nontexture inpaintings. SIAM J Appl Math 62(3):1019–1043

    Article  MathSciNet  MATH  Google Scholar 

  3. Barnes C, Shechtman E, Finkelstein A, Dan BG (2009) Patchmatch:a randomized correspondence algorithm for structural image editing. ACM Trans Gr (TOG) 28(3):1–11

    Article  Google Scholar 

  4. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: International conference on neural information processing systems, pp 2672–2680

  5. Yeh RA, Chen C, Lim TY, Schwing AG, Hasegawa-Johnson M, Do MN (2017) Semantic image inpainting with deep generative models. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5485–5493

  6. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of the IEEE international conference on computer vision, pp 3730–3738

  7. Krause J, Stark M, Jia D, Li FF (2014) 3d object representations for fine-grained categorization. In: IEEE international conference on computer vision workshops, pp 554–561

  8. Efros AA, Leung TK (1999) Texture synthesis by non-parametric sampling. In: The proceedings of the seventh IEEE international conference on computer vision, 1999, vol 2. IEEE, pp 1033–1038

  9. Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., pp 417–424

  10. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: IEEE conference on computer vision and pattern recognition, pp 815–823

  11. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: British machine vision conference, pp 41.1–41.12

  12. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587

  13. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  14. Hong C, Yu J, Wan J, Tao D, Wang M (2015) Multimodal deep autoencoder for human pose recovery. IEEE Trans Image Process 24(12):5659–5670

    Article  MathSciNet  MATH  Google Scholar 

  15. Hong C, Yu J, Tao D, Wang M (2015) Image-based three-dimensional human pose recovery by multiview locality-sensitive sparse retrieval. IEEE Trans Ind Electron 62(6):3742–3751

    Google Scholar 

  16. Hong C, Chen X, Wang X, Tang C (2016) Hypergraph regularized autoencoder for image-based 3d human pose recovery. Signal Process 124:132–140

    Article  Google Scholar 

  17. Yu J, Hong C, Rui Y, Tao D (2018) Multitask autoencoder model for recovering human poses. IEEE Trans Ind Electron 65(6):5060–5068

    Article  Google Scholar 

  18. Yang M, Liu Y, You Z (2017) The euclidean embedding learning based on convolutional neural network for stereo matching. Neurocomputing 267:195–200

    Article  Google Scholar 

  19. Zhang J, Yu J, Tao D (2018) Local deep-feature alignment for unsupervised dimension reduction. IEEE Trans Image Process 27(5):2420–2432

    Article  MathSciNet  MATH  Google Scholar 

  20. Qian S, Liu H, Liu C, Wu S, San Wong H (2018) Adaptive activation functions in convolutional neural networks. Neurocomputing 272:204–212

    Article  Google Scholar 

  21. Yu J, Zhang B, Kuang Z, Lin D, Fan J (2017) iprivacy: image privacy protection by identifying sensitive objects via deep multi-task learning. IEEE Trans Inf Forensics Secur 12(5):1005–1016

    Article  Google Scholar 

  22. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544

  23. Radford A, Metz L, Chintala S Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434

  24. Hays J, Efros AA (2008) Scene completion using millions of photographs. Commun ACM 51(10):87–94

    Article  Google Scholar 

  25. Whyte O, Sivic J, Zisserman A (2009) Get out of my picture! internet-based inpainting. In: British machine vision conference, BMVC 2009, London, 7–10 Sept 2009. Proceedings

  26. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  27. Johnson J, Alahi A, Li FF (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision, pp 694–711

  28. Liu W, Zhang H, Tao D, Wang Y, Lu K (2016) Large-scale paralleled sparse principal component analysis. Multimed Tools Appl 75(3):1481–1493

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 61673402, 61273270, and 60802069, in part by the Natural Science Foundation of Guangdong under Grants 2017A030311029, 2016B010109002, 2015B090912001, 2016B010123005, and 2017B090909005, in part by the Science and Technology Program of Guangzhou under Grants 201704020180 and 201604020024, and in part by the Fundamental Research Funds for the Central Universities of China.

Author information

Authors and Affiliations

  1. School of Electronics and Information Technology, Sun Yat-Sen University, Guangzhou, 510006, China

    Yizhen Chen & Haifeng Hu

Authors
  1. Yizhen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haifeng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haifeng Hu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Hu, H. An Improved Method for Semantic Image Inpainting with GANs: Progressive Inpainting. Neural Process Lett 49, 1355–1367 (2019). https://doi.org/10.1007/s11063-018-9877-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-018-9877-6

Keywords

Search

Navigation