Skip to main content
SpringerLink
Log in

Generative image completion with image-to-image translation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

A Correction to this article was published on 24 July 2020

This article has been updated

Abstract

Though many methods have been proposed, image completion still remains challenge; besides textured patterns completion, it often requires high-level understanding of scenes and objects being completed. More recently, deep convolutional generative adversarial networks have been turned into an efficient tool for image completion. Manually specified transformation methods are having been replaced with training neural nets. Hand-engineered loss calculations for training the generator are replaced by the loss function provided by the discriminator. With existing deep learning-based approaches, image completion results in high quality but may still lack high-level feature details or contain artificial appearance. In our completion architecture, we leverage a fully convolutional generator with two subnetworks as our basic completion approach and divide the problem into two steps: The first subnetwork generates the outline of a completed image in a new domain, and the second subnetwork translates the outline to a visually realistic output with image-to-image translation. The feedforward fully convolutional network can complete images with holes of any size at any location. We compare our method with several existing ones on representative datasets such as CelebA, ImageNet, Places2 and CMP Facade. The evaluations demonstrate that our model significantly improves the completion results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Change history

  • 24 July 2020

    Unfortunately, the corresponding author of this paper was incorrectly published as Shuzhen Xu in the original publication. The correct corresponding author should be Jin Wang.

References

  1. Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graph (ToG) 28(3):24

    Article  Google Scholar 

  2. Huang J-B, Kang SB, Ahuja N, Kopf J (2014) Image completion using planar structure guidance. ACM Trans Graph (TOG) 33(4):129

    Google Scholar 

  3. Hays J, Efros AA (2007) Scene completion using millions of photographs. ACM Trans Graph (TOG) 26:4

    Article  Google Scholar 

  4. Mirza M, Osindero S (2014) Conditional generative adversarial nets. CoRR. arXiv:abs/1411.1784

  5. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434

  6. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems. pp 2672–2680

  7. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  8. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  9. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  10. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, pp 694–711

  11. Gatys LA, Ecker AS, Bethge M (2015) A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576

  12. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of the IEEE international conference on computer vision. pp 3730–3738

  13. Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A (2018) Places: A 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell 99:1–1

    Google Scholar 

  14. Tyleček R, Šára R (2013) Spatial pattern templates for recognition of objects with regular structure. In: German conference on pattern recognition. Springer, pp 364–374

  15. Zhang H, Sun Y, Liu L, Wang X, Li L, Liu W (2018) ClothingOut: a category-supervised GAN model for clothing segmentation and retrieval. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3691-y

    Article  Google Scholar 

  16. Ji Y, Zhang H, Wu QJ (2018) Saliency detection via conditional adversarial image-to-image network. Neurocomputing 316:357–368

    Article  Google Scholar 

  17. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint

  18. Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co, pp 417–424

  19. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2536–2544

  20. Yang C, Lu X, Lin Z, Shechtman E, Wang O, Li H (2017) High-resolution image inpainting using multi-scale neural patch synthesis. In: The IEEE conference on computer vision and pattern recognition (CVPR). vol 1, p 3

  21. Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Trans Graph (TOG) 36(4):107

    Article  Google Scholar 

  22. Pérez P, Gangnet M, Blake A (2003) Poisson image editing. ACM Trans Graph (TOG) 22(3):313–318

    Article  Google Scholar 

  23. Song Y, Yang C, Lin Z, Li H, Huang Q, Kuo C-CJ (2017) Image inpainting using multi-scale feature image translation. arXiv preprint arXiv:1711.08590

  24. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. arXiv preprint

  25. Li Y, Liu S, Yang J, Yang M-H (2017) Generative face completion. In: The IEEE conference on computer ision and pattern recognition (CVPR). vol 1, p 3

  26. Ulyanov D, Vedaldi A, Lempitsky V (2017) Deep image prior. arXiv preprint arXiv:1711.10925

  27. Yang J, Price B, Cohen S, Lee H, Yang M-H (2016) Object contour detection with a fully convolutional encoder−decoder network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 193–202

  28. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). pp 770–778

  29. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein gan. arXiv preprint arXiv:1701.07875

  30. Mao X, Li Q, Xie H, Lau R, Wang Z, Smolley S (2017) Least squares generative adversarial networks. In: Proceedings—2017 IEEE international conference on computer vision, ICCV 2017. pp 2813–2821

  31. Liu G, Reda FA, Shih KJ, Wang T-C, Tao A, Catanzaro B (2018) Image inpainting for irregular holes using partial convolutions. arXiv preprint arXiv:1804.07723

  32. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International conference on international conference on machine learning. pp 448–456

  33. Kingma D, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  34. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  35. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems. pp 6626–6637

  36. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2818–2826

  37. Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. arXiv preprint

  38. Dolhansky B, Canton Ferrer C (2018) Eye in-painting with exemplar generative adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7902–7911

  39. Oord Avd, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759

Download references

Acknowledgements

This work was supported by Beijing Natural Science Foundation (4164079, 4152008), National Key Research and Development Plan of China (2017YFF0211801) and the Opening Project of Beijing Key Laboratory of Internet Culture and Digital Dissemination Research.

Author information

Authors and Affiliations

  1. Information Department, Beijing University of Technology, Beijing, 100124, People’s Republic of China

    Shuzhen Xu, Qing Zhu & Jin Wang

Authors
  1. Shuzhen Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qing Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shuzhen Xu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 28090 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, S., Zhu, Q. & Wang, J. Generative image completion with image-to-image translation. Neural Comput & Applic 32, 7333–7345 (2020). https://doi.org/10.1007/s00521-019-04253-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04253-2

Keywords

Search

Navigation