Skip to main content
SpringerLink
Log in

Occluded Face In-painting Using Generative Adversarial Networks—A Review

  • Conference paper
  • First Online:
Intelligent Systems (BRACIS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14196))

Included in the following conference series:

Abstract

Face image de-occlusion and inpainting is a challenging problem in computer vision with several practical uses and is employed in many image preprocessing applications. The impressive results achieved by generative adversarial networks in image processing increased the attention of the scientific community in recent years around facial de-occlusion and inpainting. Recent network architecture developments are the two-stage networks using coarse to fine approach, landmarks, semantic segmentation map, and edge maps that guide the inpainting process. Moreover, improved convolutions enlarge the receptive field and filter the values passed to the next layer, and attention layers create relationships between local and distant information. This article presents a brief review of recent developments in GAN-based techniques for de-occlusion and inpainting of face images. In addition, it describes and analyzes network architectures and building blocks. Finally, we identify current limitations and propose directions for future research.

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    See protocol at https://github.com/vivamoto/bracis-2023.

  2. 2.

    Python code available at https://github.com/znxlwm/pytorch-pix2pix/blob/3059f2af53324e77089bbcfc31279f01a38c40b8/network.py.

  3. 3.

    Python code is available at https://github.com/brain-research/self-attention-gan.

References

  1. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, pp. 417–424. ACM Press/Addison-Wesley Publishing Co., USA (2000)

    Google Scholar 

  2. Burgos-Artizzu, X.P., Perona, P., Dollár, P.: Robust face landmark estimation under occlusion. In: 2013 IEEE International Conference on Computer Vision, pp. 1513–1520 (2013)

    Google Scholar 

  3. Cai, J., Han, H., Cui, J., Chen, J., Liu, L., Kevin Zhou, S.: Semi-supervised natural face de-occlusion. IEEE Trans. Inf. Forensics Secur. 16, 1044–1057 (2021)

    Article  Google Scholar 

  4. Cao, S., Sakurai, K.: Face completion with pyramid semantic attention and latent codes. In: Proceedings - 2020 8th International Symposium on Computing and Networking, CANDAR 2020, pp. 1–8. Institute of Electrical and Electronics Engineers Inc. (2020)

    Google Scholar 

  5. Chen, M., Liu, Z., Ye, L., Wang, Y.: Attentional coarse-and-fine generative adversarial networks for image inpainting. Neurocomputing 405, 259–269 (2020)

    Article  Google Scholar 

  6. Chen, Y.A., Chen, W.C., Wei, C.P., Wang, Y.C.: Occlusion-aware face inpainting via generative adversarial networks. In: Proceedings - International Conference on Image Processing, ICIP, vol. 2017-September, pp. 1202–1206. IEEE Computer Society (2018)

    Google Scholar 

  7. Cheung, Y.M., Li, M., Zou, R.: Facial structure guided GAN for identity-preserved face image de-occlusion. In: ICMR 2021 - Proceedings of the 2021 International Conference on Multimedia Retrieval, pp. 46–54. Association for Computing Machinery, Inc. (2021)

    Google Scholar 

  8. Din, N., Javed, K., Bae, S., Yi, J.: Effective removal of user-selected foreground object from facial images using a novel GAN-based network. IEEE Access 8, 109648–109661 (2020)

    Article  Google Scholar 

  9. Dong, J., Zhang, L., Zhang, H., Liu, W.: Occlusion-aware GAN for face de-occlusion in the wild. In: Proceedings - IEEE International Conference on Multimedia and Expo, vol. 2020-July. IEEE Computer Society (2020)

    Google Scholar 

  10. Fang, Y., Li, Y., Tu, X., Tan, T., Wang, X.: Face completion with hybrid dilated convolution. Signal Process. Image Commun. 80, 115664 (2020)

    Article  Google Scholar 

  11. Ge, S., Li, C., Zhao, S., Zeng, D.: Occluded face recognition in the wild by identity-diversity inpainting. IEEE Trans. Circuits Syst. Video Technol. 30(10), 3387–3397 (2020)

    Article  Google Scholar 

  12. Goodfellow, I.J., et al.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS 2014, vol. 2, pp. 2672–2680. MIT Press, Cambridge (2014)

    Google Scholar 

  13. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.: Improved training of Wasserstein GANs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, pp. 5769–5779. Curran Associates Inc., Red Hook (2017)

    Google Scholar 

  14. Guo, D., Feng, J., Zhou, B.: Structure-aware image expansion with global attention. In: SIGGRAPH Asia 2019 Technical Briefs, SA 2019, pp. 13–16. Association for Computing Machinery, Inc. (2019)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  16. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)

    Google Scholar 

  17. Horé, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369 (2010)

    Google Scholar 

  18. Hu, J., Shen, L., Albanie, S., Sun, G., Wu, E.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 2011–2023 (2020)

    Article  Google Scholar 

  19. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4), 1–14 (2017)

    Article  Google Scholar 

  20. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976 (2017)

    Google Scholar 

  21. Jabbar, A., Li, X., Iqbal, M., Malik, A.: FD-stackGAN: face de-occlusion using stacked generative adversarial networks. KSII Trans. Internet Inf. Syst. 15(7), 2547–2567 (2021)

    Google Scholar 

  22. Jam, J., Kendrick, C., Drouard, V., Walker, K., Hsu, G.S., Yap, M.: R-MNet: a perceptual adversarial network for image inpainting. In: Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021, pp. 2713–2722. Institute of Electrical and Electronics Engineers Inc. (2021)

    Google Scholar 

  23. Jam, J., Kendrick, C., Drouard, V., Walker, K., Hsu, G.S., Yap, M.: Symmetric skip connection Wasserstein GAN for high-resolution facial image inpainting. In: VISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 4, pp. 35–44. SciTePress (2021)

    Google Scholar 

  24. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  25. Khan, M., Ud Din, N., Bae, S., Yi, J.: Interactive removal of microphone object in facial images. Electronics 8(10), 1115 (2019)

    Article  Google Scholar 

  26. Li, X., Hu, G., Zhu, J., Zuo, W., Wang, M., Zhang, L.: Learning symmetry consistent deep CNNs for face completion. IEEE Trans. Image Process. 29, 7641–7655 (2020)

    Article  MATH  Google Scholar 

  27. Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5892–5900 (2017)

    Google Scholar 

  28. Li, Z., Zhu, H., Cao, L., Jiao, L., Zhong, Y., Ma, A.: Face inpainting via nested generative adversarial networks. IEEE Access 7, 155462–155471 (2019)

    Article  Google Scholar 

  29. Lie, Y., Li, L.: Image inpainting using multi-scale neural network and shift-net. In: Proceedings - 2020 7th International Conference on Information Science and Control Engineering, ICISCE 2020, pp. 704–709. Institute of Electrical and Electronics Engineers Inc. (2020)

    Google Scholar 

  30. Lin, J., Yang, H., Chen, D., Zeng, M., Wen, F., Yuan, L.: Face parsing with ROI tanh-warping. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5647–5656 (2019)

    Google Scholar 

  31. Liu, J., Jung, C.: Facial image inpainting using multi-level generative network. In: Proceedings - IEEE International Conference on Multimedia and Expo, vol. 2019-July, pp. 1168–1173. IEEE Computer Society (2019)

    Google Scholar 

  32. Liu, J., Jung, C.: Facial image inpainting using attention-based multi-level generative network. Neurocomputing 437, 95–106 (2021)

    Article  Google Scholar 

  33. Luo, X., He, X., Qing, L., Chen, X., Liu, L., Xu, Y.: Eyesgan: synthesize human face from human eyes. Neurocomputing 404, 213–226 (2020)

    Article  Google Scholar 

  34. Maggipinto, M., Masiero, C., Beghi, A., Susto, G.A.: A convolutional autoencoder approach for feature extraction in virtual metrology. Procedia Manufacturing 17, 126–133 (2018). 28th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM2018), 11–14 June 2018, Columbus, OH, USAGlobal Integration of Intelligent Manufacturing and Smart Industry for Good of Humanity

    Google Scholar 

  35. Maharjan, R., Ud Din, N., Yi, J.: Image-to-image translation based face de-occlusion. In: Jiang X., F.H. (ed.) Proceedings of SPIE - The International Society for Optical Engineering, vol. 11519. SPIE (2020)

    Google Scholar 

  36. Maheshwari, U., Turlapati, V., Kiruthika, U.: Lucid-GAN: an adversarial network for enhanced image inpainting. In: CIVEMSA 2021 - IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications, Proceedings. Institute of Electrical and Electronics Engineers Inc. (2021)

    Google Scholar 

  37. Mathai, J., Masi, I., Abdalmageed, W.: Does generative face completion help face recognition? In: 2019 International Conference on Biometrics, ICB 2019. Institute of Electrical and Electronics Engineers Inc. (2019)

    Google Scholar 

  38. Maulana, A., Fatichah, C., Suciati, N.: Facial inpainting using generative adversarial network with feature reconstruction and landmark loss to preserve spatial consistency in unaligned face images. Int. J. Intell. Eng. Syst. 13(6), 219–228 (2020)

    Google Scholar 

  39. Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks (2018)

    Google Scholar 

  40. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  41. Sadiq, M., Shi, D.: Attentive occlusion-adaptive deep network for facial landmark detection. Pattern Recognit. 125, 108510 (2022)

    Article  Google Scholar 

  42. Wang, F., Li, W., Liu, Y., Gong, Y., Gao, Z., Lu, J.: Face inpainting combining structured forest edge information and gated convolution. In: Proceedings - 2021 3rd International Conference on Natural Language Processing, ICNLP 2021, pp. 213–217. Institute of Electrical and Electronics Engineers Inc. (2021)

    Google Scholar 

  43. Wu, Y., Singh, V., Kapoor, A.: From image to video face inpainting: spatial-temporal nested GAN (STN-GAN) for usability recovery. In: Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020, pp. 2385–2394. Institute of Electrical and Electronics Engineers Inc. (2020)

    Google Scholar 

  44. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE International Conference on Computer Vision, vol. 2019-October, pp. 4470–4479. Institute of Electrical and Electronics Engineers Inc. (2019)

    Google Scholar 

  45. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 5505–5514. IEEE Computer Society (2018)

    Google Scholar 

  46. Yu, L., Zhu, D., He, J.: Semantic segmentation guided face inpainting based on SN-PatchGAN. In: Proceedings - 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics, CISP-BMEI 2020, pp. 110–115. Institute of Electrical and Electronics Engineers Inc. (2020)

    Google Scholar 

  47. Zhang, H., Li, T.: Semantic face image inpainting based on generative adversarial network. In: Proceedings - 2020 35th Youth Academic Annual Conference of Chinese Association of Automation, YAC 2020, pp. 530–535. Institute of Electrical and Electronics Engineers Inc. (2020)

    Google Scholar 

  48. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 7354–7363. PMLR (2019)

    Google Scholar 

  49. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

    Google Scholar 

  50. Zhu, W., Wang, X., Wu, Y., Zou, G.: A face occlusion removal and privacy protection method for IoT devices based on generative adversarial networks. Wirel. Commun. Mob. Comput. 2021 (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of São Paulo, São Paulo, SP, 03828-000, Brazil

    Victor Ivamoto, Rodolfo Simões, Bruno Kemmer & Clodoaldo Lima

Authors
  1. Victor Ivamoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rodolfo Simões
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bruno Kemmer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Clodoaldo Lima
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Clodoaldo Lima .

Editor information

Editors and Affiliations

  1. Federal University of São Carlos, São Carlos, Brazil

    Murilo C. Naldi

  2. Centro Universitario da FEI, São Bernardo do Campo, Brazil

    Reinaldo A. C. Bianchi

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 535 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ivamoto, V., Simões, R., Kemmer, B., Lima, C. (2023). Occluded Face In-painting Using Generative Adversarial Networks—A Review. In: Naldi, M.C., Bianchi, R.A.C. (eds) Intelligent Systems. BRACIS 2023. Lecture Notes in Computer Science(), vol 14196. Springer, Cham. https://doi.org/10.1007/978-3-031-45389-2_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45389-2_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45388-5

  • Online ISBN: 978-3-031-45389-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation