Abstract
Existing image inpainting methods have shown promising results for regular and small-area breaks. However, restoration of irregular and large-area damage is still tricky and achieves mediocre results due to the lack of restrictions on the center of the hole. In contrast, face inpainting is also problematic due to facial structure and texture complexity, which always results in structural confusion and texture blurring. We propose an attention embedded adversarial generative network (AE-GAN) in the paper to solve this problem. Overall our framework is a U-shape GAN model. To enable the network to capture the practical features faster to reconstruct the content of the masked region in the face image, we also embed the attention mechanism that simplifies the Squeeze-and-Excitation channel attention mechanism and then set it reasonably in our generator. Our generator is chosen the U-net structure as a backbone. Because the structure can encode information from low-level pixels context features to high-level semantic features. And it can decode the features back into an image. Experiments on CelebA-HQ datasets demonstrate that our proposed method generates higher quality inpainting, results in consistent and harmonious facial structures and appearance than existing methods, and achieves state-of-the-art performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aleem, S., Yang, P., Masood, S., Li, P., Sheng, B.: An accurate multi-modal biometric identification system for person identification via fusion of face and finger print. World Wide Web 23(2), 1299–1317 (2019). https://doi.org/10.1007/s11280-019-00698-6
Badatia, P., Tasgaonkar, P.P.: Crowd counting and density estimation using multicolumn discriminator in GAN. In: 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 1179–1183 (2018). https://doi.org/10.1109/ICACCI.2018.8554372
Chen, Z., et al.: Structure-aware image inpainting using patch scale optimization. J. Vis. Commun. Image Represent. 40, 312–323 (2016). https://doi.org/10.1016/j.jvcir.2016.06.029, https://www.sciencedirect.com/science/article/pii/S1047320316301262
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423 (2016). https://doi.org/10.1109/CVPR.2016.265
Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020). https://doi.org/10.1145/3422622, https://doi.org/10.1145/3422622
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANS trained by a two time-scale update rule converge to a local nash equilibrium. In: 30th Proceedings Conference on Advances in Neural Information Processing System (2017)
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4) (2017). https://doi.org/10.1145/3072959.3073659, https://doi.org/10.1145/3072959.3073659
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8107–8116 (2020). https://doi.org/10.1109/CVPR42600.2020.00813
Ke, Q., Ming, L.D., Daxing, Z.: Image steganalysis via multi-column convolutional neural network. In: 2018 14th IEEE International Conference on Signal Processing (ICSP), pp. 550–553 (2018). https://doi.org/10.1109/ICSP.2018.8652324
Li, J., Wang, N., Zhang, L., Du, B., Tao, D.: Recurrent feature reasoning for image inpainting. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7757–7765 (2020). https://doi.org/10.1109/CVPR42600.2020.00778
Li, L., Tang, J., Ye, Z., Sheng, B., Mao, L., Ma, L.: Unsupervised face super-resolution via gradient enhancement and semantic guidance. The Vis. Comput. 37(9), 2855–2867 (2021). https://doi.org/10.1007/s00371-021-02236-w
Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5892–5900 (2017). https://doi.org/10.1109/CVPR.2017.624
Liu, B., Li, P., Sheng, B., Nie, Y., Wu, E.: Structure-preserving image completion with multi-level dynamic patches. The Vis. Comput. 35(1), 85–98 (2019). https://doi.org/10.1007/s00371-017-1454-x
Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_6
Liu, H., Jiang, B., Xiao, Y., Yang, C.: Coherent semantic attention for image inpainting. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4169–4178 (2019). https://doi.org/10.1109/ICCV.2019.00427
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3730–3738 (2015). https://doi.org/10.1109/ICCV.2015.425
Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536–2544 (2016). https://doi.org/10.1109/CVPR.2016.278
Peng, J., Liu, D., Xu, S., Li, H.: Generating diverse structure for image inpainting with hierarchical vq-vae. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10770–10779 (2021). https://doi.org/10.1109/CVPR46437.2021.01063
Sagong, M.c., Shin, Y.g., Kim, S.w., Park, S., Ko, S.j.: Pepsi : fast image inpainting with parallel decoding network. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11352–11360 (2019). https://doi.org/10.1109/CVPR.2019.01162
Sheng, B., Li, P., Gao, C., Ma, K.L.: Deep neural representation guided face sketch synthesis. IEEE Trans. Visual Comput. Graphics 25(12), 3216–3230 (2019). https://doi.org/10.1109/TVCG.2018.2866090
Song, L., Cao, J., Song, L., Hu, Y., He, R.: Geometry-aware face completion and editing. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 2506–2513 (2019). https://doi.org/10.1609/aaai.v33i01.33012506
Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861
Wu, X., Xu, K., Hall, P.: A survey of image synthesis and editing with generative adversarial networks. Tsinghua Sci. Technol. 22(6), 660–674 (2017). https://doi.org/10.23919/TST.2017.8195348
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018). https://doi.org/10.1109/CVPR.2018.00577
Zeng, Y., Fu, J., Chao, H., Guo, B.: Learning pyramid-context encoder network for high-quality image inpainting. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1486–1494 (2019). https://doi.org/10.1109/CVPR.2019.00158
Zhou, T., Ding, C., Lin, S., Wang, X., Tao, D.: Learning oracle attention for high-fidelity face completion. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7677–7686 (2020). https://doi.org/10.1109/CVPR42600.2020.00770
Acknowledgments
We would like to thank the anonymous reviewers for their valuable suggestions. This paper is supported by the Shandong Provincial Natural Science Foundation (ZR2020MF132), the National Natural Science Foundation of China (No. 62072020), the Leading Talents in Innovation and Entrepreneurship of Qingdao (19-3-2-21-zhc), and the Key-Area Research and Development Program of Guangdong Province (No. 209B010150001).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bao, Y., Xiao, X., Qi, Y. (2022). AE-GAN: Attention Embedded GAN for Irregular and Large-Area Mask Face Image Inpainting. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2022. Lecture Notes in Computer Science, vol 13443. Springer, Cham. https://doi.org/10.1007/978-3-031-23473-6_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-23473-6_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23472-9
Online ISBN: 978-3-031-23473-6
eBook Packages: Computer ScienceComputer Science (R0)