Abstract
The photorealism of synthetic face images has been significantly improved by generative adversarial networks (GANs). Besides of the realism, more accurate control on the properties of face images. While sketches convey the desired shapes, attributes describe appearance. However, it remains challenging to jointly exploit sketches and attributes, which are in different modalities, to generate high-resolution photorealistic face images. In this paper, we propose a novel joint sketch-attribute learning approach to synthesize photo-realistic face images with conditional GANs. A hybrid generator is proposed to learn a unified embedding of shape from sketches and appearance from attributes for synthesizing images. We propose an attribute modulation module, which transfers user-preferred attributes to reinforce sketch representation with appearance details. Using the proposed approach, users could flexibly manipulate the desired shape and appearance of synthesized face images with fine-grained control. We conducted extensive experiments on the CelebA-HQ dataset [16]. The experimental results have demonstrated the effectiveness of the proposed approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, T., Cheng, M., Tan, P., Shamir, A., Hu, S.: Sketch2Photo: Internet image montage. ACM Trans. Graph. 28(5), 124:1–124:10 (2009)
Chen, W., Hays, J.: SketchyGAN: towards diverse and realistic sketch to image synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9416–9425 (2018)
Choi, Y., Choi, M., Kim, M., Ha, J., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: IEEE Computer Vision and Pattern Recognition, pp. 8789–8797 (2018)
Denton, E.L., Chintala, S., Szlam, A., Fergus, R.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: Advances in Neural Information Processing Systems, pp. 1486–1494 (2015)
Dong, X., Yan, Y., Ouyang, W., Yang, Y.: Style aggregated network for facial landmark detection. In: IEEE Computer Vision and Pattern Recognition, pp. 379–388 (2018)
Eitz, M., Richter, R., Hildebrand, K., Boubekeur, T., Alexa, M.: Photosketcher: Interactive sketch-based image synthesis. IEEE Comput. Graph. Appl. 31(6), 56–66 (2011)
Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in Annual Conference on Neural Information Processing Systems 2014, pp. 2672–2680 (2014)
He, Z., Zuo, W., Kan, M., Shan, S., Chen, X.: AttGAN: facial attribute editing by only changing what you want. IEEE Trans. Image Process. 28(11), 5464–5478 (2019)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017)
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018)
Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 1857–1865 (2017)
Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 105–114 (2017)
Lee, D., Kim, J., Moon, W.J., Ye, J.C.: CollaGAN: collaborative GAN for missing image data imputation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2487–2496 (2019)
Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances in Neural Information Processing Systems, pp. 700–708 (2017)
Liu, S., et al.: Face aging with contextual generative adversarial nets. In: ACM International Conference on Multimedia, pp. 82–90 (2018)
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)
Lu, Y., Tai, Y., Tang, C.: Attribute-guided face generation using conditional CycleGAN. In: ECCV, pp. 293–308 (2018)
Park, M., Kim, H.G., Ro, Y.M.: Photo-realistic facial emotion synthesis using multi-level critic networks with multi-level generative model. In: MultiMedia Modeling, pp. 3–15 (2019)
Pumarola, A., Agudo, A., MartĂnez, A.M., Sanfeliu, A., Moreno-Noguer, F.: GANimation: anatomically-aware facial animation from a single image. In: ECCV, pp. 835–851 (2018)
Wang, T., Liu, M., Zhu, J., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8798–8807 (2018)
Wang, X., Li, W., Mu, G., Huang, D., Wang, Y.: Facial expression synthesis by u-net conditional generative adversarial networks. In: ACM International Conference on Multimedia Retrieval, pp. 283–290 (2018)
Xiao, T., Hong, J., Ma, J.: ELEGANT: exchanging latent encodings with GAN for transferring multiple face attributes. In: ECCV, pp. 172–187 (2018)
Xu, T., et al.: AttnGAN: fine-grained text to image generation with attentional generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1316–1324 (2018)
Yi, Z., Zhang, H., Tan, P., Gong, M.: DualGAN: unsupervised dual learning for image-to-image translation. In: IEEE International Conference on Computer Vision, pp. 2868–2876 (2017)
Zhang, G., Kan, M., Shan, S., Chen, X.: Generative adversarial network with spatial attention for face attribute editing. In: ECCV, pp. 422–437 (2018)
Zhang, H., Xu, T., Li, H.: StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: IEEE International Conference on Computer Vision, pp. 5908–5916 (2017)
Zhang, H., et al.: StackGAN++: realistic image synthesis with stacked generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 1947–1962 (2019)
Zhang, R., et al.: Style separation and synthesis via generative adversarial networks. In: ACM International Conference on Multimedia, pp. 183–191 (2018)
Zhang, Z., Xie, Y., Yang, L.: Photographic text-to-image synthesis with a hierarchically-nested adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6199–6208 (2018)
Zhao, Y., Deng, B., Huang, J., Lu, H., Hua, X.S.: Stylized adversarial autoencoder for image generation. In: ACM International Conference on Multimedia, pp. 244–251 (2017)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision, pp. 2242–2251 (2017)
Acknowledgements
This work was supported by the National Key Research & Development Plan of China under Grant 2016YFB1001402, the National Natural Science Foundation of China (NSFC) under Grants 61632006, 61622211, and 61620106009, as well as the Fundamental Re-search Funds for the Central Universities under Grants WK3490000003 and WK2100100030.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, B., Chen, X., Hong, R., Chen, Z., Li, Y., Zha, ZJ. (2020). Joint Sketch-Attribute Learning for Fine-Grained Face Synthesis. In: Ro, Y., et al. MultiMedia Modeling. MMM 2020. Lecture Notes in Computer Science(), vol 11961. Springer, Cham. https://doi.org/10.1007/978-3-030-37731-1_64
Download citation
DOI: https://doi.org/10.1007/978-3-030-37731-1_64
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37730-4
Online ISBN: 978-3-030-37731-1
eBook Packages: Computer ScienceComputer Science (R0)