Abstract
The average user may have little to no artistic skills but can describe what they envision in words. The user-provided text can be instantly transformed into a realistic image with the aid of generative neural architectures. This study intends to propose a novel approach to generate a facial image based on a user-given textual description. Prior works focus less on the manipulation aspects, hence the approach also emphasizes on manipulating and modifying the image generated, based on additional textual descriptions as required to further refine the expected face. It consists of a multi-level Vector-Quantized Variational Auto Encoder (VQVAE) that provides the image encodings, the Contrastive Language-Image Pre-Training (CLIP) module to interpret the texts and compute how close the final image encodings and the text are with each other within a common space, and a StyleGAN2 to decode and generate the required image output. The combination of such components within the architecture is unseen in previous studies and yields promising results, capturing the context of the text and generating realistic good quality images of human faces.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Razavi, A., Van den Oord, A., Vinyals, O.: Generating diverse high-fidelity images with VQ-VAE-2. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. PMLR (2021)
Karras, T., et al.: Analyzing and improving the image quality of StyleGAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)
Van Den Oord, A., Vinyals, O.: Neural discrete representation learning. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
Hinz, T., Heinrich, S., Wermter, S.: Semantic object accuracy for generative text-to-image synthesis. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
Xia, W., et al.: TediGAN: text-guided diverse face image generation and manipulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
Patashnik, O., et al.: StyleCLIP: text-driven manipulation of StyleGAN imagery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)
Di, X., Patel, V.M.: Face synthesis from visual attributes via sketch using conditional VAEs and GANs. arXiv preprint: arXiv:1801.00077 (2017)
Ramesh, A., et al.: Zero-shot text-to-image generation. In: International Conference on Machine Learning. PMLR (2021)
Deorukhkar, K., Kadamala, K., Menezes, E.: FGTD: face generation from textual description. In: Ranganathan, G., Fernando, X., Shi, F. (eds.) Inventive Communication and Computational Technologies. Lecture Notes in Networks and Systems, vol. 311. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-5529-6_43
Wang, Z., et al.: CLIP-GEN: language-free training of a text-to-image generator with CLIP. arXiv preprint: arXiv:2203.00386 (2022)
Khan, M.Z., et al.: A realistic image generation of face from text description using the fully trained generative adversarial networks. IEEE Access 9, 1250–1260 (2020)
Zhang, C., Peng, Y.: Stacking VAE and GAN for context-aware text-to-image generation. In: 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM). IEEE (2018)
Liao, W., et al.: Text to image generation with semantic-spatial aware GAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
Nichol, A., et al.: GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint: arXiv:2112.10741 (2021)
Sanh, V., et al.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint: arXiv:1910.01108 (2019)
Serengil, S.I., Ozpinar, A.: LightFace: a hybrid deep face recognition framework. In: 2020 Innovations in Intelligent Systems and Applications Conference (ASYU), pp. 1–5 (2020). https://doi.org/10.1109/ASYU50717.2020.9259802
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Srinivasa, A., Praveen, A., Mavathur, A., Pothumarthi, A., Arya, A., Agarwal, P. (2023). Text Guided Facial Image Synthesis Using StyleGAN and Variational Autoencoder Trained CLIP. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2023. Lecture Notes in Computer Science(), vol 14126. Springer, Cham. https://doi.org/10.1007/978-3-031-42508-0_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-42508-0_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42507-3
Online ISBN: 978-3-031-42508-0
eBook Packages: Computer ScienceComputer Science (R0)