Abstract
The dissemination of open-source text-to-image generative models and the increasing quality of their output has led to a growth in interest in the field. The quality of the images greatly depends on the prompt used, i.e. a phrase that includes descriptive terms to be used as input on text-to-image model. However, choosing the right prompt is a complex task, often relying on a trial-and-error approach. In this paper, we introduce an evolutionary approach to prompt generation where users begin by creating a blueprint for what might be a candidate prompt and then initiate an evolutionary process to interactively explore the space of prompts encoded by the initial blueprint and according to their preferences. Our work is a step towards a more dynamic and interactive way to generate prompts that lead to a wide variety of visual outputs, with which users can easily obtain prompts that match their goals.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
https://promptbase.com/ accessed 2023.
- 3.
See examples in https://beta.openai.com/examples accessed 2023.
- 4.
https://openart.ai/promptbook accessed 2023.
- 5.
- 6.
https://promptomania.com/stable-diffusion-prompt-builder/ accessed 2023.
- 7.
https://github.com/krea-ai/open-prompts accessed 2023.
- 8.
- 9.
https://github.com/MagnusPetersen/EvoGen-Prompt-Evolution accessed 2023.
- 10.
The source code of the presented approach can be found at: https://cdv.dei.uc.pt/metaprompter.
References
Crowson, K., et al.: VQGAN-CLIP: open domain image generation and editing with natural language guidance. In: Avidan, S., Brostow, G., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. ECCV 2022. LNCS, vol. 13697, pp. 88–105. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19836-6_6
Cunha, J.M.: Generation of concept representative symbols: towards visual conceptual blending, Ph. D. thesis, University of Coimbra (2022)
Cunha, J.M., Martins, P., Machado, P.: Let’s figure this out: a roadmap for visual conceptual blending. In: Proceedings of the Eleventh International Conference on Computational Creativity (2020)
Daras, G., Dimakis, A.G.: Discovering the hidden vocabulary of DALLE-2. CoRR abs/2206.00169 (2022)
Ge, S., Parikh, D.: Visual conceptual blending with large-scale language and vision models. In: Proceedings of the 12th International Conference on Computational Creativity (2021)
Heikkila, M.: This artist is dominating AI-generated art. And he’s not happy about it. https://www.technologyreview.com/2022/09/16/1059598/ (2022). Accessed Jan 2023
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. CoRR abs/2107.13586 (2021). https://arxiv.org/abs/2107.13586
Liu, V., Chilton, L.B.: Design guidelines for prompt engineering text-to-image generative models. In: Barbosa, S.D.J., Lampe, C., Appert, C., Shamma, D.A., Drucker, S.M., Williamson, J.R., Yatani, K. (eds.) CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022–5 May 2022, pp. 1–23. ACM (2022)
Liu, V., Vermeulen, J., Fitzmaurice, G., Matejka, J.: 3DALL-E: Integrating text-to-image AI in 3D design workflows. arXiv preprint arXiv:2210.11603 (2022)
Oppenlaender, J.: The creativity of text-to-image generation. In: 25th International Academic Mindtrek conference, Academic Mindtrek 2022, Tampere, Finland, 16–18 November 2022, pp. 192–202. ACM (2022)
Oppenlaender, J.: A taxonomy of prompt modifiers for text-to-image generation. arXiv preprint arXiv:2204.13988 (2022)
Pavlichenko, N., Ustalov, D.: Best prompts for text-to-image models and how to find them. CoRR abs/2209.11711 (2022)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18–24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 8748–8763. PMLR (2021)
Ramesh, A., et al.: Zero-shot text-to-image generation. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18–24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 8821–8831. PMLR (2021)
Reynolds, L., McDonell, K.: Prompt programming for large language models: beyond the few-shot paradigm. In: Kitamura, Y., Quigley, A., Isbister, K., Igarashi, T. (eds.) ACM CHI Conference on Human Factors in Computing Systems. ACM (2021)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, 18–24 June 2022, pp. 10674–10685. IEEE (2022)
Smith, E.: A traveler’s guide to the latent space. https://sweet-hall-e72.notion.site/A-Traveler-s-Guide-to-the-Latent-Space-85efba7e5e6a40e5bd3cae980f30235f (2022). Accessed Jan 2023
Wang, Z.J., Montoya, E., Munechika, D., Yang, H., Hoover, B., Chau, D.H.: DiffusionDB: a large-scale prompt gallery dataset for text-to-image generative models. CoRR abs/2210.14896 (2022)
Acknowledgements
This research was partially funded by the FCT - Foundation for Science and Technology, I.P./MCTES through national funds (PIDDAC), within the scope of CISUC R &D Unit - UIDB/00326/2020 or project code UIDP/00326/2020.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Martins, T., Cunha, J.M., Correia, J., Machado, P. (2023). Towards the Evolution of Prompts with MetaPrompter. In: Johnson, C., Rodríguez-Fernández, N., Rebelo, S.M. (eds) Artificial Intelligence in Music, Sound, Art and Design. EvoMUSART 2023. Lecture Notes in Computer Science, vol 13988. Springer, Cham. https://doi.org/10.1007/978-3-031-29956-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-29956-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-29955-1
Online ISBN: 978-3-031-29956-8
eBook Packages: Computer ScienceComputer Science (R0)