Customizable Text-to-Image Modeling by Contrastive Learning on Adjustable Word-Visual Pairs

Abstract

Co-creation with AI is trending and AI-generation of images from textual descriptions has shown advanced and attractive capabilities. However, commonly trained machine-learning models or built AI-based systems may have deficient points to generate satisfied results for personal usage or novice users of painting or AI co-creation, maybe because of deficient understanding of personal textual expressions or low customization of trained text-to-image machine learning models. Therefore, we assist in creating flexible and diverse visual contents from textual descriptions, by developing neural-networks models with machine learning. In modeling, we generate synthesized images using word-visual co-occurrence by Transformer model and synthesize images by decoding visual tokens. To improve visual and textual expressions and their relevance with more diversities, we utilize contrastive learning applying on texts, images, or pairs of texts and images. In the experimental results of a dataset of birds, we showed that the rendering quality was required of models with some scale neural-networks, and necessary training process with fined training by applying relatively low learning rates until the end of training. We further showed contrastive learning was possible for improvement of visual and textual expressions and their relevance.