Kota Izumi
ABSTRACT
Recently, a large-scale language-image multi-modal model, CLIP, has been used to realize language-based image translation in a zero-shot manner without training. In this study, we attempted to generate language-based decorative fonts for font images using CLIP. By the existing image style transfer methods using CLIP, stylized font images are usually only surrounded by decorations, and the characters themselves do not change significantly. On the other hand, in this study, we use CLIP and vector graphics image representation using a differentiable renderer to achieve a style transfer of text images that matches the input text. The experimental results show that the proposed method transfers the style of font images to match the given texts. In addition to text images, we confirmed that the proposed method was also able to transform the style of simple logo patterns based on the given texts.
- Gantugs Atarsaikhan, Brian Kenji Iwana, and Seiichi Uchida. 2018. Contained Neural Style Transfer for Decorated Logo Generation. In 13th IAPR International Workshop on Document Analysis Systems (2018).Google Scholar
- Kevin Frans, LB Soros, and Olaf Witkowski. 2021. CLIPDraw: Exploring text-to-drawing synthesis through language-image encoders. arXiv preprint arXiv:2106.14843 (2021).Google Scholar
- Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. 2021. StyleGAN-NADA: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00946 (2021).Google Scholar
- Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. Proc. of IEEE Computer Vision and Pattern Recognition (2016), 2414--2423.Google ScholarCross Ref
- Kwon Gihyun and Ye Jong Chul. 2022. CLIPStyler: Image style transfer with a single text condition. Proc. of IEEE Computer Vision and Pattern Recognition (2022), 18062--18071.Google Scholar
- Tero Karras, Samuli Laine, , and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. Proc. of IEEE Computer Vision and Pattern Recognition (2019), 4401--4410.Google ScholarCross Ref
- Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. 2020. Differentiable Vector Graphics Rasterization for Editing and Learning. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020), 193:1--193:15.Google Scholar
- Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021. Styleclip: Text-driven manipulation of stylegan imagery. arXiv preprint arXiv:2103.17249 (2021).Google Scholar
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Jack Clark Pamela Mishkin, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021).Google Scholar
- Peter Schaldenbrand, Zhixuan Liu, and Jean Oh. 2021. StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation. arXiv preprint arXiv:2202.12362 (2021).Google Scholar
- Wenjing Wang, Jiaying Liu, Shuai Yang, and Zongming Guo. 2019. Typography with decor: Intelligent text style transfer. Proc. of IEEE Computer Vision and Pattern Recognition (2019), 5889--5897.Google ScholarCross Ref
- Shuai Yang, Zhangyang Wang, Zhaowen Wang, Ning Xu, Jiaying Liu, and Zongming Guo. 2019. Controllable artistic text style transfer via shape-matching GAN. Proc. of IEEE International Conference on Computer Vision (2019), 4442--4451.Google ScholarCross Ref
Index Terms
- Zero-Shot Font Style Transfer with a Differentiable Renderer
Recommendations
Ribbon Font Neural Style Transfer for OpenType-SVG Font
SA '22: SIGGRAPH Asia 2022 PostersWe use existing machine learning neural style transfer model, differential rasterizer, for colored font design. The input of the proposed system is an existing TrueType font and the output is an neural style transferred OpenType-SVG color font. Each ...
Conditional Fast Style Transfer Network
ICMR '17: Proceedings of the 2017 ACM on International Conference on Multimedia RetrievalIn this paper, we propose a conditional fast neural style transfer network. We extend the network proposed as a fast neural style transfer network by Johnson et al. [1] so that the network can learn multiple styles at the same time. To do that, we add a ...
Laplacian-Steered Neural Style Transfer
MM '17: Proceedings of the 25th ACM international conference on MultimediaNeural Style Transfer based on Convolutional Neural Networks (CNN) aims to synthesize a new image that retains the high-level structure of a content image, rendered in the low-level texture of a style image. This is achieved by constraining the new ...
Comments