Abstract
Recent advances in conditional image generation have led to powerful personalized generation models that generate high-resolution artistic images based on simple text descriptions through tuning. However, the abuse of personalized generation models may also increase the risk of plagiarism and the misuse of artists’ painting styles. In this paper, we propose a novel method called Protecting Artworks from Personalizing Image Generative Models framework (PAG) to safeguard artistic images from the malicious use of generative models. By injecting learned target perturbations into the original artistic images, we aim to disrupt the tuning process and introduce the distortions that protect the authenticity and integrity of the artist’s style. Furthermore, human evaluations suggest that our PAG model offers a feasible and effective way to protect artworks, preventing the personalized generation models from generating similar images to the given artworks.
Z. Tan and S. Wang—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Crowson, K., et al.: VQGAN-CLIP: open domain image generation and editing with natural language guidance. http://arxiv.org/abs/2204.08583
Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873–12883 (2021)
Gal, R., Patashnik, O., Maron, H., Bermano, A.H., Chechik, G., Cohen-Or, D.: StyleGAN-NADA: clip-guided domain adaptation of image generators. ACM Trans. Graph. (TOG) 41(4), 1–13 (2022)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423 (2016)
Hu, E.J., et al.: LoRA: low-rank adaptation of large language models. http://arxiv.org/abs/2106.09685
Kwon, G., Ye, J.C.: CLIPstyler: image style transfer with a single text condition. http://arxiv.org/abs/2112.00374
Pinkney, J.N.M.: Pokemon blip captions. http://huggingface.co/datasets/lambdalabs/pokemon-blip-captions/ (2022)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)
Ramesh, A., et al.: Zero-shot text-to-image generation. http://arxiv.org/abs/2102.12092
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674–10685. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.01042, http://ieeexplore.ieee.org/document/9878449/
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: DreamBooth: fine tuning text-to-image diffusion models for subject-driven generation. http://arxiv.org/abs/2208.12242
Ryu, S.: Low-rank adaptation for fast text-to-image diffusion fine-tuning. http://github.com/cloneofsimo/lora, original-date: 2022–12-08T00:09:05Z
Saleh, B., Elgammal, A.: Large-scale classification of fine-art paintings: learning the right metric on the right feature. http://arxiv.org/abs/1505.00855
Shan, S., Cryan, J., Wenger, E., Zheng, H., Hanocka, R., Zhao, B.Y.: GLAZE: protecting artists from style mimicry by text-to-image models. http://arxiv.org/abs/2302.04222
Van Le, T., Phung, H., Nguyen, T.H., Dao, Q., Tran, N., Tran, A.: Anti-DreamBooth: protecting users from personalized text-to-image synthesis. http://arxiv.org/abs/2303.15433, version: 1
Acknowledgement
This research was funded by the National Natural Science Foundation of China under no. 62206225; Jiangsu Science and Technology Programme (Natural Science Foundation of Jiangsu Province) under no. BE2020006-4; Natural Science Foundation of the Jiangsu Higher Education Institutions of China under no. 22KJB520039; Xi’an Jiaotong-Liverpool University’s Research Development Fund in XJTLU under no. RDF-19-01-21.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Tan, Z., Wang, S., Yang, X., Huang, K. (2024). PAG: Protecting Artworks from Personalizing Image Generative Models. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14450. Springer, Singapore. https://doi.org/10.1007/978-981-99-8070-3_33
Download citation
DOI: https://doi.org/10.1007/978-981-99-8070-3_33
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8069-7
Online ISBN: 978-981-99-8070-3
eBook Packages: Computer ScienceComputer Science (R0)