research-article

Interactive Image Style Transfer Guided by Graffiti

Authors:

Guorui FengAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 6685 - 6694

https://doi.org/10.1145/3581783.3612203

Published: 27 October 2023 Publication History

Abstract

Neural style transfer (NST) can quickly produce impressive artistic images, which allows ordinary people to become painter. The brushstrokes of stylized images created by the current NST methods are often unpredictable, which does not conform to the logic of the artist's drawing. At the same time, the style distribution of the generated stylized image texture differs from the real artwork. In this paper, we propose an interactive image style transfer network (IIST-Net) to overcome the above limitations. Our IIST-Net can generate stylized results for brushstrokes in arbitrary directions guided by graffiti curves. The style distribution of these stylized results is closer to the real-life artwork. Specifically, we design an Interactive Brush-texture Generation (IBG) module in IIST-Net to progressively generate controllable brush-textures. Then, two encoders are introduced to embed the interactive brush-textures into the content image in the deep space for producing the fused content feature map. The Multilayer Style Attention (MSA) module is proposed to further distill multi-scale style features and transfer them to the fused content feature map for obtaining the final stylized feature map with controllable brushstrokes. Additionally, we adopt the content loss, style loss, adversarial loss and contrastive loss to jointly supervise the proposed network. Experimental comparisons have demonstrated the effectiveness of our proposed method for creating controllable and realistic stylized images.

References

[1]

2011. Wiki Art Gallery, Inc.: A Case for Critical Thinking. Issues in Accounting Education 26, 3 (2011), 593?608.

[2]

Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu. 2021a. Artistic Style Transfer with Internal-external Learning and Contrastive Learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. 26561--26573.

[3]

Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu. 2021b. DualAST: Dual Style-Learning Networks for Artistic Style Transfer. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE, 872--881.

[4]

Yingying Deng, Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang, and Changsheng Xu. 2022. StyTr(^mbox2 ): Image Style Transfer with Transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 11316--11326.

[5]

Yingying Deng, Fan Tang, Weiming Dong, Wen Sun, Feiyue Huang, and Changsheng Xu. 2020. Arbitrary Style Transfer via Multi-Adaptation Network. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020. ACM, 2719--2727.

Digital Library

[6]

Oriel Frigo, Neus Sabater, Julie Delon, and Pierre Hellier. 2016. Split and Match: Example-Based Adaptive Patch Sampling for Unsupervised Style Transfer. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 553--561.

[7]

Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015. Texture Synthesis Using Convolutional Neural Networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada. 262--270.

[8]

Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 2414--2423.

[9]

Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. 2017. Controlling Perceptual Factors in Neural Style Transfer. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 3730--3738.

[10]

Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David Salesin. 2001. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001, Los Angeles, California, USA, August 12-17, 2001, Lynn Pocock (Ed.). ACM, 327--340.

Digital Library

[11]

Xun Huang and Serge J. Belongie. 2017. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, 1510--1519.

[12]

Yongcheng Jing and Yang Liu and. 2018. Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII (Lecture Notes in Computer Science, Vol. 11217). Springer, 244--260.

[13]

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 9906). Springer, 694--711.

[14]

Henry Kang, Seungyong Lee, and Charles K. Chui. 2009. Flow-Based Image Abstraction. IEEE Trans. Vis. Comput. Graph., Vol. 15, 1 (2009), 62--76.

Digital Library

[15]

Nicholas I. Kolkin, Jason Salavon, and Gregory Shakhnarovich. 2019. Style Transfer by Relaxed Optimal Transport and Self-Similarity. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 10051--10060.

[16]

Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang, and Bjö rn Ommer. 2019. Content and Style Disentanglement for Artistic Style Transfer. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 4421--4430.

[17]

Hochang Lee, Sanghyun Seo, and Kyunghyun Yoon. 2011. Directional texture transfer with edge enhancement. Comput. Graph., Vol. 35, 1 (2011), 81--91.

Digital Library

[18]

Chuan Li and Michael Wand. 2016. Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 2479--2486.

[19]

Shaohua Li, Xinxing Xu, Liqiang Nie, and Tat-Seng Chua. 2017c. Laplacian-Steered Neural Style Transfer. In Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017. ACM, 1716--1724.

Digital Library

[20]

Xueting Li, Sifei Liu, Jan Kautz, and Ming-Hsuan Yang. 2019. Learning Linear Transformations for Fast Image and Video Style Transfer. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 3809--3817.

[21]

Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. 2017a. Universal Style Transfer via Feature Transforms. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. 386--396.

Digital Library

[22]

Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. 2017b. Demystifying Neural Style Transfer. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, Carles Sierra (Ed.). ijcai.org, 2230--2236.

[23]

Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding. 2021. AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. IEEE, 6629--6638.

[24]

Xiao-Chang Liu, Ming-Ming Cheng, Yu-Kun Lai, and Paul L. Rosin. 2017. Depth-aware neural style transfer. In 15th International Symposium on Non-Photorealistic Animation and Rendering, NPAR@Expressive 2017, Los Angeles, CA, USA, July 29-30, 2017, Proceedings. ACM, 4:1--4:10.

[25]

Dae Young Park and Kwang Hee Lee. 2019. Arbitrary Style Transfer With Style-Attentional Networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 5880--5888.

[26]

Yurui Ren, Xiaoqing Fan, Ge Li, Shan Liu, and Thomas H. Li. 2022. Neural Texture Extraction and Distribution for Controllable Person Image Synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 13525--13534.

[27]

Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Björn Ommer. 2018. A Style-Aware Content Loss for Real-Time HD Style Transfer. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII (Lecture Notes in Computer Science, Vol. 11212). Springer, 715--731.

[28]

Ahmed Selim, Mohamed A. Elgharib, and Linda Doyle. 2016. Painting style transfer for head portraits using convolutional neural networks. ACM Trans. Graph., Vol. 35, 4 (2016), 129:1--129:18.

Digital Library

[29]

Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. 2018. Avatar-Net: Multi-Scale Zero-Shot Style Transfer by Feature Decoration. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 8242--8250.

[30]

Jan Svoboda, Asha Anoosheh, Christian Osendorfer, and Jonathan Masci. 2020. Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. Computer Vision Foundation / IEEE, 13813--13822.

[31]

Zhengyan Tong, Xuanhong Chen, Bingbing Ni, and Xiaohang Wang. 2021. Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. AAAI Press, 609--616.

[32]

Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S. Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 (JMLR Workshop and Conference Proceedings, Vol. 48), Maria-Florina Balcan and Kilian Q. Weinberger (Eds.). JMLR.org, 1349--1357.

[33]

Quan Wang, Sheng Li, Xinpeng Zhang, and Guorui Feng. 2022. Multi-granularity Brushstrokes Network for Universal Style Transfer. ACM Trans. Multim. Comput. Commun. Appl., Vol. 18, 4 (2022), 107:1--107:17.

Digital Library

[34]

Xin Wang, Geoffrey Oxholm, Da Zhang, and Yuan-Fang Wang. 2017. Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 7178--7186.

[35]

Joachim Weickert and Hanno Scharr. 2002. A Scheme for Coherence-Enhancing Diffusion Filtering with Optimized Rotation Invariance. J. Vis. Commun. Image Represent., Vol. 13, 1--2 (2002), 103--118.

Digital Library

[36]

Hao Wu, Zhengxing Sun, and Weihang Yuan. 2018. Direction-aware Neural Style Transfer. In 2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22-26, 2019, Susanne Boll, Kyoung Mu Lee, Jiebo Luo, Wenwu Zhu, Hyeran Byun, Chang Wen Chen, Rainer Lienhart, and Tao Mei (Eds.). ACM, 1163--1171.

[37]

Lingchen Yang, Lumin Yang, Mingbo Zhao, and Youyi Zheng. 2018. Controlling Stroke Size in Fast Style Transfer with Recurrent Convolutional Neural Network. Comput. Graph. Forum, Vol. 37, 7 (2018), 97--107.

[38]

Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, and Jun Wang. 2019. Attention-Aware Multi-Stroke Style Transfer. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 1467--1475.

[39]

Yulun Zhang, Chen Fang, Yilin Wang, Zhaowen Wang, Zhe Lin, Yun Fu, and Jimei Yang. 2019. Multimodal Style Transfer via Graph Cuts. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 5942--5950.

[40]

Yuxin Zhang, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, Tong-Yee Lee, and Changsheng Xu. 2022. Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning. In SIGGRAPH '22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, BC, Canada, August 7 - 11, 2022. ACM, 12:1--12:8.

Digital Library

[41]

Yexun Zhang, Ya Zhang, and Wenbin Cai. 2020. A Unified Framework for Generalizable Style Transfer: Style and Content Separation. IEEE Trans. Image Process., Vol. 29 (2020), 4085--4098.

Digital Library

[42]

Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018. Places: A 10 Million Image Database for Scene Recognition. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 40, 6 (2018), 1452--1464.

Cited By

Zhou XZheng YYang J(2025)Bridging the metrics gap in image style transfer: A comprehensive survey of models and criteriaNeurocomputing10.1016/j.neucom.2025.129430624(129430)Online publication date: Apr-2025
https://doi.org/10.1016/j.neucom.2025.129430
Zhang HWang QFeng G(2023)Artistic image adversarial attack via style perturbationMultimedia Systems10.1007/s00530-023-01183-x29:6(3745-3755)Online publication date: 29-Sep-2023
https://dl.acm.org/doi/10.1007/s00530-023-01183-x

Index Terms

Interactive Image Style Transfer Guided by Graffiti
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation

Recommendations

Rethinking Neural Style Transfer: Generating Personalized and Watermarked Stylized Images
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Neural style transfer (NST) has attracted many research interests recent years. The existing NST schemes could only generate one stylized image from a content-style image pair. They are weak in creating diverse and personalized artistic styles. On the ...
Multi-granularity Brushstrokes Network for Universal Style Transfer
Neural style transfer has been developed in recent years, where both performance and efficiency have been greatly improved. However, most existing methods do not transfer the brushstrokes information of style images well. In this article, we address this ...
Reimagining Animation Making through Style Transfer
SA '23: SIGGRAPH Asia 2023 Art Papers

"Dissolution” and “Cunabula” are experimental animation projects that explore the potential of artistic style transfer with neural networks in animation filmmaking. The objective is to discover novel production methods using fast, arbitrary, and stroke-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Shanghai Science and Technology Program

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
149
Total Downloads

Downloads (Last 12 months)87
Downloads (Last 6 weeks)6

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou XZheng YYang J(2025)Bridging the metrics gap in image style transfer: A comprehensive survey of models and criteriaNeurocomputing10.1016/j.neucom.2025.129430624(129430)Online publication date: Apr-2025
https://doi.org/10.1016/j.neucom.2025.129430
Zhang HWang QFeng G(2023)Artistic image adversarial attack via style perturbationMultimedia Systems10.1007/s00530-023-01183-x29:6(3745-3755)Online publication date: 29-Sep-2023
https://dl.acm.org/doi/10.1007/s00530-023-01183-x

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten