Abstract
This paper mainly studies the Neural Style Transfer (NST) problem based on convolutional neural networks (CNN). Existing deep style migration algorithms do not mimic the styles to a reasonable position. To solve the problem, this paper proposes a multi-scale style transfer algorithm based on deep semantic matching. For purpose of guiding the correct migration of the style, we use the priori spatial segmentation and illumination information of the input image to integrate the deep semantic information. First, we find that spatial division and illumination analysis are two important visual understanding approaches for artists to make each painting decision. In order to simulate these two visual understanding approaches, this paper defines the DSS (deep semantic space), which contains spatial segmentation and contextual illumination information. The semantic exists in the form of CNN (convolution neural network) characteristic graph. Second, we propose a deep semantic loss function based on DSS matching and nearest neighbor search to optimize the effect of deep style migration. Third, we propose a multi-scale optimization strategy for improving the speed of our method. The experiments show that our method can reasonably synthesize images in spatial structures. The placement of each style is more reasonable and has a good visual aesthetic.
Similar content being viewed by others
References
Ashikhmin N (2003) Fast texture transfer. IEEE Comput Graph Appl 23(4):38–43
Barnes C, Shechtman E, Finkelstein A et al (2009) PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):24
Baxter W, Govindaraju N (2010) Simple data-driven modeling of brushes. In: Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. ACM, 135-142
Champandard AJ (2015) Deep forger: Paint photos in the style of famous artists. https://deepforger.com/
Champandard AJ (2016) Semantic style transfer and turning two-bit doodles into fine artworks.arXiv preprintarXiv:1603.01768
Chen ZL, Kim B, Ito D et al (2015) Wetbrush: GPU-based 3D painting simulation at the bristle level. ACM Trans Graph 34(6):200:1-200:11
Collomosse J (2012) Image and video-based artistic stylisation, vol 42. Springer Science & Business Media
Drori I, Cohen-Or D, Yeshurun H (2003) Example based style synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol 2. IEEE, New York, pp II–143
Efros AA, Freeman WT (2001) Image quilting for texture synthesis and transfer. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 341-346
Elad M, Milanfar P (2017) Style transfer via texture synthesis. IEEE Trans Image Process 26(5):2338–2351
Fišer J, Jamriška O, Lukáč M et al (2016) StyLit: illumination-guided example-based stylization of 3D renderings[J]. ACM Trans Graph 35(4):92
Frigo O, Sabater N, Delon J, Hellier P (2016) Split and match: Example-based adaptive patch sampling for unsupervised style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 553–561
Gatys L, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. Advances in Neural Information Processing Systems, 262–270
Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. Computer Vision and Pattern Recognition (CVPR), 2016 Conference on. IEEE, New York, pp 2414-2423
Gatys LA, Ecker AS, Bethge M et al (2017) Controlling perceptual factors in neural style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3985-3993
Gooch B, Gooch A (2001) Non-photorealistic rendering. A. K. Peters, Ltd., Natick
He K, Sun J, Tang X (2010) Guided image filtering. European Conference on Computer Vision. Springer, Berlin, 1-14
Heckbert PS (1990) Adaptive radiosity textures for bidirectional ray tracing[J]. ACM SIGGRAPH Comput Graph 24(4):145–154
Hertzmann A (1998) Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, 453-460
Hertzmann A, Jacobs CE, Oliver N et al (2001) Image analogies//Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, New York, pp 327-340
Huang X, Belongie S (2017) Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. In: Proceedings of the IEEE International Conference on Computer Vision(ICCV), v 2017-October, p 1510-1519
Jing Y, Yang Y, Feng Z et al (2019) Neural style transfer: A Review//Computer Vision and Pattern Recognition (CVPR), 2019 Conference on. IEEE, arXiv:1705.04058
Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. Springer, Cham, 694-711
Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: Proc. European Conference on Computer Vision (ECCV), pp 694–711
Lee H, Seo S, Ryoo S et al (2010) Directional texture transfer. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering. ACM, 43-48
Li C, Wand M (2016) Combining markov random fields and convolutional neural networks for image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2479-2486
Li Y, Wang N, Liu J et al (2017) Demystifying neural style transfer. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press, Palo Alto, pp 2230-2236
Li S, Xu X, Nie L, Chua T-S (2017) Laplacian-steered neural style transfer. In: ACM Multimedia Conference (MM)
Liao J, Yao Y, Yuan L et al (2017) Visual attribute transfer through deep image analogy. ACM Trans Graph 36(4):120
Liu XC, Cheng MM, Lai YK et al (2017) Depth-aware neural style transfer. In: Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. ACM, New York, 4
Prisma L, Prisma (2016) Turn memories into art using artificial intelligence. https://prisma-ai.com
Sanakoyeu A, Kotovenko D, Lang S et al (2018) A style-aware content loss for real-time HD style transfer[J]. arXiv:180710201v2, ECCV
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Strothotte T, Schlechtweg S (2002) Non-photorealistic computer graphics: modeling, rendering, and animation. Morgan Kaufmann
Ulyanov D, Lebedev V, Lempitsky V (2016) Texture networks: feed-forward synthesis of textures and stylized images. In: International Conference on Machine Learning. 1349-1357
Wang CM, Wang RJ (2007) Image-based color ink diffusion rendering. IEEE Trans Vis Comput Graph 13(2):235–246
Wang M, Wang B, Fei Y et al (2014) Towards photo watercolorization with artistic verisimilitude. IEEE Trans Vis Comput Graph 10(20):1451–1460
Wilmot P, Risser E, Barnes C (2017) Stable and controllable neural texture synthesis and style transfer using histogram losses.arXiv preprintarXiv:1701.08893
Yosinski J, Clune J, Fuchs T et al (2015) Understanding neural networks through deep visualization. In: ICML Workshop on Deep Learning.
Zeng K, Zhao M, Xiong C et al (2009) From image parsing to painterly rendering. ACM Trans Graph 29(1):2
Acknowledgements
This work was also supported in part by the key project of Trico-Robot plan of NSFC under grant No. 91748208, key project of Shaanxi province No.2018ZDCXL-GY-0607, the Fundamental Research Funds for the Central Universities No. XJJ2018254, and China Postdoctoral Science Foundation No. 2018M631164.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Yu, J., Jin, L., Chen, J. et al. Deep semantic space guided multi-scale neural style transfer. Multimed Tools Appl 81, 3915–3938 (2022). https://doi.org/10.1007/s11042-021-11694-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-021-11694-2