Skip to main content
Log in

Deep semantic space guided multi-scale neural style transfer

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper mainly studies the Neural Style Transfer (NST) problem based on convolutional neural networks (CNN). Existing deep style migration algorithms do not mimic the styles to a reasonable position. To solve the problem, this paper proposes a multi-scale style transfer algorithm based on deep semantic matching. For purpose of guiding the correct migration of the style, we use the priori spatial segmentation and illumination information of the input image to integrate the deep semantic information. First, we find that spatial division and illumination analysis are two important visual understanding approaches for artists to make each painting decision. In order to simulate these two visual understanding approaches, this paper defines the DSS (deep semantic space), which contains spatial segmentation and contextual illumination information. The semantic exists in the form of CNN (convolution neural network) characteristic graph. Second, we propose a deep semantic loss function based on DSS matching and nearest neighbor search to optimize the effect of deep style migration. Third, we propose a multi-scale optimization strategy for improving the speed of our method. The experiments show that our method can reasonably synthesize images in spatial structures. The placement of each style is more reasonable and has a good visual aesthetic.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Ashikhmin N (2003) Fast texture transfer. IEEE Comput Graph Appl 23(4):38–43

    Article  Google Scholar 

  2. Barnes C, Shechtman E, Finkelstein A et al (2009) PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):24

    Article  Google Scholar 

  3. Baxter W, Govindaraju N (2010) Simple data-driven modeling of brushes. In: Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. ACM, 135-142

  4. Champandard AJ (2015) Deep forger: Paint photos in the style of famous artists. https://deepforger.com/

  5. Champandard AJ (2016) Semantic style transfer and turning two-bit doodles into fine artworks.arXiv preprintarXiv:1603.01768

  6. Chen ZL, Kim B, Ito D et al (2015) Wetbrush: GPU-based 3D painting simulation at the bristle level. ACM Trans Graph 34(6):200:1-200:11

  7. Collomosse J (2012) Image and video-based artistic stylisation, vol 42. Springer Science & Business Media

  8. Drori I, Cohen-Or D, Yeshurun H (2003) Example based style synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol 2. IEEE, New York, pp II–143

  9. Efros AA, Freeman WT (2001) Image quilting for texture synthesis and transfer. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 341-346

  10. Elad M, Milanfar P (2017) Style transfer via texture synthesis. IEEE Trans Image Process 26(5):2338–2351

    Article  MathSciNet  Google Scholar 

  11. Fišer J, Jamriška O, Lukáč M et al (2016) StyLit: illumination-guided example-based stylization of 3D renderings[J]. ACM Trans Graph 35(4):92

    Article  Google Scholar 

  12. Frigo O, Sabater N, Delon J, Hellier P (2016) Split and match: Example-based adaptive patch sampling for unsupervised style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 553–561

  13. Gatys L, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. Advances in Neural Information Processing Systems, 262–270

  14. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. Computer Vision and Pattern Recognition (CVPR), 2016 Conference on. IEEE, New York, pp 2414-2423

  15. Gatys LA, Ecker AS, Bethge M et al (2017) Controlling perceptual factors in neural style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3985-3993

  16. Gooch B, Gooch A (2001) Non-photorealistic rendering. A. K. Peters, Ltd., Natick

  17. He K, Sun J, Tang X (2010) Guided image filtering. European Conference on Computer Vision. Springer, Berlin, 1-14

  18. Heckbert PS (1990) Adaptive radiosity textures for bidirectional ray tracing[J]. ACM SIGGRAPH Comput Graph 24(4):145–154

    Article  Google Scholar 

  19. Hertzmann A (1998) Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, 453-460

  20. Hertzmann A, Jacobs CE, Oliver N et al (2001) Image analogies//Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, New York, pp 327-340

  21. Huang X, Belongie S (2017) Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. In: Proceedings of the IEEE International Conference on Computer Vision(ICCV), v 2017-October, p 1510-1519

  22. Jing Y, Yang Y, Feng Z et al (2019) Neural style transfer: A Review//Computer Vision and Pattern Recognition (CVPR), 2019 Conference on. IEEE, arXiv:1705.04058

  23. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. Springer, Cham, 694-711

  24. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: Proc. European Conference on Computer Vision (ECCV), pp 694–711

  25. Lee H, Seo S, Ryoo S et al (2010) Directional texture transfer. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering. ACM, 43-48

  26. Li C, Wand M (2016) Combining markov random fields and convolutional neural networks for image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2479-2486

  27. Li Y, Wang N, Liu J et al (2017) Demystifying neural style transfer. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press, Palo Alto, pp 2230-2236

  28. Li S, Xu X, Nie L, Chua T-S (2017) Laplacian-steered neural style transfer. In: ACM Multimedia Conference (MM)

  29. Liao J, Yao Y, Yuan L et al (2017) Visual attribute transfer through deep image analogy. ACM Trans Graph 36(4):120

    Article  Google Scholar 

  30. Liu XC, Cheng MM, Lai YK et al (2017) Depth-aware neural style transfer. In: Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. ACM, New York, 4

  31. Prisma L, Prisma (2016) Turn memories into art using artificial intelligence. https://prisma-ai.com

  32. Sanakoyeu A, Kotovenko D, Lang S et al (2018) A style-aware content loss for real-time HD style transfer[J]. arXiv:180710201v2, ECCV

  33. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  34. Strothotte T, Schlechtweg S (2002) Non-photorealistic computer graphics: modeling, rendering, and animation. Morgan Kaufmann

  35. Ulyanov D, Lebedev V, Lempitsky V (2016) Texture networks: feed-forward synthesis of textures and stylized images. In: International Conference on Machine Learning. 1349-1357

  36. Wang CM, Wang RJ (2007) Image-based color ink diffusion rendering. IEEE Trans Vis Comput Graph 13(2):235–246

    Article  Google Scholar 

  37. Wang M, Wang B, Fei Y et al (2014) Towards photo watercolorization with artistic verisimilitude. IEEE Trans Vis Comput Graph 10(20):1451–1460

    Article  Google Scholar 

  38. Wilmot P, Risser E, Barnes C (2017) Stable and controllable neural texture synthesis and style transfer using histogram losses.arXiv preprintarXiv:1701.08893

  39. Yosinski J, Clune J, Fuchs T et al (2015) Understanding neural networks through deep visualization. In: ICML Workshop on Deep Learning.

  40. Zeng K, Zhao M, Xiong C et al (2009) From image parsing to painterly rendering. ACM Trans Graph 29(1):2

    Article  Google Scholar 

Download references

Acknowledgements

This work was also supported in part by the key project of Trico-Robot plan of NSFC under grant No. 91748208, key project of Shaanxi province No.2018ZDCXL-GY-0607, the Fundamental Research Funds for the Central Universities No. XJJ2018254, and China Postdoctoral Science Foundation No. 2018M631164.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Li Jin.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, J., Jin, L., Chen, J. et al. Deep semantic space guided multi-scale neural style transfer. Multimed Tools Appl 81, 3915–3938 (2022). https://doi.org/10.1007/s11042-021-11694-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11694-2

Keywords

Navigation