Skip to main content
SpringerLink
Log in

Photographic painting style transfer using convolutional neural networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We propose a novel automatic photographic painting style technique with a single example image by using Convolutional Neural Networks (CNN). The photographic painting style is a challenging problem in the research community. Even though, researchers have been trying to obtain good results on painting style, but not much has been done on photographic stylization. Portrait painting techniques are mainly designed for the graphite style and/or are based on image analogies; an example painting as well as its original unpainted version are required. This preceding issue is a motivation of our proposed methods. As a result, our method extends the limits of their domain of applicability. We present a novel multi-convolutional-learning technique that is developed for both images (NPR/PR) labeling, style transmission and elevating a particular unified CNN model per weight sharing. A new painting technique is generated that follows the example style in the example image and maintains the integrity of facial structures. We believe this novel interpretation connects these two important research fields and could enlighten future researches. Moreover, our proposed technique is not restricted to headshot images or specific styles as our method can also change the photographic painting style in the wild.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Helen (2016) http://www.ifp.illinois.edu/vuongle2/helen/

  2. Printerest (2017) https://www.pinterest.com/

  3. Ashikhmin M (2001) Synthesizing natural textures. In: Proceedings of the 2001 symposium on interactive 3D graphics. ACM, pp 217–226

  4. Ashikhmin N (2003) Fast texture transfer. IEEE Comput Graph Appl 23(4):38–43

    Article  Google Scholar 

  5. Chen H, Liang L, Xu Y-Q, Shum H-Y, Zheng N-N (2003) Example-based automatic portraiture. Chinese journal of computers-chinese edition- 26 2:147–152

    Google Scholar 

  6. Chen H, Liu Z, Rose C, Xu Y, Shum H-Y, Salesin D (2004) Example-based composite sketching of human portraits. In: Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering. ACM, pp 95–153

  7. Chen H, Xu Y-Q, Shum H-Y, Zhu S-C, Zheng N-N (2001) Example-based facial sketch generation with non-parametric sampling. In: Proceedings 8th IEEE international conference on computer vision, ICCV 2001, vol 2. IEEE, pp 433–438

  8. Chen H, Zheng N-N, Liang L, Li Y, Xu Y-Q, Shum H-Y (2002) Pictoon: a personalized image-based cartoon system. In: Proceedings of the tenth ACM international conference on multimedia. ACM, pp 171–178

  9. Collomosse JP, Hall PM (2005) Genetic paint: A search for salient paintings. In: Workshops on applications of evolutionary computation. Springer, pp 437–447

  10. Efros AA, Freeman WT (2001) Image quilting for texture synthesis and transfer. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, pp 341–346

  11. Efros AA, Leung TK (1999) Texture synthesis by non-parametric sampling. In: The proceedings of the 7th IEEE international conference on computer vision, 1999, vol 2. IEEE, pp 1033–1038

  12. Farabet C, Couprie C, Najman L, LeCun Y (2013) Learning hierarchical features for scene labeling. IEEE Trans Pattern Anal Mach Intell 35(8):1915–1929

    Article  Google Scholar 

  13. Gao L, Guo Z, Zhang H, Xu X, Shen HT (2017) Video captioning with attention-based lstm and semantic consistency. IEEE Trans Multimed 19(9):2045–2055

    Article  Google Scholar 

  14. Gatys L, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Advances in neural information processing systems, pp 262–270

  15. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2414– 2423

  16. Gooch B, Reinhard E, Gooch A (2004) Human facial illustrations creation and psychophysical evaluation. ACM Trans Graph 23(1):27–44

    Article  Google Scholar 

  17. Gu S, Chen C, Liao J, Yuan L (2018) Arbitrary style transfer with deep feature reshuffle. CoRR: 1805.04103

  18. Hertzmann A (1998) Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, pp 453– 460

  19. Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin DH (2001) Image analogies. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, pp 327–340

  20. Huang X, Belongie S (2017) Arbitrary style transfer in real-time with adaptive instance normalization. In: IEEE international conference on computer vision (ICCV)

  21. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, pp 694–711

  22. Kim SY, Maciejewski R, Isenberg T, Andrews WM, Chen W, Sousa MC, Ebert DS (2009) Stippling by example. In: Proceedings of the 7th international symposium on non-photorealistic animation and rendering. ACM, pp 41–50

  23. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25. Curran Associates, Inc., pp 1097–1105

  24. Kwatra V, Essa I, Bobick A, Kwatra N (2005) Texture optimization for example-based synthesis. ACM Trans Graph 24(3):795–802

    Article  Google Scholar 

  25. Kyprianidis JE, Collomosse J, Wang T, Isenberg T (2013) State of the art: a taxonomy of artistic stylization techniques for images and video. IEEE Trans Vis Comput Graph 19(5):866–885

    Article  Google Scholar 

  26. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2016) Photo-realistic single image super-resolution using a generative adversarial network. arXiv:1609.04802

  27. Lee H, Seo S, Ryoo S, Yoon K (2010) Directional texture transfer. In: Proceedings of the 8th international symposium on non-photorealistic animation and rendering. ACM, pp 43–48

  28. Li C, Wand M (2016) Combining markov random fields and convolutional neural networks for image synthesis. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 2479– 2486

  29. Li Y, Fang C, Yang J, Wang Z, Lu X, Yang M-H (2017) Universal style transfer via feature transforms. In: Advances in neural information processing systems

  30. Liao J, Yao Y, Yuan L, Hua G, Kang SB (2017) Visual attribute transfer through deep image analogy. ACM Trans Graph 36(4):120:1–120:15

    Article  Google Scholar 

  31. Lu M, Zhao H, Yao A, Xu F, Chen Y, Zhang L (2017) Decoder network over lightweight reconstructed feature for fast semantic style transfer. In: 2017 IEEE international conference on computer vision (ICCV), pp 2488–2496

  32. McKone E, Kanwisher N, Duchaine BC (2007) Can generic expertise explain special processing for faces? Trends Cogn Sci 11(1):8–15

    Article  Google Scholar 

  33. Meng M, Zhao M, Zhu S-C (2010) Artistic paper-cut of human portraits. In: Proceedings of the 18th ACM international conference on multimedia. ACM, pp 931–934

  34. Pinheiro PHO, Collobert R (2013) Recurrent convolutional neural networks for scene parsing. CoRR 1306.2795

  35. Ruder M, Dosovitskiy A, Brox T (2016) Artistic style transfer for videos. In: German conference on pattern recognition. Springer, pp 26–36

  36. Selim A, Elgharib M, Doyle L (2016) Painting style transfer for head portraits using convolutional neural networks. ACM Trans Graph 35(4):129

    Article  Google Scholar 

  37. Shih Y, Paris S, Barnes C, Freeman WT, Durand F (2014) Style transfer for headshot portraits. ACM Trans Graph 33(4):148:1–148:14

    Article  Google Scholar 

  38. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  39. Sinha P, Balas B, Ostrovsky Y, Russell R (2006) Face recognition by humans Nineteen results all computer vision researchers should know about. Proc IEEE 94(11):1948–1962

    Article  Google Scholar 

  40. Song J, Gao L, Nie F, Shen HT, Yan Y, Sebe N (2016) Optimized graph learning using partial tags and multiple features for image and video annotation. IEEE Trans Image Process 25(11):4999– 5011

    Article  MathSciNet  MATH  Google Scholar 

  41. Song J, Guo Y, Gao L, Li X, Hanjalic A, Shen HT (2018) From deterministic to generative: multi-modal stochastic RNNs for video captioning. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2018.2851077

  42. Song J, Zhang H, Li X, Gao L, Wang M, Hong R (2018) Self-supervised video hashing with hierarchical binary auto-encoder. IEEE Trans Image Process 27 (7):3210–3221

    Article  MathSciNet  MATH  Google Scholar 

  43. Tompson J, Jain A, LeCun Y, Bregler C (2014) Joint training of a convolutional network and a graphical model for human pose estimation. In: Proceedings of the 27th international conference on neural information processing systems, NIPS’14. MIT Press, Cambridge, pp 1799–1807

  44. Ulyanov D, Lebedev V, Vedaldi A, Lempitsky V (2016) Texture networks Feed-forward synthesis of textures and stylized images. In: Interational conference on machine learning (ICML)

  45. Wang B, Wang W, Yang H, Sun J (2004) Efficient example-based painting and synthesis of 2d directional texture. IEEE Trans Vis Comput Graph 10(3):266–277

    Article  Google Scholar 

  46. Wang T, Collomosse JP, Hunter A, Greig D (2013) Learnable stroke models for example-based portrait painting. In: British machine vision conference

  47. Wang X, Gao L, Song J, Shen H (2017) Beyond frame-level cnn Saliency-aware 3-d cnn with lstm for video action recognition. IEEE Signal Process Lett 24(4):510–514

    Article  Google Scholar 

  48. Wang X, Gao L, Wang P, Sun X, Liu X Two-stream 3-d convnet fusion for action recognition in videos with arbitrary size and length. 1–1

  49. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967

    Article  Google Scholar 

  50. Zeng K, Zhao M, Xiong C, Zhu S-C (2009) From image parsing to painterly rendering. ACM Trans Graph 29(1):2

    Article  Google Scholar 

  51. Zhao M, Zhu S-C (2011) Portrait painting using active templates. In: Proceedings of the ACM SIGGRAPH/Eurographics symposium on non-photorealistic animation and rendering. ACM, pp 117– 124

  52. Zhu C, Byrd RH, Lu P, Nocedal J (1997) Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans Math Softw 23 (4):550–560

    Article  MathSciNet  MATH  Google Scholar 

  53. Zhu X, Li X, Zhang S (2016) Block-row sparse multiview multilabel learning for image classification. IEEE Trans Cybern 46:450–461

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 61202296, 61872153) and the National Science Foundation of Guangdong province No. 2018A030313318.

Author information

Authors and Affiliations

  1. School of Computer Science, South China Normal University, Guangzhou, People’s Republic of China

    Asad Khan & Jing Xiao

  2. Institute of Robotics, Computer Science Department, Innopolis University, Innopolis, Russia

    Muhammad Ahmad

  3. School of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, People’s Republic of China

    Nuzhat Naqvi

  4. Military College of Engineering, National University of Sciences and Technology (NUST), Islamabad, Pakistan

    Faisal Yousafzai

Authors
  1. Asad Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nuzhat Naqvi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Faisal Yousafzai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jing Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing Xiao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, A., Ahmad, M., Naqvi, N. et al. Photographic painting style transfer using convolutional neural networks. Multimed Tools Appl 78, 19565–19586 (2019). https://doi.org/10.1007/s11042-019-7270-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7270-8

Keywords

Search

Navigation