Skip to main content
SpringerLink
Log in

Contour-enhanced CycleGAN framework for style transfer from scenery photos to Chinese landscape paintings

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Image style transfer based on the generative adversarial network model has become an important research field. Among these generative adversarial network models, a distinct advantage of CycleGAN is that it can transfer between multiple domains when the data is not paired. To approximate the effects of the texturing method with the characteristics of traditional Chinese painting—"Cun method", this paper proposes an image style transfer framework to realize the transfer from scenery photos to Chinese landscape paintings. We design a contour-enhancing translation branch, which effectively guides the transfer from photos to paintings with edge detection operators computing the gradient maps. Simulation results show that this method can convert real scenery photos to Chinese landscape paintings. The Inception Score shows that contour enhancement can make the generated set performs better on sensitivity to image edges. The Kernel Inception distance and Inception-based Structural Similarity between the generated image and the "Cun method" data set shows that contour enhancement can make the generated image closer to the "Cun method" effect. Compared with Kernel Inception distance and Frechet-Inception Distance, the Inception-based Structural Similarity proposed in this paper directly focuses on similarity, the similarities between the mean features of images generated by our model, and the "Cun method" set is 97.89%, and the composite similarity metric being 0.92. The method also performs better than the MUNIT, NiceGAN, CycleGAN, and U-GAT-IT reference models under the Neural Image Assessment metric. This indicates that the introduction of the edge operator makes the generated landscape paintings more aesthetic, especially in situations where scenery photos are rich in edge information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Huang SW, Way DL, Shih ZC (2003) Physical-based model of ink diffusion in Chinese ink paintings. J World Soc Comput Graph 10:520

    Google Scholar 

  2. Huang L, Hou Z, Zhao Y, Zhang D (2019) Research progress on and prospects for virtual brush modeling in digital calligraphy and painting. Front Inf Technol Electron Eng 20:1307

    Article  Google Scholar 

  3. Li XX, Li Y (2006) Simulation of Chinese ink-wash painting based on landscapes and trees. In: Fourcaud T, Zhang XP (eds) 2006 Second international symposium on plant growth modeling and applications, vol 328. IEEE, Los Alamitos

  4. Chen TD, Yu CH (2009) Hairy brush model interactive simulation in Chinese ink painting style. In: The 2009 international symposium on information processing (ISIP), vol 184. Citeseer

  5. Chen T (2009) Non-photorealistic rendering of ink painting style diffusion. In: Lin TY, Hu XH, Xia JL, Hong TP, Shi ZZ, Han JC, Tsumoto S, Shen ZJ (eds) 2009 IEEE international conference on granular computing, vol 78. IEEE, New York

  6. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial. Networks 27:2672

    Google Scholar 

  7. Zheng Z, Yang X, Yu Z, Zheng L, Yang Y, Kautz J (2019) Joint discriminative and generative learning for person re-identification. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, vol 2138

  8. Reed S, Akata Z, Yan X, Logeswaran L, Schiele B, Lee H (2016) Generative adversarial text to image synthesis. In: International conference on machine learning, vol 1060. PMLR

  9. Zhu J, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, vol 2223

  10. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434

  11. Brock A, Donahue J, Simonyan K (2018) Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1511.06434

  12. Karras T, Laine S, Aila T (2019) A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, vol 4401

  13. Zhang H, Xu T, Li H, Zhang S, Wang X, Huang X, Metaxas D (2017) StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision, vol 5907

  14. Antipov G, Baccouche M, Dugelay JL (2017) Face aging with conditional generative adversarial networks. In: 2017 IEEE international conference on image processing (ICIP), vol 2089. IEEE

  15. Choi Y, Choi M, Kim M, Ha JW, Choo J (2018) StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, vol 8789

  16. Kim J, Kim M, Kang H, Lee K (2019) U-GAT-IT: unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. arXiv preprint arXiv:1907.10830

  17. Huang X, Liu M, Belongie S, Kautz J (2018) Multimodal unsupervised image-to-image translation. In: Proceedings of the European conference on computer vision (ECCV), vol 172

  18. Chen R, Huang W, Huang B, Sun F, Fang B (2020) Reusing discriminators for encoding towards unsupervised image-to-image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, vol 8168

  19. Chen M (2018) The modern meaning of traditional Cun method. Fine Arts 1:58-61

  20. Talebi H, Milanfar P (2018) NIMA: neural image assessment. IEEE Trans Image Process 27:3998

    Article  MathSciNet  Google Scholar 

  21. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, vol 1125

  22. Zhang F, Gao H, Lai Y (2020) Detail-preserving CycleGAN-AdaIN framework for image-to-ink painting translation. IEEE Access 8:132002

    Article  Google Scholar 

  23. Kittler J (1983) On the accuracy of the Sobel edge detector. Image Vis Comput 1:37

    Article  Google Scholar 

  24. Chen A, Xing H, Wang F (2020) A facial expression recognition method using deep convolutional neural networks based on edge computing. IEEE Access 8:49741

    Article  Google Scholar 

  25. Karatsiolis S, Christos S (2020) Modular domain-to-domain translation network. Neural Comput Appl 32:6779

    Article  Google Scholar 

  26. Barratt S, Sharma R (2018) A note on the inception score. arXiv preprint arXiv:1801.01973

  27. Dowson DC, Landau BV (1982) The Fréchet distance between multivariate normal distributions. J Multivar Anal 12:450

    Article  Google Scholar 

  28. Bińkowski M, Sutherland DJ, Arbel M, Gretton A (2018) Demystifying MMD GANs. arXiv preprint arXiv:1801.01401

  29. Hung SK, Gan JQ (2021) Facial image augmentation from sparse line features using small training data. In: International work-conference on artificial neural networks, vol 547. Springer

  30. Shmelkov K, Schmid C, Alahari K (2018) How good is my GAN?. In: Proceedings of the European conference on computer vision (ECCV), vol 213

  31. Devries T, Romero A, Pineda L, Taylor GW, Drozdzal M (2019) On the evaluation of conditional GANs. arXiv preprint arXiv:1907.08175

  32. Zhou W, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600

    Article  Google Scholar 

  33. Devan P, Khare N (2020) An efficient XGBoost–DNN-based classification model for network intrusion detection system. Neural Comput Appl 32:12499–12514

    Article  Google Scholar 

  34. Ravuri S, Vinyals O (2019) Seeing is not necessarily believing: limitations of biggans for data augmentation. In: ICLR workshop on international conference on learning representations

  35. He N, Xie K, Li T, Ye Y (2017) Overview of image quality assessment. J Beijing Inst Graph Commun 25:47

    Google Scholar 

  36. Wu J, Xia Z, Zhang H, Li H (2018) Blind quality assessment for screen content images by combining local and global features. Digit Signal Process 91:31

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant 62006191, in part by the Key RD Program of Shaanxi under Grant 2021ZDLGY15-03, 2021ZDLGY15-04, in part by Changjiang Scholars and Innovative Research Team in University under Grant IRT-17R87, in part by the Xi’an Key Laboratory of Intelligent Perception and Cultural Inheritance under grant 2019219614SYS011CG033 and in part by the Shaanxi Provincial Department of Education Special Scientific Research Project 20JK0940.

Author information

Authors and Affiliations

  1. Art School, Northwest University, Xi’an, 710127, China

    Xianlin Peng

  2. College of Information Science and Technology, Northwest University, Xi’an, 710127, China

    Shenglin Peng, Qiyao Hu, Jinye Peng & Jiaxin Wang

  3. Network and Information Management Office, Xi’an Polytechnic University, Xi’an, 710048, China

    Xinyu Liu

  4. Deptartment of Computer Science, University of North Carolina at Charlotte, Charlotte, 28223, USA

    Jianping Fan

Authors
  1. Xianlin Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shenglin Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiyao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinye Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiaxin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xinyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jianping Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Shenglin Peng, Qiyao Hu or Jinye Peng.

Ethics declarations

Conflict of interest

Xianlin Peng, Shenglin Peng, Qiyao Hu, Jinye Peng, Jiaxin Wang, Xinyu Liu, and Jianping Fan declare that they have no conflict of interest

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peng, X., Peng, S., Hu, Q. et al. Contour-enhanced CycleGAN framework for style transfer from scenery photos to Chinese landscape paintings. Neural Comput & Applic 34, 18075–18096 (2022). https://doi.org/10.1007/s00521-022-07432-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07432-w

Keywords

Search

Navigation