Skip to main content
SpringerLink
Log in

Enhancing pencil drawing patterns via using semantic information

  • 1168: Deep Pattern Discovery for Big Multimedia Data
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Pencil drawing is recognized as a typical non-photorealistic visual art form. It is attractive to automatically generate high-quality pencil drawings from real-world photographs. Traditional model-based methods are highlighted in their interpretability, but are limited in modeling scene semantics, leading to weak sketch lines or limited pencil tone rendering effects. To address this issue, we propose a novel pencil drawing generation method via explicitly leveraging image semantic information. Specifically, the sketch lines generated from the weak boundaries can be boosted by fusing the segmented boundaries. In addition, the segmentation mask provides the spatial guidance for diversifying the tone shading effects. By combining the enhanced pencil drawing patterns, the visual quality of the generated pencil drawings can be therefore strengthened. We conduct several experiments to validate the effectiveness of our method, including ablation studies and comparison with other methods, in which promising results can be obtained.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. http://www.cse.cuhk.edu.hk/~leojia/projects/pencilsketch/pencil_drawing.htm

  2. https://colab.research.google.com/github/tensorflow/models/blob/master/research/deeplab/deeplab_demo.ipynb

  3. https://github.com/HFUT-NUIETP/sempencil

  4. Meitu Xiu Xiu V8.8.80, Meitu Inc., https://mt.meipai.com

  5. Photo To Sketch - Drawing book V2.8.17, Pexel force private limited

References

  1. Agrawal A (2009) Non-photorealistic rendering: Unleashing the artist’s imagination. IEEE Comput Graph Appl 29(04):81–85

    Article  Google Scholar 

  2. Al-nasrawi M, Deng G, Waheed W (2019) Structure extraction of images using anisotropic diffusion with directional second neighbour derivative operator. Multimed Tools Appl 78(5):6385–6407

    Article  Google Scholar 

  3. Bychkovsky V, Paris S, Chan E, Durand F (2011) Learning photographic global tonal adjustment with a database of input/output image pairs. In: IEEE conference on computer vision and pattern recognition, pp 97–104

  4. Chen LC, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587

  5. Chen L, Wu L, Hu Z, Wang M (2019) Quality-aware unpaired image-to-image translation. IEEE Trans Multimed 21(10):2664–2674

    Article  Google Scholar 

  6. Chen Z, Zhou J, Gao X, Li L, Liu J (2008) A novel method for pencil drawing generation in non-photo-realistic rendering. In: Pacific-rim conference on multimedia, pp 931–934

  7. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88 (2):303–338

    Article  Google Scholar 

  8. Ge Q, Wei Z, Liu M, Yu J, Wen C (2020) Adaptive quantized estimation fusion using strong tracking filtering and variational bayesian. IEEE Trans Syst Man Cybern Syst 50(3):899–910

    Article  Google Scholar 

  9. Hao Y, Ngo CW, Huet B (2020d) Neighbourhood structure preserving cross-modal embedding for video hyperlinking. IEEE Trans Multimed 22 (1):188–200

    Article  Google Scholar 

  10. Hao S, Zhou Y, Guo Y (2020a) A brief survey on semantic segmentation with deep learning. Neurocomputing 406:302–321

    Article  Google Scholar 

  11. Hao S, Zhou Y, Guo Y, Hong R (2020b) Bi-direction context propagation network for real-time semantic segmentation. arXiv:2005.11034

  12. Hao S, Zhou Y, Zhang Y, Guo Y (2020c) Contextual attention refinement network for real-time semantic segmentation. IEEE Access 8:55,230–55,240

    Article  Google Scholar 

  13. Inoue N, Ito D, Xu N, Yang J, Price B, Yamasaki T (2019) Learning to trace: Expressive line drawing generation from photographs. Comput Graph Forum 38(7):69–80

    Article  Google Scholar 

  14. Kong Q, Sheng Y, Zhang G (2018) Hybrid noise for lic-based pencil hatching simulation. In: IEEE international conference on multimedia and expo, pp 1–6

  15. Lee H, Kwon S, Lee S (2006) Real-time pencil rendering. In: Proceedings of the international symposium on non-photorealistic animation and rendering, pp 37–45

  16. Li G, Bi S, Wang J, Xu YQ, Yu Y (2017) Colorsketch: A drawing assistant for generating color sketches from photos. IEEE Comput Graph Appl 37(3):70–81

    Article  Google Scholar 

  17. Li Y, Fang C, Hertzmann A, Shechtman E, Yang MH (2019b) Im2pencil: Controllable pencil illustration from photographs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1525–1534

  18. Li X, Li X, Zhang L, Cheng G, Shi J, Lin Z, Tan S, Tong Y (2020) Improving semantic segmentation via decoupled body and edge supervision. arXiv:2007.10035

  19. Li H, Xiong P, Fan H, Sun J (2019a) Dfanet: Deep feature aggregation for real-time semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9522–9531

  20. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) European conference on computer vision, pp 740–755

  21. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  22. Lu P, Sheng B, Luo S, Jia X, Wu W (2015) Image-based non-photorealistic rendering for realtime virtual sculpting. Multimed Tools Appl 74(21):9697–9714

    Article  Google Scholar 

  23. Lu C, Xu L, Jia J (2012) Combining sketch and tone for pencil drawing production. In: Proceedings of the symposium on non-photorealistic animation and rendering, pp 65–73

  24. Mao X, Nagasaka Y, Imamiya A (2002) Automatic generation of pencil drawing using lic. In: ACM SIGGRAPH 2002 conference abstracts and applications, p 149

  25. Mukundan R (2015) Multi-level stroke textures for sketch based non-photorealistic rendering. In: International conference and workshop on computing and communication, pp 1–7

  26. Paszke A, Chaurasia A, Kim S, Culurciello E (2016) Enet: A deep neural network architecture for real-time semantic segmentation. arXiv:1606.02147

  27. Qiu J, Liu B, He J, Liu C, Li Y (2019) Parallel fast pencil drawing generation algorithm based on gpu. IEEE Access 7:83,543–83,555

    Article  Google Scholar 

  28. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241

  29. Rother C, Kolmogorov V, Blake A (2004) “rabcut”: Interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314

    Article  Google Scholar 

  30. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  31. Son M, Kang H, Lee Y, Lee S (2007) Abstract line drawings from 2d images. In: Pacific conference on computer graphics and applications, pp 333–342

  32. Winnemöller H (2011) Xdog: advanced image stylization with extended difference-of-gaussians. In: Proceedings of the ACM SIGGRAPH/eurographics symposium on non-photorealistic animation and rendering, pp 147–156

  33. Xie J, Bian H, Wu Y, Zhao Y, Shan L, Hao S (2020) Semantically-guided low-light image enhancement. Pattern Recogn Lett 138:308–314

    Article  Google Scholar 

  34. Yan D, Sheng Y, Mao X (2019) Pencil drawing video rendering using convolutional networks. Comput Graph Forum 38(7):91–102

    Article  Google Scholar 

  35. Yeh JW, Ouhyoung M (2009) Non-photorealistic rendering in chinese painting of animals. J Syst Simul 14:1220–1224. 1262

    Google Scholar 

  36. Yu J, Huang D, Wei Z (2018) Unsupervised image segmentation via stacked denoising auto-encoder and hierarchical patch indexing. Signal Process 143:346–353

    Article  Google Scholar 

  37. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  38. Zheng Y, Chen Y, Sarem M (2019) A novel pencil drawing algorithm based on non-symmetry and anti-packing pattern representation model. IEEE Access 7:184,950–184,962

    Article  Google Scholar 

Download references

Acknowledgment

The research was supported by the National Undergraduate Innovation and En-trepreneurship Training Program with Grant No. 202010359089.

Author information

Authors and Affiliations

  1. Hefei University of Technology, Hefei, 230009, China

    Teng Li, Jianyu Xie, Hongliang Niu & Shijie Hao

Authors
  1. Teng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianyu Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongliang Niu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shijie Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shijie Hao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, T., Xie, J., Niu, H. et al. Enhancing pencil drawing patterns via using semantic information. Multimed Tools Appl 81, 34245–34262 (2022). https://doi.org/10.1007/s11042-021-11028-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11028-2

Keywords

Search

Navigation