Skip to main content
SpringerLink
Log in

Face sketch synthesis: a survey

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Face sketch synthesis (FSS) has been widely applied to various computer vision tasks, such as criminal detection, information security, digital entertainment, etc. In the past several years, various FSS models with promising performance have been proposed. However, an in-depth understanding of these models in this topic remains lacking. The current survey: i) investigates few models; ii) classifies the models abstractly and monotonously; iii) lacks analysis of existing databases. iv) evaluates models in single evaluation metric. In this paper, we provide a comprehensive survey of the 50 state-of-the-art (SOTA) FSS models. Then we further describe the typical models objectively and analyze the results subjectively. Moreover, we divide these models into two main categories: traditional models and deep learning models. In addition, a novel classification is proposed: coefficient models and regression models. Finally, for the aforementioned problems, we discuss several challenges and highlight some directions of FSS for future research about new database and evaluation strategy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. Tech. rep. Yale University New Haven United States

  2. Bhatt HS, Bharadwaj S, Singh R et al (2012) Memetically optimized MCWLD for matching sketches with digital face images. IEEE Trans Inf Forens Secur (TIFS) 7(5):1522–1535

    Article  Google Scholar 

  3. Bi H, Li N, Guan H, Lu D, Yang L (2019) A Multi-Scale Conditional Generative Adversarial Network for Face Sketch Synthesis. In: 2019 IEEE International Conference on Image Processing (ICIP). https://doi.org/10.1109/ICIP.2019.8803629, pp 3876–3880

  4. Cai W, Wei Z (2020) PiiGAN: generative adversarial networks for pluralistic image inpainting. IEEE Access 8:48451–48463

    Article  Google Scholar 

  5. Chang L, Zhou M, Han Y, et al (2010) Face sketch synthesis via sparse representation. In: International conference on pattern recognition (ICPR), pp 2146–2149

  6. Chen LF, Liao HYM, Ko MT, et al (2000) A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recognit 33(10):1713–1726

    Article  Google Scholar 

  7. Chen H, Xu YQ, Shum HY et al (2001) Example-based facial sketch generation with non-parametric sampling. In: IEEE international conference on computer vision (ICCV), vol 2, pp 433–438

  8. Chen B-C, Chen C-S, Hsu WH (2014) Cross-age reference coding for age-invariant face recognition and retrieval. In: European conference on computer vision (ECCV). Springer, pp 768–783

  9. Chen C, Tan X, Wong K-YK (2018) Face sketch synthesis with style transfer using pyramid column feature. In: IEEE Winter conference on applications of computer vision. Lake Tahoe, USA, 18

  10. Chellappa R, Wilson CL, Sirohey S (1995) Human and machine recognition of faces: a survey. Proc IEEE 83(5):705–741

    Article  Google Scholar 

  11. Fan D, Zhang S, Wu Y, et al (2018) Face sketch synthesis style similarity: a new structure co-occurrence texture measure. CoRR arXiv:1804.02975

  12. Fan D -P, Zhang S, Wu Y -H, et al (2019) Scoot: a perceptual metric for facial sketches. In: IEEE international conference on computer vision (ICCV), pp 5612–5622

  13. Fang Y, Deng W, Du J, et al (2020) Identity-aware CycleGAN for face photo-sketch synthesis and recognition. Pattern Recognit 102

  14. Gastal ESL, Oliveira MM (2011) Domain transform for edge-aware image and video processing. ACM Trans Graph 30(4):1–12

    Article  Google Scholar 

  15. Gao X, Zhong J, Tao D, et al (2008) Local face sketch synthesis learning. Neurocomputing 71(10–12):1921–1930

    Article  Google Scholar 

  16. Gao X, Zhong J, Li J, et al (2008) Face sketch synthesis algorithm based on E-HMM and selective ensemble. IEEE Trans Circuits Syst Video Technol (TCSTV) 18(4):487–496

    Article  Google Scholar 

  17. Gao X, Wang N, Tao D, et al (2012) Face sketch–photo synthesis and retrieval using sparse representation. IEEE Trans Circuits Syst Video Technol (TCSTV) 22(8):1213–1226

    Article  Google Scholar 

  18. Gao F, Shi S, Yu J et al (2017) Composition-aided sketch-realistic portrait generation. CoRR arXiv:1712.00899

  19. Isola P, Zhu J-Y, Zhou T, et al (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1125–1134

  20. Jiang J, Yu Y, Wang Z, et al (2018) Residual Learning for Face Sketch Synthesis. In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 1952–1956

  21. Jiao L, Zhang S, Li L, et al (2018) A modified convolutional neural network for face sketch synthesis. Pattern Recognit 76:125–136

    Article  Google Scholar 

  22. Jiang J, Yu Y, Wang Z et al (2019) Graph-regularized locality-constrained joint dictionary and residual learning for face sketch synthesis. IEEE Trans Image Process (TIP) 28:628–641

    Article  MathSciNet  MATH  Google Scholar 

  23. Kim M, Kumar S, Pavlovic V, et al (2008) Face tracking and recognition with visual constraints in real-world videos. In: Computer vision and pattern recognition (CVPR). IEEE, pp 1–8

  24. Lei Y, Du W, Hu Q (2020) Face sketch-to-photo transformation with multi-scale self-attention GAN. Neurocomputing 396:13–23

    Article  Google Scholar 

  25. Li J, Yu X, Peng C, et al (2017) Adaptive representation-based face sketch-photo synthesis. Neurocomputing 269:152–159

    Article  Google Scholar 

  26. Liu Q, Tang X, Jin H, et al (2005) A nonlinear approach for face sketch synthesis and recognition. In: Computer vision and pattern recognition (CVPR), vol 1. IEEE, pp 1005–1010

  27. Liu W, Tang X, Liu J (2007) Bayesian tensor inference for sketch-based facial photo hallucination. In: International joint conference on artificial intelligence (IJCAI), pp 2141–2146

  28. Liu P, Yu H, Cang S (2019) Adaptive neural network tracking control for underactuated systems with matched and mismatched disturbances. Nonlinear Dyn 98:1447–1464

    Article  Google Scholar 

  29. Lu C, Xu L, Jia J (2012) Combining sketch and tone for pencil drawing production. 65–73

  30. Lu D, Chen Z, Wu QMJ, et al (2019) FCN based preprocessing for exemplar-based face sketch synthesis. Neurocomputing 365:113–124

    Article  Google Scholar 

  31. Martínez A, Benavente R (1998) The AR face database. CVC Technical Report 24

  32. Messer K (1999) XM2VTSDB : the extended M2VTS database. In: Proceedings of the international conference on audio video based biometric person authentication, vol 964, pp 965–966

  33. Moghaddam B, Pentland A (1997) Probabilistic visual learning for object representation. IEEE Trans Pattern Anal Mach Intell 19(7):696–710

    Article  Google Scholar 

  34. Park U, Jain AK (2010) Face matching and retrieval using soft biometrics. IEEE Trans Inf Forens Secur (TIFS) 5(3):406–415

    Article  Google Scholar 

  35. Peng C, Gao X, Wang N, et al (2016) Multiple representations-based face sketch–photo synthesis. IEEE Trans Neural Netw Learn Syst (TNNLS) 27(11):2201–2215

    Article  Google Scholar 

  36. Peng C, Gao X, Wang N, et al (2017) Superpixel-based face sketch–photo synthesis. IEEE Trans Circuits Syst Video Technol (TCSTV) 27(2):288–299

    Article  Google Scholar 

  37. Peng C, Wang N, Gao X, et al (2016) Face recognition from multiple stylistic sketches: Scenarios, datasets, and evaluation. In: European conference on computer vision (ECCV). Springer, pp 3–18

  38. Petpairote C, Madarasmi S (2014) Improved face recognition with expressions by warping to the best neutral face. In: International conference on emerging trends in computer and image processing (ICETCIP). pp 5–10

  39. Phillips PJ, Moon H, Rizvi SA, et al (2000) The FERET evaluation methodology for face-recognition algorithms. IEEE Trans Pattern Anal Mach Intell 22(10):1090–1104

    Article  Google Scholar 

  40. Rothe R, Timofte R, Van Gool L (2015) Dex: deep expectation of apparent age from a single image. In: Proceedings of the IEEE international conference on computer vision workshops. pp 10–15

  41. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326

    Article  Google Scholar 

  42. Sankarasrinivasan S (2014) Compressed measurement of structural similarity index. NIT Roukela

  43. Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process (TIP) 15(2):430–444

    Article  Google Scholar 

  44. Song Y, Bao L, Yang Q, et al (2014) Real-time exemplar-based face sketch synthesis. In: European conference on computer vision (ECCV), vol 8694. Springer, pp 800–813

  45. Sun L, Zhao C, Yan Z, et al (2019) A novel weakly-supervised approach for rgb-d-based nuclear waste object detection. IEEE Sens J 19:3487–3500

    Article  Google Scholar 

  46. Tang X, Wang X (2002) Face photo recognition using sketch. In: 2002 International conference on image processing (2002) Proceedings, 1, I–I. IEEE

  47. Tang X, Wang X (2003) Face sketch synthesis and recognition. In: Ninth IEEE international conference on computer vision, 2003. Proceedings, vol 14. IEEE, pp 687–694

  48. Tang X, Wang X (2004) Face sketch recognition. IEEE Trans Circuits Syst Video Technol (TCSTV) 14(1):50–57

    Article  Google Scholar 

  49. Tang Z, Yu H, Lu C, et al (2019) Single-trial classification of different movements on one arm based on ERD/ERS and corticomuscular coherence. IEEE Access 7:128185–128197

    Article  Google Scholar 

  50. Tang Z-C, Li C, Wu J-F, et al (2019) Classification of EEG-based single-trial motor imagery tasks using a B-CSP method for BCI. Front Inf Technol Electron Eng 20:1087–1098

    Article  Google Scholar 

  51. Tu C-T, Chan Y-H, Chen Y-C (2016) Facial sketch synthesis using 2D direct combined model-based face-specific Markov network. IEEE Trans Image Process (TIP) 25(8):3546–3561

    Article  MathSciNet  MATH  Google Scholar 

  52. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cognit Neurosci 3(1):71–86

    Article  Google Scholar 

  53. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84

    Article  Google Scholar 

  54. Wang X, Tang X (2004) Dual-space linear discriminant analysis for face recognition. In: Computer vision and pattern recognition (CVPR), vol 2, pp II–564–II–569

  55. Wang X, Tang X (2008) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967

    Article  Google Scholar 

  56. Wang X, Tang X (2004) Random sampling LDA for face recognition. In: Computer vision and pattern recognition (CVPR). IEEE, II–II, p 2

  57. Wang N, Gao X, Tao D, et al (2011) Face sketch-photo synthesis under multi-dictionary sparse representation framework. In: International conference on image and graphics (ICIG). IEEE, pp 82–87

  58. Wang S, Zhang L, Liang Y, et al (2012) Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: Computer vision and pattern recognition (CVPR). IEEE, pp 2216–2223

  59. Wang N, Tao D, Gao X et al (2013) Transductive face sketch-photo synthesis. IEEE Trans Neural Netw Learn Syst (TNNLS) 24(9):1364–1376

    Article  Google Scholar 

  60. Wang N, Tao D, Gao X, et al (2014) A comprehensive survey to face hallucination. Int J Comput Vis 106(1):9–30

    Article  Google Scholar 

  61. Wang N, Gao X, Li J, et al (2016) Evaluation on synthesized face sketches. Neurocomputing 214:991–1000

    Article  Google Scholar 

  62. Wang N, Gao X, Sun L, et al (2017) Bayesian face sketch synthesis. IEEE Trans Image Process (TIP) 26(3):1264–1274

    Article  MathSciNet  MATH  Google Scholar 

  63. Wang N, Zhang S, Gao X et al (2017) Unified framework for face sketch synthesis. Signal Process 130:1–11

    Article  Google Scholar 

  64. Wang N, Zhu M, Li J, et al (2017) Data-driven vs. model-driven: fast face sketch synthesis. Neurocomputing 257:214–221

    Article  Google Scholar 

  65. Wang L, Sindagi V, Patel VM (2018) High-quality facial photo-sketch synthesis using multi-adversarial networks. In: 13th IEEE international conference on automatic face & gesture recognition, vol 83–90. IEEE Computer Society

  66. Wang N, Gao X, Li J (2018) Random sampling for fast face sketch synthesis. Pattern Recognit 76:215–227

    Article  Google Scholar 

  67. Wang N, Gao X, Sun L, et al (2018) Anchored neighborhood index for face sketch synthesis. IEEE Trans Circuits Syst Video Technol (TCSTV) 28:2154–2163

    Article  Google Scholar 

  68. Wang N, Zha W, Li J, Gao X (2017) Back projection: An effective postprocessing method for GAN-based face sketch synthesis. Pattern Recognition Letters 107:59–65

    Article  Google Scholar 

  69. Xiao B, Gao X, Tao D, et al (2010) Photo-sketch synthesis and recognition based on subspace learning. Neurocomputing 73(4–6):840–852

    Article  Google Scholar 

  70. Yang Z-L, Guo X-Q, Chen Z-M, et al (2019) RNN-Stega: linguistic steganography based on recurrent neural networks. IEEE Trans Inf Forens Secur (TIFS) 14:1280–1295

    Article  Google Scholar 

  71. Ye L, Zhang B, Yang M, et al (2019) Triple-translation GAN with multi-layer sparse representation for face image synthesis. Neurocomputing 358:294–308

    Article  Google Scholar 

  72. Zhang W, Wang X, Tang X (2010) Lighting and pose robust face sketch synthesis. In: European conference on computer vision (ECCV), vol 6316. Springer, pp 420–433

  73. Zhang S, Gao X, Wang N, et al (2015) Face sketch synthesis via sparse representation-based greedy search. IEEE Trans Image Process (TIP) 24(8):2466–2477

    Article  MathSciNet  MATH  Google Scholar 

  74. Zhang Y, Wang N, Zhang S, et al (2016) Fast face sketch synthesis via kd-tree search. In: European conference on computer vision (ECCV), vol 9913. Springer, pp 64–77

  75. Zhang S, Gao X, Wang N, et al (2016) Robust face sketch style synthesis. IEEE Trans Image Process (TIP) 25(1):220–232

    Article  MathSciNet  MATH  Google Scholar 

  76. Zhang S, Gao X, Wang N, et al (2017) Face sketch synthesis from a single photo–sketch pair. IEEE Trans Circuits Syst Video Technol (TCSTV) 27(2):275–287

    Article  Google Scholar 

  77. Zhao W, Chellappa R, Phillips PJ, et al (2003) Face recognition: a literature survey. ACM Comput Surv (CSUR) 35(4):399–458

    Article  Google Scholar 

  78. Zhang W, Wang X, Tang X (2011) Coupled information-theoretic encoding for face photo-sketch recognition. In: Computer vision and pattern recognition (CVPR). IEEE, pp 513–520

  79. Zhang L, Zhang L, Mou X, et al (2011) FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Process (TIP) 20(8):2378–2386

    Article  MathSciNet  MATH  Google Scholar 

  80. Zhang L, Lin L, Wu X, et al (2015) End-to-end photo-sketch generation via fully convolutional representation learning. In: International conference on multimedia retrieval (ICMR). ACM, pp 627–634

  81. Zhang D, Lin L, Chen T, et al (2017) Content-adaptive sketch portrait generation by decompositional representation learning. IEEE Trans Image Process (TIP) 26(1):328–339

    Article  MathSciNet  MATH  Google Scholar 

  82. Zhang S, Ji R, Hu J, et al (2019) Face Sketch Synthesis by Multidomain Adversarial Learning. IEEE Trans Neural Netw Learn Syst (TNNLS) 30:1419–1428

    Article  Google Scholar 

  83. Zhang M, Li J, Wang N, et al (2018) Compositional model-based sketch generator in facial entertainment. IEEE Trans Cybern 48(3):904–915

    Article  Google Scholar 

  84. Zhang M, Wang N, Gao X, et al (2018) Markov Random Neural Fields for Face Sketch Synthesis. In: International joint conference on artificial intelligence (IJCAI), J Lang, Ed., pp 1142–1148

  85. Zhang M, Li Y, Wang N, et al (2020) Cascaded face sketch synthesis under various illuminations. IEEE Trans Image Process (TIP) 29:1507–1521

    Article  MathSciNet  Google Scholar 

  86. Zhang M, Wang N, Li Y, et al (2020) Neural probabilistic graphical model for face sketch synthesis. IEEE Trans Neural Netw Learn Syst (TNNLS) 31:2623–2637

    Article  MathSciNet  Google Scholar 

  87. Zhou H, Kuang Z, Wong K-YK (2012) Markov weight fields for face sketch synthesis. In: Computer vision and pattern recognition (CVPR), vol 1091–1097. IEEE

  88. Zhu M, Wang N, Gao X, et al (2017) Deep graphical feature learning for face sketch synthesis. In: International joint conference on artificial intelligence (IJCAI), 3574–3580. AAAI Press

  89. Zhu M, Li J, Wang N, et al (2019) A deep collaborative framework for face photo–sketch synthesis. IEEE Trans Neural Netw Learn Syst (TNNLS) 30(10):3096–3108

    Article  Google Scholar 

Download references

Funding

This study was funded by the NEPU Natural Science Foundation under Grant No. 2017PY ZL − 05, JY CX_CX06_2020 and JY CX_JG06_2020.

Author information

Authors and Affiliations

  1. School of Electrical and Information Engineering, Northeast Petroleum University, Daqing, China

    Hongbo Bi, Ziqi Liu, Lina Yang & Kang Wang

  2. Chengdu Lead Science & Technology Co. Ltd, Chengdu, China

    Ning Li

Authors
  1. Hongbo Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ziqi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lina Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ning Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ning Li.

Ethics declarations

Conflict of Interests

Author Hongbo Bi declares that he has no conflict of interest. Author Ziqi Liu declares that she has no conflict of interest. Author Lina Yang declares that she has no conflict of interest. Author Kang Wang declares that he has no conflict of interest. Author Ning Li declares that he has no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bi, H., Liu, Z., Yang, L. et al. Face sketch synthesis: a survey. Multimed Tools Appl 80, 18007–18026 (2021). https://doi.org/10.1007/s11042-020-10301-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10301-0

Keywords

Search

Navigation