Skip to main content

Advertisement

Springer Nature Link
Log in

BiLSTM regression model for face sketch synthesis using sequential patterns

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Face sketch synthesis from an input photo is a challenging task for law enforcement applications. Prior methods generally have required neighbor selection process for candidate sketch patches on a large-scale of training data, which is tedious and time consuming. This paper proposes a new face sketch synthesis method by means of Bidirectional Long Short Term Memory (BiLSTM) recurrent neural network, that is commonly utilized for sequence regression. BiLSTM neural networks can learn spatial and texture patterns within a single regression model, which does not only significantly speeds up the synthesis process but also improves the synthesis performance. The main thought behind the proposed method is to transfer the input photo into sequential patterns on the basis of regular overlapping patches. Then, the synthesized sketch patch corresponding to each input photo patch is predicted through BiLSTM Regression Model (BiLSTM-RM) learned from training photo-sketch pairs. Experimental results on public face sketch databases demonstrated that the proposed BiLSTM-RM method outperforms state-of-the-art methods, and also it can be generalized well to face sketch synthesis for real-world applications that work on photos with pose variance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Notes

  1. The source code and obtained results will be available online based on publication.

  2. All synthesized sketches are obtained from the authors themselves or their websites.

  3. Pose variation photos were downloaded from https://pan.baidu.com/s/1i4CYLpF.

References

  1. Zhang M, Li J, Wang N, Gao X (2017) Compositional model-based sketch generator in facial entertainment. IEEE Trans Cybern 48(3):904

    Article  Google Scholar 

  2. Zhang Y, Dong W, Ma C, Mei X, Li K, Huang F, Hu BG, Deussen O (2016) Data-driven synthesis of cartoon faces using different styles. IEEE Trans Image Process 26(1):464

    Article  MathSciNet  Google Scholar 

  3. Güçlütürk Y, Güçlü U, van Lier R, van Gerven MA (2016) European conference on computer vision, pp 810–824

  4. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955

    Article  MathSciNet  Google Scholar 

  5. Best-Rowden L, Han H, Otto C, Klare BF, Jain AK (2014) Unconstrained face recognition: identifying a person of interest from a media collection. IEEE Trans Inf Forensics Secur 9(12):2144

    Article  Google Scholar 

  6. Setumin S, Suandi SA (2019) Cascaded static and dynamic local feature extractions for face sketch to photo matching. IEEE Access 7:27135

    Article  Google Scholar 

  7. Zhu M, Li J, Wang N, Gao X (2019) A deep collaborative framework for face photo-sketch synthesis. IEEE Trans Neural Netw Learn Syst 30(10):3096–3108

    Article  Google Scholar 

  8. Wan W, Gao Y, Lee HJ (2019) Transfer deep feature learning for face sketch recognition. Neural Comput Appl 31(12):9175

    Article  Google Scholar 

  9. Samma H, Suandi SA, Mohamad-Saleh J (2019) Face sketch recognition using a hybrid optimization model. Neural Comput Appl 31(10):6493

    Article  Google Scholar 

  10. Radman A, Suandi SA (2018) Robust face pseudo-sketch synthesis and recognition using morphological-arithmetic operations and HOG-PCA. Multimed Tools Appl 79(19):25311

    Article  Google Scholar 

  11. Jiang J, Yu Y, Wang Z, Liu X, Ma J (2019) Graph-regularized locality-constrained joint dictionary and residual learning for face sketch synthesis. IEEE Trans Image Process 28(2):628

    Article  MathSciNet  Google Scholar 

  12. Radman A, Suandi SA (2019) Markov random fields and facial landmarks for handling uncontrolled images of face sketch synthesis. Pattern Anal Appl 22(1):259

    Article  MathSciNet  Google Scholar 

  13. Zhou H, Kuang Z, Wong KYK (2012) Markov weight fields for face sketch synthesis. In: IEEE conference on computer vision and pattern recognition, pp 1091–1097

  14. Wang N, Gao X, Sun L, Li J (2018) Random sampling for fast face sketch synthesis. Pattern Recognit 76:215

    Article  Google Scholar 

  15. Wang J, Yang J, Yu K, Lv F, Huang T, Gong Y (2010) Locality-constrained linear coding for image classification. In: IEEE conference on computer vision and pattern recognition, pp 3360–3367

  16. Gao X, Wang N, Tao D, Li X (2012) Face sketch-photo synthesis and retrieval using sparse representation. IEEE Trans Circuits Syst Video Technol 22(8):1213

    Article  Google Scholar 

  17. Zhang S, Gao X, Wang N, Li J (2017) Face sketch synthesis from a single photo-sketch pair. IEEE Trans Circuits Syst Video Technol 27(2):275

    Article  Google Scholar 

  18. Wang N, Zhu M, Li J, Song B, Li Z (2017) Data-driven vs. model-driven: fast face sketch synthesis. Neurocomputing 257:214

    Article  Google Scholar 

  19. Zhang S, Gao X, Wang N, Li J (2016) Robust face sketch style synthesis. IEEE Trans Image Process 25(1):220

    Article  MathSciNet  Google Scholar 

  20. Zhang S, Gao X, Wang N, Li J, Zhang M (2015) Face sketch synthesis via sparse representation-based greedy search. IEEE Trans Image Process 24(8):2466

    Article  MathSciNet  Google Scholar 

  21. Zhang M, Wang R, Gao X, Li J, Tao D (2019) Dual-transfer face sketch-photo synthesis. IEEE Trans Image Process 28(2):642

    Article  MathSciNet  Google Scholar 

  22. Wang N, Gao X, Sun L, Li J (2018) Anchored neighborhood index for face sketch synthesis. IEEE Trans Circuits Syst Video Technol 28(9):2154

    Article  Google Scholar 

  23. Zhang L, Lin L, Wu X, Ding S, Zhang L (2015) In: Proceedings of the 5th ACM on international conference on multimedia retrieval, pp 627–634

  24. Zhang D, Lin L, Chen T, Wu X, Tan W, Izquierdo E (2016) Content-adaptive sketch portrait generation by decompositional representation learning. IEEE Trans Image Process 26(1):328

    Article  MathSciNet  Google Scholar 

  25. Isola P, Zhu JY, Zhou T, Efros A (2017) In: Proceedings pf the 30th IEEE conference on computer vision and pattern recognition, pp 5967–5976

  26. Wang N, Zha W, Li J, Gao X (2018) Back projection: an effective postprocessing method for GAN-based face sketch synthesis. Pattern Recognit Lett 107:59

    Article  Google Scholar 

  27. Zhang S, Ji R, Hu J, Lu X, Li X (2018) Face sketch synthesis by multidomain adversarial learning. IEEE Trans Neural Netw Learn Syst 30(5):1419

    Article  Google Scholar 

  28. Lokesh S, Kumar PM, Devi MR, Parthasarathy P, Gokulnath C (2019) An automatic Tamil speech recognition system by using bidirectional recurrent neural network with self-organizing map. Neural Comput Appl 31(5):1521

    Article  Google Scholar 

  29. Jiang M, Liang Y, Feng X, Fan X, Pei Z, Xue Y, Guan R (2018) Text classification based on deep belief network and softmax regression. Neural Comput Appl 29(1):61

    Article  Google Scholar 

  30. Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673

    Article  Google Scholar 

  31. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735

    Article  Google Scholar 

  32. Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw 18(5–6):602

    Article  Google Scholar 

  33. Byeon W, Breuel TM, Raue F, Liwicki M (2015) In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3547–3555

  34. Kingma DP, Ba J (2014) In: Proceedings of the 3rd international conference on learning representations (ICLR), pp 1–15

  35. Dai J, Zhang P, Wang D, Lu H, Wang H (2018) Video person re-identification by temporal residual learning. IEEE Trans Image Process 28(3):1366

    Article  MathSciNet  Google Scholar 

  36. Zhang W, Wang X, Tang X (2011) Coupled information-theoretic encoding for face photo-sketch recognition. In: IEEE conference on computer vision and pattern recognition, pp 513–520

  37. Tang X, Wang X (2002) In: Proceedings of international conference on image processing, vol 1, pp 257–260

  38. Martinez AM (1998) CVC Technical Report24

  39. Messer K, Matas J, Kittler J, Luettin J, Maitre G (1999) In: Second international conference on audio and video-based biometric person authentication, vol 964, pp 965–966

  40. Zhang W, Wang X, Tang X (2010) Lighting and Pose Robust Face Sketch Synthesis. In: 11th European conference on computer vision, vol 6316, LNCS, pp 420–433

  41. Wang N, Gao X, Li J, Song B, Li Z (2016) Evaluation on synthesized face sketches. Neurocomputing 214:991

    Article  Google Scholar 

  42. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600

    Article  Google Scholar 

  43. Zhang L, Zhang L, Mou X, Zhang D (2011) FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Process 20(8):2378

    Article  MathSciNet  Google Scholar 

  44. Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15(2):430

    Article  Google Scholar 

  45. Liu A, Lin W, Narwaria M (2012) Image quality assessment based on gradient similarity. IEEE Trans Image Process 21(4):1500

    Article  MathSciNet  Google Scholar 

  46. Chen LF, Liao HYM, Ko MT, Lin JC, Yu GJ (2000) A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recognit 33(10):1713

    Article  Google Scholar 

Download references

Acknowledgements

The authors highly acknowledge the Universiti Sains Malaysia for its fund Universiti Sains Malaysia Research University (RUI) Grant no.1001.PELECT.8014056. The authors also gratefully acknowledge the support of Associate Professor Dr. Nannan Wang from Xidian University, Xi’an, China, by providing the results of his recent publications for face sketch synthesis.

Author information

Authors and Affiliations

  1. AL-Saeed Faculty for Engineering and Information Technology, Taiz University, Taiz, Yemen

    Abduljalil Radman

  2. Intelligent Biometric Group, School of Electrical and Electronic Engineering, Engineering Campus, Universiti Sains Malaysia, 14300, Nibong Tebal, Pulau Pinang, Malaysia

    Shahrel Azmin Suandi

Authors
  1. Abduljalil Radman
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Shahrel Azmin Suandi
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Shahrel Azmin Suandi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Radman, A., Suandi, S.A. BiLSTM regression model for face sketch synthesis using sequential patterns. Neural Comput & Applic 33, 12689–12702 (2021). https://doi.org/10.1007/s00521-021-05916-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-05916-9

Keywords