Skip to main content
SpringerLink
Log in

Matching a composite sketch to a photographed face using fused HOG and deep feature models

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, we focus on the research of matching a computer-generated composite face sketch to a photograph. This is of great importance in the field of criminal investigation. To blend the different facial representation modalities, we propose a robust feature model by combining pixel-level features extracted from multi-scale key face patches and high-level features learned from a pre-trained deep learning-based model. At first, texture features are captured by a two-level histogram of oriented gradient descriptor, considering both the overall structure and local details. The semantic-level facial characteristics are analyzed through the high-level features of the Visual Geometry Group-Face (VGG-Face) network. Next, feature similarities between each sketch/photograph pair are measured by feature distance. Then, adaptive weights are assigned to each feature similarity, and score level fused according to their visual saliency contribution. Finally, the fused feature similarity is evaluated for matching purposes. After experimenting on the Pattern Recognition and Image Processing-Viewed Software-Generated Composite (PRIP-VSGC) database and the expanded University of Malta Composite Face Sketch (UoM-SGFS) database, it is found that this framework could achieve more satisfying results compared to the existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Tang, X., Wang, X.: Face photo recognition using sketch. Proceedings of International Conference on Image Processing, pp. 257–260 (2002)

  2. Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. Int. Conf. Comput. Vis. Pattern Recogn. 1, 1005–1010 (2005)

    Google Scholar 

  3. Tang, X., Wang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009)

    Article  MathSciNet  Google Scholar 

  4. Gao, X., Wang, N., Tao, D., Li, X.: Face sketch-photo synthesis and retrieval using sparse representation. IEEE Trans. Circuits Syst. Video Technol. 22(8), 1213–1226 (2012)

    Article  Google Scholar 

  5. Wang, N., Gao, X., Li, J.: Random sampling for fast face sketch synthesis. Pattern Recogn. 76, 215–227 (2018)

    Article  Google Scholar 

  6. Zhang, L., Lin, L., Wu, X., Ding, S., Zhang, L.: End-to-end photo-sketch generation via fully convolutional representation learning. Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 627-634 (2015)

  7. Kazemi, H., Taherkhani, F., Nasrabadi, N.M.: Unsupervised facial geometry learning for sketch to photo synthesis. International Conference of the Biometrics Special Interest Group, pp. 1–5 (2018)

  8. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

  9. Pallavi, S., Sannidhan, M.S., Sudeepa, K.B., Bhandary, A.: A novel approach for generating composite sketches from mugshot photographs. IEEE International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 460–465 (2018)

  10. Klare, B., Jain, A.K.: Sketch-to-photo matching: a feature-based approach. In Proceedings of SPIE - The International Society for Optical Engineering, pp. 7667–7702 (2010)

  11. Klare, B., Li, Z., Jain, A.K.: Matching forensic sketches to mug shot photos. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 639–646 (2011)

    Article  Google Scholar 

  12. Klare, B., Jain, A.K.: Heterogeneous face recognition using kernel prototype similarities. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1410–1422 (2013)

    Article  Google Scholar 

  13. Mittal, P., Jain, A., Goswami, G., Singh, R., Vatsa, M.: Recognizing composite sketches with digital face images via SSD dictionary. In International Joint Conference on Biometrics, pp. 1–6 (2014)

  14. Han, H., Klare, B.F., Bonnen, K., Jain, A.K.: Matching composite sketches to face photos: a component-based approach. IEEE Trans. Inf. Forensics Secur. 8(1), 191–204 (2013)

    Article  Google Scholar 

  15. Liu, D., Li, J., Wang, N., Peng, C., Gao, X.: Composite components-based face sketch recognition. Neurocomputing 302, 46–54 (2018)

    Article  Google Scholar 

  16. Mittal, P., Jain, A., Goswami, G., Vatsa, M., Singh, R.: Composite sketch recognition using saliency and attribute feedback. Inf. Fusion 33, 86–99 (2017)

    Article  Google Scholar 

  17. Mittal, P., Vatsa, M., Singh, R.: Composite sketch recognition via deep network-a transfer learning approach. International Conference on Biometrics, pp. 251–256 (2015)

  18. Galea, C., Farrugia, R.A.: Matching software-generated sketches to face photographs with a very deep CNN, morphed faces, and transfer learning. IEEE Trans. Inf. Forensics Secur. 13(6), 1421–1431 (2017)

    Article  Google Scholar 

  19. O.M. Parkhi, A. Vedaldi, and A. Zisserman, Deep face recognition. British Machine Vision Conference (BMVC), 2015

  20. Chugh, T., Singh, M., Nagpal, S., Singh, R., Vatsa, M.: Transfer learning based evolutionary algorithm for composite face sketch recognition. International Conference on Computer Vision and Pattern Recognition Workshops, pp. 117–125 (2017)

  21. Kazemi, H., Soleymani, S., Dabouei, A., Iranmanesh, M., Nasrabadi, N.M.: Attribute-centered loss for soft-biometrics guided face sketch-photo recognition. International Conference on Computer Vision and Pattern Recognition Workshops, pp. 499–507 (2018)

  22. Wan, W., Gao, Y., Lee, H.J.: Transfer deep feature learning for face sketch recognition. Neural Comput. Appl. pp. 1–10 (2019)

  23. Peng, C., Gao, X., Wang, N., Li, J.: Sparse graphical representation based discriminant analysis for heterogeneous face recognition. Sig. Process. 156, 46–61 (2019)

    Article  Google Scholar 

  24. Peng, C., Wang, N., Li, J., Gao, X.: DLFace: Deep local descriptor for cross-modality face recognition. Pattern Recogn. 90, 131–171 (2019)

    Article  Google Scholar 

  25. Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. (2015)

  26. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. Int. Conf. Comput. Vis. Pattern Recogn. 1, 886–893 (2005)

    Google Scholar 

  27. Chen, L.F., Liao, H.Y.M., Ko, M.T., Lin, J.C., Yu, G.J.: A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recogn. 33(10), 1713–1726 (2000)

    Article  Google Scholar 

  28. Klum, S., Han, H., Klare, B., Jain, A.K.: The FaceSketchID system: Matching facial composites to mugshots. IEEE Trans. Inf. Forens. Secur. 9(12), 2248–2263 (2014)

    Article  Google Scholar 

  29. Galea, C., Farrugia, R.A.: A large-scale software-generated face composite sketch database. International Conference of the Biometrics Special Interest Group, pp. 1–5 (2016)

  30. Martinez, A., Benavente, R.: The AR face database. CVC, Barcelona, Spain, Tech. Rep. 24, Jun. (1998)

  31. Phillips, P.J., Wechsler, H., Huang, J., Rauss, P.J.: The FERET database and evaluation procedure for face-recognition algorithms. Image Vis. Comput. 16(5), 295–306 (1998)

    Article  Google Scholar 

  32. Ouyang, S., Hospedales, T.M., Song, Y.Z., Li, X.: Forgetmenot: Memory-aware forensic facial sketch matching, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5571–5579, (2018)

  33. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision, (2015)

  34. FEI. http://fei.edu.br/~cet/facedatabase.html

  35. MEDS-II. http://www.nist.gov/itl/iad/ig/sd32.cfm

  36. Dlib. http://dlib.net/

  37. Berg, T.L, Berg, A.C., Edwards, J., Maire, M., White, R., Teh, Y.W., Forsyth, D.A.: Names and faces in the news. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p. 2 (2004)

  38. Xue, X., Xu, J., Mao, X.: Composite sketch recognition using multi-scale HOG features and semantic attributes. Cyberworlds, (2018)

  39. Identi-kit, identi-kit solutions. http://www.identikit.net/

  40. Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 7, 711–720 (1997)

    Article  Google Scholar 

  41. Peng, C., Gao, X., Wang, N., Li, J.: Graphical representation for heterogeneous face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(2), 301–312 (2017)

    Article  Google Scholar 

  42. C. Galea, and R.A. Farrugia, Face photo-sketch recognition using local and global texture descriptors. In 2016 24th European Signal Processing Conference (EUSIPCO), pp. 2240–2244, (2016)

Download references

Acknowledgements

This paper was sponsored by the Public Welfare Research Project of Zhejiang Province, China (Grant No. LGF18F020015), JSPS Grants–in–Aid for Scientific Research, Japan (Grant No. 17H00737), and Opening Foundation of Key Laboratory of Fundamental Science for National Defense on Vision Synthetization, Sichuan University, China (Grant No. 2020SCUVS007).

Author information

Authors and Affiliations

  1. Hangzhou Dianzi University, Hangzhou, China

    Jiayi Xu, Xinying Xue & Yitiao Wu

  2. University of Yamanashi, Kofu, Japan

    Xiaoyang Mao

Authors
  1. Jiayi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinying Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yitiao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoyang Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoyang Mao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, J., Xue, X., Wu, Y. et al. Matching a composite sketch to a photographed face using fused HOG and deep feature models. Vis Comput 37, 765–776 (2021). https://doi.org/10.1007/s00371-020-01976-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01976-5

Keywords

Search

Navigation