Skip to main content
SpringerLink
Log in

Fine Tuning Dual Streams Deep Network with Multi-scale Pyramid Decision for Heterogeneous Face Recognition

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

In this paper, we propose a novel method called fine tuning dual streams deep network (FTDSDN) with multi-scale pyramid decision (MsPD) for solving heterogeneous face recognition task. As an extension of classical CNNs, FTDSDN can remove highly non-linear modality information and reserve the discriminative information using Rayleigh quotient objective function. Furthermore, we develop a powerful joint decision strategy called MsPD to adaptively adjust the weight of sub structure and obtain more robust classification performance. Experimental results show our proposed method achieves better performance on the challenging CASIA NIR-VIS 2.0 database, the heterogeneous face biometrics database, the CUHK face sketch FERET database, and the CUHK face sketch database, which demonstrates the effectiveness of our proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Alex AT, Asari VK, Mathew A (2013) Local difference of Gaussian binary pattern: robust features for face sketch recognition. In: IEEE international conference on systems, man, and cybernetics, pp 1211–1216

  2. Ding C, Tao D (2018) Trunk-branch ensemble convolutional neural networks for video-based face recognition. IEEE Trans Pattern Anal Mach Intell 40(4):1002–1014

    Article  Google Scholar 

  3. Fisher RA (1936) The use of multiple measurements in taxonomic problems. Ann Hum Genet 7(2):179–188

    Google Scholar 

  4. Fukunaga K (2013) Introduction to statistical pattern recognition. Academic press, Cambridge

    MATH  Google Scholar 

  5. Gong D, Li Z, Huang W, Li X, Tao D (2017) Heterogeneous face recognition: a common encoding feature discriminant approach. IEEE Trans Image Process 26(5):2079–2089

    Article  MathSciNet  Google Scholar 

  6. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference computer vision and pattern recognition, pp 770–778

  7. He R, Wu X, Sun Z, Tan T (2017) Learning invariant deep representation for NIR-VIS face recognition. In: AAAI conference on artificial intelligence, vol 4, p 7

  8. Hou CA, Yang M, Wang YCF (2014) Domain adaptive self-taught learning for heterogeneous face recognition. In: International conference on pattern recognition. IEEE, pp 3068–3073

  9. Hu S, Short N, Riggan BS, Chasse M, Sarfraz MS (2017) Heterogeneous face recognition: recent advances in infrared-to-visible matching. In: IEEE international conference on automatic face and gesture recognition, pp 883–890

  10. Huang X, Lei Z, Fan M, Wang X, Li SZ (2013) Regularized discriminative spectral regression method for heterogeneous face matching. IEEE Trans Image Process 22(1):353–362

    Article  MathSciNet  Google Scholar 

  11. Huo J, Gao Y, Shi Y, Yang W, Yin H (2018) Heterogeneous face recognition by margin-based cross-modality metric learning. IEEE Trans Cybern 48(6):1814–1826

    Article  Google Scholar 

  12. Jin Y, Lu J, Ruan Q (2015) Large margin coupled feature learning for cross-modal face recognition. In: International conference on biometrics. IEEE, pp 286–292

  13. Kan M, Shan S, Chen X (2016) Multi-view deep network for cross-view classification. In: IEEE conference computer vision and pattern recognition, pp 4847–4855

  14. Kan M, Shan S, Zhang H, Lao S, Chen X (2016) Multi-view discriminant analysis. IEEE Trans Pattern Anal Mach Intell 38(1):188–194

    Article  Google Scholar 

  15. Karlpearson FRS (1901) LIII. On lines and planes of closest fit to systems of points in space. Philos Mag 2(11):559–572

    Article  Google Scholar 

  16. Klare BF, Li Z, Jain AK (2011) Matching forensic sketches to mug shot photos. IEEE Trans Pattern Anal Mach Intell 33(3):639

    Article  Google Scholar 

  17. Lei Z, Li SZ (2009) Coupled spectral regression for matching heterogeneous faces. In: IEEE conference computer vision and pattern recognition, pp 1123–1128

  18. Lezama J, Qiu Q, Sapiro G (2017) Not afraid of the dark: NIR-VIS face recognition via cross-spectral hallucination and low-rank embedding. In: IEEE conference computer vision and pattern recognition, pp 6807–6816

  19. Li J, Hao P, Zhang C, Dou M (2008) Hallucinating faces from thermal infrared images. In: IEEE international conference on image processing, pp 465–468

  20. Li S, Yi D, Lei Z, Liao S (2013) The CASIA NIR-VIS 2.0 face database. In: IEEE conference computer vision and pattern recognition workshops, pp 348–353

  21. Li SZ, Lei Z, Ao M (2009) The HFB face database for heterogeneous face biometrics research. In: IEEE conference computer vision and pattern recognition, pp 1–8

  22. Lin D, Tang X (2006) Inter-modality face recognition. In: European conference on computer vision, pp 13–26

  23. Liu X, Kan M, Wu W, Shan S, Chen X (2017) VIPLFaceNet: an open source deep face recognition SDK. Front Comput Sci 11(2):208–218

    Article  Google Scholar 

  24. Liu X, Song L, Wu X, Tan T (2016) Transferring deep representation for NIR-VIS heterogeneous face recognition. In: International conference on biometrics, pp 1–8

  25. Lu J, Erin LV, Zhou J (2017) Simultaneous local binary feature learning and encoding for homogeneous and heterogeneous face recognition. IEEE Trans Pattern Anal Mach Intell PP(99):1–1

    Google Scholar 

  26. Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: International conference on international conference on machine learning, pp 807–814

  27. Parkhi OM, Vedaldi A, Zisserman A, et al (2015) Deep face recognition. In: British machine vision conference, vol 1, p 6

  28. Peng C, Gao X, Wang N, Li J (2017) Graphical representation for heterogeneous face recognition. IEEE Trans Pattern Anal Mach Intell 39(2):301–312

    Article  Google Scholar 

  29. Reale C, Lee H, Kwon H (2017) Deep heterogeneous face recognition networks based on cross-modal distillation and an equitable distance metric. In: IEEE conference computer vision and pattern recognition workshops, pp 32–38

  30. Saxena S, Verbeek J (2016) Heterogeneous face recognition with CNNs. In: European conference on computer vision, pp 483–491

  31. Sharma A, Jacobs DW (2011) Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch. In: IEEE conference computer vision and pattern recognition, pp 593–600

  32. Shi H, Wang X, Yi D, Lei Z, Zhu X, Li SZ (2017) Cross-modality face recognition via heterogeneous joint bayesian. IEEE Signal Process Lett 24(1):81–85

    Article  Google Scholar 

  33. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems, pp 568–576

  34. Song Y, Bao L, Yang Q, Yang MH (2014) Real-time exemplar-based face sketch synthesis. In: European conference on computer vision. Springer, pp 800–813

  35. Tenenbaum J (2000) Separating style and content with bilinear models. Neural Comput 12:1247–1283

    Article  Google Scholar 

  36. Tian Y, Yan C, Bai X, Zhou J (2017) Heterogeneous face recognition via Grassmannian based nearest subspace search. In: IEEE international conference on image processing, pp 1077–1081

  37. Wang N, Tao D, Gao X, Li X, Li J (2014) A comprehensive survey to face hallucination. Int J Comput Vis 106(1):9–30

    Article  Google Scholar 

  38. Wang S, Huang D, Wang Y, Tang Y (2017) 2D–3D heterogeneous face recognition based on deep canonical correlation analysis. In: Chinese conference on biometric recognition. Springer, pp 77–85

  39. Wang S, Zhang L, Liang Y, Pan Q (2012) Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: IEEE conference computer vision and pattern recognition, pp 2216–2223

  40. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967

    Article  Google Scholar 

  41. Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: European conference on computer vision. Springer, pp 499–515

  42. Wu X, Song L, He R, Tan T (2017) Coupled deep learning for heterogeneous face recognition. arXiv preprint arXiv:1704.02450

  43. Yan S, Xu D, Zhang B, Zhang HJ, Yang Q, Lin S (2006) Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell 29(1):40

    Article  Google Scholar 

  44. Yi D, Lei Z, Li SZ (2015) Shared representation learning for heterogenous face recognition. In: IEEE international conference and workshops on automatic face and gesture recognition, vol 1, pp 1–7

  45. Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. arXiv preprint arXiv:1411.7923

  46. Yi D, Liu R, Chu R, Lei Z, Li SZ (2007) Face matching between near infrared and visible light images. In: International conference on biometrics. Springer, pp 523–530

  47. Zhang W, Shu Z, Samaras D, Chen L (2017) Improving heterogeneous face recognition with conditional adversarial networks. arXiv preprint arXiv:1709.02848

  48. Zhang W, Wang X, Tang X (2011) Coupled information-theoretic encoding for face photo-sketch recognition. In: IEEE conference computer vision and pattern recognition, pp 513–520

  49. Zhong J, Gao X, Tian C (2007) Face sketch synthesis using E-HMM and selective ensemble. In: IEEE international conference on acoustics, speech and signal processing, pp 485–488

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 61673402, 61273270, and 60802069, in part by the Natural Science Foundation of Guangdong under Grants 2017A030311029, 2016B010109002, 2015B090912001, 2016B010123005, and 2017B090909005, in part by the Science and Technology Program of Guangzhou under Grants 201704020180 and 201604020024, and in part by the Fundamental Research Funds for the Central Universities of China.

Author information

Authors and Affiliations

  1. School of Electronics and Information Engineering, Sun Yat-sen University, Guangzhou, Guangdong, China

    Weipeng Hu & Haifeng Hu

Authors
  1. Weipeng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haifeng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haifeng Hu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, W., Hu, H. Fine Tuning Dual Streams Deep Network with Multi-scale Pyramid Decision for Heterogeneous Face Recognition. Neural Process Lett 50, 1465–1483 (2019). https://doi.org/10.1007/s11063-018-9942-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-018-9942-1

Keywords

Search

Navigation