Skip to main content
SpringerLink
Log in

Real-time Stereo Face Recognition by Fusing Appearance and Depth Fisherfaces

  • Published:
The Journal of VLSI Signal Processing Systems for Signal, Image, and Video Technology Aims and scope Submit manuscript

Abstract

Reported 3D face recognition techniques assume the use of active 3D measurement for 3D facial capture. However, active method employ structured illumination (structure projection, phase shift, gray-code demodulation, etc) or laser scanning, which is not desirable in many applications. A major problem of using passive stereo is its lower 3D face resolution and thus no passive method for 3D face recognition has been reported. In this paper, a real-time passive stereo face recognition system is presented. Entire face detection, tracking, pose estimation and face recognition are investigated. We used SRI Stereo engine that outputs sub-pixel disparity automatically. An investigation is carried out in combining 3D and 2D information for face recognition. The straightforward two-stage principal component analysis plus linear discriminant analysis is carried out in appearance and depth face images respectively. A probe face is identified using sum of the weighted appearance and depth linear discriminant distances. We investigate the complete range of linear combinations to reveal the interplay between these two paradigms. The improvement of the face recognition rate using this combination is verified. The recognition rate by the combination is higher than that of either appearance alone or depth alone. We then discuss the implementation of the algorithm on a stereo vision system. A hybrid face and facial features detection/tracking approach is proposed which collects near-frontal views for face recognition. Our face detection/tracking approach automatically initializes without user intervention and can be re-initialized automatically if the tracking of the 3D face pose is lost. The experiments include two parts. Firstly, the performance of the proposed algorithm is verified on XM2VTS database; Secondly, the algorithm is demonstrated on a real-time stereo vision system. It is able to detect, track and recognize a person while walking toward a stereo camera.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. P. Belhumeur, J. Hespanha, and D. Kriegman, “Eigenface vs Fisherfaces: Recognition using Class Specific Linear Projection,” in Proceedings of Fourth European Conference on Computer Vision, Cambridge, 1996, pp. 45–56.

  2. C. Beumier and M. Acheroy, “Face Verification from 3D and Gray Level Clues,” Pattern Recogn. Lett., vol. 22, 2001, pp. 1321–1329.

    Article  MATH  Google Scholar 

  3. C. Beumier and M. Acheroy, “Automatic Face Verification from 3D Surface,” in Proceedings of British Machine Vision Conference, Southampton, 1998, pp. 449–458.

  4. R. Bruneli and D. Falavingna, “Person Identification using Multiple Cues,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 17, no. 10, 1995, pp. 955–966.

    Article  Google Scholar 

  5. R. Chellpa, C.L. Wilson, and S. Sirohey, “Human and Machine Recognition of Faces,” Proc. IEEE, vol. 83, no. 5, 1995, pp. 705–740.

    Article  Google Scholar 

  6. K. Etemad and R. Chellappa, “Face Recognition using Discriminant Eigenvectors,” in Proceedings of International Conference on Acoustics, Speech and Signal Processing, Atlanta, GA, 1996, pp. 2148–2151.

  7. G. Gordon, “Face Recognition Based on Depth Maps and Surface Curvature,” in SPIE Proceedings International Workshop on Automatic Face and Gesture Recognition, 1996, pp. 176–181.

  8. J. Kittler, M. Hatef, P.W. Duin and J. Matas, “On Combining Classifiers,” IEEE Transactions on PAMI, vol. 20, no. 3, 1998, pp. 226–239.

    Article  Google Scholar 

  9. Videre Design. “MEGA-D Megapixel Digital Stereo Head,” http://users.rcn.com/mclaughl.dnai/sthmdcs.htm.

  10. C. Liu and H. Wechsler, “A shape- and texture-based enhanced fisher classifier for face recognition,” IEEE Trans. Image Process, vol. 10, no. 4, 2001, pp. 598–608.

    Article  MATH  Google Scholar 

  11. D.A. Socolinsky, A. Selinger and J.D. Neuheisel, “Face Recognition with Visible and Thermal Infrared Imagery,” Comput. Vis. Image Underst., vol. 91, 2003, pp. 72–114.

    Article  Google Scholar 

  12. M. Turk and A. Pentland, “Eigenfaces for Recognition,” J. Cogn. Neursci., vol. 3, no. 1, 1991, pp. 71–86.

    Article  Google Scholar 

  13. J.-G. Wang and E. Sung, “Frontal-View Face Detection and Facial Feature Extraction using Color and Morphological Operations,” Pattern Recogn. Lett., vol. 20, no. 10, 1999, pp. 1053–1068.

    Article  Google Scholar 

  14. W.S. Yambor, B.A. Draper and J.R. Beveridge, “Analyzing PCA-Based Face Recognition Algorithms: Eigen Vector Selection And Distance Measures,” in Proceedings of the 2nd Workshop on Empirical Evaluation in Computer Vision, Dublin, 2000.

  15. J.-Y. Bouguet, B. Girod, S.B. Gokturk and C. Tomasi, “Model-Based Face Tracking For View Independent Facial Expression Recognition,” in Proceedings Fifth IEEE International Conference on Automatic Face and Gesture Recognition, Washington, DC, 2002, pp. 272–278.

  16. T. Choudhury, B. Clarkson, T. Jebara and A. Pentland, “Multimodal, Person Recognition using Unconstrained Audio and Video,” in Proceedings International Conference on Audio and Video Based Person Authentication, 1999, pp. 176–181.

  17. Y. Li, S. Gong and H. Liddell, “Video-based Online Face Recognition using Identify Surfaces,” in Proceedings of Second International Workshop on Recognition, Analysis and Tracking of Faces and Gestures in Real-time, Vancouver, 2001, pp. 40–46.

  18. S. Birchfield, “An Elliptical Head Tracking using Intensity Gradients and Color Histograms,” in Proceedings IEEE International Conference on Computer Vision and Pattern Recognition, Santa Barbara, 1998, pp. 232–237.

  19. K.I. Chang, K.W. Bowyer and P.J. Flynn, “Multi-Biometrics using Facial Appearance, Shape and Temperature,” in Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition, 2004, pp. 43–48.

  20. K. Masser, J. Matas, J. Kittler, J. Luettin and G. Maitre, “XM2VTSDB: The Extended M2VTS Database,” in Second International Conference on Audio, and Video-based Biometric Person Authentication, Washington, DC, 1999.

  21. F. Tsalakanidou, D. Tzovaras and M.G. StrintzisAfzal, “Use of Depth and Colour Eigenfaces for Face Recognition,” Pattern Recogn. Lett., vol. 24, no. 9–10, 2003, pp. 1427–1435.

    Article  MATH  Google Scholar 

  22. J.-G. Wang, H. Kong and R. Venkateswarlu, “Improving Face Recognition Rates by Combining Color and Depth Fisherfaces,” 6th Asian Conference on Computer Vision, Jeju, 2004, pp. 126–131.

  23. J.-G. Wang, K.-A. Toh and R. Venkateswarlu, “Fusion of Appearance and Depth Information for Face Recognition,” Proceedings of 5th International Conference on Audio- and Video-Based Biometric Person Authentication, New York, 2005, pp. 919–928.

  24. InterSense Inc. “InertiaCube2,” http://www.intersense.com.

Download references

Author information

Authors and Affiliations

  1. Institute for Infocomm Research, 21 Heng Mui Keng Terrace, Singapore, 119613, Singapore

    Jian-Gang Wang, Eng Thiam Lim, Xiang Chen & Ronda Venkateswarlu

Authors
  1. Jian-Gang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eng Thiam Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ronda Venkateswarlu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian-Gang Wang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wang, JG., Lim, E.T., Chen, X. et al. Real-time Stereo Face Recognition by Fusing Appearance and Depth Fisherfaces. J VLSI Sign Process Syst Sign Im 49, 409–423 (2007). https://doi.org/10.1007/s11265-007-0093-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-007-0093-2

Keywords

Search

Navigation