Skip to main content
SpringerLink
Log in

Conditional Gabor phase–based disparity estimation applied to facial tracking for person–specific facial action recognition: a preliminary study

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Within the affective computing research field, researchers are still facing a big challenge to establish automated systems to recognize human emotions from video sequences. Performances are quite dependent on facial feature localization and tracking. In this paper, we present a method based on a coarse–to–fine paradigm to characterize a set of facial fiducial points using a bank of Gabor filters. When the first face image is captured, the coarsest level is used to estimate a rough position for each facial feature. Afterward, a coarse–to–fine displacement refinement on an image pyramid is performed. The positions are then tracked over the subsequent frames using a modification of a fast Gabor–phase based technique. This includes a redefinition of the confidence measure and introduces an iterative conditional disparity estimation procedure. We used the proposed tracking process to implement a personalized feature–based facial action recognition framework, motivated by the fact that the same facial expression may vary differently across humans. Experimental results show that the facial feature points can be localized with high accuracy and tracked with sufficient precision leading to a better facial action recognition performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. Facial Action Coding System.

  2. Along this work, we use, interchangeably, the words “disparity” and “displacement”

References

  1. Bashyal S, Venayagamoorthy GK (2008) Recognition of facial expressions using Gabor wavelets and learning vector quantization. Eng Appl Artif Intel 21(7):1056–1064

    Article  Google Scholar 

  2. Chuan-Yu C, Yan-Chiang H, Chi-Lu Y (2009) Personalized facial expression recognition in color image. In: Proceedings of international conference on innovative computing, information and control, pp 1164–1167

  3. Cohn JF, Zlochower A, Lien J, Kanade T (1999) Face analysis by feature point tracking has concurrent validity with manual FACS coding. Psychophysiology 36(1):35–43

    Article  Google Scholar 

  4. Cohn JF, Schmidt K, Gross R, Ekman P (2001) Individual differences in facial expression: stability over time, relation to self–reported emotion, and ability to inform person identification. In: Proceedings of IEEE international conference on multimodal interfaces, pp 491–496

  5. Cottrell GW, Dailey MN, Padgett C (2003) Is all faces processing holistic? the view from UCSD. In: Wenger M, Twnsend J (eds) Computational, geometric and process perspectives on facial recognition, contexts and challenges. Contexts and Challenges, Erlbaum

    Google Scholar 

  6. Dahmane M, Meunier J (2008) Enhanced phase–based displacement estimation - an application to facial feature extraction and tracking. In: Proceedings of international conference on computer vision theory and applications. Madeira, Portugal, pp 427–433

    Google Scholar 

  7. Dahmane M, Meunier J (2011) Individual feature–appearance for facial action recognition. In: Proceedings of international conference on image analysis and recognition, pp 233–242

  8. Dantone M, Gall J, Fanelli G, Van Gool L (2012) Real-time facial feature detection using conditional regression forests. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2578–2585

  9. Fasel B (2002) Robust face analysis using convolutional neural networks, pp 40–43. doi:10.1109/ICPR.2002.1048231

  10. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recog 36(1):259–275

    Article  MATH  Google Scholar 

  11. Flaton K, Toborg S (1989) An approach to image recognition using sparse filter graphs. In: Proceedings of international joint conference on neural networks 1, pp 313–320

  12. Fleet D, Jepson A (1993) Stability of phase information. IEEE Trans Pattern Anal Mach Intel 15(12):1253–1268

    Article  Google Scholar 

  13. Hai H, Neven H, von der Malsburg C (1998) Online facial expression recognition based on personalized galleries. In: Proceedings of IEEE international conference on automatic face and gesture recognition, pp 354–359

  14. Hammal Z, Couvreur L, Caplier A, Rombaut M (2007) Facial expression classification: An approach based on the fusion of facial deformation unsing the transferable belief model. Int J Approx Reason 46(3):542-567

    Article  Google Scholar 

  15. Hu Y, Chen L, Zhou Y, Zhang H (2004) Estimating face pose by facial asymmetry and geometry. In: Proceedings of IEEE international conference on automatic face and gesture recognition, pp 651–656

  16. Wiskott L, Fellous J-M, Kruger N, von derMalsburg C (1999) Face Recognition by Elastic Bunch Graph Matching. In Intelligent Biometric Techniques in Fingerprint and Face Recognition. In: LC, Jain et al., ISBN 0-8493-2055-0, Chapter 11. CRC Press, pp 355–396

  17. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of IEEE international conference on automatic face and gesture recognition, pp 46–53

  18. Kawulok M, Szymanek J (2012) Precise multi-level face detector for advanced analysis of facial images. Image Proces IET 6(2):95–103

    Article  MathSciNet  Google Scholar 

  19. Kim DJ, Bien Z (2008) Design of personalized classifier using soft computing techniques for personalized facial expression recognition. IEEE Trans Fuzzy Syst 16(4):874–885

    Article  Google Scholar 

  20. Koelstra S, Pantic M, Patras I (2010) A dynamic texture-based approach to recognition of facial actions and their temporal models. IEEE Trans Pattern Anal Mach Intel 32(11):1940–1954

    Article  Google Scholar 

  21. Lades M, Vorbruuggen J, Buhmann J, Lange J, Konen W, von der Malsburg C, Wurtz R (1993) Distortion invariant object recognition in the dynamic link architecture. IEEE Trans Comput 42(3):300–311

    Article  Google Scholar 

  22. Liu C, Wechsler H (2003) Independent component analysis of Gabor features for face recognition. IEEE Trans Neural Netw 14(4):919–928

    Article  Google Scholar 

  23. Lucey S, Ashraf AB, Cohn J (2007) Investigating spontaneous facial action recognition through AAM representations of the face. In: Kurihara K (ed) Face recognition book. Pro Literatur Verlag

  24. Maurer T, von der Malsburg C (1996) Tracking and learning graphs and pose on image sequences of faces. In: Proceedings of IEEE international conference on automatic face and gesture recognition, p 76

  25. McKenna SJ, Gong S, Würtz RP, Tanner J, Banin D (1997) Tracking facial feature points with Gabor wavelets and shape models. In: Proceedings of international conference on audio– and video–based biometric person authentication. Springer Verlag, pp 35–42

  26. Michal U, Franc V, Hlavac V (2012) Detector of facial landmarks learned by the structured output SVM VISAPP, pp 547–556

  27. Milborrow S, Nicolls F (2014) Active shape models with sift descriptors and MARS. In: VISAPP

  28. Nicolle J, Bailly K, Rapp V, Chetouani M (2013) Locating facial landmarks with binary map cross-correlations. In: IEEE international conference on image processing (ICIP), pp 501–508

  29. Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Delac K., Grgic M. (eds) Face recognition. I-Tech Education and Publishing, Vienna, Austria, pp 377–416

    Google Scholar 

  30. Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intel 22(12):1424–1445. http://pubs.doc.ic.ac.uk/Pantic-PAMI00/

    Article  Google Scholar 

  31. Shan C, Gong S, McOwan P (2009) Facial expression recognition based on Local Binary Patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  32. Schuller B, Valstar M, Eyben F, McKeown G, Cowie R, Pantic M (2011) AVEC 2011 - The first international audio/visual emotion challenge, LCNS. Springer Verlag

  33. Shen L, Bai L (2006) A review on Gabor wavelets for face recognition. Pattern Anal. Appl. 9(2):273–292

    Article  MathSciNet  Google Scholar 

  34. Stefanos Z, Pitas I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimedia 10(8):1528–1540

    Article  Google Scholar 

  35. Theimer W, Mallot V (1994) Phase–based binocular vergence control and depth reconstruction using active vision. CVGIP: Image Underst 60(3):343–358

    Article  Google Scholar 

  36. Tian Y, Kanade T, Cohn J (2002) Evaluation of Gabor wavelet–based facial action unit recognition in image sequences of increasing complexity. In: Proceedings of IEEE international conference on automatic face and gesture recognition, pp 229–234

  37. Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In: Computer vision and pattern recognition workshop, p 149

  38. Vezzetti E, Marcolin F (2012) Geometry-based 3D face morphology analysis: soft-tissue landmark formalization. Multimedia Tools Appl, ISSN, 1380–7501

  39. Vezzetti E, Moos S, Marcolin F, Stola V (2012) A pose-independent method for 3D face landmark formalization. Comput Methods Progr Biomed 108(3):1078–1096

    Article  Google Scholar 

  40. Wang J, Yin L, Wei X, Sun Y (2006) 3D facial expression recognition based on primitive surface feature distribution. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1399–1406

  41. Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J (2009) Toward practical smile detection. IEEE Trans Pattern Anal Mach Intel 11(31):2106–2111

    Article  Google Scholar 

  42. Wiskott L, Fellous JM, N NK, von der Malsburg C (1997) Face recognition by elastic bunch graph matching. IEEE Trans Pattern Anal Mach Intel 19(7):775–779

    Article  Google Scholar 

  43. Young IT, van Vliet LJ, van Ginkel R (2002) Recursive Gabor filtering. IEEE Trans Signal Proces 50(11):2798–2805

    Article  Google Scholar 

  44. Zhang B, Shan S, Chen X, Gao W (2007) Histogram of Gabor phase patterns (HGPP): a novel object representation approach for face recognition. IEEE Trans Image Proces 16(1):57–68

    Article  MathSciNet  Google Scholar 

  45. Zhao G, Pietikinen M (2009) Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recog Lett 30(12):1117–1127

    Article  Google Scholar 

  46. Zhu Z, Ji Q (2006) Robust pose invariant facial feature detection and tracking in real-time. In: Proceedings of international conference on pattern recognition, pp 1092–1095

Download references

Acknowledgments

This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).

Author information

Authors and Affiliations

  1. Department of Computer and Operations Research, University of Montreal, Pavillon André–Aisenstadt, CP 6128 succ Centre–Ville, H3C 3J7, Montreal, Canada

    Mohamed Dahmane

  2. Faculty of Nursing, Pavillon Marguerite-d’Youville, 2375, ch. de la Côte Ste–Catherine, H3T 1A8, Montreal, Canada

    Sylvie Cossette

  3. Department of Computer and Operations Research, University of Montreal, Pavillon André–Aisenstadt, CP 6128 succ Centre–Ville, H3C 3J7, Montreal, Canada

    Jean Meunier

Authors
  1. Mohamed Dahmane
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sylvie Cossette
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean Meunier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohamed Dahmane.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dahmane, M., Cossette, S. & Meunier, J. Conditional Gabor phase–based disparity estimation applied to facial tracking for person–specific facial action recognition: a preliminary study. Multimed Tools Appl 74, 7111–7130 (2015). https://doi.org/10.1007/s11042-014-1954-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-1954-x

Keywords

Search

Navigation