Skip to main content
SpringerLink
Log in

Simultaneous Facial Action Tracking and Expression Recognition in the Presence of Head Motion

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

The recognition of facial gestures and expressions in image sequences is an important and challenging problem. Most of the existing methods adopt the following paradigm. First, facial actions/features are retrieved from the images, then the facial expression is recognized based on the retrieved temporal parameters. In contrast to this mainstream approach, this paper introduces a new approach allowing the simultaneous retrieval of facial actions and expression using a particle filter adopting multi-class dynamics that are conditioned on the expression. For each frame in the video sequence, our approach is split into two consecutive stages. In the first stage, the 3D head pose is retrieved using a deterministic registration technique based on Online Appearance Models. In the second stage, the facial actions as well as the facial expression are simultaneously retrieved using a stochastic framework based on second-order Markov chains. The proposed fast scheme is either as robust as, or more robust than existing ones in a number of respects. We describe extensive experiments and provide evaluations of performance to show the feasibility and robustness of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ahlberg, J. (2002). An active model for facial feature tracking. EURASIP Journal on Applied Signal Processing, 2002(6), 566–571.

    Article  Google Scholar 

  • Bartlett, M., Littlewort, G., Lainscsek, C., Fasel, I., & Movellan, J. (2004). Machine learning methods for fully automatic recognition of facial expressions and facial actions. In IEEE international conference on systems, man and cybernetics.

  • Bascle, B., & Black, A. (1998). Separability of pose and expression in facial tracking and animation. In Proceedings of the IEEE international conference on computer vision.

  • Blake, A., & Isard, M. (2000). Active contours. Berlin: Springer.

    Google Scholar 

  • Blanz, V., & Vetter, T. (2003). Face recognition based on fitting a 3D morphable model. In IEEE transactions on pattern analysis and machine intelligence (pp. 1–12), September 2003.

  • Chandrasiri, N. P., Naemura, T., & Harashima, H. (2004). Interactive analysis and synthesis of facial expressions based on personal facial expression space. In IEEE international conference on automatic face and gesture recognition.

  • Cohen, I., Sebe, N., Garg, A., Chen, L., & Huang, T. S. (2003). Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding, 91(1–2), 160–187.

    Article  Google Scholar 

  • Dornaika, F., & Davoine, F. (2005a). Simultaneous facial action tracking and expression recognition using a particle filter. In IEEE international conference on computer vision.

  • Dornaika, F., & Davoine, F. (2005b). View- and texture-independent facial expression recognition in videos using dynamic programming. In IEEE international conference on image processing.

  • Dornaika, F., & Davoine, F. (2006). On appearance based face and facial action tracking. IEEE Transactions on Circuits and Systems for Video Technology, 16(9), 2006.

  • Ekman, P., & Friesen, W. V. (1977). Facial action coding system. Palo Alto: Consulting Psychology Press.

    Google Scholar 

  • Fasel, B., & Luettin, J. (2003). Automatic facial expression analysis: a survey. Pattern Recognition, 36(1), 259–275.

    Article  MATH  Google Scholar 

  • Gokturk, S. B., Bouguet, J. Y., Tomasi, C., & Girod, B. (2002). Model-based face tracking for view-independent facial expression recognition. In IEEE international conference on automatic face and gesture recognition.

  • Huang, Y., Huang, T. S., & Niemann, H. (2002). A region-based method for model-free object tracking. In 16th international conference on pattern recognition.

  • Huber, P. J. (1981). Robust statistics. New York: Wiley.

    MATH  Google Scholar 

  • Isard, M., & Blake, A. (1998). A mixed-state condensation tracker with automatic model-switching. In Proceedings of the IEEE international conference on computer vision.

  • Jepson, A. D., Fleet, D. J., & El-Maraghi, T. F. (2003). Robust online appearance models for visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10), 1296–1311.

    Article  Google Scholar 

  • Lee, D. (2005). Effective Gaussian mixture learning for video background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 827–832.

    Article  Google Scholar 

  • Liao, W.-K., & Cohen, I. (2005). Classifying facial gestures in presence of head motion. In IEEE workshop on vision for human-computer interaction.

  • Ljung, L. (1987). System identification: theory for the user. New York: Prentice Hall.

    MATH  Google Scholar 

  • Lu, L., Zhang, Z., Shum, H. Y., Liu, Z., & Chen, H. (2001). Model- and exemplar-based robust head pose tracking under occlusion and varying expression. In Proceedings of the IEEE workshop on models versus exemplars in computer vision (CVPR’01).

  • Lu, X., Jain, A. K., & Colbry, D. (2006). Matching 2.5D face scans to 3D models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(1), 31–43.

    Article  Google Scholar 

  • Lyons, M. J., Budynek, J., & Akamatsu, S. (1999). Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(12), 1357–1362.

    Article  Google Scholar 

  • Moreno, F., Tarrida, A., Andrade-Cetto, J., & Sanfeliu, A. (2002). 3D real-time tracking fusing color histograms and stereovision. In IEEE international conference on pattern recognition.

  • North, B., Blake, A., Isard, M., & Rittscher, J. (2000). Learning and classification of complex dynamics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(9), 1016–1034.

    Article  Google Scholar 

  • Pantic, M., & Rothkrantz, L. J. M. (2000). Automatic analysis of facial expressions: the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), 1424–1445.

    Article  Google Scholar 

  • Perez, P., & Vermaak, J. (2005). Bayesian tracking with auxiliary discrete processes. Application to detection and tracking of objects with occlusions. In IEEE ICCV workshop on dynamical vision, Beijing, China.

  • Tian, Y., Kanade, T., & Cohn, J. F. (2001). Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 97–115.

    Article  Google Scholar 

  • Wang, Y., Ai, H., Wu, B., & Huang, C. (2004). Real time facial expression recognition with Adaboost. In IEEE international conference on pattern recognition.

  • Wen, Z., & Huang, T. S. (2003). Capturing subtle facial motions in 3D face tracking. In IEEE international conference on computer vision.

  • Yacoob, Y., & Davis, L. S. (1996). Recognizing human facial expressions from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(6), 636–642.

    Article  Google Scholar 

  • Yilmaz, A., Shafique, K. H., & Shah, M. (2002). Estimation of rigid and non-rigid facial motion using anatomical face model. In IEEE international conference on pattern recognition.

  • Zhang, Y., & Ji, Q. (2005). Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 699–714.

    Article  MathSciNet  Google Scholar 

  • Zhou, S., Krueger, V., & Chellappa, R. (2003). Probabilistic recognition of human faces from video. Computer Vision and Image Understanding, 91(1–2), 214–245.

    Article  Google Scholar 

  • Zhou, S., Chellappa, R., & Mogghaddam, B. (2004). Visual tracking and recognition using appearance-adaptive models in particle filters. IEEE Transactions on Image Processing, 13(11), 1473–1490.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institut Géographique National, Laboratoire MATIS, 2 avenue Pasteur, 94165, Saint-Mandé, Cedex, France

    Fadi Dornaika

  2. Heudiasyc Mixed Research Unit, CNRS/UTC, 60205, Compiegne, France

    Franck Davoine

Authors
  1. Fadi Dornaika
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Franck Davoine
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fadi Dornaika.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Dornaika, F., Davoine, F. Simultaneous Facial Action Tracking and Expression Recognition in the Presence of Head Motion. Int J Comput Vis 76, 257–281 (2008). https://doi.org/10.1007/s11263-007-0059-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-007-0059-7

Keywords

Search

Navigation