Skip to main content
SpringerLink
Log in

Information extraction from image sequences of real-world facial expressions

  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract.

Information extraction of facial expressions deals with facial-feature detection, feature tracking, and capture of the spatiotemporal relationships among features. It is a fundamental task in facial expression analysis and will ultimately determine the performance of expression recognition. For a real-world facial expression sequence, there are three challenges: (1) detection failure of some or all facial features due to changes in illumination and rapid head movement; (2) nonrigid object tracking resulting from facial expression change; and (3) feature occlusion due to out-of-plane head rotation. In this paper, a new approach is proposed to tackle these challenges. First, we use an active infrared (IR) illumination to reliably detect pupils under variable lighting conditions and head orientations. The pupil positions are then used to guide the entire information-extraction process. The simultaneous use of a global head motion constraint and Kalman filtering can robustly track individual facial features even in condition of rapid head motion and significant expression change. To handle feature occlusion, we propose a warping-based reliability propagation method. The reliable neighbor features and the spatial semantics among these features are used to detect and infer occluded features through an interframe warping transformation. Experimental results show that accurate information extraction can be achieved for video sequences with real-world facial expressions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abd-Almageed W, Fadali MS, Bebis G (2002) A non-intrusive Kalman filter-based tracker for pursuit eye movement. In: Proceedings of the 2002 American Control conference

  2. Ahlberg J (2000) Real-time facial feature tracking using an active model with fast image warping. http://citeseer.ist.psu.edu/538871.html

  3. Bookstein F (1989) Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans Pattern Anal Mach Intell 11(6):567-585

    Google Scholar 

  4. Bookstein F (1991) Morphometric tools for landmark data. Oxford: Cambridge University Press, Cambridge, UK

  5. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509-522

    Google Scholar 

  6. Beier T, Neely S (1992) Feature-based image metamorphosiss. In: Computer Graphics (Proceedings of SIGGRAPH 92) 26(2):35-42

  7. Bourel F, Chibelushi CC, Low AA (2000) Robust facial feature tracking. Proceedings of the 11th British machine vision conference, 1:232-241

  8. Bretzner L, Lindeberg T (1998) Feature tracking with automatic selection of spatial scales. Comput Vis Image Understand 71(3):385-392

    Google Scholar 

  9. Chetverikov D, Verest’oy J (1998) Tracking feature points: a new algorithm. In: Proceedings of the international conference on pattern recognition, pp 1436-1438

  10. Choi CS, Takebe T (1994) Analysis and synthesis of facial image sequences in model-based image coding. IEEE Trans Video Technol 4(6):257-275

    Google Scholar 

  11. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273-297

    Article  Google Scholar 

  12. Daugman J (1988) Complete discrete 2-d gabor transforms by neural networks for image analysis and compression. IEEE Trans ASSP 36:1169-1179

    Google Scholar 

  13. Eisert P, Girod B (1998) Analyzing facial expressions for virtual conferencing IEEE Comput Graph Appl 18(5):70-78

    Google Scholar 

  14. Fieguth P (1997) Color-based tracking of heads and other mobile objects at video frame rates. In: Proceedings of the conference on computer vision and pattern recognition

  15. Gorodnichy D (2002) On importance of nose for face tracking. In: Proceedings of the international conference on automatic face and gesture recognition (FG’2002)

  16. Haro A, Flickner M, Essa I (2000) Detecting and tracking eyes by using their physiological properties, dynamics, and appearance. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 163-168

  17. Huang J, Ii D, Shao X, Wechsler H (1998) Pose discrimination and eye detection using support vector machines (svms), In: Proceedings of NATO-ASI on face recognition: from theory to applications, pp 528-536

  18. Hwang V (1998) Tracking feature points in time-varying images using an opportunistic selection approach. Pattern Recog 22:247-256

    Google Scholar 

  19. Ji Q, Yang X (2001) Real time visual cues extraction for monitoring driver vigilance. In: Schiele B, Sagerer G (eds) Lecture notes in computer science, vol 2095. Springer, Berlin Heidelberg New York, p 107

  20. Kalman RE (1960) A new approach to linear filtering and prediction problems. Trans ASME J Basic Eng 85:35-45

    Google Scholar 

  21. Kapoor A, Picard RW (2002) Real-time, fully automatic upper facial feature tracking. In: Proceedings of the 5th IEEE international conference on automatic face and gesture recognition, pp 10-16

  22. Kass M, Witkin A, Terzopoulos D (1987) Snakes: active contour models. Int J Comput Vis 1:321-332

    Google Scholar 

  23. Lee T (1996) Image representation using 2d Gabor wavelets. IEEE Trans Pattern Anal Mach Intell 18(10):959-971

    Google Scholar 

  24. Li H, Roivainen P, Forchheimer R (1993) 3-D motion estimation in model-based facial image coding. IEEE Trans Pattern Anal Mach Intell 15(6):545-555

    Google Scholar 

  25. Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: Proceedings of the international joint conference on artificial intelligence, pp 674-679

  26. Luettin J, Thacker NA, Beet SW (1996) Locating and tracking facial speech features. In: Proceedings of the international conference on pattern recognition, pp 652-656

  27. Malciu M, Preteux F (2000) Tracking facial features in video sequences using a deformable model-based approach. http://citeseer.nj.nec.com/394625.html

  28. Manjunath B, Chellappa R, Malsburg C (1992) A feature based approach to face recognition. Proceedings of IEEE international conference on computer vision and pattern recognition, pp 373-378

  29. Maurer T, Malsburg C (1996) Tracking and learning graphs on image sequences of faces. In: Proceedings of the international conference on artificial neural networks, pp 373-378

  30. McKenna S, Gong S, Wurtz R, Tanner J, Bannin D (1997) Tracking facail feature points with gabor wavelets and shape models. In: Proceedings of the international conference on audio-and video-based biometric person authentication, pp 35-42

  31. Meyer F, Bouthemy P (1994) Region-based tracking using affine motion models in long image sequences. CVGIP Image Understand 60:119-140

    Google Scholar 

  32. Morimoto C, Koons D, Amir A, Flickner M (1999) Framer-ate pupil detector and gaze tracker. In: Proceedings of IEEE ICCV, frame-rate workshop

  33. Moriyama T, Kanade T, Cohn J, Xiao J, Ambadar Z, Gao J, Imanura M (2002) Automatic recognition of eye blinking in spontaneously occurring behavior. In: Proceedings of the international conference on pattern recognition (ICPR ‘2002)

  34. Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: The state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424-1445

    Google Scholar 

  35. Petajan E, Graf H (1996) Robust face feature analysis fo automatic speech-reading and character animation. In: Proceedings of the international conference on automatic face and gesture recognition, pp 357-362

  36. Rangarajan K, Shah M (1991) Establishing motion correspondence. CVGIP: Image Understanding 54:56-73

  37. Salari V, Sethi IK (1990) Feature point correspondence in the presence of occlusion. IEEE Trans Pattern Anal Mach Intell 12:87-91

    Google Scholar 

  38. Sethi IK, Jain R (1987) Finding trajectories of feature points in a monocular image sequence. IEEE Trans Pattern Anal Mach Intell 9:56-73

    Google Scholar 

  39. Shapiro L,Wang H, Brady J (1992) A matching and tracking strategy for independently-moving, non-rigid objects. In: Proceedings of the 3rd British machine vision conference, pp 306-315

  40. Shi J, Tomasi C (1994) Good features to track. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 593-600

  41. Sorenson HW (1970) Least-squares estimation: from Gauss to Kalman. IEEE Spectrum 7:63-68

    Google Scholar 

  42. Tomasi C, Kanade T (1991) Detection and tracking of point features. Technical Report, Carnegie Mellon University, Pittsburgh, PA

  43. Thompson W, Lechleider P, Stuck E (1993) Detecting moving objects using the rigidity constraint. IEEE Trans Pattern Anal Mach Intell 15:162-166

    Google Scholar 

  44. Torresani L, Bregler C (2002) Space-time tracking. In: Proceedings of ECCV, pp 801-812

  45. Tomasi C, Kanade T (1992) Shape and motion from image streams: a factorization method. Carnegie Mellon CMU-CS-92-104, pp 1-36

    Google Scholar 

  46. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71-86

    Google Scholar 

  47. Yang J, Stiefelhagen R, Meier U, Waibel A (1998) Real-time face and facial feature tracking and applications. In: Proceedings of the international conference on auditory-visual speech processing (AVSP’98)

  48. Yuille A, Cohen DS, Hallinan PW (1989) Feature extraction from faces using deformable templates. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 104-109

  49. Yuille A, Hallinan P (1993) Deformable templates. In: Blake A, Yuille A (eds) Active vision. MIT Press, Cambridge, MA, pp 21-38

  50. Zhang L (1998) Automatic adaptation of a face model using action units for semantic coding of videophone sequences. IEEE Trans Circuits Syst Video Technol 8(6):781-795

    Google Scholar 

  51. Zheng Q, Chellappa R (1995) Automatic feature point extraction and tracking in image sequences for arbitrary camera motion. Int J Comput Vis 15:31-76

    Google Scholar 

  52. Zhang Z (1998) Feature-based facial expression recognition: Experiments with a multi-layer perception. Technical report INRIA, no 3354

  53. Zhong Y, Jain AK, Dubuisson-Jolly M (2000) Object tracking using deformable templates. IEEE Trans Pattern Anal Mach Intell 22(5):544-549

    Google Scholar 

  54. Zhu Z, Ji Q, Fujimura K, Lee K (2002) Combining Kalman filtering and mean shift for real time eye tracking under active IR illumination. In: Proceedings of the international conference on pattern recognition, pp 373-378

  55. Wang H, Brady J (1992) Corner detection with subpixel accuracy. Technical Report OUEL, Department of Engineering Science, University of Oxford, no 1925/92

  56. Wiskott L, Fellous J, Krieger N, Malsburg C (1995) Face recognition and gender determination. In: Proceedings of the international workshop on automatic face and gesture recognition, pp 92-97

  57. Fleet D, Jepson A (1993) Stability of phase information. IEEE Trans Pattern Anal Mach Intell 15(12):1253-1268

    Google Scholar 

  58. Wei X, Zhu Z, Yin L, Ji Q (2004) A real time face tracking and animation system In: 1st IEEE workshop on face processing in video, in conjunction with IEEE international conference on computer vision and pattern recognition

    Google Scholar 

  59. Gu H, Ji Q (2004) An automated face reader for fatigue detection, In: 6th international conference on automatic face and gesture recognition, 17-19 May 2004, Seoul, Korea

  60. Li X, Ji Q (2003) Active affective state detection and assistance with dynamic Bayesian networks. 3rd workshop on affective and attitude user modeling assessing and adapting to user attitudes and affect: why, when and how?, in conjunction with 10th international conference on user modeling, Pittsburgh, PA

    Google Scholar 

  61. Wiskott L, Fellous J-M, Kruger N, von der Malsburg C (1997) Face recognition by elastic bunch graph matching. IEEE Trans Pattern Anal Mach Intell 19(7):775-779

    Google Scholar 

  62. Hutchinson TE (1990) Eye movement detection with improved calibration and speed. US patent 4,950,069

  63. Cristinacce D, Cootes TF (2003) Facial feature detection using ADABOOST with shape constraints. In: Proceedings of BMVC, 1:231-240

  64. Cristinacce D, Cootes TF (2004) A comparison of shape constrained facial feature detectors. In: Proceedings of the international conference on face and gesture recognition, pp 375-380

  65. Cristinacce D, Cootes TF, Scott I (2004) A multistage approach to facial feature detection. In: Proceedings of the British machine vision conference, 1:277-286

  66. Cootes TF, Edwards GJ, Taylor CJ (1998) Active appearance models. In: Proceedings of the European conference on computer vision 2:484-498

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Nevada-Reno, NV 89557, Reno, USA

    Haisong Gu

  2. Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, NY 12180, Troy, USA

    Qiang Ji

Authors
  1. Haisong Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiang Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiang Ji.

Additional information

Received: 16 August 2003, Accepted: 20 September 2004, Published online: 20 December 2004

Correspondence to: Qiang Ji

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gu, H., Ji, Q. Information extraction from image sequences of real-world facial expressions. Machine Vision and Applications 16, 105–115 (2005). https://doi.org/10.1007/s00138-004-0161-6

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-004-0161-6

Keywords:

Search

Navigation