Skip to main content
SpringerLink
Log in

Recognizing Facial Expressions Automatically from Video

  • Chapter
Book cover Handbook of Ambient Intelligence and Smart Environments

Abstract

Facial expressions, resulting from movements of the facial muscles, are the face changes in response to a person’s internal emotional states, intentions, or social communications. There is a considerable history associated with the study on facial expressions. Darwin [22] was the first to describe in details the specific facial expressions associated with emotions in animals and humans, who argued that all mammals show emotions reliably in their faces. Since that, facial expression analysis has been a area of great research interest for behavioral scientists [27]. Psychological studies [48, 3] suggest that facial expressions, as the main mode for nonverbal communication, play a vital role in human face-to-face communication. For illustration, we show some examples of facial expressions in Fig. 1.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ahonen T, Hadid A, Pietikäinen M (2004) Face recognition with local binary patterns. In: European Conference on Computer Vision (ECCV), pp 469–481

    Google Scholar 

  2. Ambadar Z, Schooler JW, Cohn JF (2005) Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions. Psychological Science 16(5):403–410

    Article  Google Scholar 

  3. Ambady N, Rosenthal R (1992) Thin slices of expressive behaviour as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin 111(2):256–274

    Article  Google Scholar 

  4. Amin MA, Afzulpurkar NV, Dailey MN, Esichaikul VE, Batanov DN (2005) Fuzzy-c-mean determines the principle component pairs to estimate the degree of emotion from facial expressions. In: International Conference on Natural Computation and International Conference on Fuzzy Systems and Knowledge Discovery, pp 484–493

    Google Scholar 

  5. Argyle M (1988) Bodily Communication (2nd ed.). Methuen & Co. Ltd., New York

    Google Scholar 

  6. Balomenos T, Raouzaiou A, Ioannou S, Drosopoulos A, Karpouzis K, Kollias S (2005) Emotion analysis in man-machine interaction systems. In: Machine Learning for Multimodal Interaction, LNCS 3361, pp 318–328

    Google Scholar 

  7. Bartlett M, Littlewort G, Frank M, Lainscseck C, Fasel I, Movellan J (2006) Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1(6):22–35

    Article  Google Scholar 

  8. Bartlett MS, Movellan JR, Sejnowski TJ (2002) Face recognition by independent component analysis. IEEE Transactions on Neural Networks 13(6):1450–1464

    Article  Google Scholar 

  9. Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression: Machine learning and application to spotaneous behavior. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 568–573

    Google Scholar 

  10. Bassili JN (1979) Emotion recognition: The role of facial movement and the relative importance of upper and lower area of the face. Journal of Personality and Social Psychology 37(11):2049–2058

    Article  Google Scholar 

  11. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7):711–720

    Article  Google Scholar 

  12. Braathen B, Bartlett M, Littlewort G, Smith E, Movellan JR (2002) An approach to automatic recognition of spontaneous facial actions. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 231–235

    Google Scholar 

  13. Burgoon JK, Buller DB, Woodall WG (1996) Nonverbal Communication: The Unspoken Dialogue. McGraw-Hill, New York

    Google Scholar 

  14. Chang Y, Hu C, Turk M (2003) Mainfold of facial expression. In: IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG), pp 28–35

    Google Scholar 

  15. Chang Y, Hu C, Turk M (2004) Probabilistic expression analysis on manifolds. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 520–527

    Google Scholar 

  16. Cohen I, Sebe N, Garg A, Chen L, Huang TS (2003) Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding 91:160–187

    Article  Google Scholar 

  17. Cohn JF, Schmidt KL (2004) The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Information Processing 2:1–12

    Article  MathSciNet  Google Scholar 

  18. Cohn JF, Reed LI, Ambadar Z, Xiao J, Moriyama T (2004) Automatic analysis and recognition of brow actions in spontaneous facial behavior. In: IEEE International Conference on Systems, Man, and Cybernetics, p 610

    Google Scholar 

  19. Costa M, Dinsbach W, Manstead ASR, Bitti PER (2001) Social presence, embarrassment, and nonverbal behavior. Journal of Nonverbal Behavior 25(4):225–240

    Article  Google Scholar 

  20. Coulson M (2004) Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence.Journal of Nonverbal Behavior 28(2):117–139

    Article  MathSciNet  Google Scholar 

  21. Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schroder M (2000) ’feeltrace’: An instrument for recording perceived emotion in real time. In: Proceeding of the ISCA Workshop on Speech and Emotion, pp 19–24

    Google Scholar 

  22. Darwin C (1872) The Expression of the Emotions in Man and Animals. John Murray, London

    Book  Google Scholar 

  23. Donato G, Bartlett M, Hager J, Ekman P, Sejnowski T (1999) Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(10):974–989

    Article  Google Scholar 

  24. Douglas-Cowie E, Cowie R, Schroder M (2000) A new emotion database: Considerations, sources and scope. In: Proceeding of the ISCA Workshop on Speech and Emotion: A Conceptual Framework for Research, pp 39–44

    Google Scholar 

  25. Ekman P, Friesen W (1971) Constants across cultures in the face and emotion. J Personality Social Psychol 17(2):124–129

    Article  Google Scholar 

  26. Ekman P, Rosenberg E (1997) What the Face Reveals: Basic and Applied Studies of Spontaneous Expression using the Facial Action Coding System (FACS). New York: Oxford Univ. Press

    Google Scholar 

  27. Ekman P, Friesen WV, Hager JC (2002) The Facial Action Coding System: A Technique for the Measurement of Facial Movement. San Francisco: Consulting Psychologist

    Google Scholar 

  28. Essa I, Pentland A (1997) Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7):757–763

    Article  Google Scholar 

  29. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recognition 36:259–275

    Article  MATH  Google Scholar 

  30. de Gelder B (2006) Towards the neurobiology of emotional body language. Nature Reviews Neuroscience 7:242–249

    Article  Google Scholar 

  31. Gritti T, Shan C, Jeanne V, Braspenning R (2008) Local features based facial expression recognition with face registration errors. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), Amsterdam, The Netherlands

    Google Scholar 

  32. Gu H, Ji Q (2005) Information extraction from image sequences of real-world facial expressions. Machine Vision and Applications 16:105–115

    Article  Google Scholar 

  33. Gunes H, Piccardi M (2006) Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications (In press)

    Google Scholar 

  34. Gunes H, Piccardi M (2006) A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In: International Conference on Pattern Recognition (ICPR), vol 1, pp 1148–1153

    Google Scholar 

  35. Guo G, Dyer CR (2003) Simultaneous feature selection and classifier training via linear programming: A case study for face expression recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 346–352

    Google Scholar 

  36. He X, Niyogi P (2003) Locality preserving projections. In: Advances in Neural Information Processing Systems (NIPS)

    Google Scholar 

  37. Hess U, Blairy S, Kleck RE (1997) The intensity of emotional facial expression and decoding accuracy. Journal of Nonverbal Behavior 21(4):241–257

    Article  Google Scholar 

  38. Kaliouby RE, Robinson P (2004) Real-time inference of complex mental states from facial expressions and head gestures. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp 154–154

    Google Scholar 

  39. Kanade T, Cohn J, Tian Y (2000) Comprehensive database for facial expression analysis. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 46–53

    Google Scholar 

  40. Kapoor A, Picard RW (2005) Multimodal affect recognition in learning environments. In: ACM International Conference on Multimedia, pp 677–682

    Google Scholar 

  41. Kimura S, Yachida M (1997) Facial expression recognition and its degree estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 295–300

    Google Scholar 

  42. Lee CS, Elgammal A (2005) Facial expression analysis using nonlinear decomposable generative models. In: IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG), pp 17–31

    Google Scholar 

  43. Lien JJ, Kanade T, Cohn JF, Li C (1998) Subtly different facial expression recognition and expression intensity estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 853–859

    Google Scholar 

  44. Littlewort G, Bartlett M, Fasel I, Susskind J, Movellan J (2006) Dynamics of facial expression extracted automatically from video. Image and Vision Computing 24(6):615–625

    Article  Google Scholar 

  45. Littlewort G, Bartlett MS, Lee K (2006) Faces of pain: Automated measurement of spontaneous facial expressions of genuine and posed pain. In: Joint Symposium on Neural Computation, pp 1–1

    Google Scholar 

  46. Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(12):1357–1362

    Article  Google Scholar 

  47. Meeren H, Heijnsbergen C, Gelder B (2005) Rapid perceptual integration of facial expression and emotional body language. Procedings of the National Academy of Sciences of USA 102(45):16,518–16,523

    Google Scholar 

  48. Mehrabian A (1968) Communication without words. Psychology Today 2(4):53–56

    Google Scholar 

  49. Nayak V, Turk M (2005) Emotional expression in virtual agents through body language. In: International Symposium on Visual Computing, pp 313–320

    Google Scholar 

  50. Neagle RJ, Ng K, Ruddle RA (2003) Studying the fidelity requirements for a virtual ballet dancer. In: Vision, Video and Graphics, pp 181–188

    Google Scholar 

  51. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distribution. Pattern Recognition 29(1):51–59

    Article  Google Scholar 

  52. Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7):971–987

    Article  Google Scholar 

  53. Oliver N, Pentland A, Berard F (2000) Lafter: a real-time face and lips tracker with facial expression recognition. Pattern Recognition 33:1369–1382

    Article  Google Scholar 

  54. O’Toole AJ, Harms J, Snow SL, Hurst DR, Pappas MR, Ayyad JH, Abdi H (2005) A video database of moving faces and people. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(5):812–816

    Article  Google Scholar 

  55. Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Kurihara K (ed) Face Recognition, Advanced Robotics Systems, Vienna, Austria, pp 377–416

    Google Scholar 

  56. Pantic M, Patras I (2006) Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics 36(2):433–449

    Article  Google Scholar 

  57. Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: the state of art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1424–1445

    Article  Google Scholar 

  58. Pantic M, Rothkrantz L (2000) Expert system for automatic analysis of facial expression. Image and Vision Computing 18(11):881–905

    Article  Google Scholar 

  59. Pantic M, Rothkrantz L (2003) Toward an affect-sensitive multimodal human-computer interaction. In: Proceeding of the IEEE, vol 91, pp 1370–1390

    Google Scholar 

  60. Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Transactions on Systems, Man, and Cybernetics 34(3):1449–1461

    Article  Google Scholar 

  61. Pantic M, Sebe N, Cohn J, Huang T (2005) Affective multimodal human-computer interaction. In: ACM International Conference on Multimedia, pp 669–676

    Google Scholar 

  62. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE International Conference on Multimedia and Expo (ICME), pp 317–321

    Google Scholar 

  63. Pavlovic VI, Sharma R, Huang TS (1997) Visual interpretation of hand gestures for human-computer interaction: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7):677–695

    Article  Google Scholar 

  64. Ravindra De Silva P, Osano M, Marasinghe A (2006) Towards recognizing emotion with affective dimensions through body gestures. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 269–274

    Google Scholar 

  65. Russell JA (1994) Is there univeral recognition of emotion from facial expression. Psychological Bulletin 115(1):102–141

    Article  Google Scholar 

  66. Schapire RE, Singer Y (1999) Improved boosting algorithms using confidence-rated predictions. Maching Learning 37(3):297–336

    Article  MATH  Google Scholar 

  67. Sebe N, Lew MS, Cohen I, Sun Y, Gevers T, Huang TS (2004) Authentic facial expression analysis. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 517–522

    Google Scholar 

  68. Shan C (2007) Inferring facial and body language. PhD thesis, Queen Mary, University of London

    Google Scholar 

  69. Shan C, Gritti T (2008) Learning discriminative lbp-histogram bins for facial expression recognition. In: British Machine Vision Conference (BMVC), Leeds, UK

    Google Scholar 

  70. Shan C, Gong S, McOwan PW (2005) Appearance manifold of facial expression. In: Sebe N, Lew MS, Huang TS (eds) Computer Vision in Human-Computer Interaction, Lecture Notes in Computer Science, vol 3723, Springer, pp 221–230

    Google Scholar 

  71. Shan C, Gong S, McOwan PW (2005) Conditional mutual information based boosting for facial expression recognition. In: British Machine Vision Conference (BMVC), Oxford, UK, vol 1, pp 399–408

    Google Scholar 

  72. Shan C, Gong S, McOwan PW (2005) Recognizing facial expressions at low resolution. In: IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS), Como, Italy, pp 330–335

    Google Scholar 

  73. Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE International Conference on Image Processing (ICIP), Genoa, Italy, vol 2, pp 370–373

    Google Scholar 

  74. Shan C, Gong S, McOwan PW (2006) A comprehensive empirical study on linear subspace methods for facial expression analysis. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, New York, USA, pp 153–158

    Google Scholar 

  75. Shan C, Gong S, McOwan PW (2006) Dynamic facial expression recognition using a bayesian temporal manifold model. In: British Machine Vision Conference (BMVC), Edinburgh, UK, vol 1, pp 297–306

    Google Scholar 

  76. Shan C, Gong S, McOwan PW (2007) Beyond facial expressions: Learning human emotion from body gestures. In: British Machine Vision Conference (BMVC), Warwick, UK

    Google Scholar 

  77. Shan C, Gong S, McOwan PW (2007) Capturing correlations among facial parts for facial expression analysis. In: British Machine Vision Conference (BMVC), Warwick, UK

    Google Scholar 

  78. Shan C, Gong S, McOwan PW (2008) Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing

    Google Scholar 

  79. Skelley J, Fischer R, Sarma A, Heisele B (2006) Recognizing expressions in a new database containing played and natural expressions. In: International Conference on Pattern Recognition (ICPR), pp 1220–1225

    Google Scholar 

  80. Tian Y (2004) Evaluation of face resolution for expression analysis. In: International Workshop on Face Processing in Video, pp 82–82

    Google Scholar 

  81. Tian Y, Kanade T, Cohn J (2000) Eye-state action unit detection by gabor wavelets. In: International Conference on Multimodal Interfaces (ICMI), pp 143–150

    Google Scholar 

  82. Tian Y, Kanade T, Cohn J (2001) Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2):97–115

    Article  Google Scholar 

  83. Tian Y, Brown L, Hampapur A, Pankanti S, Senior A, Bolle R (2003) Real world real-time automatic recognition of facial expression. In: IEEE workshop on performance evaluation of tracking and surveillance (PETS), Australia

    Google Scholar 

  84. Tian Y, Kanade T, Cohn J (2005) Handbook of Face Recognition, Springer, chap 11. Facial Expression Analysis

    Google Scholar 

  85. Tong Y, Liao W, Ji Q (2006) Inferring facial action units with causal relations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1623–1630

    Google Scholar 

  86. Turk M, Pentland AP (1991) Face recognition using eigenfaces. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    Google Scholar 

  87. Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, p 149

    Google Scholar 

  88. Valstar M, Pantic M, Ambadar Z, Cohn JF (2006) Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. In: International Conference on Multimodal Interfaces (ICMI), pp 162–170

    Google Scholar 

  89. Wallhoff F (2006) Facial expressions and emotion database. http://www.mmk.ei.tum.de/waf/fgnet/feedtum.html

  90. Yacoob Y, Davis LS (1996) Recognizing human facial expression from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(6):636–642

    Article  Google Scholar 

  91. Yeasin M, Bullot B, Sharma R (2004) From facial expression to level of interests: A spatio-temporal approach. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 922–927

    Google Scholar 

  92. Yin L, Wei X, Sun Y, Wang J, Rosato M (2006) A 3d facial expression database for facial behavior research. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG)

    Google Scholar 

  93. Zeng Z, Fu Y, Roisman GI, Wen Z, Hu Y, Huang TS (2006) Spontaneous emotional facial expression detection. Journal of Multimedia (JMM) 1(5):1–8

    Google Scholar 

  94. Zhang G, Huang X, Li SZ, Wang Y, Wu X (2004) Boosting local binary pattern (lbp)-based face recognition. In: Chinese Conference on Biometric Recognition, pp 179–186

    Google Scholar 

  95. Zhang Y, Ji Q (2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(5):1–16

    Article  MATH  MathSciNet  Google Scholar 

  96. Zhang Z, Lyons MJ, Schuster M, Akamatsu S (1998) Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 454–461

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Philips Research, Eindhoven, The Netherlands

    Caifeng Shan & Ralph Braspenning

Authors
  1. Caifeng Shan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ralph Braspenning
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Caifeng Shan or Ralph Braspenning .

Editor information

Editors and Affiliations

  1. Future University Hakodate, Kameda-Nakano 116-2, Hakodate, Hokkaido, 041-8655, Japan

    Hideyuki Nakashima

  2. Department of Electrical Engineering, Stanford University, 350 Serra Mall, Stanford, CA, 94305-9515, USA

    Hamid Aghajan

  3. School of Computing & Mathematics, University of Ulster at Jordanstown, Shore Road, Newtownabbey, Co. Antrim, UK, BT37 0QB

    Juan Carlos Augusto

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Shan, C., Braspenning, R. (2010). Recognizing Facial Expressions Automatically from Video. In: Nakashima, H., Aghajan, H., Augusto, J.C. (eds) Handbook of Ambient Intelligence and Smart Environments. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-93808-0_18

Download citation

  • DOI: https://doi.org/10.1007/978-0-387-93808-0_18

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-0-387-93807-3

  • Online ISBN: 978-0-387-93808-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation