Skip to main content
SpringerLink
Log in

Automatic Facial Action Unit Recognition by Modeling Their Semantic And Dynamic Relationships

  • Chapter
Book cover Affective Information Processing

Abstract

A system that could automatically analyze the facial actions in real-time has applications in a wide range of different fields. The previous facial action unit (AU) recognition approaches often recognize AUs or certain AU combinations in dividually and statically, ignoring the semantic relationships among AUs and the dynamics of AUs. Hence, these approaches cannot always recognize AUs reliably, robustly, and consistently due to the richness, ambiguity, and the dynamic nature of facial actions. In this work, a novel AU recognition system is proposed to sys tematically account for the relationships among AUs and their temporal evolutions based on a dynamic Bayesian network (DBN). The DBN provides a coherent and unified hierarchical probabilistic framework to represent probabilistic relationships among various AUs and to account for the temporal changes in facial action devel opment. Within the proposed system, robust computer vision techniques are used to obtain AU measurements. And such AU measurements are then applied as evidence to the DBN for inferring various AUs. The experiments show that the integration of AU relationships and AU dynamics with AU measurements yields significant improvement of AU recognition, especially under realistic environments including illumination variation, face pose variation, and occlusion.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bartlett, M. S., Littlewort, G., Frank, M. G., Lainscsek, C., Fasel, I., & Movellan, J. R. (2005). Recognizing facial expression: Machine learning and application to spontaneous behavior.Pro ceedings of CVPR052, pp. 568–573

    Google Scholar 

  2. Bartlett, M. S., Littlewort, G., Movellan, J. R., Frank, M. G. (2007). Auto facs coding URLhttp://mplab.ucsd.edu/grants/project1/research/Fully-Auto-FACS-Coding.html

  3. Bassili, J. N. (1979). Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face.Journal of Personality and Social Psychology 37(11), 2049–2058.

    Article  Google Scholar 

  4. Bazzo, J. J., Lamar, M. V. (2004). Recognizing facial actions using gabor wavelets with neutral face average difference.Proceedings of FGR04, pp. 505–510.

    Google Scholar 

  5. Cohen, I., Cozman, F. G., Sebe, N., Cirelo, M. C., & Huang, T. S. (2004). Semisupervised learning of classifiers: Theory, algorithms, and their application to human-computer interaction.IEEE Transactions on PAMI26(12), 1553–1567.

    Google Scholar 

  6. Cohen, I., Sebe, N., Garg, A., Chen, L., & Huang, T. (2003). Facial expression recognition from video sequences: Temporal and static modeling.Computer Vision and Image Understanding 91(1–2), 160–187.

    Article  Google Scholar 

  7. Cohn, J. F., Reed, L. I., Ambadar, Z., Xiao, J., & Moriyama, T. (2004) Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior.In Proceedings of IEEE Int'l Conf on Systems, Man, and Cybernetics1, pp. 610–616

    Google Scholar 

  8. Cohn, J. F., & Zlochower, A. (1995). A computerized analysis of facial expression: Feasibility of automated discrimination.American Psychological Society, New York, New York.

    Google Scholar 

  9. Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Classifying facial actions.IEEE Transactions, on PAMI 21(10), 974–989.

    Google Scholar 

  10. Ekman, P., & Friesen, W. V. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.

    Google Scholar 

  11. Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system: The manual. Re search Nexus, Div. Salt Lake City, UT: Network Information Research Corp.

    Google Scholar 

  12. El Kaliouby, R., & Robinson, P. K. (2004). Real-time inference of complex mental states from facial expressions and head gestures.CVPRW'04 10, 154.

    Google Scholar 

  13. Fasel, B., & Luettin, J. (2000) Recognition of asymmetric facial action unit activities and intensi ties.Proceedings of ICPR00 1, pp. 1100–1103.

    Google Scholar 

  14. Gu, H., & Ji, Q. (2004). Facial event classification with task oriented dynamic bayesian network.Proceedings of CVPR042, pp. 870–875.

    Google Scholar 

  15. Heckerman, D. (1995). A tutorial on learning with Bayesian networks. Tech Report MSR-TR-95-06, Microsoft Research.

    Google Scholar 

  16. Heckerman, D., Geiger, D., Chickering, D. M. (1995). Learning Bayesian networks: The combi nation of knowledge and statistical data.Machine Learning 20(3), 197–243.

    MATH  Google Scholar 

  17. Heller, M., & Haynal, V. (1997). Depression and suicide faces. In: P., Ekman Rosenberg & E (Eds.)What the face reveals, (pp. 339–407). Oxford University Press. New York.

    Google Scholar 

  18. Kanade, T., Cohn, J. F., Tian, Y. (2000). Comprehensive database for facial expression analysis.Proceedings of FGR00, pp. 46–53.

    Google Scholar 

  19. Kapoor, A., Qi, Y., & Picard, R. W. (2003). Fully automatic upper facial action recognition.IEEE Int'l Workshop on Analysis and Modeling of Faces and Gestures, pp. 195–202.

    Google Scholar 

  20. Kjaerulff, U. (1995) dhugin: A computational system for dynamic time-sliced bayesian networks.International Journal of Forecasting, Special issue on Probability Forecasting11, 89–111.

    Article  Google Scholar 

  21. Korb, K. B., & Nicholson, A. E. (2004).Bayesian artificial intelligence. London; Chapman and Hall/CRC.

    MATH  Google Scholar 

  22. Lanitis, A., Taylor, C. J., Cootes, T. F. (1997). Automatic interpretation and coding of face images using flexible models.IEEE Transactions on PAMI 19(7), 743–756.

    Google Scholar 

  23. Lien, J. J., Kanade, T., Cohn, J. F., & Li, C. (2000). Detection, tracking, and classification of action units in facial expression.Journal of Robotics and Autonomous Systems 31, 131–146.

    Article  Google Scholar 

  24. Pantic M, Bartlett, M (2007). Machine analysis of facial expressions. In K. Delac, Grgic M. (Eds.)Face recognition, (pp. 377–416), Vienna: I-Tech Education.

    Google Scholar 

  25. Pantic, M., & Rothkrantz, L. J. M. (2004). Facial action recognition for facial expression analysis from static face images.IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cyber netics 34(3),1449–1461.

    Article  Google Scholar 

  26. Pantic, M., Valstar, M., Rademaker, R., Maat, L. (2005). Web-based database for facial expression analysis.Proceedings of IEEE Int'l Conf on Multmedia and Expo (ICME05)pp. 317–321.

    Google Scholar 

  27. Phillips, P. J., Flynn, P. J., Scruggs, T, Bowyer, K. W., Chang, J., Hoffman, K., Marques, J., Min, J., & Worek, W. (2005). Overview of the face recognition grand challenge.Proceedings of CVPR051, pp. 947–954

    Google Scholar 

  28. Schwarz, G. (1978). Estimating the dimension of a model.The Annals of Statistics 6, 461–464.

    Article  MATH  MathSciNet  Google Scholar 

  29. Smith, E., Bartlett, M. S., & Movellan, J. R. (2001). Computer recognition of facial actions: A study of co-articulation effects.Proceedings of the 8th Annual Joint Symposium on Neural Computation.

    Google Scholar 

  30. Spiegelhalter, D., & Lauritzen, S. (1990). Sequential updating of conditional probabilities on di rected graphical structures.Networks 20, 579–605.

    Article  MATH  MathSciNet  Google Scholar 

  31. Tian, Y., Kanade, T., & Cohn, J. F. (2001). Recognizing action units for facial expression analysis.IEEE Transact. on PAMI 23(2), 97–115.

    Google Scholar 

  32. Tian, Y., Kanade, T., & Cohn, J. F. (2002). Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity.Proceedings of FGR02(pp. 218– 223).

    Google Scholar 

  33. Tong, Y., Liao, W., & Ji, Q. (2007). Intelligent systems lab (isl) database URLhttp://www.ecse. rpi.edu/homepages/cvrl/database/database.html

  34. Valstar, M. F., Patras, I., & Pantic, M. (2005). Facial action unit detection using probabilistic ac tively learned support vector machines on tracked facial point data.Proceedings of CVPRW'05 on Vision for Human-Computer Interactionpp. 76.

    Google Scholar 

  35. Wang, P., Green, M. B., Ji, Q., & Wayman, J. (2005). Automatic eye detection and its validation.IEEE Workshop on Face Recognition Grand Challenge Experiments (with CVPR)3.

    Google Scholar 

  36. Williams, A. C. (2002). Facial expression of pain: An evolutionary account.Behavioral & Brain Sciences 25(4), 439–488.

    Google Scholar 

  37. Zhang, Y., & Ji, Q. (2005). Active and dynamic information fusion for facial expression under standing from image sequences.IEEE Transactions on PAMI 27(5), 699–714.

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. General Electric Global Research Center, Schenectady, NY

    Yan Tong

  2. The Thomson Corporation, Eagan, MN

    Wenhui Liao

  3. Department of Electrical, Computer and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY

    Qiang Ji

Authors
  1. Yan Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenhui Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiang Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute of Automation, Chinese Academy of Sciences, 95 Zhongguancun East Road, Haidian, Beijing, 100080, P.R. China

    Jianhua Tao  & Tieniu Tan BSc, MSc, PhD  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag London Limited

About this chapter

Cite this chapter

Tong, Y., Liao, W., Ji, Q. (2009). Automatic Facial Action Unit Recognition by Modeling Their Semantic And Dynamic Relationships. In: Tao, J., Tan, T. (eds) Affective Information Processing. Springer, London. https://doi.org/10.1007/978-1-84800-306-4_10

Download citation

  • DOI: https://doi.org/10.1007/978-1-84800-306-4_10

  • Publisher Name: Springer, London

  • Print ISBN: 978-1-84800-305-7

  • Online ISBN: 978-1-84800-306-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation