Skip to main content
SpringerLink
Log in

Towards the Usage of Optical Flow Temporal Features for Facial Expression Classification

  • Conference paper
Advances in Visual Computing (ISVC 2012)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 7432))

Included in the following conference series:

  • 2811 Accesses

Abstract

Psychological evidence suggests that the human ability to recognize facial expression improves with the addition of temporal stimuli. While the facial action coding community has largely migrated towards temporal information, the facial expression recognition community has been slow to utilize facial dynamics. This paper contrasts the contributions of static vs. temporal features, including both dense and sparse facial tracking methodologies in combination with sparse representation classification. The temporal methods of facial feature point tracking, motion history images, free form deformation, and SIFT flow are adapted for facial expression classification. Dense optical flow for facial expression recognition is successfully utilized. We show that when used in isolation, the best temporal methods are just as good as static methods. However, when fusing temporal dynamics with static imagery significant increases in facial expression classification are achieved.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Lew, M., Bakker, E.M., Sebe, N., Huang, T.S.: Human-Computer Intelligent Interaction: A Survey. In: Lew, M., Sebe, N., Huang, T.S., Bakker, E.M. (eds.) HCI 2007. LNCS, vol. 4796, pp. 1–5. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  2. Pantic, M., Pentland, A., Nijholt, A., Huang, T.S.: Human computing and machine understanding of human behavior: a survey. In: ICMI 2006 and IJCAI 2007 International Workshops, Artifical Intelligence for Human Computing, Berlin, Germany (2007)

    Google Scholar 

  3. Pantic, M., Rothkrantz, L.U.M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1424–1445 (2000)

    Article  Google Scholar 

  4. Zhihong, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 39–58 (2009)

    Article  Google Scholar 

  5. Shuai-Shi, L., Yan-Tao, T., Dong, L.: New research advances of facial expression recognition. In: Eighth International Conference on Machine Learning and Cybernetics (2009)

    Google Scholar 

  6. Martin, C., Werner, U., Gross, H.M.: A real-time facial expression recognition system based on active appearance models using gray images and edge images. In: 8th IEEE International Conference on Automatic Face & Gesture Recognition (2008)

    Google Scholar 

  7. Buciu, I., Kotropoulos, C., Pitas, I.: ICA and Gabor representation for facial expression recognition. In: Proceedings of International Conference on Image Processing (2003)

    Google Scholar 

  8. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on Local Binary Patterns: A comprehensive study. Image and Vision Computing 27, 803–816 (2009)

    Article  Google Scholar 

  9. Zafeiriou, S., Petrou, M.: Sparse representations for facial expressions recognition via l1 optimization. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010, San Francisco, CA (2010)

    Google Scholar 

  10. Ekman, P., Friesen, W.V.: The Facial Action Coding System. Consulting Psychologists Press, Inc., San Francisco (1978)

    Google Scholar 

  11. Koelstra, S., Pantic, M., Patras, I.: A dynamic texture-based approach to recognition of facial actions and their temporal models. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 1940–1954 (2010)

    Article  Google Scholar 

  12. Curio, C., Bulthoff, H., Giese, M.: Dynamic Faces: Insights from Experiments and Computation, 1st edn. The MIT Press (2010)

    Google Scholar 

  13. Jiang, B., Valstar, M.F., Pantic, M.: Action Unit Detection Using Sparse Appearance Descriptors in Space-Time Video Volumes. In: Face and Gesture Recognition Face and Gesture Recognition, Santa Barbara, CA (2011)

    Google Scholar 

  14. Lucey, P., Cohn, J., Matthews, I., Prkachin, K., Solomon, P.: Painful Data: The UNBC-McMaster Shoulder Pain Expression Archive Database. In: Face and Gesture Recognition, Santa Barbara, CA (2011)

    Google Scholar 

  15. Bhaskaran, N., Nwogu, I., Frank, M., Govindaraju, V.: Lie to Me: Deceit Detection via Online Behavioral Learning. In: Face and Gesture Recognition, Santa Barbara, CA (2011)

    Google Scholar 

  16. Cohn, J.F., et al.: Detecting depression from facial actions and vocal prosody. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, ACII 2009 (2009)

    Google Scholar 

  17. Saragih, J., Lucey, S., Cohn, J.: Real-time Avatar Animation from a Single Image. In: Face and Gesture Recognition, Santa Barbara, CA (2011)

    Google Scholar 

  18. Bobick, A.F., Davis, J.W.: The recognition of human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 257–267 (2001)

    Article  Google Scholar 

  19. Ce, L., Yuen, J., Torralba, A.: SIFT flow: dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 978–994 (2011)

    Article  Google Scholar 

  20. Songfan, Y., Bhanu, B.: Facial expression recognition using emotion avatar image. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (2011)

    Google Scholar 

  21. Ptucha, R., Tsagkatakis, G., Savakis, A.: Manifold Based Sparse Representation for Robust Expression Recognition without Neutral Subtraction. In: Presented at the BeFIT 2011 Workshop, International Conference on Computer Vision, Barcelona, Spain (2011)

    Google Scholar 

  22. Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Research 37, 3311–3325 (1997)

    Article  Google Scholar 

  23. Olshausen, B.A., Field, D.J.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996)

    Article  Google Scholar 

  24. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Yi, M.: Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 210–227 (2009)

    Article  Google Scholar 

  25. He, X., Niyogi, P.: Locality Preserving Projections. In: Advances in Neural Information Processing Systems, Vancouver, Canada, vol. 16 (2003)

    Google Scholar 

  26. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models - their training and application. Computer Vision and Image Understanding 61, 38–59 (1995)

    Article  Google Scholar 

  27. Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing 16, 172–187 (2007)

    Article  MathSciNet  Google Scholar 

  28. Rueckert, D., Sonoda, L.I., Hayes, C., Hill, D.L.G., Leach, M.O., Hawkes, D.J.: Nonrigid registration using free-form deformations: Application to breast MR images. IEEE Transactions on Medical Imaging 18, 712–721 (1999)

    Article  Google Scholar 

  29. Valstar, M., Jiang, B., Mehu, M., Pantic, M., Scherer, K.R.: The First Facial Expression Recognition and Analysis Challenge. In: Face and Gesture Recognition, Santa Barbara, CA (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computing and Information Sciences and Computer Engineering, Rochester Institute of Technology, Rochester, USA, NY

    Raymond Ptucha & Andreas Savakis

Authors
  1. Raymond Ptucha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Savakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of Nevada, 89557, Reno, NV, USA

    George Bebis

  2. NASA Ames Research Center, 94035, Moffett Field, CA, USA

    Richard Boyle

  3. Lawrence Berkeley National Laboratory, 94720, Berkeley, CA, USA

    Bahram Parvin

  4. Desert Research Institute, 89512, Reno, NV, USA

    Darko Koracin

  5. Department of Computer Science, University of California at Irvine, 92697-3435, Irvine, CA, USA

    Charless Fowlkes

  6. Eastman Kodak Company, 14650-2102, Rochester, NY, USA

    Sen Wang

  7. Department of Computer Science and Engineering, University of Colorado at Denver, 80217, Denver, CO, USA

    Min-Hyung Choi

  8. VRVis Zentrum für Virtual Reality and Visualisierung, 1220, Vienna, Austria

    Stephan Mantler

  9. California Institute for Telecommunications and Information Technology, University of California,,, San Diego, 92093, La Jolla, CA, USA

    Jürgen Schulze

  10. KAUST Visualizatioin Core Lab., 23955-6900, Thurwal, Saudi Arabia

    Daniel Acevedo

  11. Stony Brook University, 11794-4400, NY, USA

    Klaus Mueller

  12. Argonne National Laboratory, 60439, IL, USA

    Michael Papka

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Ptucha, R., Savakis, A. (2012). Towards the Usage of Optical Flow Temporal Features for Facial Expression Classification. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2012. Lecture Notes in Computer Science, vol 7432. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33191-6_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-33191-6_38

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-33190-9

  • Online ISBN: 978-3-642-33191-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation