Skip to main content
SpringerLink
Log in

Dynamic Facial Expression Recognition Using Boosted Component-Based Spatiotemporal Features and Multi-classifier Fusion

  • Conference paper
Advanced Concepts for Intelligent Vision Systems (ACIVS 2010)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 6475))

Abstract

Feature extraction and representation are critical in facial expression recognition. The facial features can be extracted from either static images or dynamic image sequences. However, static images may not provide as much discriminative information as dynamic image sequences. On the other hand, from the feature extraction point of view, geometric features are often sensitive to the shape and resolution variations, whereas appearance based features may contain redundant information. In this paper, we propose a component-based facial expression recognition method by utilizing the spatiotemporal features extracted from dynamic image sequences, where the spatiotemporal features are extracted from facial areas centered at 38 detected fiducial interest points. Considering that not all features are important to the facial expression recognition, we use the AdaBoost algorithm to select the most discriminative features for expression recognition. Moreover, based on median rule, mean rule, and product rule of the classifier fusion strategy, we also present a framework for multi-classifier fusion to improve the expression classification accuracy. Experimental studies conducted on the Cohn-Kanade database show that our approach that combines both boosted component-based spatiotemporal features and multi-classifier fusion strategy provides a better performance for expression recognition compared with earlier approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Fasel, B., Luettin, J.: Automatic facial expression analysis: a survey. Pattern Recognition 36, 259–275 (2003)

    Article  MATH  Google Scholar 

  2. Zeng, Z., Pantic, M., Roisman, G., Huang, T.: A survey of affective recognition methods: Audio, visual and spontaneous expression. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(1), 39–58 (2009)

    Article  Google Scholar 

  3. Tian, T., Kanade, T., Cohn, J.: Facial expression analysis. In: Li, S., Jain, A.K. (eds.) Handbook of Face Recognition. Spinger, Heidelberg (2004)

    Google Scholar 

  4. Tian, Y., Kanade, T., Cohn, J.: Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2), 97–115 (1999)

    Article  Google Scholar 

  5. Kobayashi, H., Hara, F.: Facial interaction between animated 3D face robot and human being. In: Systems, Man and Cybernetics, pp. 3732–3737. IEEE Press, New York (1997)

    Google Scholar 

  6. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Automatic Face and Gesture Recognition, pp. 200–205. IEEE Press, New York (1998)

    Google Scholar 

  7. Yaser, Y., Larry, S.: Recognizing human facial expressions from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(6), 636–642 (1996)

    Article  Google Scholar 

  8. Chen, F., Kotani, K.: Facial expression recognition by supervised independent component analysis using MAP estimation. IEICE - Transactions on Information and Systems 2, 341–350 (2008)

    Article  Google Scholar 

  9. Penev, P., Atick, J.: Local feature analysis: a general statistical theory for object representation. Network: Computation in Neural Systems 7(3), 477–500 (1996)

    Article  MATH  Google Scholar 

  10. Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing 16(1), 172–187 (2007)

    Article  MathSciNet  Google Scholar 

  11. Donato, G., Bartlett, M., Hager, J., Ekman, P., Sejnowski, T.: Classifing facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(10), 974–989 (1999)

    Article  Google Scholar 

  12. Feng, X., Pietikäinen, M., Hadid, A.: Facial expression recognition based on local binary patterns and linear programming. Pattern Recognition and Image Analysis 15(2), 546–548 (2005)

    Google Scholar 

  13. Lanitis, A., Taylor, C., Cootes, T.: Automatic interpretation and coding of face images using flexible models. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7), 743–756 (1997)

    Article  Google Scholar 

  14. Yesin, M., Bullot, B., Sharma, R.: From facial expression to level of interest: a spatio-temporal approach. In: Computer Vision and Pattern Recognition, pp. 922–927. IEEE Press, New York (2004)

    Google Scholar 

  15. Heisele, B., Koshizen, B.: Components for face recognition. In: Automatic Face and Gesture Recognition, pp. 153–158. IEEE Press, New York (2004)

    Google Scholar 

  16. Ivanov, I., Heisele, B., Serre, T.: Using component features for face recognition. In: Automatic Face and Gesture Recognition, pp. 421–426. IEEE Press, New York (2004)

    Google Scholar 

  17. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.G.: Active shape models - their training and application. Computer Vision and Image Understanding 61, 38–59 (1995)

    Article  Google Scholar 

  18. Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to Boosting. In: Computational Learning Theory: Eurocolt 1995, pp. 23–37 (1995)

    Google Scholar 

  19. Kittler, J., Hatef, M., Duin, R.P.W., Matas, J.: On combining classifiers. IEEE Transaction on Pattern Analysis and Machine Intelligence 20(3), 226–239 (1998)

    Article  Google Scholar 

  20. David, M.J., Martijin, B.V., Robert, P.W.D., Josef, K.: Combining multiple classifiers by averaging or by multiplying? Pattern Recognition 33, 1475–1485 (2000)

    Article  Google Scholar 

  21. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)

    Article  Google Scholar 

  22. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Automatic Face Recognition and Gesture Recognition, pp. 46–53. IEEE Press, New York (2000)

    Google Scholar 

  23. Pantic, M., Patras, I.: Dynamic of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics-Part B. 36(2), 433–449 (2006)

    Article  Google Scholar 

  24. Zhang, Z., Lyons, M., Schuster, M., Akamatsu, S.: Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: International Workshop on Automatic Face and Gesture Recognition, pp. 454–459. IEEE Press, New York (1998)

    Google Scholar 

  25. Milborrow, S., Nicolls, F.: Locating facial features with extended active shape model. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 504–513. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  26. Serrano, A., Diego, I.M., Conde, C., Cabello, E.: Influence of wavelet frequency and orientation in an SVM-based parallel Gabor PCA face verification system. In: Yin, H., Tino, P., Corchado, E., Byrne, W., Yao, X. (eds.) IDEAL 2007. LNCS, vol. 4881, pp. 219–228. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  27. MMI Database, http://www.mmifacedb.com

  28. Zhao, G., Pietikäinen, M.: Dynamic texture recognition using local binary pattern with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6), 915–928 (2007)

    Article  Google Scholar 

  29. Shan, C., Gong, S., McOwan, P.W.: Robust facial expression recognition using local binary patterns. In: Image Processing, pp. 370–373. IEEE Press, New York (2005)

    Google Scholar 

  30. Aleksic, S., Katsaggelos, K.: Automatic facial expression recognition using facial animation parameters and multi-stream HMMS. IEEE Transactions on Information Forensics and Security 1(1), 3–11 (2006)

    Article  Google Scholar 

  31. Littlewort, G., Bartlett, M., Fasel, I., Susskind, J., Movellan, J.: Dynamics of facial expression extracted automatically from video. In: IEEE Workshop Face Processing in Video. IEEE Press, New York (2004)

    Google Scholar 

  32. Zhao, G., Pietikäinen, M.: Boosted multi-resolution spatio temporal descriptors for facial expression recognition. Pattern Recognition Letters 30, 1117–1127 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Machine Vision Group, Department of Electrical and Information Engineering, University of Oulu, Finland

    Xiaohua Huang, Guoying Zhao & Matti Pietikäinen

  2. Research Center for Learning Science, Southeast University, China

    Xiaohua Huang & Wenming Zheng

Authors
  1. Xiaohua Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guoying Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matti Pietikäinen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenming Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. DGA/D4S/MRIS, CEP/GIP, 16 bis avenue Prieur de la côte d’or, 94114, Arcueil, France

    Jacques Blanc-Talon

  2. Canon Information Systems Research Australia, Sydney, Australia

    Don Bone

  3. Telecommunications and Information processing (TELIN), Ghent University, St.-Pietersnieuwstraat 41, B9000, Ghent, Belgium

    Wilfried Philips

  4. CSIRO ICT Centre, ICT Centre, Epping, NSW 1710, Po Box 76, 1710, Sydney, Australia

    Dan Popescu

  5. Physics, University of Antwerp, Universiteitsplein 1; Building N., 2610, Wilrijk, Belgium

    Paul Scheunders

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Huang, X., Zhao, G., Pietikäinen, M., Zheng, W. (2010). Dynamic Facial Expression Recognition Using Boosted Component-Based Spatiotemporal Features and Multi-classifier Fusion. In: Blanc-Talon, J., Bone, D., Philips, W., Popescu, D., Scheunders, P. (eds) Advanced Concepts for Intelligent Vision Systems. ACIVS 2010. Lecture Notes in Computer Science, vol 6475. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17691-3_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-17691-3_29

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-17690-6

  • Online ISBN: 978-3-642-17691-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation