Skip to main content
SpringerLink
Log in

Posed and spontaneous expression recognition through modeling their spatial patterns

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

This paper presents a new method to recognize posed and spontaneous expressions through modeling their spatial patterns. Gender and expression categories are employed as privileged information to further improve the recognition. The proposed approach includes three steps. First, geometric features about facial shape and Action Unit variations are extracted from the differences between apex and onset facial images to capture the spatial facial variation. Second, statistical hypothesis testings are conducted to explore the differences between posed and spontaneous expressions using the defined geometric features from three aspects: all samples, samples given the gender information, and samples given expression categories. Third, several Bayesian networks are built to capture posed and spontaneous spatial facial patterns respectively given gender and expression categories. The statistical analysis results on the USTC-NVIE and SPOS databases both demonstrate the effectiveness of the proposed geometric features. The recognition results on the USTC-NVIE database indicate that the privileged information of gender and expression can help model the spatial patterns caused by posed and spontaneous expressions. The recognition results on both databases outperform those of the state of the art.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Zhang, L., Tjondronegoro, D., Chandran, V.: Geometry vs. appearance for discriminating between posed and spontaneous emotions. In: Neural Information Processing, pp. 431–440. Springer, Berlin (2011)

  2. Pfister, T., Li, X., Zhao, G., Pietikainen, M.: Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 868–875. IEEE (2011)

  3. Lithari, C., Frantzidis, C.A., Papadelis, C., Vivas, A.B., Klados, M.A., Kourtidou-Papadeli, C., Pappas, C., Ioannides, A.A., Bamidis, P.D.: Are females more responsive to emotional stimuli? a neurophysiological study across arousal and valence dimensions. Brain Topogr. 23(1), 27–40 (2010)

    Article  Google Scholar 

  4. Yunus, S., Christopher, T.: Cascaded classification of gender and facial expression using active appearance models. In: IEEE international conference on automatic face and gesture recognition, pp. 393–400 (2006)

  5. Vapnik, V., Vashist, A.: A new learning paradigm: learning using privileged information. Neural Netw. Off. J. Int. Neural Netw. Soc. 22(5–6), 544 (2009)

    Article  Google Scholar 

  6. Chen, J., Liu, X., Lyu, S.: Boosting with side information. In: Computer vision-ACCV 2012, pp. 563–577. Springer, Berlin (2013)

  7. Seckington, M.: Using dynamic bayesian networks for posed versus spontaneous facial expression recognition. Mater Thesis, Department of Computer Science, Delft University of Technology (2011)

  8. Ekman, P., Hager, J.C., Friesen, W.V.: The symmetry of emotional and deliberate facial actions. Psychophysiology 18(2), 101–106 (1981)

  9. Ekman, P., Rosenberg, E.L.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, USA (1997)

    Google Scholar 

  10. Ekman, P., Friesen, W.V.: Felt, false, and miserable smiles. J. Nonverbal Behav. 6(4), 238–252 (1982)

    Article  Google Scholar 

  11. Ekman, P.: Darwin, deception, and facial expression. Ann. N. Y. Acad. Sci. 1000(1), 205–221 (2003)

    Article  Google Scholar 

  12. Cohn, J.F., Schmidt, K.L.: The timing of facial motion in posed and spontaneous smiles. Int. J. Wavelets Multiresolut. Inform. Process. 2(02), 121–132 (2004)

    Article  Google Scholar 

  13. Valstar, M.F., Gunes, H., Pantic, M.: How to distinguish posed from spontaneous smiles using geometric features. In: Proceedings of the 9th international conference on multimodal interfaces, pp. 38–45. ACM, Nagoya (2007)

  14. Valstar, M.F., Pantic, M., Ambadar, Z., Cohn, J.F.: Spontaneous vs. posed facial behavior: automatic analysis of brow actions. In: Proceedings of the 8th international conference on multimodal interfaces, pp. 162–170. ACM, New York (2006)

  15. Littlewort, G.C., Bartlett, M.S., Lee, K.: Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis. Comput. 27(12), 1797–1803 (2009)

    Article  Google Scholar 

  16. Dibeklioğlu, H., Salah, A.A., Gevers, T.: Are you really smiling at me? spontaneous versus posed enjoyment smiles. In: Computer vision-ECCV 2012, pp. 525–538. Springer, Berlin (2012)

  17. Tong, Y., Wang, Y., Zhu, Z., Ji, Q.: Robust facial feature tracking under varying face pose and facial expression. Pattern Recognit. 40(11), 3195–3208 (2007)

    Article  MATH  Google Scholar 

  18. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process. 29(6), 1153–1160 (1981)

  19. Mitchell, D.P., Netravali, A.N.: Reconstruction filters in computer-graphics. In: ACM Siggraph Computer Graphics, vol. 22, pp. 221–228. ACM, Newyork(1988)

  20. He, M., Wang, S., Liu, Z., Chen, X.: Analyses of the differences between posed and spontaneous facial expressions. In: 2013 Humaine association conference on affective computing and intelligent interaction (ACII), pp. 79–84, Sept 2013

  21. Zhang, Y., Ji, Q.: Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 699–714 (2005)

  22. Wilcoxon, F.: Individual comparisons by ranking methods. Biom. Bull. 1(6), 80–83 (1945)

    Article  Google Scholar 

  23. Goodman, S.N.: Toward evidence-based medical statistics. 1: The p value fallacy. Ann. Intern. Med. 130(12), 995–1004 (1999)

    Article  Google Scholar 

  24. Lowry, R.: Concepts and Applications of Inferential Statistics. R. Lowry, Poughkeepsie (1998)

  25. Cooper, G.F., Herskovits, E.: A bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9, 309–347 (1992)

    MATH  Google Scholar 

  26. Murphy, K.: How to use bayes net toolbox. Avaliable: https://code.google.com/p/bnt/ (2004)

  27. Paul, E.: Bbc-dataset. http://www.bbc.co.uk/science/humanbody/mind/surveys/smiles/

  28. Petridis, S., Martinez, B., Pantic, M.: The mahnob laughter database. Image Vis. Comput. 31(2), 186–202 (2013)

  29. Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., Wang, X.: A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans. Multimed. 12(7), 682–691 (2010)

    Article  Google Scholar 

  30. Schmidt, K.L., Ambadar, Z., Cohn, J.F., Reed, L.I.: Movement differences between deliberate and spontaneous facial expressions: Zygomaticus major action in smiling. J. Nonverbal Behav. 30(1), 37–52 (2006)

    Article  Google Scholar 

Download references

Acknowledgments

This paper is supported by the NSFC (61175037, 61228304, 61473270), Special Innovation Project on Speech of Anhui Province (11010202192), and Project from Anhui Science and Technology Agency (1106c0805008).

Author information

Authors and Affiliations

  1. Key Lab of Computing and Communication Software of Anhui Province, School of Computer Science and Technology, University of Science and Technology of China, Hefei, 230027, Anhui, People’s Republic of China

    Shangfei Wang, Chongliang Wu, Menghua He & Jun Wang

  2. Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY, 12180, USA

    Qiang Ji

Authors
  1. Shangfei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chongliang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Menghua He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiang Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shangfei Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, S., Wu, C., He, M. et al. Posed and spontaneous expression recognition through modeling their spatial patterns. Machine Vision and Applications 26, 219–231 (2015). https://doi.org/10.1007/s00138-015-0657-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-015-0657-2

Keywords

Search

Navigation