Skip to main content
SpringerLink
Log in

Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics

  • Published:
Frontiers of Information Technology & Electronic Engineering Aims and scope Submit manuscript

Abstract

As a typical biometric cue with great diversities, smile is a fairly influential signal in social interaction, which reveals the emotional feeling and inner state of a person. Spontaneous and posed smiles initiated by different brain systems have differences in both morphology and dynamics. Distinguishing the two types of smiles remains challenging as discriminative subtle changes need to be captured, which are also uneasily observed by human eyes. Most previous related works about spontaneous versus posed smile recognition concentrate on extracting geometric features while appearance features are not fully used, leading to the loss of texture information. In this paper, we propose a region-specific texture descriptor to represent local pattern changes of different facial regions and compensate for limitations of geometric features. The temporal phase of each facial region is divided by calculating the intensity of the corresponding facial region rather than the intensity of only the mouth region. A mid-level fusion strategy of support vector machine is employed to combine the two feature types. Experimental results show that both our proposed appearance representation and its combination with geometry-based facial dynamics achieve favorable performances on four baseline databases: BBC, SPOS, MMI, and UvA-NEMO.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Ambadar, Z., Cohn, J.F., Ian Reed, L., 2009. All smiles are not created equal: morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. J. Nonverb. Behav., 33(1): 17–34. https://doi.org/10.1007/s10919-008-0059-5

    Article  Google Scholar 

  • Baron-Cohen, S., Ring, H.A., Bullmore, E.T., et al., 2000. The amygdala theory of autism. Neurosci. Biobehav. Rev., 24(3): 355–364. https://doi.org/10.1016/S0149-7634(00)00011-7

    Article  Google Scholar 

  • Burgos-Artizzu, X.P., Perona, P., Dollár, P., 2013. Robust face landmark estimation under occlusion. Proc. IEEE Int. Conf. on Computer Vision, p.1513–1520. https://doi.org/10.1109/ICCV.2013.191

    Google Scholar 

  • Calvo, M.G., Nummenmaa, L., 2011. Time course of discrimination between emotional facial expressions: the role of visual saliency. Vis. Res., 51(15): 1751–1759. https://doi.org/10.1016/j.visres.2011.06.001

    Article  Google Scholar 

  • Calvo, M.G., Gutiérrez-García, A., Avero, P., et al., 2013. Attentional mechanisms in judging genuine and fake smiles: eye-movement patterns. Emotion, 13(4): 792–802. http://dx.doi.org/10.1037/a0032317

    Article  Google Scholar 

  • Cao, X., Wei, Y., Wen, F., et al., 2014. Face alignment by explicit shape regression. Int. J. Comput. Vis., 107(2): 177–190. https://doi.org/10.1007/s11263-013-0667-3

    Article  MathSciNet  Google Scholar 

  • Cohn, J.F., Schmidt, K.L., 2004. The timing of facial motion in posed and spontaneous smiles. Int. J. Wavel. Multiresol. Inform. Process., 2(2): 121–132. https://doi.org/10.1142/S021969130400041X

    Article  Google Scholar 

  • Cootes, T.F., Edwards, G.J., Taylor, C.J., 2001. Active appearance models. IEEE Trans. Patt. Anal. Mach. Intell., 23(6): 681–685. https://doi.org/10.1109/34.927467

    Article  Google Scholar 

  • Dalal, N., Triggs, B., 2005. Histograms of oriented gradients for human detection. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.886–893. https://doi.org/10.1109/CVPR.2005.177

    Google Scholar 

  • Dibeklioğlu, H., Valenti, R., Salah, A.A., et al., 2010. Eyes do not lie: spontaneous versus posed smiles. Proc. Int. Conf. on Multimedia, p.703–706. https://doi.org/10.1145/1873951.1874056

    Google Scholar 

  • Dibeklioğlu, H., Salah, A., Gevers, T., 2012. Are you really smiling at me? Spontaneous versus posed enjoyment smiles. Proc. European Conf. on Computer Vision, p.525–538. https://doi.org/10.1007/978-3-642-33712-3_38

    Google Scholar 

  • Dibeklioğlu, H., Salah, A., Gevers, T., 2015. Recognition of genuine smiles. IEEE Trans. Multimed., 17(3): 279–294. https://doi.org/10.1109/TMM.2015.2394777

    Article  Google Scholar 

  • Dollár, P., Welinder, P., Perona, P., 2010. Cascaded pose regression. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1078–1085. https://doi.org/10.1109/CVPR.2010.5540094

    Google Scholar 

  • Ekman, P., 2009. Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W. W. Norton & Company, New York, p.140–143.

    Google Scholar 

  • Ekman, P., Rosenberg, E.L., 1997. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press.

    Google Scholar 

  • Frank, M.G., Ekman, P., 1993. Not all smiles are created equal: the differences between enjoyment and nonenjoyment smiles. Humor, 6(1): 9–26. https://doi.org/10.1515/humr.1993.6.1.9

    Article  Google Scholar 

  • Hoque, M., McDuff, D., Picard, R., 2012. Exploring temporal patterns in classifying frustrated and delighted smiles. IEEE Trans. Affect. Comput., 3(3): 323–334. https://doi.org/10.1109/T-AFFC.2012.11

    Article  Google Scholar 

  • Khokher, M.R., Bouzerdoum, A., Phung, S.L., 2014. Crowd behavior recognition using dense trajectories. Proc. Int. Conf. on Digital lmage Computing: Techniques and Applications, p.1–7. https://doi.org/10.1109/DICTA.2014.7008098

    Google Scholar 

  • Le, V., Brandt, J., Lin, Z., et al., 2012. Interactive facial feature localization. Proc. European Conf. on Computer Vision, p.679–692. https://doi.org/10.1007/978-3-642-33712-3_49

    Google Scholar 

  • Li, W.S., Zhou, C.L., Xu, J.T., 2005. A novel face recognition method with feature combination. J. Zhejiang Univ.-Sci., 6A(5):454–459. https://doi.org/10.1631/jzus.2005.A0454

    Google Scholar 

  • Liu, H., Sun, X., 2016. A partial least squares based ranker for fast and accurate age estimation. Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, p.2792–2796. https://doi.org/10.1109/ICASSP.2016.7472186

    Google Scholar 

  • Liu, H., Wu, P., 2012. Comparison of methods for smile deceit detection by training AU6 and AU12 simultaneously. Proc. IEEE Int. Conf. on Image Processing, p.1805–1808. https://doi.org/10.1109/ICIP.2012.6467232

    Google Scholar 

  • Liu, H., Gao, Y., Wang, C., 2014. Gender identification in unconstrained scenarios using self-similarity of gradients features. Proc. IEEE Int. Conf. on Image Processing, p.5911–5915. https://doi.org/10.1109/ICIP.2014.7026194

    Google Scholar 

  • Miehlke, A., Fisch, U., Eneroth, C.M., 1973. Surgery of the Facial Nerve. Saunders, Philadelphia.

    Google Scholar 

  • Ojala, T., Pietikäinen, M., Mäenpää, T., 2002. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Patt. Anal. Mach. Intell., 24(7): 971–987. https://doi.org/10.1109/TPAMI.2002.1017623

    Article  MATH  Google Scholar 

  • Peng, H., Long, F., Ding, C., 2005. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Patt. Anal. Mach. Intell., 27(8): 1226–1238. https://doi.org/10.1109/TPAMI.2005.159

    Article  Google Scholar 

  • Pfister, T., Li, X., Zhao, G., et al., 2011a. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. Proc. IEEE Int. Conf. on Computer Vision Workshops, p.868–875. https://doi.org/10.1109/ICCVW.2011.6130343

    Google Scholar 

  • Pfister, T., Li, X., Zhao, G., et al., 2011b. Recognising spontaneous facial micro-expressions. Proc. IEEE Int. Conf. on Computer Vision, p.1449–1456. https://doi.org/10.1109/ICCV.2011.6126401

    Google Scholar 

  • Rinn, W.E., 1984. The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. Psychol. Bull., 95(1): 52–77. https://doi.org/10.1037/0033-2909.95.1.52

    Article  Google Scholar 

  • Sariyanidi, E., Gunes, H., Cavallaro, A., 2015. Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans. Patt. Anal. Mach. Intell., 37(6): 1113–1133. https://doi.org/10.1109/TPAMI.2014.2366127

    Article  Google Scholar 

  • Shen, X.B., Wu, Q., Fu, X.L., 2012. Effects of the duration of expressions on the recognition of microexpressions. J. Zhejiang Univ.-Sci. B, 13(3): 221–230. http://dx.doi.org/10.1631/jzus.B1100063

    Article  Google Scholar 

  • Valstar, M., Pantic, M., 2010. Induced disgust, happiness and surprise: an addition to the MMI facial expression database. Proc. 3rd Int. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, p.65–70.

    Google Scholar 

  • Valstar, M.F., Gunes, H., Pantic, M., 2007. How to distinguish posed from spontaneous smiles using geometric features. Proc. Int. Conf. on Multimodal Interfaces, p.38–45. https://doi.org/10.1145/1322192.1322202

    Google Scholar 

  • Wang, J., Yang, J., Yu, K., et al., 2010. Locality-constrained linear coding for image classification. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.3360–3367. https://doi.org/10.1109/CVPR.2010.5540018

    Google Scholar 

  • Wang, X., Wang, L., Qiao, Y., 2012. A comparative study of encoding, pooling and normalization methods for action recognition. Proc. Asian Conf. on Computer Vision, p.572–585. https://doi.org/10.1007/978-3-642-37431-9_44

    Google Scholar 

  • Whitehill, J., Bartlett, M.S., Movellan, J.R., 2013. Automatic facial expression recognition. In: Gratch, J., Marsella, S. (Eds.), Social Emotions in Nature Artifact. Oxford Scholarship Online. https://doi.org/10.1093/acprof: oso/9780195387643.003.0007

    Google Scholar 

  • Wu, P.P., Liu, H., Zhang, X.W., 2014. Spontaneous versus posed smile recognition using discriminative local spatio-temporal descriptors. Proc. Int. IEEE Conf. on Acoustics, Speech and Signal Processing, p.1249–1253. https://doi.org/10.1109/ICASSP.2014.6853795

    Google Scholar 

  • Wu, Q., Shen, X.B., Fu, X.L., 2011. The machine knows what you are hiding: an automatic micro-expression recognition system. LNCS, 6975: 152–162. https://doi.org/10.1007/978-3-642-24571-8_16

    Google Scholar 

  • Yang, J.C., Yu, K., Gong, Y.H., et al., 2009. Linear spatial pyramid matching using sparse coding for image classification. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1794–1801. https://doi.org/10.1109/CVPR.2009.5206757

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MOE Key Laboratory of Machine Perception, Peking University, Beijing, 100871, China

    Ping-ping Wu & Hong Liu

  2. Engineering Lab on Intelligent Perception for Internet of Things, Peking University Shenzhen Graduate School, Shenzhen, 518055, China

    Ping-ping Wu, Hong Liu & Xue-wu Zhang

  3. Department of Computer Science, Christian-Albrechts University, Kiel, 24118, Germany

    Yuan Gao

Authors
  1. Ping-ping Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xue-wu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong Liu.

Additional information

Project supported by the National Natural Science Foundation of China (No. 60675025), the National High-Tech R&D Program (863) of China (No. 2006AA04Z247), the Scientific and Technical Innovation Commission of Shenzhen Municipality, China (Nos. JCYJ20130331144631730 and JCYJ20130331144716089), and the Specialized Research Fund for the Doctoral Program of Higher Education, China (No. 20130001110011)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Pp., Liu, H., Zhang, Xw. et al. Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics. Frontiers Inf Technol Electronic Eng 18, 955–967 (2017). https://doi.org/10.1631/FITEE.1600041

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1631/FITEE.1600041

Key words

CLC number

Search

Navigation