research-article

Posed and Spontaneous Expression Distinction Using Latent Regression Bayesian Networks

Authors:

Qiang JiAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 16, Issue 3

Article No.: 80, Pages 1 - 18

https://doi.org/10.1145/3391290

Published: 14 July 2020 Publication History

Abstract

Facial spatial patterns can help distinguish between posed and spontaneous expressions, but this information has not been thoroughly leveraged by current studies. We present several latent regression Bayesian networks (LRBNs) to capture the patterns existing in facial landmark points and to use those points to differentiate posed from spontaneous expressions. The visible nodes of the LRBN represent facial landmark points. Through learning, the LRBN captures the probabilistic dependencies among landmark points as well as latent variables given observations, successfully modeling the spatial patterns inherent in expressions. Current methods tend to ignore gender and expression categories, although these factors can influence spatial patterns. Therefore, we propose to incorporate this as a kind of privileged information. We construct several LRBNs to capture spatial patterns from spontaneous and posed facial expressions given expression-related factors. Facial landmark points are used during testing to classify samples as either posed or spontaneous, depending on which LRBN has the largest likelihood. We conduct experiments to showcase the superiority of the proposed approach in both modeling spatial patterns and classifying expressions as either posed or spontaneous.

References

[1]

Hervé Abdi and L. J. Williams. 2010. Normalizing data. In Encyclopedia of Research Design. 935--938.

[2]

Elisabeth Andre. 2013. Exploiting unconscious user signals in multimodal human-computer interaction. ACM Trans. Multimedia Comput. Commun. Appl. 9, 1s (2013), 48.

Digital Library

[3]

Yoshua Bengio, Li Yao, and Kyunghyun Cho. 2014. Bounding the test log-likelihood of generative models. In Proceedings of the International Conference on Learning Representations (Conference Track).

[4]

George A. Bonanno and Dacher Keltner. 1997. Facial expressions of emotion and the course of conjugal bereavement. J. Abnorm. Psychol. 106, 1 (1997), 126.

[5]

J. F. Cohn and K. L. Schmidt. 2004. The timing of facial motion in posed and spontaneous smiles. Int. J. Wavelets Multires. Inf. Process. 2, 02 (2004), 121--132.

[6]

H. Dibeklioğlu, A. Salah, and T. Gevers. 2012. Are you really smiling at me? Spontaneous versus posed enjoyment smiles. In Proceedings of the European Conference on Computer Vision (ECCV’12). Springer, 525--538.

[7]

Hamdi Dibeklioglu, Roberto Valenti, Albert Ali Salah, and Theo Gevers. 2010. Eyes do not lie: Spontaneous versus posed smiles. In Proceedings of the International Conference on Multimedia. ACM, 703--706.

Digital Library

[8]

Paul Ekman. 2003. Darwin, deception, and facial expression. Ann. N. Y. Acad. Sci. 1000, 1 (2003), 205--221.

[9]

Paul Ekman and Wallace V. Friesen. 1982. Felt, false, and miserable smiles. J. Nonverb. Behav. 6, 4 (1982), 238--252.

[10]

Paul Ekman, Joseph C. Hager, and Wallace V. Friesen. 1981. The symmetry of emotional and deliberate facial actions. Psychophysiology 18, 2 (1981), 101--106.

[11]

Byron N. Fujita, Robert G. Harper, and Arthur N. Wiens. 1980. Encoding-decoding of nonverbal emotional messages: Sex differences in spontaneous and enacted expressions. J. Nonverb. Behav. 4, 3 (1980), 131--145.

[12]

Quan Gan, Siqi Nie, Shangfei Wang, and Qiang Ji. 2017. Differentiating between posed and spontaneous expressions with latent regression Bayesian network. In Proceedings of the Annual Cconference on Artifical Intelligence (AAAI’17). 4039--4045.

[13]

Zhe Gan, Ricardo Henao, David Carlson, and Lawrence Carin. 2015. Learning deep sigmoid belief networks with data augmentation. In Proceedings of the International Conference on Artificial Intelligence and Statistics (2015).

[14]

Karol Gregor, Andriy Mnih, and Daan Wierstra. 2014. Deep AutoRegressive networks. In Proceedings of the 31st International Conference on Machine Learning (2014).

Digital Library

[15]

Geoffrey Hinton. 2010. A practical guide to training restricted Boltzmann machines. Momentum 9, 1 (2010), 926.

[16]

Geoffrey Hinton and Ruslan Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507.

[17]

Geoffrey E. Hinton, Peter Dayan, Brendan J. Frey, and Radford M. Neal. 1995. The “wake-sleep” algorithm for unsupervised neural networks. Science 268, 5214 (1995), 1158--1161.

[18]

Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR’14).

[19]

C. Lithari, C. A. Frantzidis, C. Papadelis, A. B. Vivas, M. A. Klados, C. Kourtidou-Papadeli, C. Pappas, A. A. Ioannides, and P. D. Bamidis. 2010. Are females more responsive to emotional stimuli? A neurophysiological study across arousal and valence dimensions. Brain Topogr. 23, 1 (2010), 27--40.

[20]

G. C. Littlewort, M. S. Bartlett, and K. Lee. 2009. Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis. Comput. 27, 12 (2009), 1797--1803.

Digital Library

[21]

Mohammad Mavadati, Peyten Sanger, and Mohammad H. Mahoor. 2016. Extended DISFA dataset: Investigating posed and spontaneous facial expressions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1--8.

[22]

Andriy Mnih and Karol Gregor. 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on Machine Learning (2014).

[23]

Shushi Namba, Shoko Makihara, Russell S. Kabir, Makoto Miyatani, and Takashi Nakao. 2017. Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Curr. Psychol. 36, 3 (2017), 593--605.

[24]

Siqi Nie, Yue Zhao, and Qiang Ji. 2016. Latent regression Bayesian network for data representation. In Proceedings of the 23rd International Conference on Pattern Recognition (ICPR’16). IEEE, 3494--3499.

[25]

E. Paul. [n.d.]. BBC-Dataset. Retrieved from http://www.bbc.co.uk/science/humanbody/mind/surveys/smiles/.

[26]

S. Petridis, B. Martinez, and M. Pantic. 2013. The MAHNOB laughter database. Image Vis. Comput. 31, 2 (2013), 186--202.

Digital Library

[27]

T. Pfister, X. Li, G. Zhao, and M. Pietikainen. 2011. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops’11). IEEE, 868--875.

[28]

Tomas Pfister, Xiaobai Li, Guoying Zhao, and Matti Pietikäinen. 2011. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops’11). IEEE, 868--875.

[29]

Danilo J Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning (ICML’14). 1278--1286.

[30]

Fabien Ringeval, Björn Schuller, Michel Valstar, Jonathan Gratch, Roddy Cowie, and Maja Pantic. 2018. Introduction to the special section on multimedia computing and applications of socio-affective behaviors in the wild. ACM Trans. Multimedia Comput. Commun. Appl. 14, 1s (2018), 25.

Digital Library

[31]

Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. Ann. Math. Stat. 22, 3 (1951), 400--407.

[32]

Lawrence K. Saul, Tommi Jaakkola, and Michael I. Jordan. 1996. Mean field theory for sigmoid belief networks. J. Artif. Intell. Res. 4, 61 (1996), 76.

[33]

M. Seckington. 2011. Using dynamic Bayesian networks for posed versus spontaneous facial expression recognition. Master’s Thesis, Department of Computer Science, Delft University of Technology (2011).

[34]

M. F. Valstar, M. Pantic, Z. Ambadar, and J. F. Cohn. 2006. Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. In Proceedings of the 8th International Conference on Multimodal Interfaces. ACM, 162--170.

[35]

V. Vapnik and A. Vashist. 2009. A new learning paradigm: Learning using privileged information. Neur. Netw. 22, 5–6 (2009), 544.

Digital Library

[36]

Shangfei Wang, Longfei Hao, and Qiang Ji. 2019. Facial action unit recognition and intensity estimation enhanced through label dependencies. IEEE Trans. Image Process. 28, 3 (2019), 1428--1442.

Digital Library

[37]

Shangfei Wang, Zhilei Liu, Siliang Lv, Yanpeng Lv, Guobing Wu, Peng Peng, Fei Chen, and Xufa Wang. 2010. A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans. Multimedia 12, 7 (2010), 682--691.

Digital Library

[38]

Shangfei Wang, Chongliang Wu, Menghua He, Jun Wang, and Qiang Ji. 2015. Posed and spontaneous expression recognition through modeling their spatial patterns. Mach. Vis. Appl. (2015), 1--13.

[39]

Shangfei Wang, Chongliang Wu, and Qiang Ji. 2016. Capturing global spatial patterns for distinguishing posed and spontaneous expressions. Comput. Vis. Image Understand. 147 (2016), 69--76.

Digital Library

[40]

Chongliang Wu and Shangfei Wang. 2016. Posed and spontaneous expression recognition through restricted boltzmann machine. In MultiMedia Modeling. Springer, 127--137.

[41]

Alan L. Yuille. 2005. The convergence of contrastive divergences. In Advances in Neural Information Processing Systems. 1593--1600.

[42]

S. Yunus and T. Christopher. 2006. Cascaded classification of gender and facial expression using active appearance models. In Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR’06). 393--400.

[43]

Feifei Zhang, Qirong Mao, Xiangjun Shen, Yongzhao Zhan, and Ming Dong. 2018. Spatially coherent feature learning for pose-invariant facial expression recognition. ACM Trans. Multimedia Comput. Commun. Appl. 14, 1s (2018), 27.

Digital Library

[44]

L. Zhang, D. Tjondronegoro, and V. Chandran. 2011. Geometry vs. appearance for discriminating between posed and spontaneous emotions. In Neural Information Processing. Springer, 431--440.

Cited By

Feng RChen H(2025)Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in videoNeurocomputing10.1016/j.neucom.2025.129581626(129581)Online publication date: Apr-2025
https://doi.org/10.1016/j.neucom.2025.129581
Bisogni CCascone LNappi MPero C(2024)POSERImage and Vision Computing10.1016/j.imavis.2024.104952144:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.imavis.2024.104952
Gong WZhang YWang WCheng PGonzàlez J(2023)Meta-MMFNet: Meta-learning-based Multi-model Fusion Network for Micro-expression RecognitionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/353957620:2(1-20)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3539576
Show More Cited By

Index Terms

Posed and Spontaneous Expression Distinction Using Latent Regression Bayesian Networks
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Biometrics

Recommendations

Posed and spontaneous expression recognition through modeling their spatial patterns

This paper presents a new method to recognize posed and spontaneous expressions through modeling their spatial patterns. Gender and expression categories are employed as privileged information to further improve the recognition. The proposed approach ...
Capturing global spatial patterns for distinguishing posed and spontaneous expressions

A method to differentiate posed and spontaneous expressions by capturing their global spatial patterns using RBM is proposed.Partition function of RBM is estimated by extending AIS to RBM with continuous visible units.Gender and expression category ...
Posed and Spontaneous Expression Recognition Through Restricted Boltzmann Machine
MMM 2016: Proceedings, Part I, of the 22nd International Conference on MultiMedia Modeling - Volume 9516

This paper presents a new method to recognize posed and spontaneous expression through modeling their global spatial patterns in Restricted Boltzmann Machine RBM. First, the displacements of facial feature points between apex and onset facial images are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 16, Issue 3

August 2020

364 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3409646

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 July 2020

Online AM: 07 May 2020

Accepted: 01 March 2020

Revised: 01 March 2020

Received: 01 March 2019

Published in TOMM Volume 16, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Key R&D Program of China
National Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
120
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Feng RChen H(2025)Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in videoNeurocomputing10.1016/j.neucom.2025.129581626(129581)Online publication date: Apr-2025
https://doi.org/10.1016/j.neucom.2025.129581
Bisogni CCascone LNappi MPero C(2024)POSERImage and Vision Computing10.1016/j.imavis.2024.104952144:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.imavis.2024.104952
Gong WZhang YWang WCheng PGonzàlez J(2023)Meta-MMFNet: Meta-learning-based Multi-model Fusion Network for Micro-expression RecognitionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/353957620:2(1-20)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3539576
Chen DKong DLi JWang SYin B(2023)ADOSMNet: a novel visual affordance detection network with object shape mask guided feature encodersMultimedia Tools and Applications10.1007/s11042-023-16898-283:11(31629-31653)Online publication date: 18-Sep-2023
https://doi.org/10.1007/s11042-023-16898-2
Ramachandran ASangaiah A(2021)A Review on Object Detection in Unmanned Aerial Vehicle SurveillanceInternational Journal of Cognitive Computing in Engineering10.1016/j.ijcce.2021.11.005Online publication date: Dec-2021
https://doi.org/10.1016/j.ijcce.2021.11.005
Chen WLiu PLai CLin Y(2021)Identification of environmental microorganism using optimally fine-tuned convolutional neural networkEnvironmental Research10.1016/j.envres.2021.112610(112610)Online publication date: Dec-2021
https://doi.org/10.1016/j.envres.2021.112610
Ben Atitallah SDriss MBoulila WKoubaa ABen Ghézala H(2021)Fusion of convolutional neural networks based on Dempster–Shafer theory for automatic pneumonia detection from chest X‐ray imagesInternational Journal of Imaging Systems and Technology10.1002/ima.2265332:2(658-672)Online publication date: 13-Sep-2021
https://doi.org/10.1002/ima.22653
Lee KLee E(2020)Siamese Architecture-Based 3D DenseNet with Person-Specific Normalization Using Neutral Expression for Spontaneous and Posed Smile ClassificationSensors10.3390/s2024718420:24(7184)Online publication date: 15-Dec-2020
https://doi.org/10.3390/s20247184
Muhammad KKhan SKumar NDel Ser JMirjalili S(2020)Vision-based personalized Wireless Capsule Endoscopy for smart healthcare: Taxonomy, literature review, opportunities and challengesFuture Generation Computer Systems10.1016/j.future.2020.06.048Online publication date: Jun-2020
https://doi.org/10.1016/j.future.2020.06.048

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents