Shangfei Wang
Skip Abstract Section
Abstract
Facial spatial patterns can help distinguish between posed and spontaneous expressions, but this information has not been thoroughly leveraged by current studies. We present several latent regression Bayesian networks (LRBNs) to capture the patterns existing in facial landmark points and to use those points to differentiate posed from spontaneous expressions. The visible nodes of the LRBN represent facial landmark points. Through learning, the LRBN captures the probabilistic dependencies among landmark points as well as latent variables given observations, successfully modeling the spatial patterns inherent in expressions. Current methods tend to ignore gender and expression categories, although these factors can influence spatial patterns. Therefore, we propose to incorporate this as a kind of privileged information. We construct several LRBNs to capture spatial patterns from spontaneous and posed facial expressions given expression-related factors. Facial landmark points are used during testing to classify samples as either posed or spontaneous, depending on which LRBN has the largest likelihood. We conduct experiments to showcase the superiority of the proposed approach in both modeling spatial patterns and classifying expressions as either posed or spontaneous.
- Hervé Abdi and L. J. Williams. 2010. Normalizing data. In Encyclopedia of Research Design. 935--938.Google Scholar
- Elisabeth Andre. 2013. Exploiting unconscious user signals in multimodal human-computer interaction. ACM Trans. Multimedia Comput. Commun. Appl. 9, 1s (2013), 48.Google ScholarDigital Library
- Yoshua Bengio, Li Yao, and Kyunghyun Cho. 2014. Bounding the test log-likelihood of generative models. In Proceedings of the International Conference on Learning Representations (Conference Track).Google Scholar
- George A. Bonanno and Dacher Keltner. 1997. Facial expressions of emotion and the course of conjugal bereavement. J. Abnorm. Psychol. 106, 1 (1997), 126.Google ScholarCross Ref
- J. F. Cohn and K. L. Schmidt. 2004. The timing of facial motion in posed and spontaneous smiles. Int. J. Wavelets Multires. Inf. Process. 2, 02 (2004), 121--132.Google ScholarCross Ref
- H. Dibeklioğlu, A. Salah, and T. Gevers. 2012. Are you really smiling at me? Spontaneous versus posed enjoyment smiles. In Proceedings of the European Conference on Computer Vision (ECCV’12). Springer, 525--538.Google Scholar
- Hamdi Dibeklioglu, Roberto Valenti, Albert Ali Salah, and Theo Gevers. 2010. Eyes do not lie: Spontaneous versus posed smiles. In Proceedings of the International Conference on Multimedia. ACM, 703--706.Google ScholarDigital Library
- Paul Ekman. 2003. Darwin, deception, and facial expression. Ann. N. Y. Acad. Sci. 1000, 1 (2003), 205--221.Google ScholarCross Ref
- Paul Ekman and Wallace V. Friesen. 1982. Felt, false, and miserable smiles. J. Nonverb. Behav. 6, 4 (1982), 238--252.Google ScholarCross Ref
- Paul Ekman, Joseph C. Hager, and Wallace V. Friesen. 1981. The symmetry of emotional and deliberate facial actions. Psychophysiology 18, 2 (1981), 101--106.Google ScholarCross Ref
- Byron N. Fujita, Robert G. Harper, and Arthur N. Wiens. 1980. Encoding-decoding of nonverbal emotional messages: Sex differences in spontaneous and enacted expressions. J. Nonverb. Behav. 4, 3 (1980), 131--145.Google ScholarCross Ref
- Quan Gan, Siqi Nie, Shangfei Wang, and Qiang Ji. 2017. Differentiating between posed and spontaneous expressions with latent regression Bayesian network. In Proceedings of the Annual Cconference on Artifical Intelligence (AAAI’17). 4039--4045.Google Scholar
- Zhe Gan, Ricardo Henao, David Carlson, and Lawrence Carin. 2015. Learning deep sigmoid belief networks with data augmentation. In Proceedings of the International Conference on Artificial Intelligence and Statistics (2015).Google Scholar
- Karol Gregor, Andriy Mnih, and Daan Wierstra. 2014. Deep AutoRegressive networks. In Proceedings of the 31st International Conference on Machine Learning (2014).Google ScholarDigital Library
- Geoffrey Hinton. 2010. A practical guide to training restricted Boltzmann machines. Momentum 9, 1 (2010), 926.Google Scholar
- Geoffrey Hinton and Ruslan Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507.Google Scholar
- Geoffrey E. Hinton, Peter Dayan, Brendan J. Frey, and Radford M. Neal. 1995. The “wake-sleep” algorithm for unsupervised neural networks. Science 268, 5214 (1995), 1158--1161.Google Scholar
- Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR’14).Google Scholar
- C. Lithari, C. A. Frantzidis, C. Papadelis, A. B. Vivas, M. A. Klados, C. Kourtidou-Papadeli, C. Pappas, A. A. Ioannides, and P. D. Bamidis. 2010. Are females more responsive to emotional stimuli? A neurophysiological study across arousal and valence dimensions. Brain Topogr. 23, 1 (2010), 27--40.Google ScholarCross Ref
- G. C. Littlewort, M. S. Bartlett, and K. Lee. 2009. Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis. Comput. 27, 12 (2009), 1797--1803.Google ScholarDigital Library
- Mohammad Mavadati, Peyten Sanger, and Mohammad H. Mahoor. 2016. Extended DISFA dataset: Investigating posed and spontaneous facial expressions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1--8.Google Scholar
- Andriy Mnih and Karol Gregor. 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on Machine Learning (2014).Google Scholar
- Shushi Namba, Shoko Makihara, Russell S. Kabir, Makoto Miyatani, and Takashi Nakao. 2017. Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Curr. Psychol. 36, 3 (2017), 593--605.Google ScholarCross Ref
- Siqi Nie, Yue Zhao, and Qiang Ji. 2016. Latent regression Bayesian network for data representation. In Proceedings of the 23rd International Conference on Pattern Recognition (ICPR’16). IEEE, 3494--3499.Google Scholar
- E. Paul. [n.d.]. BBC-Dataset. Retrieved from http://www.bbc.co.uk/science/humanbody/mind/surveys/smiles/.Google Scholar
- S. Petridis, B. Martinez, and M. Pantic. 2013. The MAHNOB laughter database. Image Vis. Comput. 31, 2 (2013), 186--202.Google ScholarDigital Library
- T. Pfister, X. Li, G. Zhao, and M. Pietikainen. 2011. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops’11). IEEE, 868--875.Google Scholar
- Tomas Pfister, Xiaobai Li, Guoying Zhao, and Matti Pietikäinen. 2011. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops’11). IEEE, 868--875.Google ScholarCross Ref
- Danilo J Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning (ICML’14). 1278--1286.Google Scholar
- Fabien Ringeval, Björn Schuller, Michel Valstar, Jonathan Gratch, Roddy Cowie, and Maja Pantic. 2018. Introduction to the special section on multimedia computing and applications of socio-affective behaviors in the wild. ACM Trans. Multimedia Comput. Commun. Appl. 14, 1s (2018), 25.Google ScholarDigital Library
- Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. Ann. Math. Stat. 22, 3 (1951), 400--407.Google ScholarCross Ref
- Lawrence K. Saul, Tommi Jaakkola, and Michael I. Jordan. 1996. Mean field theory for sigmoid belief networks. J. Artif. Intell. Res. 4, 61 (1996), 76.Google ScholarCross Ref
- M. Seckington. 2011. Using dynamic Bayesian networks for posed versus spontaneous facial expression recognition. Master’s Thesis, Department of Computer Science, Delft University of Technology (2011).Google Scholar
- M. F. Valstar, M. Pantic, Z. Ambadar, and J. F. Cohn. 2006. Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. In Proceedings of the 8th International Conference on Multimodal Interfaces. ACM, 162--170.Google Scholar
- V. Vapnik and A. Vashist. 2009. A new learning paradigm: Learning using privileged information. Neur. Netw. 22, 5–6 (2009), 544.Google ScholarDigital Library
- Shangfei Wang, Longfei Hao, and Qiang Ji. 2019. Facial action unit recognition and intensity estimation enhanced through label dependencies. IEEE Trans. Image Process. 28, 3 (2019), 1428--1442.Google ScholarDigital Library
- Shangfei Wang, Zhilei Liu, Siliang Lv, Yanpeng Lv, Guobing Wu, Peng Peng, Fei Chen, and Xufa Wang. 2010. A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans. Multimedia 12, 7 (2010), 682--691.Google ScholarDigital Library
- Shangfei Wang, Chongliang Wu, Menghua He, Jun Wang, and Qiang Ji. 2015. Posed and spontaneous expression recognition through modeling their spatial patterns. Mach. Vis. Appl. (2015), 1--13.Google Scholar
- Shangfei Wang, Chongliang Wu, and Qiang Ji. 2016. Capturing global spatial patterns for distinguishing posed and spontaneous expressions. Comput. Vis. Image Understand. 147 (2016), 69--76.Google ScholarDigital Library
- Chongliang Wu and Shangfei Wang. 2016. Posed and spontaneous expression recognition through restricted boltzmann machine. In MultiMedia Modeling. Springer, 127--137.Google Scholar
- Alan L. Yuille. 2005. The convergence of contrastive divergences. In Advances in Neural Information Processing Systems. 1593--1600.Google Scholar
- S. Yunus and T. Christopher. 2006. Cascaded classification of gender and facial expression using active appearance models. In Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR’06). 393--400.Google Scholar
- Feifei Zhang, Qirong Mao, Xiangjun Shen, Yongzhao Zhan, and Ming Dong. 2018. Spatially coherent feature learning for pose-invariant facial expression recognition. ACM Trans. Multimedia Comput. Commun. Appl. 14, 1s (2018), 27.Google ScholarDigital Library
- L. Zhang, D. Tjondronegoro, and V. Chandran. 2011. Geometry vs. appearance for discriminating between posed and spontaneous emotions. In Neural Information Processing. Springer, 431--440.Google Scholar
Index Terms
- Posed and Spontaneous Expression Distinction Using Latent Regression Bayesian Networks
Recommendations
Occluded Facial Expression Recognition Enhanced through Privileged Information
MM '19: Proceedings of the 27th ACM International Conference on MultimediaIn this paper, we propose a novel approach of occluded facial expression recognition under the help of non-occluded facial images. The non-occluded facial images are used as privileged information, which is only required during training, but not ...
Posed and spontaneous expression recognition through modeling their spatial patterns
This paper presents a new method to recognize posed and spontaneous expressions through modeling their spatial patterns. Gender and expression categories are employed as privileged information to further improve the recognition. The proposed approach ...
Capturing global spatial patterns for distinguishing posed and spontaneous expressions
A method to differentiate posed and spontaneous expressions by capturing their global spatial patterns using RBM is proposed.Partition function of RBM is estimated by extending AIS to RBM with continuous visible units.Gender and expression category ...
Comments