research-article

Identity- and Pose-Robust Facial Expression Recognition through Adversarial Feature Learning

Authors:

Guang LiangAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 238 - 246

https://doi.org/10.1145/3343031.3350872

Published: 15 October 2019 Publication History

Abstract

Existing facial expression recognition methods either focus on pose variations or identity bias, but not both simultaneously. This paper proposes an adversarial feature learning method to address both of these issues. Specifically, the proposed method consists of five components: an encoder, an expression classifier, a pose discriminator, a subject discriminator, and a generator. An encoder extracts feature representations, and an expression classifier tries to perform facial expression recognition using the extracted feature representations. The encoder and the expression classifier are trained collaboratively, so that the extracted feature representations are discriminative for expression recognition. A pose discriminator and a subject discriminator classify the pose and the subject from the extracted feature representations respectively. They are trained adversarially with the encoder. Thus, the extracted feature representations are robust to poses and subjects. A generator reconstructs facial images to further favor the feature representations. Experiments on five benchmark databases demonstrate the superiority of the proposed method to state-of-the-art work.

References

[1]

Wen-Sheng Chu, Fernando De la Torre, and Jeffrey F Cohn. 2017. Selective transfer machine for personalized facial expression analysis. IEEE transactions on pattern analysis and machine intelligence, Vol. 39, 3 (2017), 529--545.

[2]

Abhinav Dhall, Roland Goecke, Simon Lucey, and Tom Gedeon. 2011. Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE, 2106--2112.

[3]

Changxing Ding and Dacheng Tao. 2016. A comprehensive survey on pose-invariant face recognition. ACM Transactions on intelligent systems and technology (TIST), Vol. 7, 3 (2016), 37.

[4]

Stefanos Eleftheriadis, Ognjen Rudovic, and Maja Pantic. 2015. Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition. IEEE transactions on image processing, Vol. 24, 1 (2015), 189--204.

[5]

Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. 2013. Challenges in representation learning: A report on three machine learning contests. In International Conference on Neural Information Processing. Springer, 117--124.

[6]

Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. 2010. Multi-pie. Image and Vision Computing, Vol. 28, 5 (2010), 807--813.

Digital Library

[7]

Mahdi Jampour, Vincent Lepetit, Thomas Mauthner, and Horst Bischof. 2017. Pose-specific non-linear mappings in feature space towards multiview facial expression recognition. Image and vision computing, Vol. 58 (2017), 38--46.

[8]

Ying-Hsiu Lai and Shang-Hong Lai. 2018. Emotion-preserving representation learning via generative adversarial network for multi-view facial expression recognition. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 263--270.

Digital Library

[9]

Shan Li and Weihong Deng. 2018. Deep facial expression recognition: A survey. arXiv preprint arXiv:1804.08348 (2018).

[10]

Yanwei Li, Xingang Wang, Shilei Zhang, Lingxi Xie, Wenqi Wu, Hongyuan Yu, and Zheng Zhu. 2018. Identity-Enhanced Network for Facial Expression Recognition. arXiv preprint arXiv:1812.04207 (2018).

[11]

Xiaofeng Liu, BVK Vijaya Kumar, Jane You, and Ping Jia. 2017. Adaptive deep metric learning for identity-aware facial expression recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops . 20--29.

[12]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, Vol. 9, Nov (2008), 2579--2605.

[13]

Qirong Mao, Qiyu Rao, Yongbin Yu, and Ming Dong. 2017. Hierarchical Bayesian theme models for multipose facial expression recognition. IEEE Transactions on Multimedia, Vol. 19, 4 (2017), 861--873.

Digital Library

[14]

Zibo Meng, Ping Liu, Jie Cai, Shizhong Han, and Yan Tong. 2017. Identity-aware convolutional neural network for facial expression recognition. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). IEEE, 558--565.

Digital Library

[15]

Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. 2017. Affectnet: A database for facial expression, valence, and arousal computing in the wild. arXiv preprint arXiv:1708.03985 (2017).

Digital Library

[16]

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Conference on Artificial Intelligence .

Digital Library

[17]

Can Wang and Shangfei Wang. 2018. Personalized Multiple Facial Action Unit Recognition through Generative Adversarial Recognition Network. In 2018 ACM Multimedia Conference on Multimedia Conference. ACM, 302--310.

Digital Library

[18]

Jian Wu, Zhiming Cui, Victor S Sheng, Pengpeng Zhao, Dongliang Su, and Shengrong Gong. 2013. A Comparative Study of SIFT and its Variants. Measurement science review, Vol. 13, 3 (2013), 122--131.

[19]

Jianlong Wu, Zhouchen Lin, Wenming Zheng, and Hongbin Zha. 2017. Locality-constrained linear coding based bi-layer model for multi-view facial expression recognition. Neurocomputing, Vol. 239 (2017), 143--152.

Digital Library

[20]

Huiyuan Yang, Umur Ciftci, and Lijun Yin. 2018a. Facial expression recognition by de-expression residue learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2168--2177.

[21]

Huiyuan Yang, Zheng Zhang, and Lijun Yin. 2018b. Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 294--301.

Digital Library

[22]

Lijun Yin, Xiaochen Chen, Yi Sun, Tony Worm, and Michael Reale. 2008. A high-resolution 3D dynamic facial expression database. In Automatic Face & Gesture Recognition, 2008. FG'08. 8th IEEE International Conference on. IEEE, 1--6.

[23]

Jiabei Zeng, Shiguang Shan, and Xilin Chen. 2018. Facial expression recognition with inconsistently annotated datasets. In Proceedings of the European conference on computer vision (ECCV). 222--237.

Digital Library

[24]

Feifei Zhang, Tianzhu Zhang, Qirong Mao, and Changsheng Xu. 2018. Joint pose and expression modeling for facial expression recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3359--3368.

[25]

Kaihao Zhang, Yongzhen Huang, Yong Du, and Liang Wang. 2017. Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Transactions on Image Processing, Vol. 26, 9 (2017), 4193--4203.

Digital Library

[26]

Tong Zhang, Wenming Zheng, Zhen Cui, Yuan Zong, Jingwei Yan, and Keyu Yan. 2016. A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia, Vol. 18, 12 (2016), 2528--2536.

Digital Library

Cited By

Nie WWang ZWang XChen BZhang HLiu H(2025)Diving Into Sample Selection for Facial Expression Recognition With Noisy AnnotationsIEEE Transactions on Biometrics, Behavior, and Identity Science10.1109/TBIOM.2024.34354987:1(95-107)Online publication date: Jan-2025
https://doi.org/10.1109/TBIOM.2024.3435498
Wang SYang JYang Y(2024)A collaborative filtering method by fusion of facial information featuresJournal of Intelligent & Fuzzy Systems10.3233/JIFS-232718(1-20)Online publication date: 19-Mar-2024
https://doi.org/10.3233/JIFS-232718
Win KAkhtar ZMohan C(2024)Does Hard-Negative Contrastive Learning Improve Facial Emotion Recognition?Proceedings of the 2024 7th International Conference on Machine Vision and Applications10.1145/3653946.3653971(162-168)Online publication date: 12-Mar-2024
https://dl.acm.org/doi/10.1145/3653946.3653971
Show More Cited By

Index Terms

Identity- and Pose-Robust Facial Expression Recognition through Adversarial Feature Learning
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

Pose-robust feature learning for facial expression recognition

Automatic facial expression recognition (FER) from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. Pose variations are difficult to tackle and many face analysis methods ...
Pose-Robust Facial Expression Recognition Using View-Based 2D + 3D AAM

This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D 3D active appearance model (AAM) that extends the 2D 3D AAM to the view-based approach, where one independent face model is used for a ...
Pose-robust personalized facial expression recognition through unsupervised multi-source domain adaptation
Abstract
Pose-robust personalized facial expression recognition is rather challenging, as facial expressions are subject-related and pose-dependent. Multi-source domain adaptation tries to leverage knowledge from multiple source domains to boost the ...
Highlights
- An unsupervised multi-source domain adaptation method is proposed for FER.
- Adversarial learning is used to learn a source FER encoder avoiding pose variations.
- Adversarial domain adaptation is used to train a personalized model for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Project from Anhui Science and Technology Agency
NSFC

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

61
Total Citations
View Citations
849
Total Downloads

Downloads (Last 12 months)46
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nie WWang ZWang XChen BZhang HLiu H(2025)Diving Into Sample Selection for Facial Expression Recognition With Noisy AnnotationsIEEE Transactions on Biometrics, Behavior, and Identity Science10.1109/TBIOM.2024.34354987:1(95-107)Online publication date: Jan-2025
https://doi.org/10.1109/TBIOM.2024.3435498
Wang SYang JYang Y(2024)A collaborative filtering method by fusion of facial information featuresJournal of Intelligent & Fuzzy Systems10.3233/JIFS-232718(1-20)Online publication date: 19-Mar-2024
https://doi.org/10.3233/JIFS-232718
Win KAkhtar ZMohan C(2024)Does Hard-Negative Contrastive Learning Improve Facial Emotion Recognition?Proceedings of the 2024 7th International Conference on Machine Vision and Applications10.1145/3653946.3653971(162-168)Online publication date: 12-Mar-2024
https://dl.acm.org/doi/10.1145/3653946.3653971
Cho YKim CCho HKu YKim EBoboev MLee JBaek S(2024)RMFER: Semi-supervised Contrastive Learning for Facial Expression Recognition with Reaction Mashup Video2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00581(5901-5910)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00581
Xie WPeng ZShen LLu WZhang YSong S(2024)Cross-Layer Contrastive Learning of Latent Semantics for Facial Expression RecognitionIEEE Transactions on Image Processing10.1109/TIP.2024.337845933(2514-2529)Online publication date: 2024
https://doi.org/10.1109/TIP.2024.3378459
Wang RSong PZheng W(2024)Graph-Diffusion-Based Domain-Invariant Representation Learning for Cross-Domain Facial Expression RecognitionIEEE Transactions on Computational Social Systems10.1109/TCSS.2024.335511311:3(4163-4174)Online publication date: Jun-2024
https://doi.org/10.1109/TCSS.2024.3355113
Wang SWu YChang YLi GMao M(2024)Pose-Aware Facial Expression Recognition Assisted by Expression DescriptionsIEEE Transactions on Affective Computing10.1109/TAFFC.2023.326777415:1(241-253)Online publication date: Jan-2024
https://doi.org/10.1109/TAFFC.2023.3267774
Ni SLiu SCai HWang HXiao H(2024)Facial Expression Recognition with Multi-level Integration Disentangled Generative Adversarial Network2024 IEEE International Conference on Industrial Technology (ICIT)10.1109/ICIT58233.2024.10540810(1-6)Online publication date: 25-Mar-2024
https://doi.org/10.1109/ICIT58233.2024.10540810
Zhang YJiang JZhao YLiang K(2024)SLNL: Soft Label Regularization For Semi-Supervised Facial Expression Recognition With Negative Label Learning2024 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP51287.2024.10647657(423-429)Online publication date: 27-Oct-2024
https://doi.org/10.1109/ICIP51287.2024.10647657
Sun RZhang ZLiu HZhao LZhou QLiu Z(2024)DacFER: Dual Attention Correction Learning for Efficient Facial Expression Recognition2024 7th International Conference on Electronics Technology (ICET)10.1109/ICET61945.2024.10672990(941-945)Online publication date: 17-May-2024
https://doi.org/10.1109/ICET61945.2024.10672990
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten