Abstract
Facial expression is an essential factor in conveying human emotional states and intentions. Although remarkable advancement has been made in facial expression recognition (FER) tasks, challenges due to large variations of expression patterns and unavoidable data uncertainties remain. In this paper, we propose mid-level representation enhancement (MRE) and graph embedded uncertainty suppressing (GUS) addressing these issues. On one hand, MRE is introduced to avoid expression representation learning being dominated by a limited number of highly discriminative patterns. On the other hand, GUS is introduced to suppress the feature ambiguity in the representation space. The proposed method not only has stronger generalization capability to handle different variations of expression patterns but also more robustness in capturing expression representations. Experimental evaluation on Aff-Wild2 have verified the effectiveness of the proposed method. We achieved 2nd place in the Learning from Synthetic Data (LSD) Challenge of the 4th Competition on Affective Behavior Analysis in-the-wild (ABAW). The code has been released at https://github.com/CruiseYuGH/GUS.
J. Lei and Z. Liu—Contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10, 18–31 (2017)
Cai, J., Meng, Z., Khan, A.S., O’Reilly, J., Tong, Y.: Identity-free facial expression recognition using conditional generative adversarial network. In: ICIP 2021 (2021)
Hu, P., Cai, D., Wang, S., Yao, A., Chen, Y.: Learning supervised scoring ensemble for emotion recognition in the wild. In: the 19th ACM International Conference (2017)
Huang, S., Wang, X., Dao, D.: Stochastic partial swap: enhanced model generalization and interpretability. In: ICCV (2021)
Jeong, J.Y., Hong, Y.G., Oh, J., Hong, S., Jeong, J.W., Jung, Y.: Learning from synthetic data: facial expression classification based on ensemble of multi-task networks. In: arXiv (2022)
Jiahui She, Yibo Hu, H.S.J.W.Q.S., Mei, T.: Dive into ambiguity: latent distribution mining and pairwise uncertainty estimation for facial expression recognition. In: CVPR (2021)
Kim, Y., Yoo, B., Kwak, Y., Choi, C., Kim, J.: Deep generative-contrastive networks for facial expression recognition. In: CVPR (2017)
Kipf, T.N., Welling, M.: Semi-Supervised classification with graph convolutional networks (2017)
Kollias, D.: ABAW: learning from synthetic data & multi-task learning challenges. (2022) arXiv preprint arXiv:2207.01138v2
Kollias, D.: Abaw: valence-arousal estimation, expression recognition, action unit detection & multi-task learning challenges. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2328–2336 (2022)
Kollias, D., Cheng, S., Pantic, M., Zafeiriou, S.: Photorealistic facial synthesis in the dimensional affect space. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11130, pp. 475–491. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11012-3_36
Kollias, D., Cheng, S., Ververas, E., Kotsia, I., Zafeiriou, S.: Deep neural network augmentation: Generating faces for affect analysis. Int. J. Comput. Vis. 128(5), 1455–1484 (2020)
Kollias, D., Nicolaou, M.A., Kotsia, I., Zhao, G., Zafeiriou, S.: Recognition of affect in the wild using deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1972–1979. IEEE (2017)
Kollias, D., Sharmanska, V., Zafeiriou, S.: Distribution matching for heterogeneous multi-task learning: a large-scale face study. (2021) arXiv preprint arXiv:2105.03790
Kollias, D., et al.: Deep affect prediction in-the-wild: aff-wild database and challenge, deep architectures, and beyond. Int. J. Comput. Vis. pp. 1–23 (2019)
Kollias, D., Zafeiriou, S.: Expression, affect, action unit recognition: aff-wild2, multi-task learning and arcface. (2019) arXiv preprint arXiv:1910.04855
Kollias, D., Zafeiriou, S.: VA-StarGAN: continuous affect generation. In: Blanc-Talon, J., Delmas, P., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2020. LNCS, vol. 12002, pp. 227–238. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-40605-9_20
Kollias, D., Zafeiriou, S.: Affect analysis in-the-wild: valence-arousal, expressions, action units and a unified framework (2021) arXiv preprint arXiv:2103.15792
Li, S., et al.: Facial affect analysis: learning from synthetic data and multi-task learning challenges. In: arXiv (2022)
Li, Y., Zeng, J., Shan, S., Chen, X.: Patch-gated CNN for occlusion-aware facial expression recognition. In: ICPR (2018)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. In: ICCV (2017)
Mao, S., Li, X., Chen, J., Peng, X.: Au-supervised convolutional vision transformers for synthetic facial expression recognition. arXiv (2022)
Savchenko, A.V.: HSE-NN team at the 4th ABAW competition: multi-task emotion recognition and learning from synthetic images. In: arXiv (2022)
Wang, K., Peng, X., Yang, J., Lu, S., Qiao, Y.: Suppressing uncertainties for large-scale facial expression recognition. In: CVPR (2020)
Wang, X., Zhu, M., Bo, D., Cui, P., Shi, C., Pei, J.: Am-GCN: adaptive multi-channel graph convolutional networks
Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: CVPR (2018)
Yao, A., Cai, D., Hu, P., Wang, S., Chen, Y.: HoloNet: towards robust emotion recognition in the wild. In: ICMI (2016)
Zafeiriou, S., Kollias, D., Nicolaou, M.A., Papaioannou, A., Zhao, G., Kotsia, I.: aff-wild: valence and arousal in-the-wild’challenge. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1980–1987. IEEE (2017)
Zeng, J., Shan, S., Chen, X.: Facial expression recognition with inconsistently annotated datasets. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11217, pp. 227–243. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01261-8_14
Zhao, X., et al.: Peak-piloted deep network for facial expression recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 425–442. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_27
Zhou, J., Zhang, X., Liu, Y., Lan, X.: Facial expression recognition using spatial-temporal semantic graph network. In: 2020 IEEE International Conference on Image Processing (ICIP) (2020)
Zhou, X., Liu, X., Wang, C., Zhai, D., Jiang, J., Ji, X.: Learning with noisy labels via sparse regularization. In: ICCV (2021)
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China (No. 62106226, No. 62036009), the National Key Research and Development Program of China (No. 2020YFB1707700), and Zhejiang Provincial Natural Science Foundation of China (No. LQ22F020013).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lei, J. et al. (2023). Facial Expression Recognition with Mid-level Representation Enhancement and Graph Embedded Uncertainty Suppressing. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13806. Springer, Cham. https://doi.org/10.1007/978-3-031-25075-0_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-25075-0_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25074-3
Online ISBN: 978-3-031-25075-0
eBook Packages: Computer ScienceComputer Science (R0)