Abstract
Giving machines the ability to perceive human emotions and enable them to recognize our emotional states is one of the important goals to realize human-computer interaction. In the past decades, facial expression recognition (FER) has always been a research hotspot in the field of computer vision. However, the existing facial expression datasets generally have the problems of insufficient data and unbalanced categories, leading to the phenomenon of over-fitting. To solve this problem, most methods employ the generative adversarial network (GAN) for data augmentation, and achieve good results in facial image generation. But these works focus only on facial identity or head poses, which are not robust for the transformation of facial expression recognition from the laboratory environment to unconstrained scenes. Therefore, we employ the disentangled representation learning to obtain facial feature representation, so as to reduce the impact of pose changes and identity biases on FER. Specifically, the generator uses the encoder-decoder structure to map each face image to two latent spaces: the pose space and the identity space. In each latent space, we disentangle the target attribute from other attributes, and then concatenate corresponding feature vectors to generate a new image with one person’s identity and another person’s pose. Experimental results on Multi-PIE and RAFD datasets show that the proposed method can obtain high quality generated images and effectively improve the recognition rate of facial expressions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bengio, Y., Courville, A.C., Vincent, P.: Unsupervised feature learning and deep learning: a review and new perspectives. CoRR, p. 2012 (2012)
Berglund, M., Raiko, T., Honkala, M., Kärkkäinen, L., Vetek, A., Karhunen, J.T.: Bidirectional recurrent neural networks as generative models. In: NIPS, pp. 856–864 (2015)
Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: CVPR, pp. 8789–8797 (2018)
Chu, W.S., De la Torre, F., Cohn, J.F.: Selective transfer machine for personalized facial expression analysis. TPAMI 39(3), 529–545 (2016)
Darwin, C., Prodger, P.: The Expression of the Emotions in Man and Animals. Oxford University Press, Oxford (1998)
Ding, C., Tao, D.: A comprehensive survey on pose-invariant face recognition. TIST 7(3), 1–42 (2016)
Eleftheriadis, S., Rudovic, O., Pantic, M.: Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. TIP 24(1), 189–204 (2014)
Goodfellow, I.J., et al.: Generative adversarial networks. arXiv preprint: 1406.2661 (2014)
Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image Vis. Comput. 28(5), 807–813 (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR, pp. 4700–4708 (2017)
Kaneko, T., Hiramatsu, K., Kashino, K.: Generative attribute controller with conditional filtered generative adversarial networks. In: CVPR, pp. 6089–6098 (2017)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint: 1412.6980 (2014)
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H., Hawk, S.T., Van Knippenberg, A.: Presentation and validation of the Radboud faces database. Cogn. Emotion 24(8), 1377–1388 (2010)
Li, S., Deng, W.: Deep facial expression recognition: a survey. In: IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2020.2981446
Mao, Q., Zhang, F., Wang, L., Luo, S., Dong, M.: Cascaded multi-level transformed Dirichlet process for multi-pose facial expression recognition. Comput. J. 61(11), 1605–1619 (2018)
Meng, Z., Liu, P., Cai, J., Han, S., Tong, Y.: Identity-aware convolutional neural network for facial expression recognition. In: FG, pp. 558–565. IEEE (2017)
Nirkin, Y., Keller, Y., Hassner, T.: FSGAN: subject agnostic face swapping and reenactment. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7184–7193 (2019)
Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition (2015)
Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: CVPR, pp. 815–823 (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint: 1409.1556 (2014)
Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human-level performance in face verification. In: CVPR, pp. 1701–1708 (2014)
Tian, Y.l., Kanade, T., Cohn, J.F.: Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity. In: FG, pp. 229–234. IEEE (2002)
Yan, Y., Huang, Y., Chen, S., Shen, C., Wang, H.: Joint deep learning of facial expression synthesis and recognition. TMM 22(11), 2792–2807 (2019)
Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: CVPR, pp. 2168–2177 (2018)
Yang, H., Zhang, Z., Yin, L.: Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks. In: FG, pp. 294–301. IEEE (2018)
Zhang, F., Zhang, T., Mao, Q., Xu, C.: Joint pose and expression modeling for facial expression recognition. In: CVPR, pp. 3359–3368 (2018)
Zhang, K., Huang, Y., Du, Y., Wang, L.: Facial expression recognition based on deep evolutional spatial-temporal networks. TIP 26(9), 4193–4203 (2017)
Zhang, Z., Wang, L., Zhu, Q., Chen, S.K., Chen, Y.: Pose-invariant face recognition using facial landmarks and weber local descriptor. Knowl. Based Syst. 84, 78–88 (2015)
Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. In: CVPR, pp. 5810–5818 (2017)
Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., Metaxas, D.N.: Learning active facial patches for expression analysis. In: CVPR, pp. 2562–2569 (2012)
Acknowledgment
This work was supported by the National Key Research & Development Plan of China 2020AAA0106200, the National Natural Science Foundation of China under Grant 61936005, 61872424, the Natural Science Foundation of Jiangsu Province (Grants No BK20200037).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, B., Gan, Y., Bao, BK. (2021). Multi-pose Facial Expression Recognition Based on Unpaired Images. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12889. Springer, Cham. https://doi.org/10.1007/978-3-030-87358-5_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-87358-5_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87357-8
Online ISBN: 978-3-030-87358-5
eBook Packages: Computer ScienceComputer Science (R0)