Skip to main content

Advertisement

SpringerLink
Log in

Deep-block network for AU recognition and expression migration

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Human facial behavior is an important information for communication. The study of facial behavior is one of the significant research topics in field of psychology, computer vision and artificial intelligence. In order to improve performance of facial expression and action unit recognition, a face recognition method based on deep-block network proposed in this paper. First, to improve the network performance, we preprocess the input of the network facial image, which includes two operations: face detection and face standardization. Second, deep-block network regards facial parts as the core of expression recognition rather than the whole face and key areas are in charge of specific action units to abate the weak correlation bias, which results in better classification and regression effect. Last, with the purpose of reducing impact of image independent factors, relevant feature map is applied to recognize the associated facial action units, which can promote the accuracy of detection to a certain extent. Experimental results on CK+ and MMI show that proposed method can not only capture the correlation of whole face regions globally, but also can increase network speed caused by too few pooling layers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Batista JC, Albiero V, Bellon ORP et al (2017) AUMPNet: simultaneous action units detection and intensity estimation on multipose facial images using a single convolutional neural network[C]. IEEE International Conference on Automatic Face & Gesture Recognition

    Google Scholar 

  2. Chen D, Hua G, Wen F et al (2016) Supervised transformer network for efficient face detection[C]. European Conference on Computer Vision:122–138

  3. Cheok AD, Goh KH, Liu W et al (2004) Human Pacman: a Mobile, wide-area entertainment system based on physical, social, and ubiquitous computing[J]. Pers Ubiquit Comput 8(2):71–81

    Article  Google Scholar 

  4. Cootes TF, Taylor CJ (1993) Active shape model search using local Grey-level models: a quantitative evaluation[C]. British Machine Vision Conference

  5. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection[C]. IEEE Computer Society Conference on Computer Vision & Pattern Recognition

    Book  Google Scholar 

  6. Deng J, Guo J, Ververas E et al (2020) Retinaface: single-shot multi-level face localisation in the wild[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition:5203–5212

  7. Ekman P, Friesen WV (1978) Facial action coding system (FACS): a technique for the measurement of facial actions[J]. Rivista Di Psichiatria 47(2):126–138

    Google Scholar 

  8. Ghosh S, Laksana E, Scherer S et al (2015) A multi-label convolutional neural network approach to cross-domain action unit detection[C]. International Conference on Affective Computing and Intelligent Interaction (ACII), pp 609–615

    Google Scholar 

  9. Gudi A, Tasli HE, Den Uyl TM et al (2015) Deep learning based FACS Action Unit occurrence and intensity estimation[C]. IEEE International Conference & Workshops on Automatic Face & Gesture Recognition, pp 1–5

    Google Scholar 

  10. Li S, Deng W, Du J (2017) Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2584–2593

    Google Scholar 

  11. Liu X, Kumar BVKV, You J et al (2017) Adaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition[C]. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 522–531

    Google Scholar 

  12. Lowe DG (2004) Distinctive image features from scale-invariant Keypoints[J]. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  13. Lucey P, Cohn JF, Kanade T et al (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression[C]. Computer Vision & Pattern Recognition Workshops

    Google Scholar 

  14. Maaten L, Hendriks E (2012) Action unit classification using active appearance models and conditional random fields[J]. Cogn Process 13(S2):507–518

    Article  Google Scholar 

  15. Parr LA, Waller BM, Burrows AM et al (2010) Brief communication: MaqFACS: a muscle-based facial movement coding system for the rhesus macaque[J]. Am J Phys Anthropol 143(4)

  16. Peng G, Wang S (2018) Weakly Supervised Facial Action Unit Recognition Through Adversarial Training[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2188–2196

    Google Scholar 

  17. Ruiz A, Van De Weijer J, Binefa X (2015) From Emotions to Action Units with Hidden and Semi-Hidden-Task Learning[C]. IEEE International Conference on Computer Vision (ICCV), pp 3703–3711

    Google Scholar 

  18. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition[J]. Computer Science

    Google Scholar 

  19. Tong Y, Ji Q (2008) Learning bayesian networks with qualitative constraints[C]. IEEE Conference on Computer Vision and Pattern Recognition 2008:1–8

    Google Scholar 

  20. Tran DL, Walecki R, Rudovic O et al (2017) DeepCoder: semi-parametric Variational autoencoders for automatic facial action coding[C]. IEEE International Conference on Computer Vision

    Google Scholar 

  21. Tsér Z, Jeni LA, Lrincz A et al (2016) Deep learning for facial action unit detection under large head poses[M]. Springer International Publishing

    Google Scholar 

  22. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database[J]. procinternworkshop on emotion corpora for research on emotion & affect

  23. Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions[J]. IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS PART B-CYBERNETICS

    Book  Google Scholar 

  24. Walecki R, Rudovic O, Pavlovic V et al (2017) Deep Structured Learning for Facial Action Unit Intensity Estimation[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    Book  Google Scholar 

  25. Wang C, Zeng J, Shan S et al (2019) Multi-task learning of emotion recognition and facial action unit detection with adaptively weights sharing network[C]. IEEE International Conference on Image Processing (ICIP) 2019:56–60

    Google Scholar 

  26. Wang Z, Li Y, Wang S et al (2013) Capturing global semantic relationships for facial action unit recognition[C]. IEEE International Conference on Computer Vision

    Book  Google Scholar 

  27. Wongpatikaseree K, Hnoohom N, Yuenyong S et al (2021) Facial Action Units Recognition using Deep Learning Model with Bottleneck Features[C]. 25th International Computer Science and Engineering Conference (ICSEC), pp 318–323

    Google Scholar 

  28. Yan J, Wang J, Li Q, et al. Weakly supervised regional and temporal learning for facial action unit recognition[J], 2022.

    Google Scholar 

  29. Zeng N, Zhang H, Song B et al (2018) Facial expression recognition via learning deep sparse autoencoders[J]. Neurocomputing 273:643–649

    Article  Google Scholar 

  30. Zeng Z, Pantic M, Roisman GI et al (2009) A survey of affect recognition methods[J]. IEEE Trans Pattern Anal Mach Intell

  31. Zhang KP, Zhang ZP, Li ZF et al (2016) Joint face detection and alignment using multitask cascaded convolutional networks[J]. Ieee Signal Processing Letters 23(10):1499–1503

    Article  Google Scholar 

  32. Zhang S, Zhu X, Lei Z et al (2017) FaceBoxes: a CPU real-time face detector with high accuracy[C]. IEEE International Joint Conference on Biometrics (IJCB) 2017:1–9

    Google Scholar 

  33. Zhang Y, Dong W, Hu BG et al (2018) Weakly-supervised deep convolutional neural network learning for facial action unit intensity estimation[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition 2018:2314–2323

    Google Scholar 

  34. Zhao Z, Yang Z, Luo L et al (2016) ML-CNN: A novel deep learning based disease named entity recognition architecture[C]. IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp 794–794

    Google Scholar 

  35. Zhou Y, Pi J, Shi BE (2017) Pose-independent facial action unit intensity regression based on multi-task deep transfer learning[C]. IEEE International Conference on Automatic Face & Gesture Recognition

    Book  Google Scholar 

Download references

Acknowledgements

The authors wish to thank the editor-in-chief, associate editor and reviewers for their insightful comments and suggestions. This work was supported by National Key Technology Research and Development Program of China(2017YFB1402103-3), National Natural Science Foundation of China (61901363, 61901362) and Natural Science Foundation of Shaanxi province, China (2020JQ-648,2019JM-381, 2019JQ-729) and Key Laboratory Foundation of Shaanxi Education Department (20JS086).

Data availability statements

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Xi’an University of Technology, No.5 South Jinhua Road, Xi’an, 710048, Shaanxi, China

    Minghua Zhao, Yuxing Zhi, Fei Yuan, Junhuai Li, Jing Hu, Shuangli Du & Zhenghao Shi

  2. Shaanxi Key Laboratory of Network Computing and Security Technology, No.5 South Jinhua Road, Xi’an, 710048, Shaanxi, China

    Minghua Zhao

Authors
  1. Minghua Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuxing Zhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fei Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junhuai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jing Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shuangli Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhenghao Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Minghua Zhao.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, M., Zhi, Y., Yuan, F. et al. Deep-block network for AU recognition and expression migration. Multimed Tools Appl 82, 25733–25746 (2023). https://doi.org/10.1007/s11042-023-14527-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14527-6

Keywords

Search

Navigation