Skip to main content
SpringerLink
Log in

Method of Transfer Deap Learning Convolutional Neural Networks for Automated Recognition Facial Expression Systems

  • Conference paper
  • First Online:
Lecture Notes in Computational Intelligence and Decision Making (ISDMCI 2021)

Part of the book series: Lecture Notes on Data Engineering and Communications Technologies ((LNDECT,volume 77))

Abstract

The analysis of automated solutions for recognition of human facial expression (FER) and emotion detection (ED) is based on Deep Learning (DL) of Convolutional Neural Networks (CNN). The need to develop human FER and ED systems for various platforms, both for stationary and mobile devices, is shown, which imposes additional re-strictions on the resource intensity of the DL CNN architectures used and the speed of their learning. It is proposed, in conditions of an insufficient amount of annotated data, to implement an approach to the recognition of the main motor units of facial activity (AU) based on transfer learning, which involves the use of public DL CNNs previously trained on the ImageNet set with adaptation to the problems being solved. Networks of the MobileNet family and networks of the DenseNet family were selected as the basic ones. The DL CNN model was developed to solve the FER and ED problem of a person and the training method of the proposed model was modified, which made it possible to reduce the training time and computing resources when solving the FER and ED problems of a person without losing the reliability of AU recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Almaev, T., Martinez, B., Valstar, M.: Learning to transfer: transferring latent task structures and its application to person-specific facial action unit detection. In: 2015 IEEE International Conference on Computer Vision (ICCV 2015), pp. 3774–3782 (2015). https://doi.org/10.1109/ICCV.2015.430

  2. Almaev, T., Valstar, M.: Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (2013). https://doi.org/10.1109/ACII.2013.65

  3. Arsirii, O., Antoshchuk, S., Babilunha, O., Manikaeva, O., Nikolenko, A.: Intellectual information technology of analysis of weakly-structured multi-dimensional data of sociological research. In: Lytvynenko, V., Babichev, S., Wójcik, W., Vynokurova, O., Vyshemyrskaya, S., Radetskaya, S. (eds.) Lecture Notes in Computational Intelligence and Decision Making, pp. 242–258. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-26474-1_18

  4. Baltrusaitis, T., Mahmoud, M., Robinson, P.: Cross-dataset learning and person-specific normalisation for automatic action unit detection. In: 2015 IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, vol. 6, pp. 1–6 (2015). https://doi.org/10.1109/FG.2015.7284869

  5. Bianco, S., Cadene, R., Celona, L., Napoletano, P.: Benchmark analysis of representative deep neural network architectures. IEEE Access 6, 64270–64277 (2018). https://doi.org/10.1109/ACCESS.2018.2877890

    Article  Google Scholar 

  6. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press (1978)

    Google Scholar 

  7. Ekman, P., Friesen, W.V., Hager, J.C.: Facial action coding system (facs). A Human Face (2002). https://ci.nii.ac.jp/naid/10025007347/en/

  8. Howard, A., et al.: Mobilenets: eficient convolutional neural networks for mobile vision applications (2017). https://arxiv.org/abs/1704.04861

  9. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269 (2017). https://doi.org/10.1109/CVPR.2017.243

  10. ImageNet: Imagenet overview https://image-net.org/about

  11. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 448–456. PMLR, Lille, France, 07–09 Jul 2015. http://proceedings.mlr.press/v37/ioffe15.html

  12. Jaiswal, S., Valstar, M.: Deep learning the dynamic appearance and shape of facial action units. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–8 (2016). https://doi.org/10.1109/WACV.2016.7477625

  13. Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection using sparse appearance descriptors in space-time video volumes. In: 2011 IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 314–321 (2011). https://doi.org/10.1109/FG.2011.5771416

  14. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1, 541–551 (1989)

    Article  Google Scholar 

  15. Li, S., Deng, W.: Deep facial expression recognition: asurvey. IEEE Trans. Affective Comput., 1 (2018). https://doi.org/10.1109/taffc.2020.2981446

  16. Li, W., Abtahi, F., Zhu, Z., Yin, L.: Eac-net: deep nets with enhancing and cropping for facial action unit detection. IEEE Trans. Pattern Anal. Mach. Intell. 40(11), 2583–2596 (2018). https://doi.org/10.1109/tpami.2018.2791608

    Article  Google Scholar 

  17. Li, Y., Song, Y., Luo, J.: Improving pairwise ranking for multi-label image classification. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1837–1845 (2017). https://doi.org/10.1109/CVPR.2017.199

  18. Lim, Y., Liao, Z., Petridis, S., Pantic, M.: Transfer learning for action unit recognition. ArXiv (2018) http://arxiv.org/abs/1807.07556v1

  19. Mavadati, S., Mahoor, H., Bartlett, K., Trinh, P., Cohn, J.: Disfa: a spontaneous facial action intensity database. 2013 IEEE Trans. Affective Comput. 4(2), 151–160 (2013). https://doi.org/10.1109/T-AFFC.2013.4

  20. Niu, X., Han, H., Yang, S., Huang, Y., Shan, S.: Local relationship learning with person-specific shape regularization for facial action unit detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11909–11918 (2019). https://doi.org/10.1109/CVPR.2019.01219

  21. Ntinou, I., Sanchez, E., Bulat, A., Valstar, M., Tzimiropoulos, G.: A transfer learning approach to heatmap regression for action unit intensity estimation (2020). https://arxiv.org/abs/2004.06657

  22. Samadiani, N., Huang, G., Cai, B., Luo, W., Chi, C., Xiang, Y., He, J.: A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors 19(8), 1863 (2019). https://doi.org/10.3390/s19081863

    Article  Google Scholar 

  23. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018). https://doi.org/10.1109/CVPR.2018.00474

  24. Shao, Z., Liu, Z., Cai, J., Ma, L.: Jaa-Net: joint facial action unit detection and face alignment via adaptive attention. Int. J. Comput. Vision, 1–20 (2020). https://doi.org/10.1007/s11263-020-01378-z

  25. Shao, Z., Liu, Z., Cai, J., Wu, Y., Ma, L.: Facial action unit detection using attention and relation learning. IEEE Trans. Affective Comput., 1 (2019). https://doi.org/10.1109/taffc.2019.2948635

  26. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1409.1556

  27. University of California: EMFACS-7: Emotional Facial Action Coding System. Unpublished manual (1983)

    Google Scholar 

  28. Valstar, M., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW 2006) pp. 149–149. IEEE (2006). https://doi.org/10.1109/CVPRW.2006.85

  29. Walecki, R., Rudovic, O., Pavlovic, V., Schuller, B., Pantic, M.: Deep structured learning for facial action unit intensity estimation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5709–5718 (2017). https://doi.org/10.1109/CVPR.2017.605

  30. Zhang, Z., Zhai, S., Yin, L.: Identity-based adversarial training of deep CNNs for facial action unit recognition. In: British Machine Vision Conference 2018. p. 226. BMVA Press (2018). http://www.bmva.org/bmvc/2018/contents/papers/0741.pdf

  31. Zhao, K., Chu, W.S., De la Torre, F., Cohn, J.F., Zhang, H.: Joint patch and multi-label learning for facial action unit detection. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2207–2216 (2015). https://doi.org/10.1109/CVPR.2015.7298833

  32. Zhou, Y., Pi, J., Shi, B.E.: Pose-independent facial action unit intensity regression based on multi-task deep transfer learning. In: 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017), pp. 872–877 (2017). https://doi.org/10.1109/FG.2017.112

Download references

Author information

Authors and Affiliations

  1. Odesa Polytechnic State University, Odesa, Ukraine

    Arsirii Olena, Denys Petrosiuk, Babilunha Oksana & Nikolenko Anatolii

Authors
  1. Arsirii Olena
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Denys Petrosiuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Babilunha Oksana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nikolenko Anatolii
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Physics, Kherson State University, Kherson, Ukraine

    Sergii Babichev

  2. Department of Informatics and Computer Science, Kherson National Technical University, Kherson, Ukraine

    Volodymyr Lytvynenko

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Olena, A., Petrosiuk, D., Oksana, B., Anatolii, N. (2022). Method of Transfer Deap Learning Convolutional Neural Networks for Automated Recognition Facial Expression Systems. In: Babichev, S., Lytvynenko, V. (eds) Lecture Notes in Computational Intelligence and Decision Making. ISDMCI 2021. Lecture Notes on Data Engineering and Communications Technologies, vol 77. Springer, Cham. https://doi.org/10.1007/978-3-030-82014-5_51

Download citation

Publish with us

Policies and ethics

Search

Navigation