Skip to main content

Advertisement

Springer Nature Link
Log in

Automatic facial expression localization and recognition across a large range of emotions

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Automatic facial expression recognition (AFER) has been shown to work well when restricted to subjects showing a limited range of 6-basic expressions (BE). Expression recognition in subjects showing a large range of 22-compound expressions (CE) is harder as it has been shown that CE and BE are partially similar which might lead to huge confusion in AFER. We present a discriminative system that predicts expression across a large range of emotions. We first build a fully automatic facial feature detector using Random Forest Regression Voting in a Constrained Local Models (RFRV-CLM) framework used to automatically detect facial points, and study the effect of CE on the accuracy of point localization task. Second, a set of expression recognizers is trained from the extracted features including shape, texture, and appearance, to analyze the effect of the CE on the facial features and subsequently on the performance of AFER. The performance was evaluated using the CE dataset of 22 emotions. The results show the system to be accurate and robust against a wide variety of expressions. Evaluation of point localization and expression recognition against ground truth data was obtained and compared with the existing results of alternative approaches tested on the same data. The quantitative results with 55.6 recognition rates, 2.1% error rates using manual points, and 51.8 recognition rates, 2.1% error rates using automatic points demonstrated that our system was encouraging in comparison with the state-of-the-art systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Al-Garaawi, N.: Modelling of human ageing, compound emotions, and intensity for automatic facial expression recognition. Ph.D. thesis, University of Manchester (2019)

  2. Al-Garaawi, N., Morris, T., Cootes, T.F.: Fully automated age-weighted expression classification using real and apparent age. Pattern Analysis and Applications pp. 1–16 (2022)

  3. Al-Garaawi, N., Wu, Q., Morris, T.: Brief-based face descriptor: an application to automatic facial expression recognition (afer). Signal, Image Video Process. 15(2), 371–379 (2021)

    Article  Google Scholar 

  4. Appasaheb Borgalli, R., Surve, S.: Learning framework for compound facial emotion recognition. Recent Adv. Electr. Eng. (Former Recent Patents Electr. Electr. Eng.) 16(6), 664–676 (2023)

    Google Scholar 

  5. Benitez-Quiroz, C.F., Srinivasan, R., Feng, Q., Wang, Y., Martinez, A.M.: Emotionet challenge: Recognition of facial expressions of emotion in the wild. arXiv preprint arXiv:1703.01210 (2017)

  6. Breiman, L.: Random for. Mach. learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  7. Bromiley, P., Adams, J., Cootes, T.: Localisation of vertebrae on dxa images using constrained local models with random forest regression voting. In: Recent Advances in Computational Methods and Clinical Applications for Spine Imaging, pp. 159–171. Springer (2015)

  8. Bromiley, P.A., Adams, J.E., Cootes, T.F.: Automatic localisation of vertebrae in dxa images using random forest regression voting. In: International Workshop on Computational Methods and Clinical Applications for Spine Imaging, pp. 38–51. Springer (2015)

  9. Bromiley, P.A., Kariki, E.P., Adams, J.E., Cootes, T.F.: Fully automatic localisation of vertebrae in ct images using random forest regression voting. In: International Workshop on Computational Methods and Clinical Applications for Spine Imaging, pp. 51–63. Springer (2016)

  10. Cohn, J.F., Kruez, T.S., Matthews, I., Yang, Y., Nguyen, M.H., Padilla, M.T., Zhou, F., De la Torre, F.: Detecting depression from facial actions and vocal prosody. In: Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009. 3rd International Conference on, pp. 1–7. IEEE (2009)

  11. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)

    Article  MATH  Google Scholar 

  12. Cootes, T.F., Ionita, M.C., Lindner, C., Sauer, P.: Robust and accurate shape model fitting using random forest regression voting. In: European Conference on Computer Vision, pp. 278–291. Springer (2012)

  13. Dodgson, N.A.: Variation and extrema of human interpupillary distance. In: Stereoscopic Displays and Virtual Reality Systems XI, vol. 5291, pp. 36–47. International Society for Optics and Photonics (2004)

  14. Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proceedings of the National Academy of Sciences 111(15), E1454–E1462 (2014)

  15. Hamsici, O.C., Martinez, A.M.: Active appearance models with rotation invariant kernels. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1003–1009. IEEE (2009)

  16. Hayale, W., Negi, P.S., Mahoor, M.H.: Deep siamese neural networks for facial expression recognition in the wild. IEEE Trans. Affect. Comput. 14(2), 1148–1158 (2021)

    Article  Google Scholar 

  17. Jiang, J., Wang, M., Xiao, B., Hu, J., Deng, W.: Joint recognition of basic and compound facial expressions by mining latent soft labels. Pattern Recognit. 148, 110173 (2024)

    Article  MATH  Google Scholar 

  18. Kamińska, D., Aktas, K., Rizhinashvili, D., Kuklyanov, D., Sham, A.H., Escalera, S., Nasrollahi, K., Moeslund, T.B., Anbarjafari, G.: Two-stage recognition and beyond for compound facial emotion recognition. Electronics 10(22), 2847 (2021)

    Article  Google Scholar 

  19. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), pp. 46–53. IEEE (2000)

  20. Li, X., Deng, W., Li, S., Li, Y.: Compound expression recognition in-the-wild with au-assisted meta multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5735–5744 (2023)

  21. Lindner, C., Bromiley, P.A., Ionita, M.C., Cootes, T.F.: Robust and accurate shape model matching using random forest regression-voting. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1862–1874 (2015)

    Article  Google Scholar 

  22. Lindner, C., Cootes, T.: Fully automatic cephalometric evaluation using random forest regression-voting. In: Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI) 2015–Grand Challenges in Dental X-ray Image Analysis–Automated Detection and Analysis for Diagnosis in Cephalometric X-ray Image. Citeseer (2015)

  23. Lindner, C., Thiagarajah, S., Wilkinson, J.M., Wallis, G.A., Cootes, T.F., arcOGEN Consortium, et al.: Fully automatic segmentation of the proximal femur using random forest regression voting. IEEE transactions on medical imaging 32(8), 1462–1472 (2013)

  24. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with gabor wavelets. In: Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, pp. 200–205. IEEE (1998)

  25. Martinez, B., Valstar, M.F.: Advances, challenges, and opportunities in automatic facial expression recognition. In: Advances in Face Detection and Facial Image Analysis, pp. 63–100. Springer (2016)

  26. Prkachin, K.M., Solomon, P.E.: The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain. Pain 139(2), 267–274 (2008)

    Article  MATH  Google Scholar 

  27. Rivera, S., Martinez, A.M.: Learning deformable shape manifolds. Pattern Recognit 45(4), 1792–1801 (2012)

    Article  MATH  Google Scholar 

  28. Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)

    Article  MATH  Google Scholar 

  29. Shahid, A.R., Yan, H.: Squeezexpnet: Dual-stage convolutional neural network for accurate facial expression recognition with attention mechanism. Knowle. -Based Syst. 269, 110451 (2023)

    Article  MATH  Google Scholar 

  30. Ullah, S., Xie, Y., Ou, J., Wang, Z., Tian, W.: A robust lightweight compound emotion recognition approach using depthwise separable cnn (2024)

  31. Vural, E., Cetin, M., Ercil, A., Littlewort, G., Bartlett, M., Movellan, J.: Drowsy driver detection through facial movement analysis. In: International Workshop on Human-Computer Interaction, pp. 6–18. Springer (2007)

  32. Whitehill, J., Bartlett, M., Movellan, J.: Automatic facial expression recognition for intelligent tutoring systems. In: Computer Vision and Pattern Recognition Workshops, 2008. CVPRW’08. IEEE Computer Society Conference on, pp. 1–6. IEEE (2008)

  33. Win, S.S.K., Siritanawan, P., Kotani, K.: Compound facial expressions image generation for complex emotions. Multimed. Tools Appl. 82(8), 11549–11588 (2023)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Education for Girls, Department of Computer Science, University of Kufa, Najaf, Iraq

    Nora Al-Garaawi

  2. Department of Computer Science, University of Manchester, Manchester, UK

    Tim Morris

  3. Centre for Imaging Sciences, The University of Manchester, Manchester, UK

    Tim F. Cootes

Authors
  1. Nora Al-Garaawi
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Tim Morris
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Tim F. Cootes
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nora Al-Garaawi.

Ethics declarations

Conflict of interest

The authors declare that they have no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Garaawi, N., Morris, T. & Cootes, T.F. Automatic facial expression localization and recognition across a large range of emotions. SIViP 19, 236 (2025). https://doi.org/10.1007/s11760-025-03822-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11760-025-03822-4

Keywords