Skip to main content
SpringerLink
Log in

Facial Emotions Classification Supported in an Ensemble Strategy

  • Conference paper
  • First Online:
Universal Access in Human-Computer Interaction. Novel Design Approaches and Technologies (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13308))

Included in the following conference series:

Abstract

Humans are prepared to comprehend each other’s emotions from subtle body movements or facial expressions, and from those, they change the way they deliver messages when communicating between them. Machines, user interfaces, or robots need to empower this ability, in a way to change the interaction from the traditional “human-computer interaction” to a “human-machine cooperation”, where the machine provides the “right” information and functionality, at the “right” time, and in the “right” way. This paper presents a framework for facial expression prediction supported in an ensemble of facial expression methods, being the main contribution the integration of outputs from different methods in a single prediction consistent with the expression presented by the system’s user. Results show a classification accuracy above 73% in both FER2013 and RAF-DB datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Deguchi, A., et al.: What is Society 5.0. Chapter 1 in Society 5.0 – A People-centric Super-smart Society. Hitachi-UTokyo Laboratory (eds.), pp. 1–23. Springer (2020). https://doi.org/10.1007/978-981-15-2989-4

  2. Rothfuß, S., Wörner, M., Inga, J., Hohmann, S.: A study on human-machine cooperation on decision level. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 2291–2298. IEEE (2020)

    Google Scholar 

  3. Kumar, P., Gupta, A.: Active learning query strategies for classification, regression, and clustering: a survey. J. Comput. Sci. Technol. 35(4), 913–945 (2020). https://doi.org/10.1007/s11390-020-9487-4

    Article  Google Scholar 

  4. Ardabili, S., Mosavi, A., Várkonyi-Kóczy, A.R.: Advances in machine learning modeling reviewing hybrid and ensemble methods. In: Várkonyi-Kóczy, A.R. (ed.) INTER-ACADEMIA 2019. LNNS, vol. 101, pp. 215–227. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-36841-8_21

    Chapter  Google Scholar 

  5. Zhang, F., Zhang, T., Mao, Q., Xu, C.: Joint pose and expression modeling for facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3359–3368 (2018)

    Google Scholar 

  6. Noroozi, F., Kaminska, D., Corneanu, C., Sapinski, T., Escalera, S., Anbarjafari, G.: Survey on emotional body gesture recognition. IEEE Trans. Affect. Comput. 12(2), 505–523 (2018)

    Article  Google Scholar 

  7. Filntisis, P.P., Efthymiou, N., Koutras, P., Potamianos, G., Maragos, P.: Fusing body posture with facial expressions for joint recognition of affect in child–robot interaction. IEEE Robot. Autom. Lett. 4(4), 4011–4018 (2019)

    Article  Google Scholar 

  8. Leiva, S., Margulis, L., Micciulli, A., Ferreres, A.: Dissociation between facial and bodily expressions in emotion recognition: a case study. Clin. Neuropsychol. 33(1), 166–182 (2019)

    Article  Google Scholar 

  9. Canedo, D., Neves, A.J.: Mood estimation based on facial expressions and postures. In: Proceedings of the RECPAD 2020, pp. 49–50 (2020)

    Google Scholar 

  10. Bänziger, T., Mortillaro, M., Scherer, K.R.: Introducing the geneva multimodal expression corpus for experimental research on emotion perception. Emotion 12(5), 1161 (2012)

    Article  Google Scholar 

  11. Kleinsmith, A., BianchiBerthouze, N.: Affective body expression perception and recognition: a survey. IEEE Trans. Affect. Comput. 4(1), 15–33 (2012)

    Article  Google Scholar 

  12. Senecal, S., Cuel, L., Aristidou, A., Magnenat-Thalmann, N.: Continuous body emotion recognition system during theater performances. Comput. Animat. Virtual Worlds 27(3–4), 311–320 (2016)

    Article  Google Scholar 

  13. Ahmed, F., Bari, A.H., Gavrilova, M.L.: Emotion recognition from body movement. IEEE Access 8, 11761–11781 (2019)

    Article  Google Scholar 

  14. Liang, G., Wang, S., Wang, C.: Pose-aware adversarial domain adaptation for personalized facial expression recognition. arXiv preprint arXiv:2007.05932 (2020)

  15. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971). https://doi.org/10.1037/h0030377

    Article  Google Scholar 

  16. Zavaschi, T.H., Koerich, A.L., Oliveira, L.E.S.: Facial expression recognition using ensemble of classifiers. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1489–1492 (2011). https://doi.org/10.1109/ICASSP.2011.5946775

  17. Renda, A., Barsacchi, M., Bechini, A., Marcelloni, F.: Comparing ensemble strategies for deep learning: an application to facial expression recognition. Expert Syst. Appl. 136, 1–11 (2019)

    Article  Google Scholar 

  18. Ali, G., et al.: Artificial neural network based ensemble approach for multicultural facial expressions analysis. IEEE Access 8, 134950–134963 (2020)

    Article  Google Scholar 

  19. Wang, Z., Zeng, F., Liu, S., Zeng, B.: OAENet: oriented attention ensemble for accurate facial expression recognition. Pattern Recognit. 112, 107694 (2021)

    Google Scholar 

  20. Benamara, N.K., et al.: Real-time facial expression recognition using smoothed deep neural network ensemble. Integr. Comput.-Aid. Eng. (Preprint) 28, 1–15 (2021)

    Google Scholar 

  21. Pecoraro, R., Basile, V., Bono, V., Gallo, S.: Local multi-head channel self-attention for facial expression recognition. arXiv preprint arXiv:2111.07224 (2021)

  22. Goodfellow, I., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

    Chapter  Google Scholar 

  23. Cheong, J.H., Xie, T., Byrne, S., Chang, L.J.: Py-Feat: Python facial expression analysis toolbox. arXiv preprint arXiv:2104.03509 (2021)

  24. Banerjee, R., De, S., Dey, S.: A survey on various Deep Learning algorithms for an efficient facial expression recognition system. Int. J. Image Graph., 2240005 (2021)

    Google Scholar 

  25. Revina, I.M., Emmanuel, W.S.: A survey on human face expression recognition techniques. J. King Saud Univ.-Comput. Inf. Sci. 33(6), 619–628 (2021)

    Google Scholar 

  26. LHC-NET: Local multi-head channel self-attention (code) (2021). https://github.com/bodhis4ttva/lhc_net. Accessed 28 Dec 2021

  27. Py-FEAT: Python facial expression analysis toolbox (code) (2021). https://pythonrepo.com/repo/cosanlab-py-feat-python-deep-learning. Accessed 28 Dec 2021

  28. Shenk, J.: Facial expression recognition (code) (2021). https://github.com/justinshenk/fer. Accessed 28 Dec 2021

  29. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  30. Hastie, T., Rosset, S., Zhu, J., Zou, H.: Multi-class Adaboost. Statistics and its. Interface 2(3), 349–360 (2009)

    MathSciNet  MATH  Google Scholar 

  31. Ayyadevara, V.K.: Pro Machine Learning Algorithms. Apress, Berkeley (2018)

    Book  Google Scholar 

  32. FER2013: Learn facial expressions from an image (2021). https://www.kaggle.com/msambare/fer2013. Accessed 28 Dec 2021

  33. RAF-DB: Real-world affective faces database (2021). http://www.whdeng.cn/raf/model1.html. Accessed 28 Dec 2021

  34. OpenCV: OpenCV: Cascade classifier – face detection (2021). https://docs.opencv.org/4.5.5/db/d28/tutorial_cascade_classifier.html. Accessed 28 Dec 2021

  35. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    Google Scholar 

  36. Li, S., Deng, W.: Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans. Image Process. 28(1), 356–370 (2019)

    Article  MathSciNet  Google Scholar 

  37. Cheong, J.H., Xie, T., Byrne, S., Chang, L.J.: Py-Feat: Python facial expression analysis toolbox. arXiv preprint arXiv:2104.03509 (2021)

Download references

Acknowledgements

This work was supported by the Portuguese Foundation for Science and Technology (FCT), project LARSyS - FCT Project UIDB/50009/2020.

Author information

Authors and Affiliations

  1. LARSyS & ISE, Universidade do Algarve, Faro, Portugal

    Rui Novais, Pedro J. S. Cardoso & João M. F. Rodrigues

Authors
  1. Rui Novais
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pedro J. S. Cardoso
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. João M. F. Rodrigues
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to João M. F. Rodrigues .

Editor information

Editors and Affiliations

  1. Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  2. University of Crete and Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

Appendix

Appendix

This appendix presents the parameters (Tables 3 and 4) used for Random Forest, AdaBoost and MLP/Neural Network for the results presented in Sect. 4.

Table 3. Grid search parameters (although the majority of the naming of the parameters is self-explicative, we suggest that the readers refer to the library’s documentation [35] for a more detailed explanation).
Full size table
Table 4. Sets of parameters used to obtain the results for the different models (tuned using grid search stratified cross-validation).
Full size table

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Novais, R., Cardoso, P.J.S., Rodrigues, J.M.F. (2022). Facial Emotions Classification Supported in an Ensemble Strategy. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. Novel Design Approaches and Technologies. HCII 2022. Lecture Notes in Computer Science, vol 13308. Springer, Cham. https://doi.org/10.1007/978-3-031-05028-2_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05028-2_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05027-5

  • Online ISBN: 978-3-031-05028-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation