Skip to main content
Springer Nature Link
Log in

Classification of Facial Expressions Under Partial Occlusion for VR Games

  • Conference paper
  • First Online:
Optimization, Learning Algorithms and Applications (OL2A 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1754))

Abstract

Facial expressions are one of the most common way to ex- ternalize our emotions. However, the same emotion can have different effects on the same person and has different effects on different people. Based on this, we developed a system capable of detecting the facial expressions of a person in real-time, occluding the eyes (simulating the use of virtual reality glasses). To estimate the position of the eyes, in or- der to occlude them, Multi-task Cascade Convolutional Neural Networks (MTCNN) were used. A residual network, a VGG, and the combination of both models, were used to perform the classification of 7 different types of facial expressions (Angry, Disgust, Fear, Happy, Sad, Surprise, Neutral), classifying the occluded and non-occluded dataset. The combi- nation of both models, achieved an accuracy of 64.9% for the occlusion dataset and 62.8% for no occlusion, using the FER-2013 dataset. The primary goal of this work was to evaluate the influence of occlusion, and the results show that the majority of the classification is done with the mouth and chin. Nevertheless, the results were far from the state-of-the- art, which is expect to be improved, mainly by adjusting the MTCNN.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aifanti, N., Papachristou, C., Delopoulos, A.: The MUG facial expression database. In: 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pp. 1–4. IEEE (2010)

    Google Scholar 

  2. Almeida, J., Rodrigues, F.: Facial expression recognition system for stress detection with deep learning. In: Proceedings of the 23rd International Conference on Enterprise Information Systems, pp. 256–263. SCITEPRESS - Science and Technology Publications, Online Streaming, – Select a Country – (2021). https://doi.org/10.5220/0010474202560263. https://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0010474202560263

  3. Baltrusaitis, T., Robinson, P., Morency, L.P.: OpenFace: an open source facial behavior analysis toolkit. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA, pp. 1–10. IEEE, March 2016. https://doi.org/10.1109/WACV.2016.7477553. https://ieeexplore.ieee.org/document/7477553/

  4. Barsoum, E., Zhang, C., Ferrer, C.C., Zhang, Z.: Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan, pp. 279–283. ACM, October 2016. https://doi.org/10.1145/2993148.2993165. https://dl.acm.org/doi/10.1145/2993148.2993165

  5. Bartlett, M., Littlewort, G., Lainscsek, C., Fasel, I., Movellan, J.: Machine learning methods for fully automatic recognition of facial expressions and facial actions. In: 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), The Hague, Netherlands, vol. 1, pp. 592–597. IEEE (2004). https://doi.org/10.1109/ICSMC.2004.1398364. https://ieeexplore.ieee.org/document/1398364/

  6. Cheng, Y., Jiang, B., Jia, K.: A deep structure for facial expression recognition under partial occlusion. In: 2014 Tenth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pp. 211–214 (2014). https://doi.org/10.1109/IIH-MSP.2014.59

  7. Devries, T., Biswaranjan, K., Taylor, G.W.: Multi-task Learning of Facial Landmarks and Expression. In: 2014 Canadian Conference on Computer and Robot Vision, Montreal, QC, Canada, pp. 98–103. IEEE, May 2014. https://doi.org/10.1109/CRV.2014.21. https://ieeexplore.ieee.org/document/6816830/

  8. Donaldson, M.: Plutchik’s wheel of emotions-2017. Update (2017)

    Google Scholar 

  9. Cheng, F., Yu, J., Xiong, H.: Facial expression recognition in JAFFE dataset based on gaussian process classification. IEEE Trans. Neural Netw. 21(10), 1685–1690 (2010)

    Google Scholar 

  10. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. Neural Netw. 64, 59–63 (2015)

    Google Scholar 

  11. Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-PIE. Image Vis. Comput. 28(5), 807–813 (2010)

    Google Scholar 

  12. Houshmand, B., Mefraz Khan, N.: Facial expression recognition under partial occlusion from virtual reality headsets based on transfer learning. In: 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM), New Delhi, India, pp. 70–75. IEEE, September 2020. https://doi.org/10.1109/BigMM50055.2020.00020. https://ieeexplore.ieee.org/document/9232653/

  13. Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), pp. 46–53. IEEE Comput. Soc, Grenoble, France (2000). https://doi.org/10.1109/AFGR.2000.840611. https://ieeexplore.ieee.org/document/840611/

  14. Li, R., et al.: MindLink-Eumpy: an open-source Python toolbox for multimodal emotion recognition. Front. Hum. Neurosci. 15, 621493 (2021)

    Google Scholar 

  15. Loizou, C.P.: An automated integrated speech and face imageanalysis system for the identification of human emotions. Speech Commun. 130, 15–26 (2021)

    Google Scholar 

  16. Lopes, J.C., Lopes, R.P.: A review of dynamic difficulty adjustment methods for serious games. In: Pereira, A.I., et al. (eds.) OL2A 2022, CCIS 1754, pp. xx–yy (2022)

    Google Scholar 

  17. Lopes, R.P., et al.: Digital technologies for innovative mental health rehabilitation. Electronics 10(18) (2021). https://doi.org/10.3390/electronics10182260. https://www.mdpi.com/2079-9292/10/18/2260

  18. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE (2010)

    Google Scholar 

  19. Mehrabian, A.: Communication without words. In: Mortensen, C.D. (ed.) Communication Theory, Routledge, 2 edn., pp. 193–200, September 2017. https://doi.org/10.4324/9781315080918-15. https://www.taylorfrancis.com/books/9781351527538/chapters/10.4324/9781315080918-15

  20. Poria, S., Majumder, N., Mihalcea, R., Hovy, E.: Emotion recognition in conversation: research challenges, datasets, and recent advances. IEEE Access 7, 100943–100953 (2019)

    Google Scholar 

  21. Prodger, P.: Darwin’s Camera: Art and Photography in the Theory of Evolution. Oxford University Press, Oxford (2009)

    Google Scholar 

  22. Ramirez Cornejo, J.Y., Pedrini, H.: Emotion recognition from occluded facial expressions using weber local descriptor. In: 2018 25th International Conference on Systems, Signals and Image Processing (IWSSIP), Maribor, Slovenia, pp. 1–5. IEEE, June 2018. https://doi.org/10.1109/IWSSIP.2018.8439631. https://ieeexplore.ieee.org/document/8439631/

  23. Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: 2013 IEEE International Conference on Computer Vision Workshops, Sydney, Australia, pp. 397–403. IEEE, December 2013. https://doi.org/10.1109/ICCVW.2013.59. https://ieeexplore.ieee.org/document/6755925/

  24. Saurav, S., Saini, A., Saini, R., Singh, S.: Deep learning inspired intelligent embedded system for haptic rendering of facial emotions to the blind. Neural Comput. Appl. 34(6), 4595–4623 (2022). https://doi.org/10.1007/s00521-021-06613-3. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85117830803 &doi=10.1007%2fs00521-021-06613-3 &partnerID=40 &md5=ff31d4bd7bc4190b4483b813b2837a34, publisher: Springer Science and Business Media Deutschland GmbH

  25. Singh, S., Gupta, A., Pavithr, R.S.: Automatic classroom monitoring system using facial expression recognition. In: Sanyal, G., Travieso-González, C.M., Awasthi, S., Pinto, C.M.A., Purushothama, B.R. (eds.) International Conference on Artificial Intelligence and Sustainable Engineering. LNEE, vol. 836, pp. 151–165. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-8542-2_12. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85130262593 &doi=10.1007%2f978-981-16-8542-2_12 &partnerID=40 &md5=3d727b02a4b6cbca64032f67f9156366. ISBN: 9789811685415

  26. Susskind, J.M., Anderson, A.K., Hinton, G.E.: The Toronto face database. Department of Computer Science, University of Toronto, Toronto, ON, Canada, Technical report 3 (2010)

    Google Scholar 

  27. Tammina, S.: Transfer learning using VGG-16 with deep convolutional neural network for classifying images. Int. J. Sci. Res. Publ. (IJSRP) 9(10), 9420 (2019). https://doi.org/10.29322/IJSRP.9.10.2019.p9420. https://www.ijsrp.org/research-paper-1019.php?rp=P949194

  28. Tang, Y.: Deep learning using linear support vector machines, February 2015. arXiv:1306.0239 [cs, stat]

  29. Viana, I.: Comunicação não verbal e expressões faciais das emoções básicas. Revista de Letras 13(II), 165–181 (2014)

    Google Scholar 

  30. Wood, E., Baltruaitis, T., Zhang, X., Sugano, Y., Robinson, P., Bulling, A.: Rendering of eyes for eye-shape registration and gaze estimation. In: 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, pp. 3756–3764. IEEE, December 2015. https://doi.org/10.1109/ICCV.2015.428. https://ieeexplore.ieee.org/document/7410785/

  31. Xiang, J., Zhu, G.: Joint face detection and facial expression recognition with MTCNN. In: 2017 4th International Conference on Information Science and Control Engineering (ICISCE), pp. 424–427 (2017). https://doi.org/10.1109/ICISCE.2017.95

  32. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, Washington, USA, pp. 435–442. ACM, November 2015. https://doi.org/10.1145/2818346.2830595. https://dl.acm.org/doi/10.1145/2818346.2830595

  33. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Google Scholar 

  34. Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, pp. 4511–4520. IEEE, June 2015. https://doi.org/10.1109/CVPR.2015.7299081. https://ieeexplore.ieee.org/document/7299081/

  35. Zhao, X., Zhang, S.: A review on facial expression recognition: feature extraction and classification. IETE Techn. Rev. 33(5), 505–517 (2016)

    Google Scholar 

Download references

Acknowledgment

This work is funded by the European Regional Development Fund (ERDF) through the Regional Operational Program North 2020, within the scope of Project GreenHealth - Digital strategies in biological assets to improve well-being and promote green health, Norte-01-0145-FEDER-000042. This work has been supported by FCT - Fundação para a Ciência e Tecnologia within the Project Scope: UIDB/05757/2020.

Author information

Authors and Affiliations

  1. Research Center in Digitalization and Intelligent Robotics (CeDRI), Instituto Politécnico de Bragança, Bragança, Portugal

    Ana Sofia Figueiredo Rodrigues, Júlio Castro Lopes & Rui Pedro Lopes

  2. Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, Aveiro, Portugal

    Rui Pedro Lopes

  3. INESC TEC, Faculdade de Engenharia da Universidade do Porto, Porto, Portugal

    Luís F. Teixeira

Authors
  1. Ana Sofia Figueiredo Rodrigues
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Júlio Castro Lopes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rui Pedro Lopes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Luís F. Teixeira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rui Pedro Lopes .

Editor information

Editors and Affiliations

  1. Instituto Politécnico de Bragança, Bragança, Portugal

    Ana I. Pereira

  2. University of Ljubljana, Ljubljana, Slovenia

    Andrej Košir

  3. Instituto Politécnico de Bragança, Bragança, Portugal

    Florbela P. Fernandes

  4. Instituto Politécnico de Bragança, Bragança, Portugal

    Maria F. Pacheco

  5. Instituto Politécnico de Bragança, Bragança, Portugal

    João P. Teixeira

  6. Instituto Politécnico de Bragança, Bragança, Portugal

    Rui P. Lopes

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rodrigues, A.S.F., Lopes, J.C., Lopes, R.P., Teixeira, L.F. (2022). Classification of Facial Expressions Under Partial Occlusion for VR Games. In: Pereira, A.I., Košir, A., Fernandes, F.P., Pacheco, M.F., Teixeira, J.P., Lopes, R.P. (eds) Optimization, Learning Algorithms and Applications. OL2A 2022. Communications in Computer and Information Science, vol 1754. Springer, Cham. https://doi.org/10.1007/978-3-031-23236-7_55

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23236-7_55

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23235-0

  • Online ISBN: 978-3-031-23236-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics