Skip to main content
SpringerLink
Log in

Meaningful Learning for Deep Facial Emotional Features

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Facial expression is an important aspect to recognize emotions between humans. However, this task remains difficult for machines. Several approaches have been developed aiming at strengthening the machine and endowing it, with the ability to decipher and read people’s emotions from their faces in order to interact more intelligently. In this context, many deep learning (DL) approaches have been applied due to their outstanding recognition Accuracy. Aiming to gain better performance for facial expression recognition (FER) systems, we propose a hybrid DL architecture based on convolutional neural network and Stacked AutoEncoder. The main idea of this work is to combine the two feature vectors generated by each of these architectures and feed the resulting vector to a meaningful neural network. The architecture of the latter leads to learn each feature by dedicating a set of neurons for each component of the vector before combining them all together in the last layer. The publicly available four facial expression Datasets: Japanse Female Facial Expression (JAFFE), Extended Chon-Kanade (CK+), Facial Expression Recognition 2013 (FER2013) and AffectNet, were used during this research for both training and testing. The experimental results of our proposed architecture are comparable to or better than the relevant state-of-the-art methods in term of Accuracy, Recall, Precision and F-measure. We noted that our proposed approach obtains the best accuracy of 98.65% on the CK+, 95.78% on the JAFFE, 63.14% on the AffectNet and 80.02% on the FER2013 Datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Bloch H, Chemama R, Gallo A, et al (1992) Grand dictionnaire de la psychologie

  2. Ekman P, Dalgleish T, Power M (1999) Handbook of cognition and emotion. Wiley, Chihester

    Google Scholar 

  3. Cigna M-H, Guay J-P, Renaud P (2015) La reconnaissance émotionnelle faciale: validation préliminaire de stimuli virtuels dynamiques et comparaison avec les Pictures of Facial Affect (POFA). Criminologie 48:237–263

    Article  Google Scholar 

  4. Bhowmik MK, Saha K, Majumder S et al (2011) Thermal infrared face recognition—a biometric identification technique for robust security system. In: Reviews, Refinements and New Ideas in Face Recognition, vol 7, pp 113–138

    Google Scholar 

  5. Bartlett MS, Littlewort G, Fasel I, Movellan JR (2003) Real time face detection and facial expression recognition: development and applications to human computer interaction. In: 2003 Conference on computer vision and pattern recognition workshop. IEEE, p 53

  6. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816

    Article  Google Scholar 

  7. Michel P, El Kaliouby R (2003) Real time facial expression recognition in video using support vector machines. In: Proceedings of the 5th international conference on Multimodal interfaces, pp 258–264

  8. Usman M, Latif S, Qadir J (2017) Using deep autoencoders for facial expression recognition. In: 2017 13th International conference on emerging technologies (ICET). IEEE, pp 1–6

  9. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV). IEEE, pp 1–10

  10. Kuo C-M, Lai S-H, Sarkis M (2018) A compact deep learning model for robust facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 2121–2129

  11. Kahou SE, Bouthillier X, Lamblin P et al (2016) Emonets: multimodal deep learning approaches for emotion recognition in video. J Multimodal User Interfaces 10:99–111

    Article  Google Scholar 

  12. Viola P, Jones M (2001) Robust real-time face detection. In: Proceedings eighth IEEE international conference on computer vision. ICCV 2001. IEEE, p 747

  13. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3:71–86

    Article  Google Scholar 

  14. Lee S-J, Jung S-B, Kwon J-W, Hong S-H (1999) Face detection and recognition using PCA. In: Proceedings of IEEE. IEEE region 10 conference. TENCON 99. ‘Multimedia technology for Asia-Pacific information infrastructure’ (Cat. No. 99CH37030). IEEE, pp 84–87

  15. Sun X, Wu P, Hoi SCH (2018) Face detection using deep learning: An improved faster RCNN approach. Neurocomputing 299:42–50

    Article  Google Scholar 

  16. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23:1499–1503

    Article  Google Scholar 

  17. Chen J, Shan S, Yang P, et al (2004) Novel face detection method based on gabor features. In: Chinese conference on biometric recognition. Springer, pp 90–99

  18. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recognit 29:51–59

    Article  Google Scholar 

  19. Haar A (1909) Zur theorie der orthogonalen funktionensysteme. Georg-August-Universitat, Gottingen

    MATH  Google Scholar 

  20. Dailey MN, Cottrell GW, Padgett C, Adolphs R (2002) EMPATH: A neural network that categorizes facial expressions. J Cogn Neurosci 14:1158–1173

    Article  Google Scholar 

  21. Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41:3383–3390

    Article  Google Scholar 

  22. Campos V, Jou B, Giro-i-Nieto X (2017) From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction. Image Vis Comput 65:15–22

    Article  Google Scholar 

  23. Mannepalli K, Sastry PN, Suman M (2017) A novel adaptive fractional deep belief networks for speaker emotion recognition. Alex Eng J 56:485–497

    Article  Google Scholar 

  24. Chai X, Wang Q, Zhao Y et al (2016) Unsupervised domain adaptation techniques based on auto-encoder for non-stationary EEG-based emotion recognition. Comput Biol Med 79:205–214

    Article  Google Scholar 

  25. Anderson K, McOwan PW (2006) A real-time automated system for the recognition of human facial expressions. IEEE Trans Syst Man Cybern Part B 36:96–105

    Article  Google Scholar 

  26. Wang X, Jin C, Liu W, et al (2013) Feature fusion of HOG and WLD for facial expression recognition. In: Proceedings of the 2013 IEEE/SICE international symposium on system integration. IEEE, pp 227–232

  27. Li J, Oussalah M (2010) Automatic face emotion recognition system. In: 2010 IEEE 9th international conference on cyberntic intelligent systems. IEEE, pp 1–6

  28. Liu Y, Li Y, Ma X, Song R (2017) Facial expression recognition with fusion features extracted from salient facial areas. Sensors 17:712

    Article  Google Scholar 

  29. Xing Y, Luo W (2016) Facial expression recognition using local Gabor features and adaboost classifiers. In: 2016 international conference on progress in informatics and computing (pic). IEEE, pp 228–232

  30. Gupta O, Raviv D, Raskar R (2017) Multi-velocity neural networks for facial expression recognition in videos. IEEE Trans Affect Comput 10:290–296

    Article  Google Scholar 

  31. Cai J, Meng Z, Khan AS, et al (2019) Identity-free facial expression recognition using conditional generative adversarial network. arXiv Preprint http://arxiv.org/abs/190308051

  32. Agrawal A, Mittal N (2020) Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis Comput 36:405–412

    Article  Google Scholar 

  33. Lopes AT, De Aguiar E, Oliveira-Santos T (2015) A facial expression recognition system using convolutional networks. In: 2015 28th SIBGRAPI conference on graphics, patterns and images. IEEE, pp 273–280

  34. Graham B (2015) Fractional Max-Pooling. arXiv. arXiv Preprint http://arxiv.org/abs/14126071

  35. Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp 445–450

  36. Jain N, Kumar S, Kumar A et al (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recognit Lett 115:101–106

    Article  Google Scholar 

  37. Chen L, Wu M, Wanjuan SU, Hirota K (2018) Multi-convolution neural networks-based deep learning model for emotion understanding. In: 2018 37th Chinese control conference (CCC). IEEE, pp 9545–9549

  38. Ruiz-Garcia A, Elshaw M, Altahhan A, Palade V (2018) A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots. Neural Comput Appl 29:359–373

    Article  Google Scholar 

  39. Ebrahimi Kahou S, Michalski V, Konda K, et al (2015) Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 467–474

  40. Wang Y, Li Y, Song Y, Rong X (2020) The influence of the activation function in a convolution neural network model of facial expression recognition. Appl Sci 10:1897

    Article  Google Scholar 

  41. Liu P, Han S, Meng Z, Tong Y (2014) Facial expression recognition via a boosted deep belief network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1805–1812

  42. Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions. arXiv Preprint http://arxiv.org/abs/170501842

  43. Cai J, Chang O, Tang X-L, et al (2018) Facial expression recognition method based on sparse batch normalization CNN. In: 2018 37th Chinese control conference (CCC). IEEE, pp 9608–9613

  44. Li W, Huang D, Li H, Wang Y (2018) Automatic 4D facial expression recognition using dynamic geometrical image network. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, pp 24–30

  45. Kim J-H, Kim B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285

    Article  Google Scholar 

  46. Jeong D, Kim B-G, Dong S-Y (2020) Deep joint spatiotemporal network (DJSTN) for efficient facial expression recognition. Sensors 20:1936

    Article  Google Scholar 

  47. Mohan K, Seal A, Krejcar O, Yazidi A (2020) Facial expression recognition using local gravitational force descriptor-based deep convolution neural networks. IEEE Trans Instrum Meas 70:1–12

    Article  Google Scholar 

  48. Mohan K, Seal A, Krejcar O, Yazidi A (2021) FER-net: facial expression recognition using deep neural net. Neural Comput Appl 70:1–12

    Google Scholar 

  49. Xi Z, Niu Y, Chen J et al (2021) Facial expression recognition of industrial internet of things by parallel neural networks combining texture features. IEEE Trans Ind Inform 17:2784–2793

    Article  Google Scholar 

  50. Ma F, Sun B, Li S (2021) Robust facial expression recognition with convolutional visual transformers. arXiv Preprint http://arxiv.org/abs/210316854

  51. Wang K, Peng X, Yang J, et al (2020) Suppressing uncertainties for large-scale facial expression recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6897–6906

  52. Wang K, Peng X, Yang J et al (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057–4069

    Article  Google Scholar 

  53. Lyons MJ, Akamatsu S, Kamachi M, et al (1998) The Japanese female facial expression (JAFFE) database. In: Proceedings of third international conference on automatic face and gesture recognition, pp 14–16

  54. Lucey P, Cohn JF, Kanade T, et al (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94–101

  55. Goodfellow IJ, Erhan D, Carrier PL, et al (2013) Challenges in representation learning: a report on three machine learning contests. In: International conference on neural information processing. Springer, pp 117–124

  56. Mollahosseini A, Hasani B, Mahoor MH (2017) AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput 10:18–31

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LISAC, Department of Computer Science, Faculty of Science Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez, Morocco

    Hajar Filali, Jamal Riffi, Ilyasse Aboussaleh, Adnane Mohamed Mahraz & Hamid Tairi

Authors
  1. Hajar Filali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jamal Riffi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ilyasse Aboussaleh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adnane Mohamed Mahraz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hamid Tairi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hajar Filali.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Filali, H., Riffi, J., Aboussaleh, I. et al. Meaningful Learning for Deep Facial Emotional Features. Neural Process Lett 54, 387–404 (2022). https://doi.org/10.1007/s11063-021-10636-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-021-10636-1

Keywords

Search

Navigation