Skip to main content
SpringerLink
Log in

Improvement of emotion recognition from facial images using deep learning and early stopping cross validation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Inthis paper, we present a new approach for emotion recognition from facial images. The proposed method is based on the association of a pretrained convolutional neural network (CNN) model (VGG16, ResNet50) with a multilayer perceptron (MLP) classifier. The pretrained CNN model is used as a feature extractor. For this purpose, we adapt the original architecture by adding a global average pooling layer (GAP) without any fine tuning of the network parameters. In order to avoid overfitting for the MLP classifier, we introduce the early stopping criterion. It is proved that the aforementioned elements contribute in improving the performance of our approach in terms of generalization ability. The procedure for emotion recognition from facial images is applied on the CK+ (extended Cohen-Kanad), JAFFE (Japanese Female Facial Expression) and KDEF (Karolinska Directed Emotional Faces) databases. The k-fold cross validation procedure is used for accuracy estimation. The experimental results show the effectiveness of our facial emotion recognition (FER) approach compared to the existing methods yielding to recognition rates of 100%, 96.40% and 98.78% for the CK+, JAFFE and KDEF databases, respectively. On the other hand, further improvement of our recognition performance is obtained for images from the JAFFE database by performing a data augmentation during the training phase. This allows to achieve an accuracy of 100% for this database. Four other metrics, namely the F1-score, positive and negative likelihood ratios and Mathews correlation coefficient, confirm as well the classification results obtained from accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Al Taee EJ, Jasim QM (2020) Blurred facial expression recognition system by using convolution neural network. Webology, 17(2)

  2. Bettadapura V (2012) Face expression recognition and analysis: the state of the art. arXiv:1203.6722

  3. Cai J, Chang O, Tang X, Xue C et al (2018) Wei: facial expression recognition method based on sparse batch normalization CNN. In: Proceedings of 37th Chinese control conference. Wuhan

  4. Caleanu CD (2013) Fase expression recognition: a brief overview of the last decade. In: 2013 IEEE 8th International symposium applied computational intelligence and informatics (SACI), pp 157–161

  5. Chicco D, Jurman G (2020) The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21. https://doi.org/10.1186/s12864-019-6413-7

  6. Christopher M (2006) Bishop: pattern recognition and machine learning. Springer

  7. Daniel B (2018) Cross-Validation, https://doi.org/10.1016/B978-0-12-809633-8.20349-X

  8. Dimitrievska V, Ackovska N (2020) Behavior models of emotion-featured robots: a survey. J Intell Robot Syst 100:1031–1053. https://doi.org/10.1007/s10846-020-01219-8

    Article  Google Scholar 

  9. Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  10. Eng SK, Ali H, Cheah AY, Chong YF (2019) Facial expression recognition in JAFFE and KDEF Datasets using histogram of oriented gradients and support vector machine. In: 5th International conference on man machine systems, IOP conf. series: materials science and engineering, vol 705, p 012031

  11. Furlán F, Rubio E, Sossa H, Ponce V (2020) CNN based detectors on planetary environments: a performance evaluation. Frontiers Neurorob 14:85

    Article  Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. arXiv:1512.03385v1

  13. Hinz T, Barros P, Wermter S (2016) The eects of regularization on learning facial expressions with convolutional neural networks. In: Proceedings of the 25th international conference on articial neural networks (ICANN), Part II, LNCS 9887. Barcelona, pp 80–87

  14. Hung JC, Chang J-W (2021) Multi-level transfer learning for improving the performance of deep neural networks: theory and practice from the tasks of facial emotion recognition and named entity recognition. Appl Soft Comput 109:107491. ISSN 1568-4946, https://doi.org/10.1016/j.asoc.2021.107491

    Article  Google Scholar 

  15. Iwendi C, Srivastava G, Khan SS et al (2020) Cyberbullying detection solutions based on deep learning architectures. Multimedia Systems. https://doi.org/10.1007/s00530-020-00701-5

  16. Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recogn Lett 115:101–106

    Article  Google Scholar 

  17. Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recogn Lett 120:69–74

    Article  Google Scholar 

  18. Kar NB, Babu KS, Sangaiah AK, Bakshi S (2019) Face expression recognition system based on ripplet transform type II and least square SVM. Multimed Tools Applic 78:4789–4812

    Article  Google Scholar 

  19. Kaur J, Sharma A (2017) Performance analysis of face detection by using Viola-Jones algorithm. Int J Comput Intell Res ISSN 0973-1873 13 (5):707–717

    Google Scholar 

  20. Kim J-H, Kim B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285. https://doi.org/10.1109/ACCESS.2019.2907327

    Article  Google Scholar 

  21. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR). arXiv:1412.6980v9

  22. Kwong JCT, Garcia FCC, Abu PAR, Reyes RSJ (2018) Emotion recognition via facial expression: utilization of numerous feature descriptors in different machine learning algorithms. In: Proceedings of TENCON 2018 - 2018 IEEE Region 10 conference. Jeju, pp 28–31

  23. Lauer F, Bentoumi M, Bloch G, Millérioux G, Aknin P (2004) Ho-Kashyap with early stopping vs soft margin SVM for linear classifiers - an application, lecture notes in computer science, vol 3173/2004. Springer, Heidelberg, pp 524–530

    Google Scholar 

  24. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–44. https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  25. Lekdioui K, Messoussi R, Ruichek Y, Chaabi Y, Touahni R (2017) Facial decomposition for expression recognition using texture/shape descriptors and svm classifer. Signal Processing: Image Communication. https://doi.org/10.1016/j.image.2017.08.001

  26. Li S, Deng W (2018) Deep facial expression recognition: a survey. arXiv:1804.08348v2, 22

  27. Li J, Jin K, Zhou D, Kubota N, Ju Z (2020) Attention mechanism-based CNN for facial expression recognition. Neurocomputing 411:340–350

    Article  Google Scholar 

  28. Liliana DY (2019) Emotion recognition from facial expression using deep convolutional neural network. IOP Conf. Series: Journal of Physics: Conf., Series 1193. IOP Publishing, pp 012004. https://doi.org/10.1088/1742-6596/1193/1/012004

  29. Lima E, Sun X, Dong J, Wang H, Yang Y, Liu L (2017) Learning and transferring convolutional neural network knowledge to ocean front recognition. IEEE Geosci Remote Sens Lett 14(3):354–358

    Article  Google Scholar 

  30. Lin M, Chen Q, Yan S (2014) Network in network. arXiv:1312.4400v3

  31. Liu Y, Li Y, Ma X, Song R (2017) Facial expression recognition with fusion features extracted from salient facial areas. Sensors 17:712. https://doi.org/10.3390/s17040712

    Article  Google Scholar 

  32. Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T et al (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628. https://doi.org/10.1016/j.patcog.2016.07.026

    Article  Google Scholar 

  33. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer society conference on computer vision and pattern recognition - workshops. Juin, pp 94–101. https://doi.org/10.1109/CVPRW.2010.5543262

  34. Lundqvist D, Flykt A, Ohman A (1998) The Karolinska directed emotional faces – KDEF, CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet ISBN 91-630-7164-9

  35. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings third IEEE international conference on automatic face and gesture recognition. IEEE, pp 200–205

  36. Mavani V, Raman S, Miyapuram KP (2017) Facial expression recognition using visual saliency and deep learning. IEEE International conference on computer vision workshops (ICCVW)

  37. Mohammadpour M, Khaliliardali H, Hashemi SMR, AlyanNezhadi MN et al (2017) facial emotion recognition using deep convolutional networks. In: 2017 IEEE 4th International conference on knowledge-based engineering and innovation (KBEI)

  38. Nwankpa CE, Ijomah W, Gachagan A, Marshall S (2018) Activation functions: comparison of trends in practice and research for deep learning. arXiv:1811.03378v1

  39. Oquab M, Bottou L, Laptev I, Sivic J (2014) Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1717–1724

  40. Pandey S (2014) Review and comparison of face detection algorithms. Int J Comput Sci Inf Technol 5(3):4111–4117

    Google Scholar 

  41. Punyani P, Gupta R, Kumar A (2020) Neural networks for facial age estimation: a survey on recent advances. Artif Intell Rev 53:3299–3347. https://doi.org/10.1007/s10462-019-09765-w

    Article  Google Scholar 

  42. Ruder S (2017) An overview of gradient descent optimization algorithms. arXiv:1609.04747v2

  43. Ruiz-Garcia A, Elshaw M, Altahhan A, Palade V (2018) A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots. Neural Comput Applic 29:359–373

    Article  Google Scholar 

  44. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, FeiFei L (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis (IJCV) 115 (3):211–252

    Article  MathSciNet  Google Scholar 

  45. Sabri M, Kurita T (2018) Improvement of feature localization for facial expressions by adding noise. Int J Affect Eng 17 (n∘ 1):27–37

    Article  Google Scholar 

  46. Said Y, Barr M (2021) Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-021-10918-9

  47. Sarvamangala DR, Kulkarni RV (2021) Convolutional neural networks in medical image understanding: a survey. Evolutionary Intelligence. https://doi.org/10.1007/s12065-020-00540-3

  48. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556v6

  49. Sokolova M, Japkowicz N, Szpakowicz S (2006) Beyond accuracy, F-Score and ROC: a family of discriminant measures for performance evaluation. In: Sattar A, Kang B (eds) AI 2006: advances in artificial intelligence lecture notes in computer science. https://doi.org/10.1007/11941439_114, vol 4304. Springer, Berlin

  50. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. Computer Vision and Pattern Recognition (CVPR). arXiv:1409.4842

  51. Tharwat A (2021) Classification assessment methods. Appl Comput Inform 17(1):168–192. https://doi.org/10.1016/j.aci.2018.08.003

    Article  Google Scholar 

  52. Ting KM (2011) Confusion matrix. In: Sammut C, Webb GI (eds) Encyclopedia of machine learning. https://doi.org/10.1007/978-0-387-30164-8_157. Springer, Boston

  53. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. IEEE Comput Soc Conf Comput Vis Pattern Recogn(CVPR’01) 1:511–518

    Google Scholar 

  54. Yang B, Li Z, Cao E (2020) Facial Expression Recognition Based on Multi-dataset. Neur Netw Radioeng 29 (n∘):1

    Google Scholar 

  55. Yao X, Wang X, Wang S-H, Zhang Y-D (2020) A comprehensive survey on convolutional neural network in medical image analysis. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-020-09634-7

  56. Ye Y, Zhang X, Lin Y, Wang H (2019) Facial expression recognition via region-based convolutional fusion network. J Vis Commun Image, R 62:1–11. https://doi.org/10.1016/j.jvcir.2019.04.009

    Article  Google Scholar 

  57. Yolcu G, Oztel I, Kazan S, Oz C, Palaniappan K, Lever TE, Bunyak F (2019) Facial expression recognition for monitoring neurological disorders based on convolutional neural network. Multimed Tools Applic 78:31581–31603

    Article  Google Scholar 

  58. Zhang T, Yu B (2005) Boosting with early stopping: convergence and consistency. The Annals of Statistics, Institute of Mathematical Statistics 33:1538–1579

    MathSciNet  MATH  Google Scholar 

  59. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2015) Learningdeep features for discriminative localization. arXiv:1512.04150v1

Download references

Author information

Authors and Affiliations

  1. Signals and Systems Laboratory, University Abdelhamid Ibn Badis of Mostaganem, Mostaganem, Algeria

    Mohamed Bentoumi, Mohamed Daoud & Mohamed Benaouali

  2. IEMN DOAE UMR CNRS 8520, Université Polytechniques des Hauts de France, Valenciennes, France

    Abdelmalik Taleb Ahmed

Authors
  1. Mohamed Bentoumi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Daoud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamed Benaouali
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Abdelmalik Taleb Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohamed Bentoumi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bentoumi, M., Daoud, M., Benaouali, M. et al. Improvement of emotion recognition from facial images using deep learning and early stopping cross validation. Multimed Tools Appl 81, 29887–29917 (2022). https://doi.org/10.1007/s11042-022-12058-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12058-0

Keywords

Search

Navigation