Skip to main content
SpringerLink
Log in

An efficient deep learning technique for facial emotion recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Emotion recognition from facial images is considered as a challenging task due to the varying nature of facial expressions. The prior studies on emotion classification from facial images using deep learning models have focused on emotion recognition from facial images but face the issue of performance degradation due to poor selection of layers in the convolutional neural network model.To address this issue, we propose an efficient deep learning technique using a convolutional neural network model for classifying emotions from facial images and detecting age and gender from the facial expressions efficiently. Experimental results show that the proposed model outperformed baseline works by achieving an accuracy of 95.65% for emotion recognition, 98.5% for age recognition, and 99.14% for gender recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Abbreviations

h × w × d :

Height, width, and depth of input image

K :

Filter matrix

I :

Image matrix

b :

Bias value

R :

Nonlinear ReLU function

∘:

Convolutional operation

F :

Feature map matrix

max:

Maxpooling operator

M:

Pooled feature matrix

V:

Long feature vector

n :

Net input

x i :

Input vector

w i :

Weight vector

e ˗ n :

Input parameter exponent

References

  1. Adil B, Nadjib KM, Yacine L (2019) A novel approach for facial expression recognition. In 2019 Inter Conf Net Adv Syst (ICNAS) pp 1–5. IEEE

  2. All things Image (2019) Image Preprocessing, Image Data Generator class. Retrieved from https://keras.io/preprocessing/image/

  3. All things Keras (2019) Keras Deep Learning Library Retrieved from https://keras.io/

  4. Asghar MZ, Khan A, Bibi A, Kundi FM, Ahmad H (2017) Sentence-level emotion detection framework using rule-based classification. Cogn Comput 9(6):868–894

    Article  Google Scholar 

  5. Avots E, Sapiński T, Bachmann M, Kamińska D (2019) Audiovisual emotion recognition in wild. Mach Vis Appl 30(5):975–985

    Article  Google Scholar 

  6. Balasubramanian B, Diwan P, Nadar R, Bhatia A (2019) Analysis of Facial Emotion Recognition. In 2019 3rd Inter Conf Trends Elect Inform (ICOEI) pp 945–949. IEEE

  7. Bouzakraoui MS, Sadiq A, Alaoui AY (2019) Appreciation of Customer Satisfaction Through Analysis Facial Expressions and Emotions Recognition. In 2019 4th World Conf Compl Syst (WCCS) pp 1–5. IEEE

  8. Cadayona AM, Cerilla NMS, Jurilla DMM, Balan AKD, de Goma JC (2019) Emotional State Classification: An Additional Step in Emotion Classification through Face Detection. In 2019 IEEE 6th Inter Conf Indust Eng Appl (ICIEA) pp 667–671. IEEE

  9. Chen J, Ou Q, Chi Z, Fu H (2017) Smile detection in the wild with deep convolutional neural networks. Mach Vis Appl 28(1–2):173–183

    Article  Google Scholar 

  10. Das A, Dantcheva A, Bremond F (2018) Mitigating bias in gender, age and ethnicity classification: a multi-task convolution neural network approach. In Proc Euro Conf Comput Vis (ECCV) pp 0–0

  11. Egger HL, Pine DS, Nelson E, Leibenluft E, Ernst M, Towbin KE, Angold A (2011) The NIMH Child Emotional Faces Picture Set (NIMH-ChEFS): a new set of children’s facial emotion stimuli. Int J Methods Psychiatr Res 20(3):145–156

    Article  Google Scholar 

  12. Fathallah A, Abdi L, Douik A (2017) Facial Expression Recognition via Deep Learning. In Computer Systems and Applications (AICCSA), 2017 IEEE/ACS 14th Inter Conf Comput Syst Appl pp 745–750. IEEE

  13. Guo J, Lei Z, Wan J, Avots E, Hajarolasvadi N, Baro X (2018) Dominant and Complementary Emotion Recognition From Still Images of Faces

  14. Guo JM, Huang PC, Chang LY (2019) A Hybrid Facial Expression Recognition System Based on Recurrent Neural Network. In 2019 16th IEEE Inter Conf Adv Vid Signal Based Surv (AVSS) pp 1–8. IEEE

  15. Hossain MS, Muhammad G (2019) Emotion recognition using deep learning approach from audio–visual emotional big data. Information Fusion 49:69–78

    Article  Google Scholar 

  16. Joseph A, Geetha P (2020) Facial emotion detection using modified eyemap–mouthmap algorithm on an enhanced image and classification with tensorflow. Vis Comput 36(3):529–539

    Article  Google Scholar 

  17. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis.Proceedings of the Fourth IEEE Inter Conf Auto Face Gesture Recognit (FG'00) Grenoble, France 46–53

  18. Kolodziej M, Majkowski A, Rak RJ, Tarnowski P, Pielaszkiewicz T (2018) Analysis of facial features for the use of emotion recognition. In 19th Inter Conf Comput Prob Elect Eng pp 1–4. IEEE

  19. Kumar A, Jaiswal A (2017) Image sentiment analysis using convolutional neural network. In Inter Conf Intel Syst Des Appl pp 464–473. Springer, Cham

  20. Kuo PH, Tsai TN, Luan PC (2019) CNN and LSTM Based Facial Expression Analysis Model for a Humanoid Robot. IEEE Access 7:93998–94011

    Article  Google Scholar 

  21. Li THS, Kuo PH, Tsai TN, Luan PC (2019) CNN and LSTM Based Facial Expression Analysis Model for a Humanoid Robot. IEEE Access 7:93998–94011

    Article  Google Scholar 

  22. Liu KC, Hsu CC, Wang WY, Chiang HH (2019) Real-Time facial expression recognition based on cnn. In 2019Inter Conf Syst Sci Eng (ICSSE) pp 120–123. IEEE

  23. Lopez-Martin M, Carro B, Sanchez-Esguevillas A, Lloret J (2017) Network traffic classifier with convolutional and recurrent neural networks for Internet of Things. IEEE Access 5:18042–18050

    Article  Google Scholar 

  24. Lopez-Rincon A (2019) Emotion recognition using facial expressions in children using the NAO Robot. In 2019 Inter Conf Elect Commun Comput (CONIELECOMP) pp 146–153). IEEE

  25. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) Subedi sanjay. Large Scale Face Dataset for Age and gender. UTKFace dataset. Retrieved from https://www.kaggle.com/jangedoo/utkface-new

  26. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998, April) Coding facial expressions with gabor wavelets. In Proceedings Third IEEE international conference on automatic face and gesture recognition (pp. 200–205). IEEE

  27. Mellouk W, Handouzi W (2020) Facial emotion recognition using deep learning: review and insights. Procedia Comput Sci 175:689–694

    Article  Google Scholar 

  28. Nadjib KM, Yacine L (2019) A novel approach for facial expression recognition. In 2019 Inter Conf Net Adv Syst (ICNAS) pp 1–5. IEEE

  29. Nithyashri J, Kulanthaivel G (2012) Classification of human age based on Neural Network using FG-NET Aging database and Wavelets. In 2012 Fourth Inter Conf Adv Comput (ICoAC) pp 1–5. IEEE

  30. Ozdemir MA, Elagoz B, Alaybeyoglu A, Sadighzadeh R, Akan A (2019) Real Time Emotion Recognition from Facial Expressions Using CNN Architecture. In 2019 Med Technol Cong (TIPTEKNO) pp 1–4. IEEE

  31. Papapicco C, Mininni G (2020) Impact memes: phds humor (e). Multi Tools Appl 1–22

  32. Pathar R, Adivarekar A, Mishra A, Deshmukh A (2019) Human Emotion Recognition using Convolutional Neural Network in Real Time. In 2019 1st Inter Conf Innov Inform Commun Technol (ICIICT) pp 1–7. IEEE

  33. Patil M, Veni S (2019) Driver Emotion Recognition for Enhancement of Human Machine Interface in Vehicles. In 2019Inter Conf Commun Signal Proc (ICCSP) pp 0420–0424. IEEE

  34. Prabhu R (2018) Understanding of Convolutional Neural Network (CNN) — Deep Learning. Retrieved from https://medium.com/@RaghavPrabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835f148

  35. Rao A (2019) Convolutional Neural Network Tutorial (CNN) – Developing An Image Classifier In Python Using Tensor Flow Retrieved from https://www.edureka.co/blog/convolutional-neural-network/

  36. Rasamoelina AD, Adjailia F, SinČàk P (2019) Deep Convolutional Neural Network for Robust Facial Emotion Recognition. In 2019 IEEE Inter Symp Innov Intel Syst Appl (INISTA) pp 1–6. IEEE

  37. Shan K, Guo J, You W, Lu D, Bie R (2017) Automatic facial expression recognition based on a deep convolutional-neural-network structure. In 2017 IEEE 15th Inter Conf Soft Eng Res Manage Appl (SERA) pp 123–128. IEEE

  38. Sinha S, Kumar R, Nalawade S, Ray D (2017) Facial Expression Recognition. Inter Res J Eng Technol Volume 04, p ISSN: 2395–0072. IRJET

  39. Taha B, Hatzinakos D (2019) Emotion Recognition from 2D Facial Expressions. In 2019 IEEE Can Conf Elect Comput Eng (CCECE) pp 1–4. IEEE

  40. Tripathi S, Acharya S, Sharma RD, Mittal S, Bhattacharya S (2017) Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset. In AAAI pp 4746–4752

  41. Verma A, Singh P, Alex JSR (2019) Modified Convolutional Neural Network Architecture Analysis for Facial Emotion Recognition. In 2019 Inter Conf Syst Signals Image Proc (IWSSIP) pp 169–173. IEEE

  42. Wu C, Chai L, Yang J, Sheng Y (2019) Facial Expression Recognition using Convolutional Neural Network on Graphs. In 2019 Chin Cont Conf (CCC) pp 7572-7576. IEEE

  43. Wu H, Zhao J (2018) Deep convolutional neural network model based chemical process fault diagnosis. Comput Chem Eng 115:185–197

    Article  Google Scholar 

  44. Xue D, Bagamian N, Broder A (2016) Classification of Emotion from Facial Images

  45. Yarlagadda A, Murthy JVR, Prasad MK (2015) A novel method for human age group classification based on Correlation Fractal Dimension of facial edges. J king Saud University Comput Inform Sci 27(4):468–476

    Article  Google Scholar 

  46. Yang Z, Zhang Y, Luo J (2019) Human-Centered Emotion Recognition in Animated GIFs. In 2019 IEEE Inter Conf Multi Expo (ICME) pp 1090–1095. IEEE

  47. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In 7th inter Conf Auto Face Gesture recognit (FGR06) pp 211–216. IEEE

  48. Zadeh MMT, Imani M, Majidi B (2019) Fast Facial emotion recognition Using Convolutional Neural Networks and Gabor Filters. In 2019 5th Conf Knowl Based Eng Innov (KBEI) pp 577–581. IEEE

  49. Zhong L, Bai C, Li J, Chen T, Li S, Liu Y (2019) A Graph-Structured Representation with BRNN for Static-based Facial Expression Recognition. In 2019 14th IEEE Inter Conf Auto Face Gesture Recognit (FG 2019) pp 1–5. IEEE

  50. Zhu L, Yu W, Zhang C, Zhang Z, Huang F, Yu H (2019) SVS-JOIN: Efficient Spatial Visual Similarity Join for Geo-Multimedia. IEEE Access 7:158389–158408

    Article  Google Scholar 

Download references

Acknowledgements

This Research work was supported by Zayed University Research Incentives Fund# R19096

Author information

Authors and Affiliations

  1. College of Technological Innovation, Zayed University, Abu Dhabi Campus, 144534, Abu Dhabi, UAE

    Asad Khattak

  2. Institute of Computing and Information Technology, Gomal University, DIKhan (KP), Pakistan

    Muhammad Zubair Asghar & Ulfat Batool

  3. Department of Information Technology, Hazara University Mansehra, Dhodial, Pakistan

    Mushtaq Ali

Authors
  1. Asad Khattak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Zubair Asghar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mushtaq Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ulfat Batool
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Zubair Asghar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khattak, A., Asghar, M.Z., Ali, M. et al. An efficient deep learning technique for facial emotion recognition. Multimed Tools Appl 81, 1649–1683 (2022). https://doi.org/10.1007/s11042-021-11298-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11298-w

Keywords

Search

Navigation