Skip to main content

Advertisement

SpringerLink
Log in

Facial emotion recognition based on deep transfer learning approach

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expressions play a major role in the communication of emotions through nonverbal channels. In recent years, the topic of automatic facial expression recognition (FER) has become very popular. Researchers are looking at how it may be used in education, security surveillance, smart healthcare system, and to understand the behavior of a community or a person. As long as there are variations in images, such as different poses and lighting, accurate and robust FER remains a challenge using computer models. We developed an approach to automatically classifying facial expressions based on deep transfer learning. The approach was constructed with convolutional neural networks (CNN) and VGG19, which is a transfer learning model. To train the model, we employed contemporary deep learning techniques such as optimal learning rate finder, fine-tuning, and data augmentation. On both the Extended Cohn-Kanade (CK+) and the Japanese Female Facial Expression (JAFFE) datasets, the proposed model achieved accuracy values of 94.8% and 93.7%, respectively. The developed system has already been tested on a vast database and can be used to assist online education systems, surveillance systems, and smart healthcare systems in their daily activities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

References

  1. Abdulrahman M, Eleyan A (2015) Facial expression recognition using support vector machines. 2015 23nd signal processing and communications applications conference (SIU) (IEEE), pp 276–9

  2. Abidin Z, Harjoko A (2012) A neural network based facial expression recognition using fisherface. Int J Comput Appl 59(3)

  3. Agarwal S, Mukherjee DP (2018) Synthesis of realistic facial expressions using expression map. IEEE Trans Multimed 21:902–914

    Article  Google Scholar 

  4. Busso C, Deng Z, Yildirim S, Bulut M, Lee CM, Kazemzadeh A, Lee S, Neumann U, Narayanan S (2004) Analysis of emotion recognition using facial expressions, speech and multimodal information. In: Proceedings of the 6th international conference on Multimodal interfaces, pp 205–211. https://doi.org/10.1145/1027933.1027968

  5. Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial-expression databases from movies. IEEE Multimed 19:34–41

    Article  Google Scholar 

  6. Ebrahimi Kahou S, Michalski V, Konda K, Memisevic R, Pal C (2015) Recurrent neural networks for emotion recognition in video. Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 467–74

  7. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124

  8. Erickson BJ, Korfiatis P, Kline TL, Akkus Z, Philbrick K, Weston AD (2018) Deep learning in radiology: does one size fit all? J Am Coll Radiol 15:521–526

    Article  Google Scholar 

  9. Goodfellow I J, Erhan D, Carrier P L, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D-H (2013) Challenges in representation learning: a report on three machine learning contests. International conference on neural information processing (Springer), pp 117–24

  10. Happy SL, Routray A (2014) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6:1–12

    Article  Google Scholar 

  11. He K, Zhang X, Ren S, Sun J 2016 Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  12. Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recognit Lett 115:101–106

    Article  Google Scholar 

  13. Khorrami P, Le Paine T, Brady K, Dagli C, Huang TS (2016) How deep neural networks can improve emotion recognition on video data. 2016 IEEE international conference on image processing (ICIP) (IEEE), pp 619–23

  14. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, pp 1097–1105

  15. Le QV, Jaitly N, Hinton GE (2015) A simple way to initialize recurrent networks of rectified linear units. https://doi.org/10.48550/arXiv.1504.00941

  16. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324

    Article  Google Scholar 

  17. Li M, Xu H, Huang X, Song Z, Liu X, Li X (2018) Facial expression recognition with identity and emotion joint learning. IEEE Trans Affect Comput 12:544–550

    Article  Google Scholar 

  18. Liu M, Li S, Shan S, Wang R, Chen X (2014) Deeply learning deformable facial action parts model for dynamic expression analysis. Asian conference on computer vision (Springer), pp 143–157

  19. Lopes AT, De Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628

    Article  Google Scholar 

  20. Lucey P, Cohn J F, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition - workshops, CVPRW 2010

  21. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wavelets. In: Proceedings Third IEEE international conference on automatic face and gesture recognition. IEEE, Nara, pp 200–205

  22. Mollahosseini A, Chan D, Mahoor M H (2016) Going deeper in facial expression recognition using deep neural networks. 2016 IEEE winter conference on applications of computer vision (WACV) (IEEE), pp 1–10

  23. Pramerdorfer C, Kampel M (2016) Facial expression recognition using convolutional neural networks: state of the art. https://doi.org/10.48550/arXiv.1612.02903

  24. Raghuvanshi A, Choksi V (2016) Facial expression recognition with convolutional neural networks. CS231n Course Proj, pp 362

  25. Rao P, Choudhary A, Kumar V (2019) 3D facial emotion recognition using deep learning technique. Journal homepage 6(3):64–8. http://iieta.org/journals/rces

  26. Rifai S, Bengio Y, Courville A, Vincent P, Mirza M (2012) Disentangling factors of variation for facial expression recognition. European conference on computer vision (Springer), pp 808–22

  27. Sharma S, Kumar V 2019 Transfer learning in 2.5 D face image for occlusion presence and gender classification. Handbook of research on deep learning innovations and trends (IGI global), pp 97–113

  28. Sharma S, Kumar V (2020) Voxel-based 3D face reconstruction and its application to face recognition using sequential deep learning. Multimed Tools Appl 79:17303–17330

    Article  Google Scholar 

  29. Sharma S, Kumar V (2020) Voxel-based 3D occlusion-invariant face recognition using game theory and simulated annealing. Multimed Tools Appl 79:26517–26547

    Article  Google Scholar 

  30. Sharma S, Kumar V (2021) 3D landmark-based face restoration for recognition using variational autoencoder and triplet loss. IET Biom 10:87–98

    Article  Google Scholar 

  31. Sharma S, Kumar V (2022) 3D face reconstruction in deep learning era: a survey. Arch Comput Methods Eng 29(5):3475–3507

  32. Tan L, Zhang K, Wang K, Zeng X, Peng X, Qiao Y (2017) Group emotion recognition with individual facial emotion CNNs and global image based CNNs. Proceedings of the 19th ACM international conference on multimodal interaction, pp 549–52

  33. Tian Y (2004) Evaluation of face resolution for expression analysis. In: 2004 conference on computer vision and pattern recognition workshop, Washington, DC, pp 82–82

  34. Wen Y, Zhang K, Li Z, Qiao Y 2016 A discriminative feature learning approach for deep face recognition. European conference on computer vision (Springer) pp 499–515

  35. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. ICMI 2015 - proceedings of the 2015 ACM international conference on multimodal interaction

  36. Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron. Proceedings third IEEE international conference on automatic face and gesture recognition (IEEE), pp 454–9

  37. Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2018) Spatial–temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 49:839–847

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering (CSE), Dhaka International University (DIU), Dhaka, 1205, Bangladesh

    Aziza Sultana

  2. School of Science and Technology (SST), Bangladesh Open University (BOU), Gazipur, 1705, Bangladesh

    Samrat Kumar Dey

  3. Faculty of Engineering and Technology, Multimedia University, Melaka, Malaysia

    Md. Armanur Rahman

Authors
  1. Aziza Sultana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samrat Kumar Dey
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Md. Armanur Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samrat Kumar Dey.

Ethics declarations

Declaration of interests

The authors declare that there are no conflicts of interest.

Ethical approval

Not required.

Consent to participate

Not required.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sultana, A., Dey, S.K. & Rahman, M.A. Facial emotion recognition based on deep transfer learning approach. Multimed Tools Appl 82, 44175–44189 (2023). https://doi.org/10.1007/s11042-023-15570-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15570-z

Keywords

Search

Navigation