Skip to main content

Advertisement

SpringerLink
Log in

Facial expression recognition: a review

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression recognition has become a hot issue in the field of artificial intelligence. So, we collect literature on facial expression recognition. First, methods based on machine learning are introduced in detail, which include image preprocessing, feature extraction, and image classification. Then, we review deep learning methods in detail: convolutional neural networks, deep belief networks, generative adversarial networks, and recurrent neural networks. Moreover, the advantages and limitations of different facial expression recognition methods are compared. In addition, 20 commonly used facial expression datasets are collected in this paper, and the types of expressions and the number of images contained in each dataset are summarized. Finally, the current problems and future development of facial expression recognition are concluded.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

Data availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Aamir M, Ali T, Shaf A, Irfan M, Saleem MQ (2020) ML-DCNNet: multi-level deep convolutional neural network for facial expression recognition and intensity estimation. Arab J Sci Eng 45(12):10605–10620

    Article  Google Scholar 

  2. Abdulrahman, M, Gwadabe, TR, Abdu, FJ, Eleyan, A (2014) Gabor wavelet transform based facial expression recognition using PCA and LBP. In Signal Processing & Communications Applications Conference. (Trabzon, Turkey), pp. 2265–2268

  3. Aifanti, N, Papachristou, C, Delopoulos, A (2010) The MUG facial expression database. In image analysis for Multimedia interactive services (WIAMIS), 2010 11th International workshop on

  4. Albahli S, Albattah W (2020) Deep transfer learning for COVID-19 prediction: case study for limited data problems. Curr Med Imaging Former Curr Med Imaging Rev 16:1–9

    Google Scholar 

  5. Alenazy WM, Alqahtani AS (2021) Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition. J Ambient Intell Humaniz Comput 12(2):1631–1646

    Article  Google Scholar 

  6. Ali, K (2019) Stacked support vector machine ensembles for cross-culture emotions classification. Int J Comput Sci Netw Secur, 1738-7906.

  7. Ali G, Ali A, Ali F, Draz U, Majeed F et al (2020) Artificial neural network based ensemble approach for multicultural facial expressions analysis. IEEE Access 8:134950–134963

    Article  Google Scholar 

  8. Ali, K, Hughes, CE, IEEE Comp, S.O.C (2021) Facial expression recognition by using a disentangled identity-invariant expression representation. In 25th International Conference on pattern recognition (ICPR). (Electr Network), pp. 9460–9467

  9. Alrizq M, Hamdi M, Hafeez A, Alghamdi A, Khan AH (2021) An effective combination of textures and wavelet features for facial expression recognition. Eng Technol Appl Sci Res 11(3):7172–7176

    Article  Google Scholar 

  10. Andrew, G, Bilmes, J, IEEE (2012) Sequential Deep Belief Networks. In IEEE International Conference on acoustics, speech and signal processing. (Kyoto, Japan), pp. 4265–4268

  11. Aneja, D, Colburn, A, Faigin, G, Shapiro, L, Mones, B (2016) Modeling stylized character expressions via deep learning. In 13th Asian Conference on computer vision (ACCV). (Taipei, Taiwan)

  12. Bansal S, Nagar P (2015) Emotion recognition from facial expression based on Bezier curve. Int J Adv Inf Technol 5(3):1–7

    CAS  Google Scholar 

  13. Barman A, Dutta P (2019) Influence of shape and texture features on facial expression recognition. IET Image Process 13(8):1349–1363

    Article  Google Scholar 

  14. Bashyal S, Venayagamoorthy G (2008) Recognition of facial expressions using Gabor wavelets and learning vector quantization. Eng Appl Artif Intell 21:1056–1064

    Article  Google Scholar 

  15. Bhatti UA, Huang M, Wu D, Zhang Y, Mehmood A et al (2018) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enterp Inf Syst 13(3):329–351

    Article  Google Scholar 

  16. Bhatti UA, Huang M, Wang H, Zhang Y, Mehmood A et al (2018) Recommendation system for immunization coverage and monitoring. Hum Vaccin Immunother 14(1):165–171

    Article  PubMed  Google Scholar 

  17. Bhatti UA, Zeeshan Z, Nizamani MM, Bazai S, Yu Z et al (2022) Assessing the change of ambient air quality patterns in Jiangsu Province of China pre-to post-COVID-19. Chemosphere 288(Pt 2):132569

    Article  CAS  PubMed  Google Scholar 

  18. Bhatti UA, Yu Z, Hasnain A, Nawaz SA, Yuan L et al (2022) Evaluating the impact of roads on the diversity pattern and density of trees to improve the conservation of species. Environ Sci Pollut Res 29(10):14780–14790

    Article  Google Scholar 

  19. Bhatti UA, Yu Z, Chanussot J, Zeeshan Z, Yuan L et al (2022) Local Similarity-Based Spatial–Spectral Fusion Hyperspectral Image Classification With Deep CNN and Gabor Filtering. IEEE Trans Geosci Remote Sens 60:1–15

    Article  Google Scholar 

  20. Bromley J, Guyon I, Lecun Y, Säckinger E (1993) Signature verification using a Siamese time delay neural network. Int J Pattern Recognit Artif Intell 07(4):669–669

    Article  Google Scholar 

  21. Cambridge, AL (n.d.) The ORL Database of Faces. http://www.uk.research.att.com/facedatabase.html

  22. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. Artif Intell Res 16:321–357

    Article  Google Scholar 

  23. Chen W, Meng JE, Wu S (2006) Illumination compensation and normalization for robust face recognition using discrete cosine transform in logarithm domain. IEEE Trans Syst Man Cybern Part B Cybern A Publ IEEE Syst Man Cybernet Soc 36(2):458

    Article  Google Scholar 

  24. Chen LI, Wang YJ, Liang MT (2019) Face recognition under uncontained environment based on CS-LBP combined with DBN. Comput Eng Des 40(5):1430–1434

    Google Scholar 

  25. Chengeta, K, Viriri, S (2019) Image preprocessing techniques for facial expression recognition with canny and kirsch edge detectors. In 11th International Conference on computational collective intelligence (ICCCI), Volume 11684. (Hendaye, France), pp. 85–96

  26. Choi, S, Kim, EH, Ahn, B, Sohn, JH (2012) Facial emotion recognition using k-NN and SVM. Aik Conference, 242–246

  27. Chowdary MK, Tu NN, Hemanth DJ (2021) Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Comput & Applic 4:1–18

    Google Scholar 

  28. Christou, N, Kanojiya, N (2018) Human facial expression recognition with convolution neural networks. In 3rd International Conference on information and communication technology (ICICT), Volume 797. (London, England), pp. 539–545

  29. Christou, N, Kanojiya, N (2019) Human facial expression recognition with convolution neural networks. In 3rd International Conference on information and communication technology (ICICT), Volume 797. (London, England), pp. 539–545

  30. Cohn, JF, Ambadar, Z, Ekman, P (2007) Observer-based measurement of facial expression with the facial action coding system. Neurosci Lett, 203-221

  31. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S et al (2002) Emotion recognition in human-computer interaction. IEEE Signal Process Mag 18(1):32–80

    Article  ADS  Google Scholar 

  32. Dhall A, Goecke R, Lucey S, Gedeon, T (2011) Acted Facial Expr The Wild Database

  33. Dhall A, Goecke R, Lucey S, Gedeon T (2011) Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. In IEEE International Conference on computer vision workshops. IEEE, Barcelona, Spain

    Google Scholar 

  34. Dharanya V, Raj ANJ, Gopi VP (2021) Facial expression recognition through person-wise regeneration of expressions using auxiliary classifier generative adversarial network (AC-GAN) based model. J Vis Commun Image Represent 77:103110

    Article  Google Scholar 

  35. Dubey M, Singh L (2016) Automatic emotion recognition using facial expression: a review. Int Res J Eng Technol 03(02):488–492

    Google Scholar 

  36. Ebner NC, Riediger M, Lindenberger U (2010) FACES--a database of facial expressions in young, middle-aged, and older women and men: development and validation. Behav Res Methods 42(1):351–362

    Article  PubMed  Google Scholar 

  37. Eddy SR (1996) Hidden Markov models. Curr Opin Struct Biol 6(3):361–365

    Article  CAS  PubMed  Google Scholar 

  38. Ekman P (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124–129

    Article  CAS  PubMed  Google Scholar 

  39. Fang Y, Liu J, Li J, Cheng J, Hu J et al (2022) Robust zero-watermarking algorithm for medical images based on SIFT and Bandelet-DCT. Multimed Tools Appl 81(12):16863–16879

    Article  Google Scholar 

  40. Feng X, Huang D, Cui S, Wang K (2020) Spatial-temporal attention network for facial expression recognition. J Northwest Univ Nat Sci Ed 50(3):319–327

    Google Scholar 

  41. Georghiades AS, Belhumeur PN, Kriegman DJ (2002) From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell 23(6):643–660

    Article  Google Scholar 

  42. Gopalan NP, Bellamkonda S, Chaitanya VS (2019) Facial expression recognition using geometric landmark points and convolutional neural networks. In 2018 International Conference on inventive research in computing applications (ICIRCA). IEEE, Coimbatore, India

    Google Scholar 

  43. Gross, R, Matthews, I, Cohn, J, Kanade, T, Ba Ker, S (2010) Multi-PIE Image Vis Comput, 28(5), 807–813

  44. Guo X, Zhang Y-D, Lu S, Lu Z (2021) A survey on machine learning in COVID-19 diagnosis. Comput Model Eng Sci 129(1):1–49

    Google Scholar 

  45. Hablani R (2020) Facial expression recognition using transfer learning on deep convolutional network. Biosci Biotechnol Res Commun 13(14):185–188

    Article  Google Scholar 

  46. Han ZY, Huang H GAN Based Three-Stage-Training Algorithm for Multi-view Facial Expression Recognition. Neural Process Lett 53:4189–4205

  47. Han Z, Huang H, Huang T, Cao J (2019) Face merged generative adversarial network with tripartite adversaries. Neurocomputing 368(Nov.27):188–196

    Article  Google Scholar 

  48. He Y, Chen SX (2020) Person-independent facial expression recognition based on improved local binary pattern and higher-order singular value decomposition. IEEE Access 8:190184–190193

    Article  Google Scholar 

  49. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Sci 313(5786):504–507

    Article  ADS  MathSciNet  CAS  Google Scholar 

  50. Hong, J, Lee, HJ, Kim, Y, Ro, YM (2020) Face tells detailed expression: generating comprehensive facial expression sentence through facial action units. In 26th International Conference on MultiMedia modeling (MMM), Volume 11962. (Daejeon, South Korea), pp. 100–111

  51. Hu M, Gao Y, Wu H, Wang X, Huang Z (2020) Video facial emotion recognition based on edge detection and recurrent neural network. J Electron Meas Instrum 34(7):103–111

    Google Scholar 

  52. Hu, YY, Zeng, XP, Huang, ZY, Dong, X, IEEE (2021) A preprocessing method of facial expression image under different illumination. In 13th IEEE International Conference on communication software and networks (ICCSN). (Chongqing, Peoples R China), pp. 318–322

  53. Huang, Y, Chen, F, Lv, S, Wang, X (2019) Facial Expr Recognition: A Surv Symmetry, 11(10), 1189

  54. Huang, K, Li, JJ, Cheng, SC, Yu, J, Tian, WY, et al. (2020) An efficient algorithm of facial expression recognition by TSG-RNN network. In 26th International Conference on MultiMedia modeling (MMM), Volume 11962. (Daejeon, South Korea), pp. 161–174

  55. Ilyas, BR, Mohammed, B, Khaled, M, Ahmed, AT, Ihsen, A (2019) Facial expression recognition based on DWT feature for deep CNN. In 2019 6th International Conference on control, decision and information technologies (CoDIT). (Cairo Egypt)

  56. Jabid T, Kabir MH, Chae O (2010) Local directional pattern (LDP) a robust image descriptor for object recognition. In 2010 7th IEEE International Conference on advanced video and signal based surveillance. IEEE, Boston, MA, USA

    Google Scholar 

  57. Jiang B, Gan Y, Zhang HL, Zhang QW (2019) Survey on non-frontal facial expression recognition methods. Comput Therm Sci 46(3):53–62

    Google Scholar 

  58. Jie, H, Li, S, Gang, S, Albanie, S (2017) Squeeze-and-excitation networks. In 2018 IEEE/CVF Conference on computer vision and pattern recognition. Salt Lake City, UT, USA

  59. Jin X, Sun WY, Jin Z (2020) A discriminative deep association learning for facial expression recognition. Int J Mach Learn Cybern 11(4):779–793

    Article  CAS  Google Scholar 

  60. Jung, H, Lee, S, Park, S, Kim, B, Kim, J, et al. (2015) Development of deep learning-based facial expression recognition system. In 2015 21st Korea-Japan joint workshop on Frontiers of computer vision (FCV). (Mokpo, Korea (South)), pp. 1–4

  61. Kanade T, Tian Y, Cohn JF (2002) Comprehensive database for facial expression analysis. In IEEE International Conference on Automatic Face & Gesture Recognition. IEEE, Grenoble, France

    Google Scholar 

  62. Khan SA, Qasim HSA, Azam I (2018) Feature extraction trends for intelligent facial expression recognition: a survey. Inf-J Comput Inf 42(4):507–514

    Google Scholar 

  63. Khanzada, A, Bai, C, Celepcikay, FT (2020) Facial expression recognition with deep learning. arXiv e-prints

  64. Kim, S, An, GH, Kang, SJ (2017) Facial expression recognition system using machine learning. In International SoC design Conference. (Seoul, Korea)

  65. Kołakowska A, Landowska A, Szwoch M, Szwoch W, Wróbel MR (2014) Emotion recognition and its applications. Adv Intell Syst Comput 300:51–62

    Article  Google Scholar 

  66. Kuang L, Zhang M, Pan Z (2016) Facial expression recognition with CNN ensemble. In International Conference on Cyberworlds. IEEE, Chongqing, China

    Google Scholar 

  67. Kuntzler T, Hofling TTA, Alpers GW (2021) Automatic facial expression recognition in standardized and non-standardized emotional expressions. Front Psychol 12:1–13

    Article  Google Scholar 

  68. Kuo, CM, Lai, SH, Sarkis, M (2018) A Compact Deep Learning Model for Robust Facial Expression Recognition. In 2018 IEEE/CVF Conference on computer vision and pattern recognition workshops (CVPRW). Salt Lake City, UT, USA

  69. Langner O, Dotsch R, Bijlstra G, Wigboldus DH, Hawk ST et al (2010) Presentation and validation of the Radboud faces database. Cognit Emot 24(8):1377–1388

    Article  Google Scholar 

  70. Li, SZ, Jain, AK (2011) Handbook of face recognition. In Facial Expression Recognition, pp. 305–322

  71. Li, H, Li, G (2019) Research on Facial Expression Recognition Based on LBP and DeepLearning. In 2019 International Conference on Robots & Intelligent System (ICRIS). (Haikou, China)

  72. Li C, Diao Y, Ma H, Li Y (2009) A Statistical PCA Method for Face Recognition. In 2008 Second International symposium on intelligent information technology application. Shanghai, China

    Google Scholar 

  73. Li S, Deng W, Du JP (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In 2017 IEEE Conference on computer vision and pattern recognition (CVPR). IEEE, Honolulu, HI, USA

    Google Scholar 

  74. Li, Q, Liu, XW, Gong, XY, Jing, SF, IEEE (2019) INDReview on facial expression analysis and its application in education. In Chinese automation congress (CAC). (Hangzhou, Peoples R China), pp. 4526–4530

  75. Li T, Li J, Liu J, Huang M, Chen Y-W et al (2022) Robust watermarking algorithm for medical images based on log-polar transform. EURASIP J Wirel Commun Netw 2022(1):24

    Article  Google Scholar 

  76. Liang, XC, Yang, L, Luo, SD, IOP (2019) Facial expression recognition based on Gabor multi-orientation feature fusion. In 3rd International Conference on machine vision and information technology (CMVIT), Volume 1229. (Guangzhou, Peoples R China)

  77. Lipton, ZC, Berkowitz, J, Elkan, C (2015) A critical review of recurrent neural networks for sequence learning. arXiv.org

  78. Liu, ZT, Sui, GT, Li, DY, Tan, GZ (2015) A navel facial expression recognition method based on extreme learning machine. In 2015 34th Chinese control Conference (CCC), Q. Zhao and S. Liu, eds. (Hangzhou, China), pp. 3852–3857

  79. Liu F, Wang S, Zhang Y (2018) Survey on deep belief network model and its applications. Comput Eng Appl 54(1):11–18

    ADS  Google Scholar 

  80. Liu, HY, Zeng, JB, Shan, SG (2020) Facial expression recognition for in-the-wild videos. In 15th IEEE International Conference on automatic face and gesture recognition (FG). (Buenos Aires, Argentina), pp. 615–618

  81. Liu J, Wang HX, Feng YJ (2021) An end-to-end deep model with discriminative facial features for facial expression recognition. IEEE Access 9:12158–12166

    Article  Google Scholar 

  82. Lu, LH (2021) Multi-angle face expression recognition based on generative adversarial networks. IEEE Trans Antennas Propag 1

  83. Lucey P, Cohn JF, Kanade T, Saragih J, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Computer Vision & Pattern Recognition Workshops, San Francisco, CA, USA

  84. Lundqvist, D, Flykt, A, Hman, A (1998) The Karolinska directed emotional faces – KDEF, CD ROM from Department of Clinical Neuroscience, Psychology section

  85. Luo Y, Wu CM, Zhang Y (2013) Facial expression recognition based on fusion feature of PCA and LBP with SVM. Optik - Int J Light Electron Opt 124(17):2767–2770

    Article  Google Scholar 

  86. Lyons, MJ, Kamachi, M, Gyoba, J (1997) The Japanese female facial expression (JAFFE) database

  87. Ma, SH, Lai, SM, Sun, Y, Pan, ZC, IEEE (2019) Research Status and Prospect of Face Expression Recognition. In Proceedings of the 2019 31st Chinese Control and Decision Conference. (Nanchang, China: IEEE), pp. 640–646

  88. Ma H, Celik T, Li HC (2021) Lightweight attention convolutional neural network through network slimming for robust facial expression recognition. Signal Image Vid Process 15:1507–1515

    Article  Google Scholar 

  89. Mavadati, M, Sanger, P, Mahoor, MH (2016) Extended DISFA Dataset: Investigating Posed and Spontaneous Facial Expressions. In 2016 IEEE Conference on computer vision and pattern recognition workshops (CVPRW). Las Vegas, NV, USA

  90. Mehrabian, A, Russell, JA (1974) An approach to environmental psychology. In The MIT Press, pp. 222–253

  91. Mishra, N, Bhatt, A (2021) Feature Extraction Techniques in Facial Expression Recognition. In 2021 5th International Conference on intelligent computing and control systems (ICICCS). (Madurai, India)

  92. Mohan K, Seal A, Krejcar O, Yazidi A (2021) FER-net: facial expression recognition using deep neural net. Neural Comput & Applic 33(15):9125–9136

    Article  Google Scholar 

  93. Mollahosseini A, Hasani B, Mahoor MH (1949) AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput 10(1):18–31

    Article  Google Scholar 

  94. Mungra D, Agrawal A, Sharma P, Tanwar S, Obaidat MS (2020) PRATIT: a CNN-based emotion recognition system using histogram equalization and data augmentation. Multimed Tools Appl 79(3–4):2285–2307

    Article  Google Scholar 

  95. Murugappan, M, Mutawa, AM, Sruthi, S, Hassouneh, A, Ranjana, R (2020) Facial Expression Classification using KNN and Decision Tree Classifiers. In 2020 4th International Conference on computer, communication and signal processing (ICCCSP). (Chennai, India)

  96. Nawaz SA, Li J, Bhatti UA, Bazai SU, Zafar A et al (2021) A hybrid approach to forecast the COVID-19 epidemic trend. PLoS One 16(10):e0256971

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  97. Ngo, QT, Yoon, S (2020) Facial expression recognition based on weighted-cluster loss and deep transfer learning using a highly imbalanced dataset. Sensors, 20(9)

  98. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recogn 29(1):51–59

    Article  ADS  Google Scholar 

  99. Patel K, Mehta D, Mistry C, Gupta R, Tanwar S et al (2020) Facial sentiment analysis using AI techniques: state-of-the-art, taxonomies, and challenges. IEEE Access 8:90495–90519

    Article  Google Scholar 

  100. Peng, X, Yu, X, Sohn, K, Metaxas, D, Chandraker, M (2017) Reconstruction-based disentanglement for pose-invariant face recognition. In IEEE International Conference on computer vision (ICCV), 2017. (Venice, Italy)

  101. Raghu, V.N., and Bharathi, R.S. (2020). Facial expression recognition using deep learning. arXiv.

  102. Rahimzadeh, M, Attar, A (2021) Introduction of a new dataset and method for detecting and counting the pistachios based on deep learning. arXiv.org

  103. Rajan S, Poongodi C, Devaraj S, Madian N (2020) A novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM. IET Image Process 14(7):1373–1381

    Article  Google Scholar 

  104. Rao, K, Ghosh, D (2005) Facial expression recognition using hidden Markov models. In proceedings of the 2nd Indian International Conference on artificial intelligence, Pune, India, December 20–22, 2005

  105. Rokkones, AS, Uddin, MZ, Torresen, J (2019) Facial expression recognition using robust local directional strength pattern features and recurrent neural network. In 9th IEEE International Conference on consumer electronics (ICCE-Berlin). Berlin, Germany, pp. 283–288

  106. Saito, JH, Carvalho, TVD, Hirakuri, M, Saunite, A, Abib, S (2005) Using CMU PIE human face database to a convolutional neural network - Neocognitron. In ESANN 2005, 13th European symposium on artificial neural networks, Bruges, Belgium, April 27-29, 2005, proceedings

  107. Sawyer, R, Smith, A, Rowe, J, Azevedo, R, Lester, J (2017) Enhancing student models in game-based learning with facial expression recognition. In The 25th Conference on user modeling, adaptation and personalization. (Bratislava, Slovaki), pp. 192–201

  108. Sengupta S, Chen JC, Castillo C, Patel VM, Jacobs DW (2016) Frontal to profile face verification in the wild. In 2016 IEEE winter Conference on applications of computer vision (WACV). IEEE, Lake Placid, NY, USA

    Google Scholar 

  109. Shan D, Ward R (2005) Wavelet-based illumination normalization for face recognition. In IEEE International Conference in image processing. IEEE, Genova, Italy

    Google Scholar 

  110. Shan S, Wen G, Bo C, Zhao D (2003) Illumination normalization for robust face recognition against varying lighting conditions. In analysis and modeling of faces and gestures, 2003. AMFG 2003. IEEE International workshop on. IEEE, Nice, France

    Google Scholar 

  111. Shan C, Gong S, Mcowan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  112. Sharma R, Kaushik B (2016) Facial expression recognition: a survey. Int J Comput Appl 153(10):32–36

    Google Scholar 

  113. Sharma S, Verma R, Sharma MK, Kumar V (2016) Facial expression recognition using PCA. Int J Sci Res Dev 4(2):1905–1907

    Google Scholar 

  114. Shehu, HA, Sharif, MH,Uyaver, S (2020) Facial expression recognition using deep learning. In Fourth International Conference of Mathematical Sciences (ICMS 2020). (Thailand)

  115. Shiva Prakash, B, Sanjeev, KV, Prakash, R, Chandrasekaran, K (2019) A survey on recurrent neural network architectures for sequential learning. In International Conference on soft computing for problem solving (SocProS). Indian Inst Technol Bhubaneswar, Bhubaneswar, India

  116. Singh, S, Nasoz, F (2020) Facial Expression Recognition with Convolutional Neural Networks. In 2020 10th annual computing and communication workshop and Conference (CCWC). (Las Vegas, NV, USA)

  117. Song L (2019) A survey of facial expression recognition based on convolutional neural network. In 2019 IEEE/ACIS 18th International Conference on computer and information Science (ICIS), Beijing, China, IEEE

  118. Song, KS, Kwon, DS, IEEE (2019) Accuracy improvement of facial expression recognition in speech acts: confirmation of effectiveness of information around a mouth and GAN-based data augmentation. In 28th IEEE International Conference on robot and human interactive communication (RO-MAN). New Delhi, India

  119. Sun X, Pan X (2020) Facial expression recognition using PSO and CG on optimizing the DBN parameters. J Yunnan Univ Nat Sci 42(5):877–885

    Google Scholar 

  120. Sun, JM, Pei, XS, Zhou, SS (2008) Facial emotion recognition in modern distant education system using SVM. In International Conference on machine learning & cybernetics. (Kunming, China)

  121. Supta, SR, Sahriar, MR, Rashed, MG, Das, D, Yasmin, R, et al. (2020) An effective facial expression recognition system. In 6th IEEE International women in engineering (WIE) Conference on electrical and computer engineering (WIECON-ECE). (Bhubaneswar, India), pp. 66–69

  122. Ucar, A (2017) Deep Convolutional Neural Networks for facial expression recognition. In 2017 IEEE International Conference on INnovations in intelligent SysTems and applications (INISTA). Gdynia, Poland

  123. Uddin MZ, Hassan MM, Almogren A, Alamri A, Alrubaian M et al (2017) Facial expression recognition utilizing local direction-based robust features and deep belief network. IEEE Access 5:4525–4536

    Article  Google Scholar 

  124. Ullah, A, Wang, J, Anwar, MS, Ahmad, U, Wang, J (2018) Feature Extraction based on Canonical Correlation Analysis using FMEDA and DPA for Facial Expression Recognition with RNN. In 2018 14th IEEE International Conference on signal processing (ICSP). (Beijing, China)

  125. Valstar, M, Pantic, M (2010) Induced disgust, happiness and surprise: An addition to the mmi facial expression database. Proc. Int'l Conf. Language Resources and Evaluation, Workshop Emotion, 65–70

  126. Vedantham R, Reddy ES (2020) A robust feature extraction with optimized DBN-SMO for facial expression recognition. Multimed Tools Appl 79(8):21487–21512

    Article  Google Scholar 

  127. Viola PA, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  128. Viroonluecha, P, Borisut, T, Santa, J (2020) COVID19 X-ray image classification using voting ensemble CNNs transfer learning. In The 15th International joint symposium on artificial intelligence and natural language processing (iSAI-NLP 2020). Bangkok

  129. Wang Y, Haizhou AI, Bo WU, Huang C (2004) Real time facial expression recognition with Adaboost. In proceedings of the 17th International Conference on pattern recognition, 2004. ICPR 2004, Cambridge

    Google Scholar 

  130. Wang M, Hong J, Ying L (2010) Face recognition based on DWT/DCT and SVM. In 2010 International Conference on computer application and system modeling (ICCASM 2010). IEEE, Taiyuan

    Google Scholar 

  131. Wang, WX, Sun, Q, Fu, YW, Chen, T, Cao, CJ, et al. (2019) Comp-GAN: compositional generative adversarial network in synthesizing and recognizing facial expression. In 27th ACM International Conference on Multimedia (MM). (Nice, France), pp. 211–219

  132. Wang, HN, Ding, JH, Wang, F, Ma, Z (2019) Facial expression recognition system based on deep residual fusion neural network. In Chinese intelligent automation Conference, Volume 586. (Jiangsu, Peoples R China), pp. 138–144

  133. Wei, H, Zhang, Z, IEEE (2020) A survey of facial expression recognition based on deep learning. In 15th IEEE Conference on industrial electronics and applications (ICIEA). (Electr Network), pp. 90–94

  134. Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J (2009) Toward practical smile detection. IEEE Trans Pattern Anal Mach Intell 31(11):2106–2111

    Article  PubMed  Google Scholar 

  135. Wingenbach T, Ashwin C, Brosnan M (2016) Validation of the Amsterdam dynamic facial expression set – Bath intensity variations (ADFES-BIV): a set of videos expressing low, intermediate, and high intensity emotions. PLoS One 11(1):26784347

    Article  Google Scholar 

  136. Xing Z, Yin L, Cohn JF, Canavan S, Reale M et al (2013) A high-resolution spontaneous 3D dynamic facial expression database. Image Vis Comput 32(10):692–706

    Google Scholar 

  137. Xu, Q, Zhao, N (2020) A Facial Expression Recognition Algorithm based on CNN and LBP Feature. In 2020 IEEE 4th information technology, networking, electronic and automation control Conference (ITNEC). Chongqing, China

  138. Xu Y, Zhang Z, Lu G, Yang J (2016) Approximately symmetrical face images for image preprocessing in face recognition and sparse representation based classification. Pattern Recogn 54:68–82

    Article  ADS  Google Scholar 

  139. Yadav KS, Singha J (2020) Facial expression recognition using modified Viola-John's algorithm and KNN classifier. Multimed Tools Appl 79(19):13089–13107

    Article  Google Scholar 

  140. Yaermaimaiti Y (2021) Facial expression recognition based on local feature and deep belief network. J Decis Syst 2:1–13

    Google Scholar 

  141. Yan Y, Huang Y, Chen S, Shen CH, Wang HZ (2020) Joint deep learning of facial expression synthesis and recognition. IEEE Trans Multimed 22(11):2792–2807

    Article  Google Scholar 

  142. Yang Y, Liu G, Zhang L (2014) 2DPCA and LDA based face image preprocessing and RBF neural network based human face recognition. Software 35(2):115–118

    CAS  Google Scholar 

  143. Yang, H, Ciftci, U, Yin, L (2018) Facial Expression Recognition by De-expression Residue Learning. In 2018 IEEE/CVF Conference on computer vision and pattern recognition (CVPR). Salt Lake City, UT, USA

  144. Yang B, Li ZY, Cao EG (2020) Facial expression recognition based on multi-dataset neural network. Radioeng 29(1):259–266

    Google Scholar 

  145. Yang L, Tian Y, Song YH, Yang NC, Ma K et al (2020) A novel feature separation model exchange-GAN for facial expression recognition. Knowl-Based Syst 24:106217

    Article  Google Scholar 

  146. Yao L, Wan Y, Ni HJ, Xu BG (2021) Action unit classification for facial expression recognition using active learning and SVM. Multimed Tools Appl 80(16):24287–24301

    Article  Google Scholar 

  147. Yin L, Wei X, Yi S, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In International Conference on automatic face & gesture recognition. IEEE, Southampton

    Google Scholar 

  148. Yue, CT, Liang, J, Qu, BY, Lu, ZP, Li, BL, et al. (2019) Sparse representation feature for facial expression recognition. In International Conference on extreme learning machine (ELM), Volume 10. (Yantai, Peoples R China), pp. 12–21

  149. Zhalehpour S, Onder O, Akhtar Z, Erdem CE (2017) BAUM-1: a spontaneous audio-visual face database of affective and mental states. IEEE Trans Affect Comput 8(3):300–313

    Article  Google Scholar 

  150. Zhang T (2017) Facial expression recognition based on deep learning: a survey. IEEE Trans Affect Comput 13(3):1195–1215

    Google Scholar 

  151. Zhang YP, Chen L, Hao H (2013) An improved training algorithm for quantum neural networks. J Electron Inf Technol 35(7):1630–1635

    Article  Google Scholar 

  152. Zhang T, Zheng WM, Cui Z, Zong Y, Li Y (2019) Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 49(3):839–847

    Article  PubMed  Google Scholar 

  153. Zhang SQ, Pan XZ, Cui YL, Zhao XM, Liu LM (2019) Learning affective video features for facial expression recognition via hybrid deep learning. IEEE Access 7:32297–32304

    Article  Google Scholar 

  154. Zhao XM, Zhang SQ (2016) A review on facial expression recognition: feature extraction and classification. IETE Tech Rev 33(5):505–517

    Article  MathSciNet  Google Scholar 

  155. Zhao G, Huang X, Taini M, Li SZ, Pietikäinen M (2011) Facial expression recognition from near-infrared videos. Image Vis Comput 29(9):607–619

    Article  Google Scholar 

  156. Zhao, R, Liu, TS, Xiao, J, Lun, DPK, Lam, KM, et al. (2021) Deep multi-task learning for facial expression recognition and synthesis based on selective feature sharing. In 25th International Conference on pattern recognition (ICPR). Electr Network, pp. 4412–4419

  157. Zheng H, Wang RL, Ji WT, Zong M, Wong WK et al (2020) Discriminative deep multi-task learning for facial expression recognition. Inf Sci 533:60–71

    Article  Google Scholar 

  158. Zhi, RC, Wan, M (2019) Dynamic facial expression feature learning based on sparse RNN. In IEEE 8th joint International information technology and artificial intelligence Conference (ITAIC). (Chongqing, Peoples R China), pp. 1373–1377

  159. Zhou X (2021) Video expression recognition method based on spatiotemporal recurrent neural network and feature fusion. J Inf Process Syst 17(2):337–351

    Google Scholar 

  160. Zhou J, Wang TJ (2020) FER based on the improved convex nonnegative matrix factorization feature. Multimed Tools Appl 79(35–36):26305–26325

    Article  Google Scholar 

  161. Zhou T, Lu H, Huo B (2020) Survey of deep belief network. Comput Eng Appl 56(9):24–32

    Google Scholar 

Download references

Acknowledgments

The paper is supported by Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD).

Author information

Author notes

  1. Xing Guo and Yudong Zhang contributed equally to this work.

Authors and Affiliations

  1. School of Educational Science, Nanjing Normal University, Nanjing, 210023, Jiangsu, China

    Xing Guo & Zhihai Lu

  2. School of Computing and Mathematical Sciences, University of Leicester, Leicester, LE1 7RH, UK

    Yudong Zhang & Siyuan Lu

Authors
  1. Xing Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yudong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Siyuan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhihai Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhihai Lu.

Ethics declarations

Conflict of interest

Xing Guo, Yudong Zhang, Siyuan Lu, and Zhihai Lu declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, X., Zhang, Y., Lu, S. et al. Facial expression recognition: a review. Multimed Tools Appl 83, 23689–23735 (2024). https://doi.org/10.1007/s11042-023-15982-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15982-x

Keywords

Search

Navigation