Skip to main content
SpringerLink
Log in

Facial expression recognition based on hybrid geometry-appearance and dynamic-still feature fusion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Emotion recognition through facial expression is regarded as one of the most effective methods to directly reflect a person’s inner emotional state for affective computing. However, a key issue of facial expression recognition (FER) is how to design and fuse features from videos rapidly and thus extract representative features to improve the recognition accuracy efficaciously. In this paper, we propose a novel expression recognition framework to mitigate this issue. Specifically, we first present a new descriptor, the improved Local Binary Pattern from Three Orthogonal Planes (I-LBP-TOP), which can extract both the static and dynamic features in changing expressions, and set Gabor’s magnitude feature (GMF) as texture information. Meanwhile, the facial landmarks of the peak frame are proposed to represent geometric feature (GF) and the spatiotemporal geometric feature (ST-GF) is obtained by extending it to time dimension. Then we integrate multiple features of image sequences to overcome the limitation of using one single feature descriptor. A support vector machine (SVM) with multiple kernels is applied to train three base classifiers. Finally, to realize reliable expression classification, a decision-level feature fusion method based on a relative majority voting (MV) strategy is also employed. Intensive experiments are conducted on the CK+ and Oulu-CASIA databases, where the experimental results demonstrate that our proposed method achieves an improved performance compared with the existing state-of-the-art hand-crafted approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Acevedo D, Negri P, Buemi ME, Mejail M (2016) Facial expression recognition based on static and dynamic approaches. In: 2016 23rd International conference on pattern recognition (ICPR). IEEE, pp 4124–4129

  2. Almaev TR, Valstar MF (2013) Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: 2013 Humaine association conference on affective computing and intelligent interaction. IEEE, pp 356–361

  3. Awad AI, Hassaballah M (2016) Image Feature Detectors and Descriptors. Springer International Publishing

  4. Bonab H, Can F (2019) Less is more: a comprehensive framework for the number of components of ensemble classifiers. IEEE Trans Neural Netw Learn Syst 30(9):2735–2745

    Article  MathSciNet  Google Scholar 

  5. Bougourzi F, Mokrani K, Ruichek Y, Dornaika F, Ouafi A, Taleb-Ahmed A (2019) Fusion of transformed shallow features for facial expression recognition. IET Image Process 13(9):1479–1489

    Article  Google Scholar 

  6. Chen J, Chen Z, Chi Z, Fu H (2015) Dynamic texture and geometry features for facial expression recognition in video. In: IEEE international conference on image processing (ICIP), pp 4967–4971. IEEE

  7. Chen J, Chen Z, Chi Z, Fu H (2018) Facial expression recognition in video with multiple feature fusion. IEEE Trans Affect Comput 9(1):38–50

    Article  Google Scholar 

  8. Chen L, Wu M, Zhou M, She J, Dong F, Hirota K (2018) Information-driven multirobot behavior adaptation to emotional intention in human–robot interaction. IEEE Trans Cogn Dev Syst 10(3):647–658

    Article  Google Scholar 

  9. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05). IEEE, vol 1, pp 886–893

  10. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124

    Article  Google Scholar 

  11. Fan X, Tjahjadi T (2015) A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recogn 48(11):3407–3416

    Article  Google Scholar 

  12. Feng D, Ren F (2018) Dynamic facial expression recognition based on two-stream-cnn with lbp-top. In: 2018 5th IEEE international conference on cloud computing and intelligence systems (CCIS). IEEE, pp 355–359

  13. Gao T, Lei X-M, Hu W (2017) Face recognition based on sift and lbp algorithm for decision level information fusion. In: 2017 13th International conference on natural computation, fuzzy systems and knowledge discovery (ICNC-FSKD). IEEE, pp 2242–2246

  14. Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6):7714–7734

    Article  Google Scholar 

  15. Guo Z, Wang X, Zhou J, You J (2015) Robust texture image representation by scale selective local binary patterns. IEEE Trans Image Process 25 (2):687–699

    Article  MathSciNet  MATH  Google Scholar 

  16. Guo Z, Zhang L, Zhang D (2010) A completed modeling of local binary pattern operator for texture classification. IEEE Trans Image Process 19 (6):1657–1663

    Article  MathSciNet  MATH  Google Scholar 

  17. Guojiang W, Guoliang Y (2017) A modified optical flow algorithm and its application in facial expression recognition. In: 2017 3rd IEEE international conference on computer and communications (ICCC). IEEE, pp 1601–1605

  18. Hassaballah M, Aly S (2015) Face recognition: challenges, achievements and future directions. IET Comput Vis 9:614–626

    Article  Google Scholar 

  19. Hassaballah M, Bekhet S, Rashed AAM, Zhang G (2019) Facial features detection and localization. In: Recent advances in computer vision

  20. Hassaballah M, Murakami K, Ido S (2011) An automatic eye detection method for gray intensity facial images. Int J Comput Sci Issues 8(4):272–282

    Google Scholar 

  21. Hassaballah M, Murakami K, Ido S (2013) Face detection evaluation: a new approach based on the golden ratio Φ, Signal Image & Video Processing

  22. Hu M, Yang C, Zheng Y, Wang X, He L, Ren F (2019) Facial expression recognition based on fusion features of center-symmetric local signal magnitude pattern. IEEE Access 7 :118435–118445

    Article  Google Scholar 

  23. Huang X, Zhao G, Zheng W, Pietikäinen M (2012) Towards a dynamic expression recognition system under facial occlusion. Pattern Recogn Lett 33(16):2181–2191

    Article  Google Scholar 

  24. Jan A, Meng H, Gaus YFBA, Zhang F (2017) Artificial intelligent system for automatic depression level analysis through visual and vocal expressions. IEEE Trans Cogn Dev Syst 10(3):668–680

    Article  Google Scholar 

  25. Jeong M, Ko B C (2018) Driverąŕs facial expression recognition in real-time for safe driving. Sensors 18(12):4270

    Article  Google Scholar 

  26. Jones JP, Palmer LA (1987) An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex. J Neurophysiol 58 (6):1233–1258

    Article  Google Scholar 

  27. Kim J-H, Kim B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285

    Article  Google Scholar 

  28. Laurens VDM, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(2605):2579–2605

    MATH  Google Scholar 

  29. Li Y, Zou B, Deng S, Zhou G (2020) Using feature fusion strategies in continuous authentication on smartphones. IEEE Internet Comput 24:49–56

    Article  Google Scholar 

  30. Liliana DY, Widyanto MR, Basaruddin T (2018) Geometric facial components feature extraction for facial expression recognition. In: 2018 International conference on advanced computer science and information systems (ICACSIS). IEEE, pp 391–396

  31. Liu M, Shan S, Wang R, Chen X (2014) Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1749–1756

  32. Liu Y, Zhang X, Lin Y, Wang H (2019) Facial expression recognition via deep action units graph network based on psychological mechanism. IEEE Trans on Cogn Dev Syst

  33. Liu Y, Zhang X, Lin Y, Wang H (2020) Facial expression recognition via deep action units graph network based on psychological mechanism. IEEE Trans Cogn Dev Syst 12(2):311–322

    Article  Google Scholar 

  34. Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  35. Lucey P, Cohn J F, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 ieee computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94–101

  36. Majumder A, Behera L, Subramanian VK (2018) Automatic facial expression recognition system using deep network based data fusion. IEEE Trans Cybern 48(1):103–114

    Article  Google Scholar 

  37. Mehrabian A (2008) Communication without words. Commun theory:193–200

  38. Meng D, Peng X, Wang K, Qiao Y (2019) Frame attention networks for facial expression recognition in videos. In: 2019 IEEE International Conference on Image Processing (ICIP)

  39. Ming Z, Xia J, Luqman MM, Burie J-C, Zhao K (2019) Dynamic multi-task learning for face recognition with facial expression. arXiv:1911.03281

  40. Nguyen VD, Nguyen DD, Nguyen TT, Dinh VQ, Jeon JW (2013) Support local pattern and its application to disparity improvement and texture classification. IEEE Trans Circuits Syst Vid Technol 24(2):263–276

    Article  Google Scholar 

  41. Ning X, Duan P, Li W, Zhang S (2020) Real-time 3d face alignment using an encoder-decoder network with an efficient deconvolution layer. IEEE Signal Process Lett 27:1944–1948

    Article  Google Scholar 

  42. Ning X, Xu S, Li W, Nie S (2020) Fegan: flexible and efficient face editing with pre-trained generator. IEEE Access 8:65340–65350

    Article  Google Scholar 

  43. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  44. Rathee N, Vaish A, Gupta S (2017) Emotion detection through fusion of complementary facial features. In: 2017 7th International conference on communication systems and network technologies (CSNT). IEEE, pp 163–166

  45. Sadeghi H, Raie A-A, Mohammadi M-R (2013) Facial expression recognition using geometric normalization and appearance representation

  46. Sahoo S, Routray A (2016) Emotion recognition from audio-visual data using rule based decision level fusion. In: 2016 IEEE studentsąŕtechnology symposium (TechSym). IEEE, pp 7–12

  47. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Compu 27 (6):803–816

    Article  Google Scholar 

  48. Shanthi P, Nickolas S (2020) An efficient automatic facial expression recognition using local neighborhood feature fusion. Multimed Tools Appl:1–26

  49. Sikka K, Dhall A, Bartlett M (2015) Exemplar hidden markov models for classification of facial expressions in videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 18–25

  50. Tanfous AB, Drira H, Amor BB (2020) Sparse coding of shape trajectories for facial expression and action recognition. IEEE Trans Pattern Anal Mach Intell 42(10):2594–2607

    Article  Google Scholar 

  51. Xie S, Shan S, Chen X, Chen J (2010) Fusing local patterns of gabor magnitude and phase for face recognition. IEEE Trans Image Process 19 (5):1349–1361

    Article  MathSciNet  MATH  Google Scholar 

  52. Xiong X, De la Torre F (2013) Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 532–539

  53. Yang B, Cao J, Ni R, Zhang Y (2017) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640

    Article  Google Scholar 

  54. Zhang Q, Li H, Sun Z, Tan T (2018) Deep feature fusion for iris and periocular biometrics on mobile devices. IEEE Trans Inf Forensics Secur 13:2897–2912

    Article  Google Scholar 

  55. Zhang B, Shan S, Chen X, Gao W (2006) Histogram of gabor phase patterns (hgpp): a novel object representation approach for face recognition. IEEE Trans Image Process 16(1):57–68

    Article  MathSciNet  Google Scholar 

  56. Zhao G, Huang X, Taini M, Li S Z, PietikäInen M. (2011) Facial expression recognition from near-infrared videos. Image Vis Comput 29 (9):607–619

    Article  Google Scholar 

  57. Zhao G, Pietikäinen M (2006) Dynamic texture recognition using volume local binary patterns. In: Dynamical vision. Springer, pp 165–177

  58. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  59. Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and gabor multiorientation fusion histogram. Math Probl Eng, vol 2017

  60. Zheng Q, Tian X, Yang M, Su H (2019) The email author identification system based on support vector machine (svm) and analytic hierarchy process (ahp). IAENG Int J Comput Sci 46(2):178–191

    Google Scholar 

  61. Zheng Q, Tian X, Yang M, Wu Y, Su H (2019) Pac-bayesian framework based drop-path method for 2d discriminative convolutional network pruning. Multidim Syst Sign Process:1–35

Download references

Acknowledgements

This research was supported by National Key R&D Program of China (2018YFC0831503), National Natural Science Foundation of China (61571275), Shenzhen Science and Technology Research and Development Funds (JCYJ20170818104011781).

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Shandong University, Qingdao, Shandong, 266237, China

    Ruyu Yan, Mingqiang Yang, Qinghe Zheng, Deqiang Wang & Cheng Peng

Authors
  1. Ruyu Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingqiang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinghe Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deqiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cheng Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Mingqiang Yang or Deqiang Wang.

Ethics declarations

Conflict of Interests

The authors declare that have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, R., Yang, M., Zheng, Q. et al. Facial expression recognition based on hybrid geometry-appearance and dynamic-still feature fusion. Multimed Tools Appl 82, 2663–2688 (2023). https://doi.org/10.1007/s11042-022-13327-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13327-8

Keywords

Search

Navigation