Skip to main content
SpringerLink
Log in

Facial expression recognition based on dual-feature fusion and improved random forest classifier

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression recognition (FER) is an important means for machines to understand the changes in the facial expression of human beings. Expression recognition using single-modal facial images, such as gray scale, may suffer from illumination changes and the lack of detailed expression-related information. In this study, multi-modal facial images, such as facial gray scale, depth, and local binary pattern (LBP), are used to recognize six basic facial expressions, namely, happiness, sadness, anger, disgust, fear, and surprise. Facial depth images are used for robust face detection initially. The deep geometric feature is represented by point displacement and angle variation in facial landmark points with the help of depth information. The local appearance feature, which is obtained by concatenating LBP histograms of expression-prominent patches, is utilized to recognize those expression changes that are difficult to capture by only the geometric changes. Thereafter, an improved random forest classifier based on feature selection is used to recognize different facial expressions. Results of comparative evaluations in benchmarking datasets show that the proposed method outperforms several state-of-the-art FER approaches that are based on hand-crafted features. The capability of the proposed method is comparable to that of the popular convolutional neural-network-based FER approach but with fewer demands for training data and a high-performance hardware platform.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Aly S, Trubanova A, Abbott L et al (2015) VT-KFER: A Kinect-based RGBD+ Time dataset for spontaneous and non-spontaneous facial expression recognition. International Conference on Biometrics (ICB), pp 90–97

  2. Aly S, Abbott AL, Torki M (2016) A multi-modal feature fusion framework for kinect-based facial expression recognition using Dual Kernel Discriminant Analysis (DKDA). 2016 I.E. Winter Conference on Applications of Computer Vision (WACV), pp 1–10

  3. Baltrusaitis T, Robinson P, Morency LP (2013) Constrained local neural fields for robust facial landmark detection in the wild. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, IEEE, pp 354–361

  4. Barnouti NH, Aldabbagh SSM, Matti WE et al (2016) Face Detection and Recognition Using Viola-Jones with PCA-LDA and Square Euclidean Distance. Int J Adv Comput Sci Appl 7(5):371–377

    Google Scholar 

  5. Beeler T, Bickel B, Beardsley P et al (2010) High-quality single-shot capture of facial geometry. ACM T GRAPHIC (Proc. SIGGRAPH), vol 29, no. 4, p 40

  6. Chao WL, Ding JJ, Liu JZ (2015) Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection. Signal Process 117:1–10

    Article  Google Scholar 

  7. Chen CR, Wong WS, Chiu CT (2011) A 0.6 4 mm real-time cascade face detection design based on reduced two-field extraction. IEEE Trans Very Large Scale Integr VLSI Syst. 19(11):1937–1948

    Article  Google Scholar 

  8. Chen J, Chen Z, Chi Z et al (1949) Facial expression recognition in video with multiple feature fusion. IEEE Trans Affect Comput PP(99):1–1

  9. Cheng S, Asthana A, Zafeiriou S et al (2014) Real-time generic face tracking in the wild with cuda. Proceedings of the 5th ACM Multimedia Systems Conference, ACM, Singapore, Singapore, pp 148–151

  10. Corneanu CA, Oliu M, Cohn JF et al (2016) Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568

    Article  Google Scholar 

  11. Danelakis A, Theoharis T, Pratikakis I (2016) A spatio-temporal wavelet-based descriptor for dynamic 3D facial expression retrieval and recognition. Vis Comput 32(6-8):1001–1011

    Article  Google Scholar 

  12. Demirkus M, Precup D, Clark J et al (2014) Multi-layer temporal graphical model for head pose estimation in real-world videos. IEEE International Conference on Image Processing (ICIP), Paris, France, pp 3392–3396

  13. Fan X, Jia Q, Huyan K et al (2016) 3D facial landmark localization using texture regression via conformal mapping. Pattern Recogn Lett 83:395–402

    Article  Google Scholar 

  14. Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  15. Jain S, Hu C, Aggarwal J (2011) Facial expression recognition with temporal modeling of shapes. IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, pp 1642–1649

  16. Kenji M (1991) Recognition of facial expression from optical flow. IEICE Trans Inf Syst 74(10):3474–3483

    Google Scholar 

  17. Kobayashi H, Hara F (1997) Facial interaction between animated 3d face robot and human beings. IEEE International Conference on Man and Cybernetics, vol 4, pp 3732–3737

  18. Kung SH, Zohdy MA, Bouchaffra D (2016) 3D HMM-based Facial Expression Recognition using Histogram of Oriented Optical Flow. Trans Mach Learn Artif Intell 3(6):42

    Google Scholar 

  19. Li B, Mian A, Liu W et al (2013) Using kinect for face recognition under varying poses, expressions, illumination and disguise. Workshop on Applications of Computer Vision (WACV), pp 186–192

  20. Li HB, Ding HX, Huang D et al (2015) An efficient multimodal 2D+3D feature-based approach to automatic facial expression recognition. Comput Vis Image Underst 140:83–92

    Article  Google Scholar 

  21. Liu C, Wechsler H (2002) Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Trans Image Process 11(4):467–476

    Article  Google Scholar 

  22. Lopes AT, DeAguiar E, DeSouza AF et al (2017) Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  23. Mavadati SM, Mahoor MH, Bartlett K et al (2013) Disfa: A spontaneous facial action intensity database. IEEE Trans Affect Comput 4(2):151–160

    Article  Google Scholar 

  24. Moeini A, Faez K, Sadeghi H et al (2016) 2D facial expression recognition via 3D reconstruction and feature fusion. J Vis Commun Image Represent 35:1–14

    Article  Google Scholar 

  25. Mohammadi M, Fatemizadeh E, Mahoor M (2014) Pca-based dictionary building for accurate facial expression recognition via sparse representation. J Vis Commun Image Represent 25(5):1082–1092

    Article  Google Scholar 

  26. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. Applications of Computer Vision (WACV), 2016 I.E. Winter Conference on. IEEE, pp 1-10

  27. Pu X, Fan K, Chen X et al (2015) Facial expression recognition from image sequences using twofold random forest classifier. Neurocomputing 168:1173–1180

    Article  Google Scholar 

  28. Shan CF, Gong SG, McOwan PW (2009) Facial expression recognition based on local binary patterns: A comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  29. Shao J, Gori I, Wan SH et al (2015) 3D dynamic facial expression recognition using low-resolution videos. Pattern Recogn Lett 65(1):157–162

    Article  Google Scholar 

  30. Siddiqi MH, Ali R, Sattar A, Khan AM et al (2014) Depth camera-based facial expression recognition system using mul-tilayer scheme. IETE Tech Rev 31(4):277–286

    Article  Google Scholar 

  31. Siddiqi MH, Ali R, Khan AM et al (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24(4):1386–1398

    Article  MathSciNet  Google Scholar 

  32. Valstar M, Pantic M, Patras I (2004) Motion history for facial action detection in video, vol 1. IEEE International Conference on Man and Cybernetics, pp 635–640

  33. Yu ZD, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, pp 435–442

  34. Zhang Y, Zhang L, Hossain MA (2015) Adaptive 3D facial action intensity estimation and emotion recognition. Expert Syst Appl 42(3):1446–1464

    Article  Google Scholar 

  35. Zhang W, Zhang YM, Ma L et al (2015) Multimodal learning for facial expression recognition. Pattern Recogn 48(10):3191–3202

    Article  Google Scholar 

  36. Zhang Z, Cui L, Liu X et al (2016) Emotion detection using Kinect 3D facial points. IEEE/WIC/ACM International Conference on Web Intelligence (WI), pp 407–410

  37. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  38. Zhao XM, Shi XG, Zhang SQ (2015) Facial expression recognition via deep learning. IETE Tech Rev 32(5):347–355

    Article  Google Scholar 

  39. Zhen Q, Huang D, Wang Y et al (2016) Muscular Movement Model-Based Automatic 3D/4D Facial Expression Recognition. IEEE Trans Multimedia 18(7):1438–1450

    Article  Google Scholar 

  40. Zhong L, Liu QS, Yang P et al (2015) Learning multiscale active facial patches for expression analysis. IEEE Trans Cybern 45(8):1499–1510

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported by the National Natural Science Foundation of China under Grant No. 61501060 and No. 61502058, the Natural Science Foundation of Jiangsu Province under Grant No. BK20150271 and No. BK20140266, Natural Science Foundation of Educational Committee of Jiangsu Province under Grant 15KJB520002.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Changzhou University, Changzhou, Jiangsu Province, 213164, China

    Biao Yang, Jin-Meng Cao, Da-Peng Jiang & Ji-Dong Lv

Authors
  1. Biao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin-Meng Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Da-Peng Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ji-Dong Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Biao Yang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, B., Cao, JM., Jiang, DP. et al. Facial expression recognition based on dual-feature fusion and improved random forest classifier. Multimed Tools Appl 77, 20477–20499 (2018). https://doi.org/10.1007/s11042-017-5489-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5489-9

Keywords

Search

Navigation