Skip to main content
SpringerLink
Log in

Facial expression classification using salient pattern driven integrated geometric and textual features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression classification aims to recognize human emotion via face images. The major challenge of facial expression classification is how to extract discriminative features from the human face images to differentiate various emotions. To tackle this challenge, a new feature extraction approach is proposed in this paper. The proposed approach defines a new set of salient patterns at the facial keypoint locations. This is in contrast to conventional approaches that either represent the whole face image as a regular grid or use local image patches centered at all facial key point locations. Driven by the proposed salient patterns, both the geometric and textual features are extracted and then concatenated and further incorporated into a machine learning framework to perform facial expression classification. The proposed approach is evaluated in the well-known CK + benchmark dataset to demonstrate its superior performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Acevedo D, Negri P, Buemi ME, Fernández FG, Mejail M (2017) A simple geometric-based descriptor for facial expression recognition. In: IEEE Int Conf on automatic face gesture recognition, pp 802– 808

  2. Chang C-C, Lin C-J (2011) LIBSVM: A library for support vector machines. ACM Trans Intell Syst Technol 2(3):1–17

    Article  Google Scholar 

  3. Datta S, Sen D, Balasubramanian R (2017) Integrating geometric and textural features for facial emotion classification using SVM frameworks. In: Int Conf on computer vision and image processing, Singapore, pp 619–628

  4. Deshmukh S, Patwardhan M, Mahajan A (2016) Survey on real-time facial expression recognition techniques. IET Biom 5(3):155–163

    Article  Google Scholar 

  5. Fernandes JA, Matos LN, Aragao MGS (2016) Geometrical approaches for facial expression recognition using support vector machines. In: Int Conf on graphics, patterns and images Sao Paulo, Brazil, pp 347–354

  6. Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimedia Tools and Applications 76(6):7803–7821

    Article  Google Scholar 

  7. Ghimire D, Lee J, Li Z-N, Jeong S (2017) Recognition of facial expressions based on salient geometric features and support vector machines. Multimedia Tools and Applications 76(6):7921–7946

    Article  Google Scholar 

  8. Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  9. Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: A survey. IEEE Trans Syst Man Cybern Part C Appl Rev 41(6):765– 781

    Article  Google Scholar 

  10. Hsieh C-C, Hsih M-H, Jiang M-K, Cheng Y-M, Liang E-H (2016) Effective semantic features for facial expressions recognition using SVM. Multimedia Tools and Applications 75(11):6663–6682

    Article  Google Scholar 

  11. Jain S, Hu C, Aggarwal JK (2011) Facial expression recognition with temporal modeling of shapes. In: IEEE Int Conf on computer vision workshops, pp 1642–1649

  12. Kim B-K, Roh J, Dong S-Y, Lee S-Y (2016) Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. Journal on Multimodal User Interfaces 2:173– 189

    Article  Google Scholar 

  13. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  14. Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  15. Liu C, Wechsler H (2002) Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Trans Image Process 11(4):467–476

    Article  Google Scholar 

  16. Liu M, Li S, Shan S, Wang R, Chen X (2015) Deeply learning deformable facial action parts model for dynamic expression analysis. In: Asian Conf on computer vision, Singapore, pp 143–157

  17. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (CK + ): A complete dataset for action unit and emotion-specified expression. In: IEEE Int. Conf. on computer vision and pattern recognition, San Francisco, pp 94–101

  18. Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Ana Mach Intell 21(12):1357–1362

    Article  Google Scholar 

  19. Majumder A, Behera L, Subramanian V (2018) Automatic facial expression recognition system using deep network-based data fusion. IEEE Trans on Cybernetics 28(1):103–114

    Article  Google Scholar 

  20. Martinez B, Valstar MF, Jiang B, Pantic M (2017) Automatic analysis of facial actions: A survey. IEEE Trans. on Affective Computing. accepted

  21. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  22. Sariyanidi E, Gunes H, Cavallaro A (2015) Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans Pattern Anal Mach Intell 37(6):1113–1133

    Article  Google Scholar 

  23. Siddiqi MH, Ali M, Abdelrahman ME, Khan A, Banos O, Khan AM, Lee S, Choo H (2018) Evaluating real-life performance of the state-of-the-art in facial expression recognition using a novel YouTube-based datasets. Multimedia Tools and Applications 77(1):917–937

    Article  Google Scholar 

  24. Takalkar M, Xu M, Wu Q, Chaczko Z (2017) A survey: Facial micro-expression recognition. Multimedia Tools and Applications :1–25. accepted

  25. Yang B, Cao J-M, Jiang D-P, Lv J-D (2017) Facial expression recognition based on dual-feature fusion and improved random forest classifier. Multimedia Tools and Applications :1–23. accepted

  26. Yu J, Wang Z (2017) A video-based facial motion tracking and expression recognition system. Multimedia Tools and Applications 76(13):14,653–14,672

    Article  Google Scholar 

  27. Zhang Z, Luo P, Loy CC, Tang X (2014) Facial landmark detection by deep multi-task learning. In: European Conf on computer vision, Zurich, Switzerland, pp 94–108

Download references

Author information

Authors and Affiliations

  1. Institute of Systems Science, National University of Singapore, Singapore, 119615, Singapore

    Ruiqi Li, Jing Tian & Matthew Chin Heng Chua

Authors
  1. Ruiqi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthew Chin Heng Chua
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing Tian.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, R., Tian, J. & Chua, M.C.H. Facial expression classification using salient pattern driven integrated geometric and textual features. Multimed Tools Appl 78, 28971–28983 (2019). https://doi.org/10.1007/s11042-018-6133-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6133-z

Keywords

Search

Navigation