Skip to main content
SpringerLink
Log in

Facial emotion classification using concatenated geometric and textural features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Geometric and textural features have been separately used in the literature for facial emotion classification. In this paper, we theoretically and empirically study the capture of facial expressions through geometric and texture-based features, and demonstrate that a simple concatenation of these features can lead to significant improvement in facial emotion classification. We also propose the use of the Directed Acyclic Graph SVM (DAGSVM) for facial emotion classification using the concatenated feature, by analyzing DAGSVM structures. We perform experiments using the well-known extended Cohn-Kanade (CK+), the MUG facial expression (MUG) and the Japanese Female Facial Expression (JAFFE) databases to evaluate the integration of geometric and textural features, and the use of DAGSVM for facial emotion classification. The said integration is found to be effective and DAGSVM is found to be computationally efficient in facial emotion classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: Proceedings of the international workshop on image analysis for multimedia interactive services, pp 1–4

  2. Agarwal S, Mukherjee DP (2017) Facial expression recognition through adaptive learning of local motion descriptor. Multimedia Tools and Applications 76(1):1073–1099

    Article  Google Scholar 

  3. Bartlett MS, Littlewort G, Fasel I, Movellan JR (2003) Real time face detection and facial expression recognition: development and applications to human computer interaction. In: Proceedings of the IEEE conference on computer vision and pattern recognition - workshop, pp 53–53

  4. Bashyal S, Venayagamoorthy GK (2008) Recognition of facial expressions using Gabor wavelets and learning vector quantization. Eng Appl Artif Intell 21:1056–1064

    Article  Google Scholar 

  5. Bazrafkan S, Nedelcu T, Filipczuk P, Corcoran P (2017) Deep learning for facial expression recognition: A step closer to a smartphone that knows your moods. In: Proceedings of the IEEE international conference on consumer electronics (ICCE), pp 217–220. https://www.hindawi.com/journals/ahci/2014/408953/

  6. Bickel S, Scheffer T (2004) Multi-view clustering. In: IEEE International Conference on Data Mining, pp 19–26

  7. Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: International conference on computational learning theory, pp 92–100

  8. Buciu I, Pitas I (2004) Application of non-negative and local non negative matrix factorization to facial expression recognition. In: Proceedings of the international conference on pattern recognition (ICPR), pp 288–291

  9. Busso C, Bulut M, Lee C-C, Kazemzadeh A, Mower E, Kim S, Chang JN, Lee S, Narayanan SS (2008) Iemocap: interactive emotional dyadic motion capture database. Lang Resour Eval 42(4):335

    Article  Google Scholar 

  10. Chaudhuri K, Kakade S, Livescu K, Sridharan K (2009) Multi-view clustering via canonical correlation analysis. In: International Conference on Machine Learning, pp 129–136

  11. Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models - their training and application. Comp Vision Image Underst (CVIU) 61(1):38–59

    Article  Google Scholar 

  12. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(6):681–685

    Article  Google Scholar 

  13. Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91:160–187

    Article  Google Scholar 

  14. Cruz A, Bhanu B, Yang S (2011) A psychologically-inspired match-score fusion mode for video-based facial expression recognition. In: Proceedings of the international conference on affective computing and intelligent interaction, pp 341–350

  15. Dahmane M, Meunier J (2011) Continuous emotion recognition using Gabor energy filters. In: Proceedings of the international conference on affective computing and intelligent interaction, pp 351–358

  16. Datta S, Sen D, Balasubramanian R (2016) Integrating geometric and textural features for facial emotion classification using SVM frameworks. In: Proceedings of the international conference on computer vision and image processing, pp 619–628

  17. Dhall A, Goecke R, Joshi J, Wagner M, Gedeon T (2013) Emotion recognition in the wild challenge 2013. In: Proceedings of the ACM on international conference on multimodal interaction, pp 509–516

  18. Ding C, Choi J, Tao D, Davis LS (2016) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38(3):518–531

    Article  Google Scholar 

  19. Douglas-Cowie E, Cowie R, Cox C, Amier N, Heylen DK (2008) The sensitive artificial listner: an induction technique for generating emotionally coloured conversation. In: Proceedings of the LREC workshop on corpora for research on emotion and affect, no WP 08-02, pp 1–4

  20. Farquhar J, Hardoon D, Meng H, Shawe-taylor JS, Szedmák S (2006) Two view learning: Svm-2k, theory and practice. In: Weiss Y, Schölkopf B, Platt JC (eds) Advances in neural information processing systems 18, pp 355–362

  21. Glodek M, Tschechne S, Layher G, Schels M, Brosch T, Scherer S, Kächele M., Schmidt M, Neumann H, Palm G, Schwenker F (2011) Multiple classifier systems for the classification of audio-visual emotional states. In: Proceedings of the international conference on affective computing and intelligent interaction, pp 359–368

  22. Gritti T, Shan C, Jeanne V, Braspenning R (2008) Local features based facial expression recognition with face registration errors. In: Proceedings of the IEEE international conference on automatic face gesture recognition, pp 1–8

  23. Gu W, Xiang C, Venkatesh YV, Huang D, Lin H (2012) Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recogn 45(1):80–91

    Article  Google Scholar 

  24. Huang KC, Huang SY, Kuo YH (2010) Emotion recognition based on a novel triangular facial feature extraction method. In: Proceedings of the international joint conference on neural networks (IJCNN), pp 1–6

  25. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, pp 46–53

  26. Knill O (2009) Probability and stochastic processes with applications. Overseas Press, New Delhi

    Google Scholar 

  27. Krebel U (1999) Pairwise classification and support vector machine, ser. Advances in Kernel Methods: support vector learning. MIT Press, Cambridge, pp 255–268

    Google Scholar 

  28. Kumar A, Rai P, DauméIII H (2011) Co-regularized spectral clustering with multiple kernels. In: International Conference on Neural Information Processing Systems, pp 1413–1421

  29. Jeni LA, Girard JM, Cohn JF, Torre FDL (2013) Continuous AU intensity estimation using localized, sparse facial feature space. In: Proceedings of the IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–7

  30. Jiang B, Valstar MF, Pantic M (2011) Action unit detection using sparse appearance descriptors in space-time video volumes. In: Face and Gesture, pp 314–321

  31. Jiang B, Valstar M, Martinez B, Pantic M (2014) A dynamic appearance descriptor approach to facial actions temporal modeling. IEEE Transactions on Cybernetics 44(2):161–174

    Article  Google Scholar 

  32. Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  33. Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence 21:1357–1362

    Article  Google Scholar 

  34. Liu P, Yin L (2015) Spontaneous facial expression analysis based on temperature changes and head motions. In: Proceedings of the IEEE international conference and workshops on automatic face and gesture recognition (FG), vol 1, pp 1–6

  35. Liu Y, Nie L, Han L, Zhang L, Rosenblum D (2015) Action2activity: Recognizing complex activities from sensor data. In: International Joint Conference on Artificial Intelligence, pp 1617–1623

  36. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: Sensor-based activity recognition. Neurocomputing 181:108–115

    Article  Google Scholar 

  37. Liu Y, Zhang L, Nie L, Yan Y, Rosenblum D (2016) Fortune teller predicting your career path. In: AAAI Conference on Artificial Intelligence, pp 201–207

  38. Liu W, Yang X, Tao D, Cheng J, Tang Y (2018) Multiview dimension reduction via hessian multiset canonical correlations. Information Fusion 41:119–128

    Article  Google Scholar 

  39. Long F, Wu T, Movellan JR, Bartlett MS, Littlewort G (2012) Learning spatiotemporal features by using independent component analysis with application to facial expression recognition. Neurocomputing 93:126–132

    Article  Google Scholar 

  40. Lyons MJ, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wavelets. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, pp 200–205

  41. Lucey S, Ashraf AB, Cohn J (2007) Face Recognition Germany: Pro Literatur Verlag, ch. Investigating spontaneous facial action recognition through AAM representations of the face, pp 395–406

  42. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kande dataset (ck+): A complete facial expression dataset for action unit and emotion-specified expression. In: Proceedings of the IEEE computer vision and pattern recognition - workshops, pp 94–101

  43. Luo Y, Liu T, Tao D, Xu C (2015) Multiview matrix completion for multilabel image classification. IEEE Trans Image Processing 24(8):2355–2368

    Article  MathSciNet  Google Scholar 

  44. Mariooryad S, Busso C (2013) Feature and model level compensation of lexical content for facial emotion recognition. In: Proceedings of the IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–6

  45. Mariooryad S (2016) Facial expression recognition in the presence of speech using blind lexical compensation. IEEE Trans Affect Comput 7(4):346–359

    Article  Google Scholar 

  46. McKeown G, Valstar MF, Cowie R, Pantic M (2010) The semaine corpus of emotionally coloured character interactions. In: Proceedings of the IEEE international conference on multimedia and expo, pp 1079–1084

  47. Meng H, Romera-paredes B, Bianchi-Berthouze N (2011) Emotion recognition by two view SVM_2k classifier on dynamic facial expression features. In: Face and gesture, pp 854–859

  48. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: Proceedings of the IEEE winter conference on applications of computer vision (WACV), pp 1–10

  49. Milborrow S, Nicolls F (2014) Active shape models with SIFT descriptors and MARS. In: Proceedings of the international conference on computer vision theory and applications (VISAPP), vol 2, pp 380–387

  50. Nicolle J, Rapp V, Bailly K, Prevost L, Chetouani M (2012) Robust continuous prediction of human emotions using multiscale dynamic cues. In: Proceedings of the ACM international conference on multimodal interaction, pp 501–508

  51. Niese R, Al-Hamadi A, Farag A, Neumann H, Michaelis B (2012) Facial expression recognition based on geometric and optical flow features in colour image sequences. IET Comput Vis 6(2):79–89

    Article  MathSciNet  Google Scholar 

  52. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24:971–987

    Article  Google Scholar 

  53. Pantic M, Rothkrantz JM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Transactions on Systems, Man and Cybernetics Part B 34(3):1449–1461

    Article  Google Scholar 

  54. Pantic M, Valstar MF, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: Proceedings of the IEEE international conference on multimedia and expo, Amsterdam, The Netherlands, pp 317–321

  55. Pantic M, Patras I (2005) Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. In: Proceedings of the IEEE international conference on systems, man and cybernetics, pp 3358–3363

  56. Platt JC, Cristianini N, Taylor JS (1999) Large margin dags for multiclass classification. Advances in Neural Information Processing Systems (NIPS), pp 547–553

  57. Quadrianto N, Lampert C (2011) Learning multi-view neighborhood preserving projections. In: International Conference on International Conference on Machine Learning, pp 425–432

  58. Rudovic O, Pavlovic V, Pantic M (2012) Multi-output laplacian dynamic ordinal regression for facial expression recognition and intensity estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2634–2641

  59. Saeed A, Al-Hamadi A, Niese R, Elzobi M (2012) Effective geometric features for human emotion recognition. In: Proceedings of the IEEE international conference on signal processing (ICSP), pp 623–627

  60. Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometrical features. Advances in Human-Computer Interaction, vol 2014

  61. Senechal T, Rapp V, Prevost L (2011) Facial feature tracking for emotional dynamic analysis. In: Proceedings of the international conference on advanced concepts for intelligent vision systems, pp 495–506

  62. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  63. Sariyanidi E, Gunes H, Gokmen M, Cavallaro A (2013) Local zernike moment representations for facial affect recognition. In: Proceedings of the British machine vision conference, p 108.1108.13

  64. Savran A, Cao H, Shah M, Nenkova A, Verma R (2012) Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering. In: Proceedings of the ACM international conference on multimodal interaction, pp 485–492

  65. Shojaeilangari S, Yau WY, Nandakumar K, Li J, Teoh EK (2015) Robust representation and recognition of facial emotions using extreme sparse learning. IEEE Trans Image Process 24:2140–2152

    Article  MathSciNet  Google Scholar 

  66. Sikka K, Wu T, Susskind J, Bartlett M (2012) Exploring bag of words architectures in the facial expression domain. In: Proceedings of the European conference on computer vision - workshops and demonstrations, pp 250–259

  67. Sindhwani V, Niyogi P, Belkin M (2005) A co-regularization approach to semi-supervised learning with multiple views. ICML Workshop on Learning with Multiple Views

  68. Tian YI, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2):97–115

    Article  Google Scholar 

  69. Tao H, Huang TS (1998) Connected vibrations: a modal analysis approach for non-rigid motion tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 735–740

  70. Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions. EEE Trans Syst Man Cybern B (Cybernetics) 42(1):28–43

    Article  Google Scholar 

  71. van der Schalk J, Hawk ST, Fischer AH, Doosje B (2011) Moving faces, looking places: Validation of the amsterdam dynamic facial expression set (ADFES). Emotion 11(4):907–920

    Article  Google Scholar 

  72. Vapnik V (1998) Statistical Learning Theory. Wiley, New York

    MATH  Google Scholar 

  73. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the international conference on computer vision and pattern recognition (CVPR), pp 511–518

  74. Wu T, Bartlett MS, Movellan JR (2010) Facial expression recognition using Gabor motion energy filters. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition - workshops, pp 42–47

  75. Yang P, Liu Q, Metaxas D (2008) Similarity features for facial event analysis. In: Proceedings of the European conference on computer vision, pp 685–696

  76. Yang S, Bhanu B (2011) Facial expression recognition using emotion avatar image. In: Face and gesture, pp 866–871

  77. Yang P (2011) Dynamic soft encoded patterns for facial event analysis. Comput Vis Image Underst 115(3):456–465

    Article  Google Scholar 

  78. Yang X, Liu W, Tao D, Cheng J (2017) Canonical correlation analysis networks for two-view image recognition. Inf Sci 385-386:338–352

    Article  Google Scholar 

  79. Yang S, Ramanan D (2015) Multi-scale recognition with dag-cnns. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1215–1223

  80. Zen G, Porzi L, Sangineto E, Ricci E, Sebe N (2016) Learning personalized models for facial expression analysis and gesture recognition. IEEE Transactions on Multimedia 18(4):775–788

    Article  Google Scholar 

  81. Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, pp 454–459

  82. Zhang S, Zhao X, Lei B (2012) Facial expression recognition based on local binary patterns and local fisher discriminant analysis. WSEAS Transactions on Signal Processing 8:21–31

    Google Scholar 

  83. Zhang J, Huang X, Yang L, Nie L (2016) Bridge the semantic gap between pop music acoustic feature and emotion: Build an interpretable model. Neurocomputing 208:333–341

    Article  Google Scholar 

  84. Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18(12):2528–2536

    Article  Google Scholar 

  85. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6):915–928

    Article  Google Scholar 

  86. Zhao G, Pietikinen M (2009) Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recogn Lett 30(12):1117–1127

    Article  Google Scholar 

  87. Zhu Y, la Torre FD, Cohn JF, Zhang YJ (2011) Dynamic cascades with bidirectional bootstrapping for action unit detection in spontaneous facial behavior. IEEE Trans Affect Comput 2(2):79–91

    Article  Google Scholar 

Download references

Acknowledgment

One of the authors, Debashis Sen, would like to thank Mr. Ashish Verma, a Ph.D. student at IIT Kharagpur, for his help in completing this article. The authors would also like to thank the anonymous reviewers, as their contribution has been pivotal in bringing this article to its best version.

Author information

Authors and Affiliations

  1. Department of Electronics and Electrical Communication Engineering, Indian Institute of Technology, Kharagpur, India

    Debashis Sen

  2. Department of Computer Science and Engineering, Indian Institute of Technology, Roorkee, India

    Samyak Datta & R. Balasubramanian

Authors
  1. Debashis Sen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samyak Datta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. Balasubramanian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Debashis Sen.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sen, D., Datta, S. & Balasubramanian, R. Facial emotion classification using concatenated geometric and textural features. Multimed Tools Appl 78, 10287–10323 (2019). https://doi.org/10.1007/s11042-018-6537-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6537-9

Keywords

Search

Navigation