Skip to main content
Springer Nature Link
Log in

Facial-expression recognition based on a low-dimensional temporal feature space

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper suggests a facial-expression recognition in accordance with face video sequences based on a newly low-dimensional feature space proposed. Indeed, we extract a Pyramid of uniform Temporal Local Binary Pattern representation, using only XT and YT orthogonal planes (PTLBPu2). Then, a Wrapper method is applied to select the most discriminating sub-regions, and therefore, reduce the feature space that is going to be projected on a low-dimensional feature space by applying the Principal Component Analysis (PCA). Support Vector Machine (SVM) and C4.5 algorithm have been tested for the classification of facial expressions. Experiments conducted on CK + and MMI, which are the two famous facial-expression databases, have shown the effectiveness of the approach proposed under a lab-controlled environment with more than 97% of recognition rate as well as under an uncontrolled environment with more than 92%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Branco P, Torgo L, Ribeiro R P (2016) A survey of predictive modeling on imbalanced domains. ACM Comput Surv 49(2):1–50. article No. 31

    Article  Google Scholar 

  2. Chandrashekar G, Sahin F (2014) A survey on feature selection methods. Comput Electr Eng 40(1): 16–28

    Article  Google Scholar 

  3. Chen J, Chen Z, Chi Z, Fu H (2016) Facial expression recognition in video with multiple feature fusion. IEEE Trans Affect Comput PP(99):1–12

    Google Scholar 

  4. Cheng H, Shi X (2004) A simple and effective histogram equalization approach to image enhancement. Digit Signal Process 14(2):158–170

    Article  Google Scholar 

  5. Chung K C, Kee S C, Kim SR (1999) Face recognition using principal component analysis of gabor filter responses. In: Proceedings of the international workshop on recognition, analysis, and tracking of faces and gestures in real-time systems. IEEE Computer Society, pp 53–57

  6. Cortes C, Vapnik V (1995) Support vector networks. J Mach Learn 20 (3):237–297

    MATH  Google Scholar 

  7. De La Torre F, Cohn JF (2011) Facial expression analysis. In: Visual analysis of humans: looking at people. Springer, London, pp 377–409

  8. Deng B, Jin L W, Zhen L X, Huang J C, Deng H B (2005) A new facial expression recognition method based on L ocal G abor F ilter bank and PCA plus LDA. Inf Technol IT 11:86–96

    Google Scholar 

  9. Donia M M F, Youssif A A A, Hashad A (2014) Spontaneous facial expression recognition based on histogram of oriented gradients descriptor. Comput Inf Sci 7:31–37

    Google Scholar 

  10. Doretto G, Chiuso A, Nian Y N, Soatto S (2003) Dynamic textures. Int J Comput Vis 51(2):91–109

    Article  MATH  Google Scholar 

  11. Ekman P (1972) Universals and cultural differences in facial expressions of emotion. University of Nebraska Press Lincoln

  12. Fan X, Tjahjadi T (2015) A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recogn 48(11):3407–3416

    Article  Google Scholar 

  13. Fu X, Wei W (2008) Centralized binary patterns embedded with image Euclidean distance for facial expression recognition. In: Proceedings of the 2008 fourth international conference on natural computation ICNC ’08, vol 04. IEEE Computer Society, Washington, DC, pp 115–119

  14. Fukunaga K, Olsen D R (1971) An algorithm for finding intrinsic dimensionality of data. IEEE Trans Comput C-20(2):176–183

    Article  MATH  Google Scholar 

  15. Gonzalez R C, Woods R E (2008) Digital image processing, 3rd edn. Upper Saddle River, Prentice Hall

    Google Scholar 

  16. Gritti T, Shan C, Jeanne V, Braspenning R (2008) Local features based facial expression recognition with face registration errors. In: 2008 8th IEEE International conference on automatic face gesture recognition, pp 1–8

  17. Guo Y, Zhao G, Pietikäinen M (2016) Dynamic facial expression recognition with Atlas construction and sparse representation. IEEE Trans Image Process 25 (5):1977–1992

    Article  MathSciNet  Google Scholar 

  18. Happy SL, George A, Routray A (2012) A real time facial expression classification system using local binary patterns. In: 2012 4th International conference on intelligent human computer interaction (IHCI), pp 1–5

  19. IMOTIONS - BIOMETRIC RESEARCH PLATFORM (2016) Facial expression analysis: The complete pocket guide. https://imotions.com/blog/facial-expression-analysis

  20. Ji Y, Idrissi K (2012) Automatic facial expression recognition based on spatiotemporal descriptors. Pattern Recogn Lett 33(10):1373–1380

    Article  Google Scholar 

  21. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings Fourth IEEE international conference on automatic face and gesture recognition, pp 46–53

  22. Keerthi S S, Shevade S K, Bhattacharyya C, Murthy K R K (2001) Improvements to Platt’s SMO algorithm for SVM classifier design. Neural Comput 13 (3):637–649

    Article  MATH  Google Scholar 

  23. Kégl B (2002) Intrinsic dimension estimation based on packing numbers. In: Advances in neural information processing systems. Cambridge, pp 833–840

  24. Khan R A (2013) Detection of emotions from video in non-controlled environment. University of Claude Bernard of Lyon, Phd thesis

    Google Scholar 

  25. Khan RA, Meyer A, Konik H, Bouakaz S (2012) Human vision inspired framework for facial expressions recognition. In: 2012 19th IEEE international conference on image processing, pp 2593–2596

  26. Khan R A, Meyer A, Konik H, Bouakaz S (2013) Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recogn Lett 34(10):1159–1168

    Article  Google Scholar 

  27. Kohavi R, John G H (1997) Wrappers for feature subset selection. Artif Intell 97(1):273–324

    Article  MATH  Google Scholar 

  28. Kotsia I, Zafeiriou S, Pitas I (2008) Texture and shape information fusion for facial expression and facial action unit recognition. Pattern Recog 41(3):833–851. Part Special issue: Feature Generation and Machine Learning for Robust Multimodal Biometrics

    Article  MATH  Google Scholar 

  29. Kumari J, Rajesh R, Pooja KM (2015) Facial expression recognition: a survey. Procedia Comput Sci 58:486–491. Second international symposium on computer vision and the internet

    Article  Google Scholar 

  30. Lee T Z, Bong D B L (2016) Analysis of B it-P lane images by using P rincipal C omponent on face and palmprint database. Pertanika J Sci Technol 24(1):191–203

    Google Scholar 

  31. Levina E, Bickel PJ (2004) Maximum likelihood estimation of intrinsic dimension. In: Advances in neural information processing systems. Cambridge, pp 777–784

  32. Littlewort G, Bartlett MS, Fasel I, Susskind J, Movellan J (2006) Dynamics of facial expression extracted automatically from video. Image Vis Comput 24(6):615–625. Face processing in video sequences

    Article  Google Scholar 

  33. Mayer C, Eggers M, Radig B (2014) Cross-database evaluation for facial expression recognition. Pattern Recogn Image Anal 24(1):124–132

    Article  Google Scholar 

  34. Mliki H, Hammami M, Ben-Abdallah H (2013) Mutual information-based facial expression recognition. In: 2013 Sixth international conference on machine vision, society of photo-optical instrumentation engineers (SPIE), vol 9067

  35. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: Proceedings of the 13th ACM international conference on multimedia

  36. Platt J (1998) Fast training of support vector machines using sequential minimal optimization. In: Advances in Kernel methods - support vector learning. MIT Press

  37. Pu X, Fan K, Chen X, Ji L, Zhou Z (2015) Facial expression recognition from image sequences using twofold random forest classifier. Neurocomputing 168:1173–1180

    Article  Google Scholar 

  38. Quinlan J R (1993) C4.5: programs for machine learning. Morgan Kaufmann, San Mateo

    Google Scholar 

  39. Sadeghi H, Raie A, Mohammadi M (2014) Facial expression recognition using texture description of displacement image. J Inf Syst Telecommun 2(4):205–212

    Google Scholar 

  40. Samad R, Sawada H (2011) Extraction of the minimum number of Gabor wavelet parameters for the recognition of natural facial expressions. Artif Life Robot 16(1):21–31

    Article  Google Scholar 

  41. Sánchez A, Ruiz JV, Moreno AB, Montemayor AS, Hernández J, Pantrigo JJ (2011) Differential optical flow applied to automatic facial expression recognition. Neurocomputing 74(8):1272–1282

    Article  Google Scholar 

  42. Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE International conference on image processing 2005, vol 2

  43. Shan C, Gong S, McOwan P W (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  44. Shi J, Tomasi C (1994) Good features to track. In: 1994 Proceedings of IEEE conference on computer vision and pattern recognition, pp 593–600

  45. Shin G, Chun J (2008) Spatio-temporal facial expression recognition using optical flow and HMM. In: Studies in computational intelligence, vol 149. Springer, Berlin, pp 27–38

  46. Su M, Hsieh Y, Huang D (2007) A simple approach to facial expression recognition. In: CEA’07 Proceedings of the international conference on computer engineering and applications. World Scientific and Engineering Academy and Society (WSEAS), pp 456-461

  47. Suk M, Prabhakaran B (2014) Real-time mobile facial expression recognition system – a case study. In: 2014 IEEE Conference on computer vision and pattern recognition workshops, pp 132–137

  48. Tang J, Alelyani S, Liu H (2014) Feature selection for classification: a review, D ata C lassification chapter 2. CRC, Chapman & Hall

    MATH  Google Scholar 

  49. Tian Y, Kanade T, Cohn J F (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23(2):97–115

    Article  Google Scholar 

  50. Ting K C, Tan J (2013) Face recognition by neural network using B it-P lanes extracted from an image. Inf Comput Sci 10(16):5253–5261

    Article  Google Scholar 

  51. Ting KC, Bong DBL, Wang YC (2008) Performance analysis of single and combined bit-planes feature extraction for recognition in face expression database. In: 2008 International conference on computer and communication engineering, pp 792–795

  52. Vapnik V (1995) The nature of statistical learning theory. Springer-Verlag New York, Inc, New York

    Book  MATH  Google Scholar 

  53. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 1, pp 511–518

  54. Wan S, Aggarwal J K (2014) Spontaneous facial expression recognition: a robust metric learning approach. Pattern Recogn 47(5):1859–1868

    Article  Google Scholar 

  55. Wang J, Wang S, Ji Q (2014) Early facial expression recognition using Hidden Markov Models. In: 2014 22nd International conference on pattern recognition, pp 4594–4599

  56. Wang Y, Yu H, Stevens B, Liu H (2015) Dynamic facial expression recognition using local patch and LBP-TOP. In: 2015 8th International conference on human system interaction (HSI), pp 362–367

  57. Whitney A W (1971) A direct method of nonparametric measurement selection. IEEE Trans Comput C-20(9):1100–1103

    Article  MATH  Google Scholar 

  58. Yeasin M, Bullot B (2005) Comparison of linear and non-linear data projection techniques in recognizing universal facial expressions. In: Proceedings. 2005 IEEE international joint conference on neural networks, 2005, vol 5, pp 3087–3092

  59. Yu K, Wang Z, Guan G, Wu Q, Chi Z, Feng D (2012) How many frames does facial expression recognition require? In: 2012 IEEE International conference on multimedia and expo workshops, pp 290–295

  60. Zhang L, Tjondronegoro D, Chandran V (2014) Facial expression recognition experiments with data from television broadcasts and the World Wide Web. Image Vis Comput 32(2):107–119

    Article  Google Scholar 

  61. Zhang X, Mahoor M H, Mavadati S M (2015) Facial expression recognition using l p -norm MKL multiclass-SVM. Mach Vis Appl 26(4):467–483

    Article  Google Scholar 

  62. Zhao G, Pietikäinen M (2007) Dynamic T exture R ecognition using L ocal B inary P atterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  63. Zhao G, Pietikäinen M (2009) Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recogn Lett 30(12):1117–1127. Image/video-based Pattern Analysis and HCI Applications

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful to Sofiene HADDED, teacher of English at the Faculty of Economics and Management of Sfax, Tunisia for having proofread the manuscript.

Author information

Authors and Affiliations

  1. MIR@CL Laboratory, Faculty of Economics and Management of Sfax (FSEGS), University of Sfax, Sfax, Tunisia

    Taoufik Ben Abdallah

  2. Saudi Electronic University, Riyadh, Kingdom of Saudi Arabia

    Radhouane Guermazi

  3. MIR@CL Laboratory, Faculty of Sciences of Sfax (FSS), University of Sfax, Sfax, Tunisia

    Mohamed Hammami

Authors
  1. Taoufik Ben Abdallah
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Radhouane Guermazi
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Mohamed Hammami
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Taoufik Ben Abdallah.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abdallah, T., Guermazi, R. & Hammami, M. Facial-expression recognition based on a low-dimensional temporal feature space. Multimed Tools Appl 77, 19455–19479 (2018). https://doi.org/10.1007/s11042-017-5354-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5354-x

Keywords