Skip to main content
SpringerLink
Log in

Effective semantic features for facial expressions recognition using SVM

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Most traditional facial expression-recognition systems track facial components such as eyes, eyebrows, and mouth for feature extraction. Though some of these features can provide clues for expression recognition, other finer changes of the facial muscles can also be deployed for classifying various facial expressions. This study locates facial components by active shape model to extract seven dynamic face regions (frown, nose wrinkle, two nasolabial folds, two eyebrows, and mouth). Proposed semantic facial features could then be acquired using directional gradient operators like Gabor filters and Laplacian of Gaussian. A multi-class support vector machine (SVM) was trained to classify six facial expressions (neutral, happiness, surprise, anger, disgust, and fear). The popular Cohn–Kanade database was tested and the average recognition rate reached 94.7 %. Also, 20 persons were invited for on-line test and the recognition rate was about 93 % in a real-world environment. It demonstrated that the proposed semantic facial features could effectively represent changes between facial expressions. The time complexity could be lower than the other SVM based approaches due to the less number of deployed features.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Abboud B, Davoine F, Dang M (2004) Facial expression recognition and synthesis based on an appearance model. Signal Process Image Commun 19:723–740

    Article  Google Scholar 

  2. Amjad A, Griffiths A, Patwary MN (2012) Multiple face detection algorithm using colour skin modelling. Image Process IET 6(8):1093–1101

    Article  MathSciNet  Google Scholar 

  3. Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2006) Fully automatic facial action recognition in spontaneous behavior, 7th International Conference on Automatic Face and Gesture Recognition. 223–230, doi:10.1109/FGR.2006.55

  4. Bashyal S, Venayagamoorthy GK (2008) Recognition of facial expressions using Gabor wavelets and learning vector quantization. Eng Appl Artif Intell 21:1056–1064

    Article  Google Scholar 

  5. Bettadapura V (2012) Face expression recognition and analysis: the state of the art, Technical Report, arXiv:1203.6722

  6. Calder AJ, Burton AM, Miller P, Young AW, Akamatsu S (2001) A principal component analysis of facial expressions. Vis Res 41(9):1179–1208. doi:10.1016/S0042-6989(01)00002-5, ISSN 0042–6989

    Article  Google Scholar 

  7. Castrillón-Santana M, Déniz-Suárez O, Hernández-Sosa D, Lorenzo J (2011) A comparison of face and facial feature detectors based on the Viola-Jones general object detection framework. Mach Vis Appl 22(3):481–494

    Google Scholar 

  8. Cevikalp H, Triggs B, Franc V (2013) Face and landmark detection by using cascade of classifiers, IEEE FG

  9. Chen J, Chen D, Gong Y, Yu M, Zhang K, Wang L (2012) Facial expression recognition using geometric and appearance features, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service (ICIMCS’ 12), ACM, New York, NY, USA, 29–33, doi:10.1145/2382336.2382345

  10. Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models—their training and application. Comput Vis Image Underst 61:38–59

    Article  Google Scholar 

  11. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  12. Daugman JG (1985) Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. J Opt Soc Am 2(7):1160–1169

    Article  Google Scholar 

  13. Duan KB, Keerthi SS (2005) Multiple classifier systems. LNCS 3541:278–285. doi:10.1007/11494683_28

    Google Scholar 

  14. Ekman P, Friesen WV (1978) The facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  15. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36:259–275

    Article  MATH  Google Scholar 

  16. Fellenz WA, Taylor JG, Tsapatsoulis N, Kollias S (1999) Comparing template-based, feature-base and supervised classification of facial expressions from static images. Proc. Circuits Syst Commun Comput, pp 5331–5336

  17. Gonzalez R, Woods R (1992) Digital image processing, Addison-Wesley Publishing Company, Chap. 2

  18. Hsu CW, Chang CC, Lin CJ (2004) A practical guide to support vector classification, Technical Report, Department of Computer Science and Information Engineering, National Taiwan University

  19. Huang C, Huang Y (1997) Facial expression recognition using model-based feature extraction and action parameters classification. Image Represent 8(3):278–290

    Article  Google Scholar 

  20. Jain V, Learned-Miller E (2011) Online domain adaptation of a pretrained cascade of classifiers. IEEE Conf Comp Vis Pattern Recognit (CVPR), 577–584, 20–25

  21. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG’00), Grenoble, France, 46–53

  22. Kawulok M, Szymanek J (2012) Precise multi-level face detector for advanced analysis of facial images. Image Process IET 6(2):95–103

    Article  MathSciNet  Google Scholar 

  23. Kobayashi H, Hara F (1997) Facial interaction between animated 3D face robot and human beings. Proc Int Conf Syst Man Cybern, pp 3732–3737

  24. Lanitis A, Taylor C, Cootes TF (2002) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell, 743–756

  25. Lee HC, Wu CY, Lin TM (2013) Facial expression recognition using image processing techniques and neural networks, Advances in Intelligent Systems and Applications - Volume 2. Smart Innov Syst Technol Vol 21(2013):259–267

    Article  Google Scholar 

  26. Lienhart R, Maydt J (2002) An extended set of Haar-like features for rapid object detection. Image Process, 900–903

  27. Ma R, Wang J (2005) Automatic facial expression recognition using linear and nonlinear holistic spatial analysis. Affect Comput Intell Interaction Lect Notes Comput Sci 3784:144–151

    Article  Google Scholar 

  28. Michel P, Kaliouby RE (2003) Real time facial expression recognition in video using support vector machines, Proceedings of the 5th international conference on Multimodal interfaces, 258–264

  29. Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput Vis Image Underst 115(4):541–558

    Article  Google Scholar 

  30. Ou J, Bai XB, Pei Y, Ma L, Liu W (2010) Automatic facial expression recognition using Gabor filter and expression analysis. Int Conf Comput Model Simul 2:215–218

    Google Scholar 

  31. Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometrical features, Advances in Human-Computer Interaction, Article ID 408953, 13 pages, doi:10.1155/2014/408953

  32. Sandbacha G, Zafeirioua S, Pantica M, Yin L (2012) Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput 30(10):683–697

    Article  Google Scholar 

  33. Schmidt M, Schels M, Schwenker F (2010) A hidden Markov model based approach for facial expression recognition in image sequences. Artif Neural Netw Pattern Recognit Lect Notes Comput Sci 5998:149–160

    Article  Google Scholar 

  34. Shin G, Chun J (2008) Spatio-temporal facial expression recognition using optical flow and HMM, Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. Stud Comp Intell 149:27–38

    Google Scholar 

  35. Sumathi CP, Santhanam T, Mahadevi M (2012) Automatic facial expression analysis: a survey. Int J Comput Sci Eng Survey (IJCSES) 3(6):47–59. doi:10.5121/ijcses.2012.3604

    Article  Google Scholar 

  36. Surendran N, Xie S (2009) Automated facial expression recognition—an integrated approach with optical flow analysis and support vector machines. Int J Intell Syst Technol Appl 7:316–346

    Google Scholar 

  37. Tabbone S (1994) Detecting junctions using properties of the Laplacian of Gaussian detector. Pattern Recogn 1:52–56

    Google Scholar 

  38. Tang F, Deng B (2007) Facial expression recognition using AAM and local facial features. Int Conf Nat Comput, pp 632–635

  39. Tsai HH, Lai YS, Zhang YC (2010) Using SVM to design facial expression recognition for shape and texture features. Int Conf Mach Learn Cybern (ICMLC) 5:2697–2704. doi:10.1109/ICMLC.2010.5580938

    Google Scholar 

  40. Tsao WK, Lee AJT, Liu H, Chang HW, Lin HH (2010) A data mining approach to face detection. Pattern Recogn 43(3):1039–1049

    Article  MATH  Google Scholar 

  41. Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions. IEEE Trans Syst Man Cybern Part B: Cybern 42(1):28–43. doi:10.1109/TSMCB.2011.2163710

    Article  Google Scholar 

  42. Vezzetti E, Marcolin F (2012) 3D human face description: landmarks measures and geometrical features. Image Vis Comput 30(10):698–712

    Article  Google Scholar 

  43. Vezzetti E, Marcolin F (2014) 3D Landmarking in multiexpression face analysis: a preliminary study on eyebrows and mouth. Aesthetic Plastic Surg, ISSN 0364-216X

  44. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. Comput Vis Pattern Recognit, 511–518

  45. Vukadinovic D, Pantic M (2005) Fully automatic facial feature point detection using Gabor feature based boosted classifiers. IEEE Int Conf Syst, 1692–1698

  46. Wan C, Tian Y, Chen H, Wang X (2011) Rapid face detection algorithm of color images under complex background, in Proc. 8th Int. Symp. on Neural Networks, LNCS 6676: 356–363

  47. Wang Y, Ai H, Wu B, Huang C (2004) Real time facial expression recognition with AdaBoost, Pattern Recognition, Proceedings of the 17th International Conference on ICPR, 3: 926–929

  48. Wu T, Bartlett MS, Movellan, JR (2010) Facial expression recognition using Gabor motion energy filters. IEEE Comput Soc Conf Comput Vis Pattern Recognit Workshops, 42–47

  49. Wu T, Fu S, Yang G (2012) Survey of the facial expression recognition research. Adv Brain Inspired Cogn Syst Lect Notes Comput Sci 7366:392–402

    Article  Google Scholar 

  50. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  51. Zhao G, Pietikäinen M (2009) Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recogn Lett 30(12):1117–1127

    Article  Google Scholar 

  52. Zhu X, Ramanan D (2012) Face detection, pose estimation, and landmark localization in the wild, Computer Vision and Pattern Recognition (CVPR), Providence, Rhode Island

  53. Zilu Y, Jingwen L, Youwei Z (2008) Facial expression recognition based on two dimensional feature extraction. Int. Conf. Software Process, pp 1440–1444

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Tatung University, No. 40, Sec. 3, Jhongshan N. Rd., Taipei 104, Taiwan, Republic of China

    Chen-Chiung Hsieh, Meng-Kai Jiang & Yun-Maw Cheng

  2. Department of Industrial Design, Tatung University, No. 40, Sec. 3, Jhongshan N. Rd., Taipei 104, Taiwan, Republic of China

    Mei-Hua Hsih

  3. Department of Information Management, Tamkang University, No.151, Yingzhuan Rd. Tamsui Dist., New Taipei City, 25137, Taiwan

    En-Hui Liang

Authors
  1. Chen-Chiung Hsieh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mei-Hua Hsih
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng-Kai Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yun-Maw Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. En-Hui Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chen-Chiung Hsieh.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hsieh, CC., Hsih, MH., Jiang, MK. et al. Effective semantic features for facial expressions recognition using SVM. Multimed Tools Appl 75, 6663–6682 (2016). https://doi.org/10.1007/s11042-015-2598-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2598-1

Keywords

Search

Navigation