Skip to main content
SpringerLink
Log in

Hierarchical support vector machine for facial micro-expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The sample category distribution of spontaneous facial micro-expression datasets is unbalanced, due to the experimental environment, collection equipment, and individualization of subjects, which brings great challenges to micro-expression recognition. Therefore, this paper introduces a micro-expression recognition model based on the Hierarchical Support Vector Machine (H-SVM) to reduce the interference of sample category distribution imbalance. First, we calculated the position of the apex frame in the micro-expression image sequence. To keep micro-expression frames balanced, we sparsely sample the images sequence according to the apex frame. Then, the Low-level Descriptors of the region of interest of the micro-expression image sequence and the High-level Descriptors of apex frame are extracted. Finally, the H-SVM model is used to classify the fusion features of different levels. The experimental results on SMIC, CAMSE2, SAMM, and their composite datasets show that our method can achieve superior performance in micro-expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  2. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2018) SAMM: A spontaneous micro-facial movement dataset. IEEE Transactions Affective Computing 9(1):116–129

    Google Scholar 

  3. Davison AK, Yap MH, Costen N, Tan K, Lansley C, Leightley D (2014). Micro-facial movements: an investigation on spatio-temporal descriptors. In: European conference on computer vision. Springer, pp 111–123

  4. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106

    Google Scholar 

  5. Fan DP, Zhang YH, Wu YH, Liu Y, Cheng MM, Ren B, Rosin P, Ji R (2019). Scoot: a perceptual metric for facial sketches. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE, pp 5611–5621

  6. Fernandez A, Garcia S, Herrera F, Chawla NV (2018) Smote for learning from imbalanced data: progress and challenges, marking the 15-year anniversary. J Artif Intell Res 61:863–905

    MathSciNet  MATH  Google Scholar 

  7. Gan YS, Liong ST, Yau WC, Huang YC, Tan LK (2019) OFF-ApexNet on micro-expression recognition system. Signal Process Image Commun 74:129–139

    Google Scholar 

  8. Grinsven MJ, Ginneken B, Hoyng CB, Theelen T, Sánchez CI (2016) Fast convolutional neural network training using selective data sampling: application to hemorrhage detection in color fundus images. IEEE Trans Med Imaging 35(5):1273–1284

    Google Scholar 

  9. Haggard EA, Isaacs KS (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy. Springer, pp 154–165

  10. He H, Garcia EA (2008) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21(9):1263–1284

    Google Scholar 

  11. Huang X, Wang SJ, Zhao G, Piteikainen M (2015). Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops (ICCV workshops). IEEE, pp. 1–9

  12. Huang X, Zhao G, Hong X, Zheng W, Pietikäinen M (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578

    Google Scholar 

  13. Khan SA, Naveed R, Sheeraz A, Shahzad L (2015) Facial expression recognition using computationally intelligent techniques. J Intell Fuzzy Syst 28(6):2881–2887

    Google Scholar 

  14. Kim DH, Wissam JB, Yong MR (2016). Micro-expression recognition with expression-state constrained Spatio-temporal feature representations. In: Proceedings of the 24th ACM international conference on multimedia. ACM, pp 382–386

  15. Le Ngo AC, Phan RCW, See J (2014). Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Asian conference on computer vision (ACCV). Springer, pp 33–48

  16. Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikainen M (2015). Reading hidden emotions: spontaneous micro-expression spotting and recognition. arXiv:http://arXiv.org/abs/1511.00423

  17. Li X, Pfister T, Huang X, Zhao G, Pietikainen M (2013). A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–6

  18. Li Q, Zhan S, Xu L, Wu C (2018) Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow. Multimed Tools Appl 78(20):29307–29322

    Google Scholar 

  19. Liong ST, See J, Wong K, Le Ngo AC, Oh YH, Phan R (2015). Automatic apex frame spotting in micro-expression database. In: 2015 3rd IAPR Asian conference on pattern recognition (ACPR). IEEE, pp 665–669

  20. Liong ST, See J, Wong K, Phan RCW (2018) Less is more: micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92

    Google Scholar 

  21. Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Google Scholar 

  22. Ma Y, Chen W, Ma X, Xu J, Huang X, Ross M, Anthony KH (2017) EasySVM: a visual analysis approach for open-box support vector machines. Computational Visual Media 3(2):161–175

    Google Scholar 

  23. Matsumoto D, Hwang HS (2011) Evidence for training the ability to read micro expressions of emotion. Motiv Emot 35(2):181–191

    Google Scholar 

  24. Merghani W, Davison AK, Yap M (2018). Facial micro-expressions grand challenge 2018: evaluating spatio-temporal features for classification of objective classes. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, pp 662–666

  25. Merghani W, Davison AK, Yap MH (2019) The implication of spatial temporal changes on facial micro-expression analysis. Multimed Tools Appl 78(15):21613–21628

    Google Scholar 

  26. MunirA AH, Khan SA, Muhammad N, Sadia A (2018) Illumination invariant facial expression recognition using selected merged binary patterns for real world images. Optik 158:1016–1025

    Google Scholar 

  27. Nie, GY, Cheng MM, Liu Y, Liang Z, Fan DP, Liu Y, Wang Y (2019) Multi-level context ultra-aggregation for stereo matching. In: 2019 IEEE/CVF conference on computer vision and pattern recognition. IEEE, pp 3283–3291

  28. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59

    Google Scholar 

  29. Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence 7:971–987

    MATH  Google Scholar 

  30. OSullivan M, Frank MG, Hurley CM, Tiwana J (2009) Police lie detection accuracy: the effect of lie scenario. Law Hum Behav 33(6):530–538

    Google Scholar 

  31. Patel D, Hong X, Zhao G (2016). Selective deep features for micro-expression recognition. In: 2016 23rd international conference on pattern recognition (ICPR 2016). IEEE, pp 2258–2263

  32. Peng M, Wang C, Chen T, Liu G, Fu X (2017) Dual temporal scale convolutional neural network for micro-expression recognition. Front Psychol 8:1745–1745

    Google Scholar 

  33. Pfister T, Li X, Zhao G, Pietikainen M (2011). Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops). IEEE, pp 868–875

  34. Polikovsky S, Kameda, Y, Ohta Y (2009). Facial micro-expressions recognition using high-speed camera and 3D-gradient descriptor. In: 3rd international conference on crime detection and prevention (ICDP 2009). IET, pp 1–6

  35. Russell TA, Chu E, Phillips ML (2006) A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. Br J Clin Psychol 45(4):579–583

    Google Scholar 

  36. Salter F, Grammer K, Rikowski A (2005) Sex differences in negotiating with powerful males. Hum Nat 16(3):306–321

    Google Scholar 

  37. Shabat AM, Tapamo JR (2018) Angled local directional pattern for texture analysis with an application to facial expression recognition. IET Comput Vis 12(5):603–608

    Google Scholar 

  38. Shreve M, Godavarthy S, Goldgof D, Sarkar S (2011). Macro-and micro-expression spotting in long videos using Spatio-temporal strain. In: 2011 IEEE international conference on automatic face gesture recognition and workshops (FG 2011), pp 51–56

  39. Takalkar M A, Xu M (2017). Image based facial micro-expression recognition using deep learning on small datasets. In: 2017 international conference on digital image computing: techniques and applications (DICTA 2017). IEEE, pp 1–7

  40. Wang Y, See J, Phan RCW, Oh YH (2014). Lbp with six intersection points: reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision (ACCV). Springer, pp 525–537

  41. Weiss J, Ekman P (2009) Telling lies : clues to deceit in the marketplace, politics, and marriage. New York: Norton. Am J Clin Hypn 53(4):287–288

    Google Scholar 

  42. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):1–8

    Google Scholar 

  43. Yan WJ, Wu Q, Liang J, Chen YH, Fu X (2013) How fast are the leaked facial expressions: the duration of micro-expressions. J Nonverbal Behav 37(4):217–230

    Google Scholar 

  44. Yan WJ, Wu Q, Liu YJ, Wang SJ, Fu X (2013). CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–7

  45. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis & Machine Intelligence 6:915–928

    Google Scholar 

  46. Zhou Z, Zhao G, Pietikainen M (2011). Towards a practical lipreading system. In: Proceedings of the 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 137–144

  47. Zong Y, Huang X, Zheng W, Cui Z, Zhao G (2018) Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Transactions on Multimedia 20(11):3160–3172

    Google Scholar 

Download references

Funding

This work was supported by the National Key R&D Program of China (No. 2018YFC 2001700), National Natural Science Foundation of China (No. 61672093), Beijing Municipal Natural Science Foundation (No. L192005), and Advanced Innovation Center for Intelligent Robots and Systems Open Research Project (No.2018IRS01).

Author information

Authors and Affiliations

  1. School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China

    Hang Pan, Lun Xie & Zhiliang Wang

  2. Affiliated Rehabilitation Hospital of National Research Center for Rehabilitation Technical Aids, Beijing, China

    Zeping Lv

  3. Center on Aging Psychology, Key Laboratory of Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, China

    Juan Li

Authors
  1. Hang Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lun Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zeping Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhiliang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lun Xie.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pan, H., Xie, L., Lv, Z. et al. Hierarchical support vector machine for facial micro-expression recognition. Multimed Tools Appl 79, 31451–31465 (2020). https://doi.org/10.1007/s11042-020-09475-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09475-4

Keywords

Search

Navigation