Skip to main content

Advertisement

Springer Nature Link
Log in

A survey: facial micro-expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression recognition plays a crucial role in a wide range of applications of psychotherapy, security systems, marketing, commerce and much more. Detecting a macro-expression, which is a direct representation of an ‘emotion,’ is a relatively straight-forward task. Playing a pivotal role as macro-expressions, micro-expressions are more accurate indicators of a train of thought or even subtle, passive or involuntary thoughts. Compared to macro-expressions, identifying micro-expressions is a much more challenging research question because their time spans are narrowed down to a fraction of a second, and can only be defined using a broader classification scale. This paper is an all-inclusive survey-cum-analysis of the various micro-expression recognition techniques. We analyze the general framework for micro-expression recognition system by decomposing the pipeline into fundamental components, namely face detecting, pre-processing, facial feature detection and extraction, datasets, and classification. We discuss the role of these elements and highlight the models and new trends that are followed in their design. Moreover, we provide an extensive analysis of micro-expression recognition systems by comparing their performance. We also discuss the new deep learning features that can, in the near future, replace the hand-crafted features for facial micro-expression recognition. This survey has been developed, focusing on the methodologies applied, databases used, performance regarding recognition accuracy and comparing these to distil the gaps in the efficiencies, future scope, and research potentials. Through this survey, we intend to look into this problem and develop a comprehensive and efficient recognition scheme. This study allows us to identify open issues and to determine future directions for designing real-world micro-expression recognition systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Agarwal S, Mukherjee DP (2017) Facial expression recognition through adaptive learning of local motion descriptor. Multimed Tools Appl 76:1073–1099

    Article  Google Scholar 

  2. Aifanti N, Papachristou C, Delopoulos A (2010) The MUG facial expression database. In: 11th international workshop on image analysis for multimedia interactive services WIAMIS 10, pp 1–4

  3. Asthana A, Zafeiriou S, Cheng S, Pantic M (2013) Robust discriminative response map fitting with constrained local models. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3444–3451

  4. Ben-Hur A, Weston J (2010) A user’s guide to support vector machines. In: Carugo O, Eisenhaber F (eds) Data mining techniques for the life sciences. Methods in Molecular Biology (Methods and Protocols), vol 609. Humana Press, New York

  5. Benyamin D (2012) A gentle introduction to random forests, ensembles, and performance metrics in a commercial system | Blog & Press. [online] Blog & Press. Available at: https://citizennet.com/blog/2012/11/10/random-forests-ensembles-and-performance-metrics/. Accessed 2 Nov 2015

  6. Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2:121–167

    Article  Google Scholar 

  7. Chaudhry R, Ravichandran A, Hager G, Vidal R (2009) Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: Computer vision and pattern recognition, 2009. CVPR 2009. IEEE conference on, pp 1932–1939

  8. Chavali GK, Bhavaraju SKNV, Adusumilli T, Puripanda V (2014) Micro-expression extraction for lie detection using Eulerian video (motion and color) magnification. Dissertation

  9. Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models-their training and application. Comput Vis Image Underst 61:38–59

    Article  Google Scholar 

  10. Cootes TF, Edwards GJ, Taylor CJ (1998) Active appearance models. In: European conference on computer vision, pp 484–498

  11. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23:681–685

    Article  Google Scholar 

  12. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 I.E. computer society conference on computer vision and pattern recognition (CVPR'05), pp 886–893

  13. Davison AK, Yap MH, Costen N, Tan K, Lansley C, Leightley D (2014) Micro-facial movements: an investigation on spatio-temporal descriptors. In: European conference on computer vision, pp 111–123

  14. Donia MM, Youssif AA, Hashad A (2014) Spontaneous facial expression recognition based on histogram of oriented gradients descriptor. Comput Inform Sci 7:31

    Article  Google Scholar 

  15. Eckman P (2003) Emotions revealed. St. Martin’s Griffin, New York

  16. Edwards GJ, Taylor CJ, Cootes TF (1998) Interpreting face images using active appearance models. In: Automatic face and gesture recognition, 1998. Proceedings. Third IEEE international conference on, pp 300–305

  17. Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists, Palo Alto

  18. Endres J, Laidlaw A (2009) Micro-expression recognition training in medical students: a pilot study. BMC Med Educ 9:47

    Article  Google Scholar 

  19. Guo Y, Tian Y, Gao X, Zhang X (2014) Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. In: 2014 international joint conference on neural networks (IJCNN), pp 3473–3479

  20. Guo Y, Xue C, Wang Y, Yu M (2015) Micro-expression recognition based on CBP-TOP feature with ELM. Optik Int J Light Electron Opt 126:4446–4451

    Article  Google Scholar 

  21. Horn BK, Schunck BG (1981) Determining optical flow. Artif Intell 17:185–203

    Article  Google Scholar 

  22. House C, Meyer R (2015) Preprocessing and descriptor features for facial micro-expression recognition [Online]. Available: https://web.stanford.edu/class/ee368/Project_Spring_1415/Reports/House_Meyer.pdf. Accessed 15 Oct 2016

  23. Huang X, Wang S-J, Zhao G, Piteikainen M (2015) Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops, pp 1–9

  24. Karn U (2016) An intutive explanation of convolutional neural networks [Online]. Available: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/. Accessed 2 Nov 2016

  25. Khan SA, Hussain A, Usman M (2017) Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features. Multimed Tools Appl. https://doi.org/10.1007/s11042-016-4324-z

  26. Kim I, Shim JH, Yang J (2010) Face detection. Comput Inf Sci 3:71–90

    Google Scholar 

  27. Kim DH, Baddar WJ, Ro YM (2016) Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 2016 ACM on multimedia conference, pp 382–386

  28. Lajevardi SM, Wu HR (2012) Facial expression recognition in perceptual color space. IEEE Trans Image Process 21:3721–3733

    Article  MathSciNet  MATH  Google Scholar 

  29. Le NgoAC, Oh Y-H, Phan RC-W, See J (2016) Eulerian emotion magnification for subtle expression recognition. In: acoustics, speech and signal processing (ICASSP), 2016 I.E. international conference on, pp 1243–1247

  30. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: inducement, collection and baseline. In: Automatic face and gesture recognition (FG), 2013 10th IEEE international conference and workshops on, pp 1–6

  31. Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G et al (2015) Reading hidden emotions: spontaneous micro-expression spotting and recognition. arXiv preprint arXiv:1511.00423

  32. Liong S-T, See J, Phan RC-W, Le Ngo AC, Oh Y-H, Wong K (2014) Subtle expression recognition using optical strain weighted features. In: Asian Conference on Computer Vision, pp 644–657

  33. Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput PP:1–1

    Google Scholar 

  34. Lu Z, Luo Z, Zheng H, Chen J, Li W (2014) A Delaunay-based temporal coding model for micro-expression recognition. In: Asian conference on computer vision, pp 698–711

  35. Lucas BD (1985) Generalized image matching by the method of differences

  36. Lun X, Xin L, Xiujun Y, Zhiliang W (2016) Cognitive regulation and emotion modeling for micro-expression. Int J Control Autom 9:361–372

    Article  Google Scholar 

  37. Lyons M, Kamachi M, Gyoba J (1997) Japanese female facial expressions (JAFFE). Database of Digital Images

  38. Matsugu M, Mori K, Mitari Y, Kaneda Y (2003) Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw 16:555–559

    Article  Google Scholar 

  39. Patel D, Zhao G, Pietikäinen M (2015) Spatiotemporal integration of optical flow vectors for micro-expression detection. In: International conference on advanced concepts for intelligent vision systems, pp 369–380

  40. Pfister T, Li X, Zhao G, Pietikäinen M (2011) Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: Computer vision workshops (ICCV workshops), 2011 I.E. international conference on, pp 868–875

  41. Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 international conference on computer vision, pp 1449–1456

  42. Polikovsky S, Kameda Y (2013) Facial micro-expression detection in hi-speed video based on facial action coding system (FACS). IEICE Trans Inf Syst 96:81–92

    Article  Google Scholar 

  43. Rowley HA, Baluja S, Kanade T (1998) Neural network-based face detection. IEEE Trans Pattern Anal Mach Intell 20:23–38

    Article  Google Scholar 

  44. Sampangi RV, Hawkey K (2014) Can’t) Lie to ME: Using Micro Expressions for User Authentication. In: Who are you?! Adventures in Authentication: WAY Workshop, Symposium on Usable Privacy and Security (SOUPS) 2014

  45. Shen X-b, Wu Q, Fu X-l (2012) Effects of the duration of expressions on the recognition of microexpressions. J Zhejiang Univ Sci B 13:221–230

    Article  Google Scholar 

  46. Shreve M, Godavarthy S, Goldgof D, Sarkar S (2011) Macro-and micro-expression spotting in long videos using spatio-temporal strain. In: Automatic Face & Gesture Recognition and workshops (FG 2011), 2011 I.E. international conference on, pp 51–56

  47. Song Y, Morency L-P, Davis R (2013) Learning a sparse codebook of facial and body microexpressions for emotion recognition. In: proceedings of the 15th ACM on international conference on multimodal interaction, pp 237–244

  48. Sung K-K, Poggio T (1998) Example-based learning for view-based human face detection. IEEE Trans Pattern Anal Mach Intell 20:39–51

    Article  Google Scholar 

  49. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57:137–154

    Article  Google Scholar 

  50. Wang P, Ji Q (2004) Multi-view face detection under complex scene based on combined SVMs. In: ICPR (4), pp 179–182

  51. Wang S-J, Yang J, Zhang N, Zhou C-G (2011) Tensor discriminant color space for face recognition. IEEE Trans Image Process 20:2490–2501

    Article  MathSciNet  MATH  Google Scholar 

  52. Wang S-J, Yang J, Sun M-F, Peng X-J, Sun M-M, Zhou C-G (2012) Sparse tensor discriminant color space for face verification. IEEE Trans Neural Netw Learn Syst 23:876–888

    Article  Google Scholar 

  53. Wang S-J, Chen H-L, Yan W-J, Chen Y-H, Fu X (2014) Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process Lett 39:25–43

    Article  Google Scholar 

  54. Wang S-J, Yan W-J, Zhao G, Fu X, Zhou C-G (2014) Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In: Workshop at the European conference on computer vision, pp 325–338

  55. Wang S, Yan W-J, Li X, Zhao G, Fu X (2014) Micro-expression recognition using dynamic textures on tensor independent color space. In: ICPR, pp 4678–4683

  56. Wang Y, See J, Phan RC-W, Oh Y-H (2014) Lbp with six intersection points: reducing redundant information in lbp-top for micro-expression recognition. In: Asian Conference on Computer Vision, pp 525–537

  57. Wang S-J, Yan W-J, Li X, Zhao G, Zhou C-G, Fu X et al (2015) Micro-expression recognition using color spaces. IEEE Trans Image Process 24:6034–6047

    Article  MathSciNet  Google Scholar 

  58. Wang Y, See J, Phan RC-W, Oh Y-H (2015) Efficient Spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PLoS One 10:e0124674

    Article  Google Scholar 

  59. Wang Y, See J, Oh YH et al (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76: 21665. https://doi.org/10.1007/s11042-016-4079-6

  60. Warren G, Schertler E, Bull P (2009) Detecting deception from emotional and unemotional cues. J Nonverbal Behav 33:59–69

    Article  Google Scholar 

  61. Wu Q, Shen X, Fu X (2011) The machine knows what you are hiding: an automatic micro-expression recognition system. In: international conference on affective computing and intelligent Interaction, pp 152–162

  62. Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267

  63. Yan W-J, Wu Q, Liu Y-J, Wang S-J, Fu X (2013) CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: Automatic face and gesture recognition (FG), 2013 10th IEEE international conference and workshops on, pp 1–7

  64. Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H et al (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9:e86041

    Article  Google Scholar 

  65. Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurement of levels of interest from video. IEEE Trans Multimedia 8:500–508

    Article  Google Scholar 

  66. Zhang M, Fu Q, Chen Y-H, Fu X (2014) Emotional context influences micro-expression recognition. PLoS One 9:e95018

    Article  Google Scholar 

  67. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29:915–928

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing and Communications, Faculty of Engineering and Information Technology, University of Technology Sydney, 15 Broadway, Ultimo, NSW, 2007, Australia

    Madhumita Takalkar, Min Xu, Qiang Wu & Zenon Chaczko

Authors
  1. Madhumita Takalkar
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Min Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Qiang Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Zenon Chaczko
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Madhumita Takalkar.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Takalkar, M., Xu, M., Wu, Q. et al. A survey: facial micro-expression recognition. Multimed Tools Appl 77, 19301–19325 (2018). https://doi.org/10.1007/s11042-017-5317-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5317-2

Keywords