Skip to main content
SpringerLink
Log in

A Survey of Micro-expression Recognition Methods Based on LBP, Optical Flow and Deep Learning

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Micro-expressions typically reflect suppressed feelings and they can provide an accurate indication of the real feelings and motivations of a person. Accordingly, they have been used for clinical diagnosis, business negotiations, interrogations, and security research. However, it is difficult to detect micro-expressions because of their instantaneity and imperceptibility. Therefore, micro-expression recognition is challenging. So far, various micro-expression recognition algorithms have been proposed to improve micro-expression recognition performance, which LBP, optical flow method and deep learning methods have made good progress as mainstream algorithms. In this survey, we aim to provide a review of micro-expression recognition based on LBP, optical flow, and deep learning. We first introduce the current commonly used micro-expression datasets. Then the three mainstream classical methods, LBP, optical flow, and deep learning, are described and summarized respectively. The existing methods in the last 5 years are discussed in regard to datasets, pre-processing, evaluation metrics and accuracy. Finally, we explain the shortcomings and challenges of micro-expression recognition and propose the future directions. This can help researchers to more quickly understand the current status of research, problems and future development directions, and provide a reference point for further research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Ben X, Ren Y, Zhang J, Wang SJ, Kpalma K, Meng W, Liu YJ (2021) Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans Pattern Anal. https://doi.org/10.1109/TPAMI.2021.3067464

    Article  Google Scholar 

  2. Takalkar M, Xu M, Wu Q, Chaczko Z (2017) A survey: facial micro-expression recognition. Multimed Tools Appl 77:19301–19325. https://doi.org/10.1007/s11042-017-5317-2

    Article  Google Scholar 

  3. Goh KM, Ng CH, Lim LL, Sheikh UU (2020) Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis Comput 36:445–468. https://doi.org/10.1007/s00371-018-1607-6

    Article  Google Scholar 

  4. Li XB, Hong XP, Moilanen A, Huang XH, Pfister T, Zhao GY, Pietikainen M (2018) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9:563–577. https://doi.org/10.1109/TAFFC.2017.2667642

    Article  Google Scholar 

  5. Frank MG, Svetieva E (2015) Microexpressions and deception. In: Mandal M, Awasthi A (eds) Understanding facial expressions in communication. Springer, New Delhi, pp 227–242

    Chapter  Google Scholar 

  6. Stanciu L, Albu A (2019) Analysis on emotion detection and recognition methods using facial microexpressions. A review. In: 2019 E-health and bioengineering conference (EHB). https://doi.org/10.1109/EHB47216.2019.8969925

  7. Ekman P (2004) Emotions revealed. BMJ 328:0405184. https://doi.org/10.1136/sbmj.0405184

    Article  Google Scholar 

  8. Ekman P, Friesen W (1978) Facial action coding system (FACS): a technique for the measurement of facial action. Consulting, Palo Alto

    Google Scholar 

  9. Tian YI, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal 23:97–115. https://doi.org/10.1109/34.908962

    Article  Google Scholar 

  10. Oh YH, Le Ngo AC, See J, Liong ST, Phan RCW, Ling HC (2015) Monogenic Riesz wavelet representation for micro-expression recognition. In: 2015 IEEE international conference on digital signal processing (DSP). https://doi.org/10.1109/ICDSP.2015.7252078

  11. Wu Q, Shen XB, Fu XL (2011) The machine knows what you are hiding: an automatic micro-expression recognition system. In: Affective computing and intelligent interaction—fourth international conference, ACII 2011, Memphis, TN, USA 6975, pp 152-162. https://doi.org/10.1007/978-3-642-24571-8_16

  12. Oh YH, See J, Le Ngo AC, Phan RCW, Baskaran VM (2018) A survey of automatic facial micro-expression analysis: databases, methods, and challenges. Front Psychol 9:1128. https://doi.org/10.3389/fpsyg.2018.01128

    Article  Google Scholar 

  13. Khan SA, Hussain A, Usman M (2018) Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features. Multimed Tools Appl 77:1133–1165. https://doi.org/10.1007/s11042-016-4324-z

    Article  Google Scholar 

  14. Agarwal S, Mukherjee DP (2017) Facial expression recognition through adaptive learning of local motion descriptor. Multimed Tools Appl 76:1073–1099. https://doi.org/10.1007/s11042-015-3103-6

    Article  Google Scholar 

  15. Zhang LF, Arandjelovic O (2021) Review of automatic microexpression recognition in the past decade. Mach Learn Knowl Extr 3:414–434. https://doi.org/10.3390/make3020021

    Article  Google Scholar 

  16. Zhou L, Shao XY, Mao QR (2021) A survey of micro-expression recognition. Image Vision Comput 105:104043. https://doi.org/10.1016/j.imavis.2020.104043

    Article  Google Scholar 

  17. Zhang P, Ben X, Yan R, Wu C, Guo C (2016) Micro-expression recognition system. Optik 127:1395–1400. https://doi.org/10.1016/j.ijleo.2015.10.217

    Article  Google Scholar 

  18. Wang Y, See J, Phan RCW, Oh YH (2015) Lbp with six intersection points: reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision 9003, pp 525–537. https://doi.org/10.1007/978-3-319-16865-4_34

  19. Huang X, Zhao G, Hong X, Zheng W, Pietikainen M (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578. https://doi.org/10.1016/j.neucom.2015.10.096

    Article  Google Scholar 

  20. Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7:299–310. https://doi.org/10.1109/TAFFC.2015.2485205

    Article  Google Scholar 

  21. Xu F, Zhang J, Wang JZ (2016) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8:254–267. https://doi.org/10.1109/TAFFC.2016.2518162

    Article  Google Scholar 

  22. Huang YX, Chen F, Lv SH, Wang XD (2019) Facial expression recognition: a survey. Symmetry-Basel 11:1189. https://doi.org/10.3390/sym11101189

    Article  Google Scholar 

  23. Liu L, Ouyang WL, Wang XG, Fieguth P, Chen J, Liu XW, Pietikainen M (2020) Deep learning for generic object detection: a survey. Int J Comput Vis 128:261–318. https://doi.org/10.1007/s11263-019-01247-4

    Article  MATH  Google Scholar 

  24. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: 3rd International conference on imaging for crime detection and prevention (ICDP 2009). https://doi.org/10.1049/ic.2009.0244

  25. Shreve M, Godavarthy S, Goldgof D, Sarkar S (2011) Macro-and micro-expression spotting in long videos using spatio-temporal strain. In 2011 IEEE international conference on automatic face & gesture recognition (FG). https://doi.org/10.1109/FG.2011.5771451

  26. Warren G, Schertler E, Bull P (2009) Detecting deception from emotional and unemotional cues. J Nonverbal Behav 33:59–69. https://doi.org/10.1007/s10919-008-0057-7

    Article  Google Scholar 

  27. Li X, Pfister T, Huang X, Zhao G, Pietikainen M (2013) A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–6. https://doi.org/10.1109/FG.2013.655371

  28. Yan WJ, Wu Q, Liu YJ, Wang SJ, Fu X (2013) CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–7. https://doi.org/10.1109/FG.2013.6553799

  29. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9:e86041. https://doi.org/10.1371/journal.pone.0086041

    Article  Google Scholar 

  30. Qu F, Wang SJ, Yan WJ, Li H, Wu S, Fu X (2017) CAS (ME)2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans Affect Comput 9:424–436. https://doi.org/10.1109/TAFFC.2017.2654440

    Article  Google Scholar 

  31. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2016) SAMM: a spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9:116–129. https://doi.org/10.1109/TAFFC.2016.2573832

    Article  Google Scholar 

  32. Husák P, Cech J, Matas J (2017) Spotting facial micro-expressions “in the wild”. In: Proceedings of computer vision winter workshop (Retz), pp 1–9

  33. Zhang M, Fu Q, Chen YH, Fu X (2014) Emotional context influences micro-expression recognition. PLoS ONE 9:e95018. https://doi.org/10.1371/journal.pone.0095018

    Article  Google Scholar 

  34. Shen X, Wu Q, Fu X (2012) Effects of the duration of expressions on the recognition of micro-expressions. J Zhejiang Univ Sci B 13:221–230. https://doi.org/10.1631/jzus.B1100063

    Article  Google Scholar 

  35. Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models-their training and application. Comput Vis Image Underst 61:38–59. https://doi.org/10.1006/cviu.1995.1004

    Article  Google Scholar 

  36. Matthews I, Baker S (2004) Active appearance models revisited. Int J Comput Vis 60:135–164. https://doi.org/10.1023/B:VISI.0000029666.37597.d3

    Article  Google Scholar 

  37. Cristinacce D, Cootes TF (2006) Feature detection and tracking with constrained local models. In: Bmvc, vol 1, p 3

  38. Asthana A, Zafeiriou S, Cheng S, Pantic M (2013) Robust discriminative response map fitting with constrained local models. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3444–3451

  39. Zhou E, Fan H, Cao Z, Jiang Y, Yin Q (2013) Extensive facial landmark localization with coarse-to-fine convolutional network cascade. In: Proceedings of the IEEE international conference on computer vision workshops, pp 386–391

  40. Zhang Z, Luo P, Loy CC, Tang X (2014) Facial landmark detection by deep multi-task learning. In: European conference on computer vision, vol 8694, pp 94–108. https://doi.org/10.1007/978-3-319-10599-4_7

  41. Wu Y, Hassner T, Kim K, Medioni G, Natarajan P (2017) Facial landmark detection with tweaked convolutional neural networks. IEEE Trans Pattern Anal 40:3067–3074. https://doi.org/10.1109/TPAMI.2017.2787130

    Article  Google Scholar 

  42. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Proc Lett 23:1499–1503. https://doi.org/10.1109/LSP.2016.2603342

    Article  Google Scholar 

  43. Kowalski M, Naruniec J, Trzcinski T (2017) Deep alignment network: A convolutional neural network for robust face alignment. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 88–97

  44. Wu J, Xu J, Lin D, Tu M (2020) Optical flow filtering-based micro-expression recognition method. Electronics 9:2056. https://doi.org/10.3390/electronics9122056

    Article  Google Scholar 

  45. Chen B, Zhang Z, Liu N, Tan Y, Liu X, Chen T (2020) Spatiotemporal convolutional neural network with convolutional block attention module for micro-expression recognition. Information 11:380. https://doi.org/10.3390/info11080380

    Article  Google Scholar 

  46. Chen JG, Wadhwa N, Cha YJ, Durand F, Freeman WT, Buyukozturk O (2014) Structural modal identification through high speed camera video: motion magnification. Top Modal Anal I 7:191–197. https://doi.org/10.1007/978-3-319-04753-9_19

    Article  Google Scholar 

  47. Trang Thanh Quynh L, Tran TK, Rege M (2020) Rank-pooling-based features on localized regions for automatic micro-expression recognition. Int J Multimed Data 11:25–37

    Article  Google Scholar 

  48. Xia Z, Hong X, Gao X, Feng X, Zhao G (2020) Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans Multimed 22:626–640. https://doi.org/10.1109/TMM.2019.2931351

    Article  Google Scholar 

  49. Le Ngo AC, Johnston A, Phan RC, See J (2018) Micro-expression motion magnification: global Lagrangian vs. local Eulerian approaches. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG), pp.650–656. https://doi.org/10.1109/FG.2018.00102

  50. Zhou Z, Zhao G, Pietikainen M (2011) Towards a practical lipreading system. In: CVPR 2011, pp 137–144. https://doi.org/10.1109/CVPR.2011.5995345

  51. Huang X, Zhao G (2017) Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern. In: 2017 International conference on the frontiers and advances in data science (FADS), pp 159–164. https://doi.org/10.1109/FADS.2017.8253219

  52. Annoni J, Seiler P, Jovanović MR (2016) Sparsity-promoting dynamic mode decomposition for systems with inputs. In: 2016 IEEE 55th conference on decision and control (CDC), pp 6506–6511. https://doi.org/10.1109/CDC.2016.7799270

  53. Wang SJ, Yan WJ, Li X, Zhao G, Fu X (2014) Micro-expression recognition using dynamic textures on tensor independent color space. In: 2014 22nd international conference on pattern recognition, pp 4678–4683. https://doi.org/10.1109/ICPR.2014.800

  54. Wang SJ, Yan WJ, Zhao GY, Fu X, Zhou CG (2015) Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In: European conference on computer vision, vol 8925, pp 325–338

  55. Huang X, Wang SJ, Zhao G, Piteikainen M (2015) Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops, pp 1–9

  56. Huang X, Wang S, Liu X, Zhao G, Feng X, Piteikainen M (2019) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 10:32–47. https://doi.org/10.1109/TAFFC.2017.2713359

    Article  Google Scholar 

  57. Happy SL, Routray A (2017) Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans Affect Comput 10:394–406. https://doi.org/10.1109/TAFFC.2017.2723386

    Article  Google Scholar 

  58. Chaudhry R, Ravichandran A, Hager G, Vidal R (2009) Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: 2009 IEEE conference on computer vision and pattern recognition, pp 1932–1939. https://doi.org/10.1109/CVPR.2009.5206821

  59. Allaert B, Bilasco LM, Djeraba C (2017) Consistent optical flow maps for full and micro facial expression recognition. In: VISAPP, pp 235–242. https://doi.org/10.5220/0006127402350242

  60. Liong ST, See J, Wong KS, Phan RCW (2018) Less is more: micro-expression recognition from video using apex frame. Signal Process-Image 62:82–92. https://doi.org/10.1016/j.image.2017.11.006

    Article  Google Scholar 

  61. Esmaeili V, Mohassel Feghhi M, Shahdi SO (2022) A comprehensive survey on facial micro-expression: approaches and databases. Multimed Tools Appl. https://doi.org/10.1007/s11042-022-13133-2

    Article  MATH  Google Scholar 

  62. Liu YJ, Li BJ, Lai YK (2018) Sparse MDMO: learning a discriminative feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 12:254–261. https://doi.org/10.1109/TAFFC.2018.2854166

    Article  Google Scholar 

  63. Quang NV, Chun J, Tokuyama T (2019) CapsuleNet for micro-expression recognition. In: 2019 14th IEEE international conference on automatic face & gesture recognition (FG), pp 1–7. https://doi.org/10.1109/FG.2019.8756544

  64. Xia B, Wang W, Wang S, Chen E (2020) Learning from macro-expression: a micro-expression recognition framework. In: Proceedings of the 28th ACM international conference on multimedia. https://doi.org/10.1145/3394171.3413774

  65. Xia ZQ, Peng W, Khor HQ, Feng XY, Zhao GY (2020) Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Trans Image Process 29:8590–8605. https://doi.org/10.1109/TIP.2020.3018222

    Article  MATH  Google Scholar 

  66. Kim DH, Baddar WJ, Ro YM (2016) Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: ACM on multimedia conference, pp 382–386. https://doi.org/10.1145/2964284.2967247

  67. Peng M, Wang CY, Chen T, Liu GY, Fu XL (2017) Dual temporal scale convolutional neural network for micro-expression recognition. Front Psychol 8:1745. https://doi.org/10.3389/fpsyg.2017.01745

    Article  Google Scholar 

  68. Li J, Wang YD, See J, Liu WB (2018) Micro-expression recognition based on 3D flow convolutional neural network. Pattern Anal Appl 22:1331–1339. https://doi.org/10.1007/s10044-018-0757-5

    Article  MathSciNet  Google Scholar 

  69. Jia XT, Ben XY, Yuan H, Kpalma K, Meng WX (2018) Macro-to-micro transformation model for micro-expression recognition. J Comput Sci-Neth 25:289–297. https://doi.org/10.1016/j.jocs.2017.03.016

    Article  Google Scholar 

  70. Gan YS, Liong S (2018) Bi-directional vectors from apex in CNN for micro-expression recognition. In: 2018 IEEE 3rd international conference on image, vision and computing (ICIVC), pp 168–172. https://doi.org/10.1109/ICIVC.2018.8492829

  71. Teja Reddy SP, Teja Karri S, Dubey SR, Mukherjee S (2019) Spontaneous facial micro-expression recognition using 3D spatiotemporal convolutional neural networks. In: 2019 international joint conference on neural networks (IJCNN), pp 1–8. https://doi.org/10.1109/IJCNN.2019.8852419

  72. Khor HQ, See J, Phan RCW, Lin W (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp 667–674. https://doi.org/10.1109/FG.2018.00105

  73. Song BL, Li K, Zong Y, Zhu J, Zheng WM, Shi JG, Zhao L (2019) Recognizing spontaneous micro-expression using a three-stream convolutional neural network. IEEE Access 7:184537–184551. https://doi.org/10.1109/ACCESS.2019.2960629

    Article  Google Scholar 

  74. Liong ST, Gan YS, See J, Khor HQ, Huang YC (2019) Shallow triple stream three-dimensional CNN (ststnet) for micro-expression recognition. In: 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019), pp 1–5. https://doi.org/10.1109/FG.2019.8756567

  75. Xia Z, Feng X, Hong X, Zhao G (2018) Spontaneous facial micro-expression recognition via deep convolutional network. In: 2018 eighth international conference on image processing theory, tools and applications (IPTA), pp 1–6. https://doi.org/10.1109/IPTA.2018.8608119

  76. Gupta P (2021) MERASTC: micro-expression recognition using effective feature encodings and 2D convolutional neural network. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2021.3061967

    Article  Google Scholar 

  77. Lo L, Xie HX, Shuai HH, Cheng WH (2020) MER-GCN: micro-expression recognition based on relation modeling with graph convolutional networks. In: 2020 IEEE conference on multimedia information processing and retrieval (MIPR), pp 79–84. https://doi.org/10.1109/MIPR49039.2020.00023

  78. Seymour G (1974) A predictive approach to the random effect model. 1974. Biometrika 61:101–107. https://doi.org/10.1093/biomet/61.1.101

    Article  MathSciNet  MATH  Google Scholar 

  79. Seymour G (1975) The predictive sample reuse method with applications. J Am Stat Assoc 70:320–328

    Article  MATH  Google Scholar 

  80. He JC, Hu JF, Lu X, Zheng WS (2017) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recognit 66:44–52. https://doi.org/10.1016/j.patcog.2016.11.029

    Article  Google Scholar 

  81. Hu CL, Chen JJ, Zuo X, Zou HT, Deng X, Shu YC (2019) Gender-specific multi-task micro-expression recognition using pyramid CGBP-TOP feature. CMES-Comp Model Eng Sci 118:547–559. https://doi.org/10.31614/cmes.2019.04032

    Article  Google Scholar 

  82. Niu M, Tao J, Li Y, Huang J, Lian Z (2019) Discriminative video representation with temporal order for micro-expression recognition. In: ICASSP 2019–2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 2112–2116. https://doi.org/10.1109/ICASSP.2019.8682295

  83. Yu M, Guo ZQ, Yu Y, Wang Y, Cen SX (2019) Spatiotemporal feature descriptor for micro-expression recognition using local cube binary pattern. IEEE Access 7:159214–159225. https://doi.org/10.1109/ACCESS.2019.2950339

    Article  Google Scholar 

  84. Guo CY, Liang JY, Zhan G, Liu Z, Pietikainen M (2019) Extended local binary patterns for efficient and robust spontaneous facial micro-expression recognition. IEEE Access 7:174517–174530. https://doi.org/10.1109/ACCESS.2019.2942358

    Article  Google Scholar 

  85. Pan H, Xie L, Lv ZP, Li J, Wang ZL (2020) Hierarchical support vector machine for facial micro-expression recognition. Multimed Tools Appl 79:31451–31465. https://doi.org/10.1007/s11042-020-09475-4

    Article  Google Scholar 

  86. Sun Z, Hu ZP, Zhao MY, Li SF (2020) Multi-scale active patches fusion based on spatiotemporal LBP-TOP for micro-expression recognition. J Vis Commun Image Represent 71:102862. https://doi.org/10.1016/j.jvcir.2020.102862

    Article  Google Scholar 

  87. Cen SX, Yu Y, Yan G, Yu M, Yang Q (2020) Sparse spatiotemporal descriptor for micro-expression recognition using enhanced local cube binary pattern. Sensors 20:4437. https://doi.org/10.3390/s20164437

    Article  Google Scholar 

  88. Guermazi R, Benabdallah T, Hammami M (2021) Facial micro-expression recognition based on accordion spatio-temporal representation and random forests. J Vis Commun Image Represent 79:103183. https://doi.org/10.1016/j.jvcir.2021.103183

    Article  Google Scholar 

  89. Zhao Y, Xu JC (2019) An improved micro-expression recognition method based on necessary morphological patches. Symmetry-Basel 11:497. https://doi.org/10.3390/sym11040497

    Article  Google Scholar 

  90. Gao JL, Chen HJ, Zhang XH, Guo J, Liang WY (2020) A new feature extraction and recognition method for microexpression based on local non-negative matrix factorization. Front Neurorobotics 14:579338. https://doi.org/10.3389/fnbot.2020.579338

    Article  Google Scholar 

  91. Patel D, Hong X, Zhao G (2016) Selective deep features for micro-expression recognition. In: 2016 23rd international conference on pattern recognition (ICPR), pp 2258–2263. https://doi.org/10.1109/ICPR.2016.7899972

  92. Li QY, Zhan S, Xu LF, Wu CZ (2018) Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow. Multimed Tools Appl 78:29307–29322. https://doi.org/10.1007/s11042-018-6857-9

    Article  Google Scholar 

  93. Wang CY, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing 410:354–362. https://doi.org/10.1016/j.neucom.2020.06.005

    Article  Google Scholar 

  94. Wang SJ, Li BJ, Liu YJ, Yan WJ, Ou XY, Huang XH, Xu F, Fu XL (2018) Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing 312:251–262. https://doi.org/10.1016/j.neucom.2018.05.107

    Article  Google Scholar 

  95. Gan YS, Liong ST, Yau WC, Huang YC, Tan LK (2019) OFF-ApexNet on micro-expression recognition system. Signal Process-Image 74:129–139. https://doi.org/10.1016/j.image.2019.02.005

    Article  Google Scholar 

  96. Verma M, Vipparthi SK, Singh G, Murala S (2019) LEARNet: Dynamic imaging network for micro expression recognition. IEEE Trans Image Process 29:1618–1627. https://doi.org/10.1109/TIP.2019.2912358

    Article  MathSciNet  MATH  Google Scholar 

  97. Peng M, Wang C, Bi T, Shi Y, Zhou X, Chen T (2019) A novel apex-time network for cross-dataset micro-expression recognition. In: 2019 8th international conference on affective computing and intelligent interaction (ACII), pp 1–6. https://doi.org/10.1109/ACII.2019.8925525

  98. Zhi RC, Xu HR, Wan M, Li TT (2019) Combining 3D convolutional neural networks with transfer learning by supervised pre-training for facial micro-expression recognition. IEICE Trans Inf Syst E102D:1054–1064. https://doi.org/10.1587/transinf.2018EDP7153

    Article  Google Scholar 

  99. Liu N, Liu X, Zhang Z, Xu X, Chen T (2020) Offset or onset frame: a multi-stream convolutional neural network with capsulenet module for micro-expression recognition. In: 2020 5th international conference on intelligent informatics and biomedical sciences (ICIIBMS), pp 236–240. https://doi.org/10.1109/ICIIBMS50712.2020.9336412

  100. Lai ZY, Chen RH, Jia JL, Qian YR (2020) Real-time micro-expression recognition based on ResNet and atrous convolutions. J Ambient Intel Humaniz Comput. https://doi.org/10.1007/s12652-020-01779-5

    Article  Google Scholar 

  101. Choi DY, Song BC (2020) Facial micro-expression recognition using two-dimensional landmark feature maps. IEEE Access 8:121549–121563. https://doi.org/10.1109/ACCESS.2020.3006958

    Article  Google Scholar 

  102. Buhari AM, Ooi CP, Baskaran VM, Phan RCW, Wong KK, Tan WH (2020) FACS-based graph features for real-time micro-expression recognition. J Imaging 6:130. https://doi.org/10.3390/jimaging6120130

    Article  Google Scholar 

  103. Le TTQ, Tran TK, Rege M (2020) Rank-pooling-based features on localized regions for automatic micro-expression recognition. Int J Multimed Data 11:25–37. https://doi.org/10.4018/IJMDEM.2020100102

    Article  Google Scholar 

  104. Zhu W, Chen Y (2020) Micro-expression recognition convolutional network based on dual-stream temporal-domain information interaction. In: 2020 13th international symposium on computational intelligence and design (ISCID), pp 396–400. https://doi.org/10.1109/ISCID51228.2020.00096

  105. Wang L, Jia J, Mao N (2020) Micro-expression recognition based on 2d-3d CNN. In: 2020 39th Chinese control conference (CCC), pp 3152–3157. https://doi.org/10.23919/CCC50068.2020.9188920

  106. Takalkar MA, Xu M, Chaczko Z (2020) Manifold feature integration for micro-expression recognition. Multimedia Syst 26:535–551. https://doi.org/10.1007/s00530-020-00663-8

    Article  Google Scholar 

  107. Gajjala VR, Reddy SPT, Mukherjee S, Dubey SR (2021) MERANet: facial micro-expression recognition using 3D residual attention network. In: Proceedings of the twelfth Indian conference on computer vision, graphics and image processing, pp 1–10. https://doi.org/10.1145/3490035.3490260

  108. Yang B, Cheng J, Yang YX, Zhang B, Li JX (2021) MERTA: micro-expression recognition with ternary attentions. Multimed Tools Appl 80:16125–16140. https://doi.org/10.1007/s11042-019-07896-4

    Article  Google Scholar 

  109. Li YT, Huang XH, Zhao GY (2020) Joint local and global information learning with single apex frame detection for micro-expression recognition. IEEE Trans Image Process 30:249–263. https://doi.org/10.1109/TIP.2020.3035042

    Article  Google Scholar 

  110. Verma M, Vipparthi SK, Singh G (2020) AffectiveNet: affective-motion feature learning for microexpression recognition. IEEE Multimedia 28:17–27. https://doi.org/10.1109/MMUL.2020.3021659

    Article  Google Scholar 

  111. Pan H, Xie L, Li J, Lv ZP, Wang ZL (2021) Micro-expression recognition by two-stream difference network. IET Comput Vis 15:440–448. https://doi.org/10.1049/cvi2.12030

    Article  Google Scholar 

  112. Nie X, Takalkar MA, Duan MY, Zhang HM, Xu M (2021) GEME: Dual-stream multi-task GEnder-based micro-expression recognition. Neurocomputing 427:13–28. https://doi.org/10.1016/j.neucom.2020.10.082

    Article  Google Scholar 

  113. Wang Y, Huang YK, Liu C, Gu XY, Yang DD, Wang SP, Zhang B (2021) Micro expression recognition via dual-stream spatiotemporal attention network. J Healthc Eng. https://doi.org/10.1155/2021/7799100

    Article  Google Scholar 

  114. Zhao SR, Tao HQ, Zhang YS, Xu T, Zhang K, Hao ZK, Chen EH (2021) A two-stage 3D CNN based learning method for spontaneous micro-expression recognition. Neurocomputing 448:276–289. https://doi.org/10.1016/j.neucom.2021.03.058

    Article  Google Scholar 

  115. Lei L, Chen T, Li SG, Li JF (2021) Micro-expression recognition based on facial graph representation learning and facial action unit fusion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1571–1580

  116. Wang J, Pan X, Li XY, Wei GS, Zhou YF (2021) Single trunk multi-scale network for micro-expression recognition. Graph Vis Comput. https://doi.org/10.1016/j.gvc.2021.200026

    Article  Google Scholar 

  117. Zhao Y, Chen Z, Luo S (2021) Micro-expression recognition based on pixel residual sum and cropped Gaussian pyramid. Front Neurorobotics 15:746985. https://doi.org/10.3389/fnbot.2021.746985

    Article  Google Scholar 

  118. Takalkar MA, Thuseethan S, Rajasegarar S, Chaczko Z, Xu M, Yearwood J (2021) LGAttNet: automatic micro-expression detection using dual-stream local and global attentions. Knowl-Based Syst 212:106566. https://doi.org/10.1016/j.knosys.2020.106566

    Article  Google Scholar 

  119. Wang SJ, He Y, Li JT, Fu XL (2021) MESNet: a convolutional neural network for spotting multi-scale micro-expression intervals in long videos. IEEE Trans Image Process 30:3956–3969. https://doi.org/10.1109/TIP.2021.3064258

    Article  Google Scholar 

  120. Zhou L, Mao QR, Huang XH, Zhang FF, Zhang ZH (2021) Feature refinement: an expression-specific feature learning and fusion method for micro-expression recognition. Pattern Recognit 122:108275. https://doi.org/10.1016/j.patcog.2021.108275

    Article  Google Scholar 

  121. Benlamine MS, Chaouach M, Frasson C, Dufresneet A (2016) Physiology-based recognition 0f facial micro-expressions using EEG and identification of the relevant sensors by emotion. In: International conference on physiological computing systems, vol 2, pp 130–137

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China (Nos. 61472330 and 61872301).

Author information

Authors and Affiliations

  1. School of Electronic and Information Engineering, Southwest University, Tiansheng Road, Beibei, Chongqing, 400715, China

    Xiaomei Zeng, Xingcong Zhao, Xinyue Zhong & Guangyuan Liu

  2. Institute of Affective Computing and Information Processing, Southwest University, Chongqing, China

    Xiaomei Zeng, Xingcong Zhao, Xinyue Zhong & Guangyuan Liu

  3. Chongqing Key Laboratory of Nonlinear Circuits and Intelligent Information Processing, Southwest University, Chongqing, China

    Xiaomei Zeng, Xingcong Zhao, Xinyue Zhong & Guangyuan Liu

Authors
  1. Xiaomei Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingcong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyue Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangyuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangyuan Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interests.

Ethical Approval

Not required.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, X., Zhao, X., Zhong, X. et al. A Survey of Micro-expression Recognition Methods Based on LBP, Optical Flow and Deep Learning. Neural Process Lett 55, 5995–6026 (2023). https://doi.org/10.1007/s11063-022-11123-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-11123-x

Keywords

Search

Navigation