Skip to main content
Springer Nature Link
Log in

Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Micro-expression is a kind of split-second subtle expression which could not be controlled by the autonomic nervous system. Micro-expression indicates that a person is hiding his truly emotion consciously. Because the micro-expression is closely interrelated with lie detection, micro-expression recognition has various potential applications in many domains, such as the public security, the clinical medicine, the investigation and the interrogation. Because recognizing the micro-expression through human observation is very difficult, researchers focus on the automatic micro-expression recognition. This research proposed a novel algorithm for automatic micro-expression recognition which combined a deep multi-task convolutional network for detecting the facial landmarks and a fused deep convolutional network for estimating the optical flow features of the micro-expression. Firstly, this research employed the deep multi-task convolutional network to detect facial landmarks with the manifold related tasks and divided the facial region by utilizing these facial landmarks. Furthermore, a fused convolutional network was applied for extracting the optical flow features from the facial regions which contain the muscle changes when the micro-expression presents. Finally the enhanced optical flow was applied for refining the information of the features and these refined optical flow features were classified by Support Vector Machine classifier for recognizing the micro-expression. The result of experiments on two spontaneous micro-expression database demonstrated that the method proposed in this paper achieved good performance in micro-expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ben X et al (2016) Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput & Applic 27(8):2629–2646

    Article  Google Scholar 

  2. Burgos-Artizzu XP, Perona P, Dollár P (2013) Robust face landmark estimation under occlusion. Proceedings of the IEEE international conference on computer vision

  3. Chan CH et al (2012) Local ordinal contrast pattern histograms for spatiotemporal, lip-based speaker authentication. IEEE T Inf Foren Sec 7(2):602–612

    Article  Google Scholar 

  4. Chang C-C, Chih-Jen L (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol (TIST) 2(3):27

    Google Scholar 

  5. Chaudhry R et al (2009) Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE

  6. Dollár P et al (2005) Behavior recognition via sparse spatio-temporal features. Visual surveillance and performance evaluation of tracking and surveillance, 2005. 2nd Joint IEEE International Workshop on. IEEE

  7. Dong C et al (2014) Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision. Springer, Cham

    Google Scholar 

  8. Dosovitskiy A et al (2015) Flownet: learning optical flow with convolutional networks. Proceedings of the IEEE international conference on computer vision

  9. Ekman P (2003) Darwin, deception, and facial expression. Ann N Y Acad Sci 1000(1):205–221

    Article  Google Scholar 

  10. Ekman P (2003) Micro expressions training tool. Emotionsrevealed.com

  11. Ekman P (2009) Telling lies: clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & Company

  12. Ekman P (2009) Lie catching and microexpressions. The philosophy of deception, pp 118–133

  13. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106

    Article  Google Scholar 

  14. Ekman P, Friesen WV (1977) Facial action coding system

  15. Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy. Springer, Boston, pp 154–165

    Chapter  Google Scholar 

  16. He J et al (2017) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn 66:44–52

    Article  Google Scholar 

  17. Huang X et al (2012) Spatiotemporal local monogenic binary patterns for facial expression recognition. IEEE Signal Proc Let 19(5):243–246

    Article  Google Scholar 

  18. Huang X et al (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578

    Article  Google Scholar 

  19. Ilg E et al (2016) Flownet 2.0: evolution of optical flow estimation with deep networks. arXiv preprint arXiv:1612.01925

    Google Scholar 

  20. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems

  21. Le Ngo AC, Phan RC-W, See J (2014) Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Asian conference on computer vision. Springer, Cham

    Google Scholar 

  22. Le Ngo AC, See J, Phan RC-W (2017) Sparsity in dynamics of spontaneous subtle emotion: analysis & application. IEEE Trans Affect Comput

  23. Li X, Yu J, Zhan S (2016) Spontaneous facial micro-expression detection based on deep learning. Signal Processing (ICSP), 2016 IEEE 13th International Conference on. IEEE

  24. Li X et al (2017) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE transactions on affective computing

  25. Liu Y-J et al (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  26. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition

  27. Ma L, Khashayar K (2004) Facial expression recognition using constructive feedforward neural networks. IEEE T Syst Man Cy B 34(3):1588–1595

    Article  Google Scholar 

  28. Mayer N et al (2016) A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. Proceedings of the IEEE conference on computer vision and pattern recognition

  29. Ni Z et al (2016) Gradient direction for screen content image quality assessment. IEEE Signal Proc Let 23(10):1394–1398

    Article  Google Scholar 

  30. Ni Z et al (2017) ESIM: edge similarity for screen content image quality assessment. IEEE Trans Image Process 26(10):4818–4831

    Article  MathSciNet  Google Scholar 

  31. Oh Y-H et al (2015) Monogenic Riesz wavelet representation for micro-expression recognition. Digital Signal Processing (DSP), 2015 IEEE International Conference on. IEEE

  32. Patel D, Hong X, Zhao G (2016) Selective deep features for micro-expression recognition. Pattern Recognition (ICPR), 2016 23rd International Conference on. IEEE

  33. Pfister T et al (2011) Recognising spontaneous facial micro-expressions. Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE

  34. Pfister T et al (2011) Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE

  35. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor, pp 16–16

  36. Ren S et al (2015) Faster R-CNN: towards real-time object detection with region proposal networks. Advances in neural information processing systems

  37. Ruder S (2017) An overview of multi-task learning in deep neural networks[J]. arXiv preprint arXiv:1706.05098

    Google Scholar 

  38. Sun Y, Wang X, Tang X (2013) Deep convolutional network cascade for facial point detection. Proceedings of the IEEE conference on computer vision and pattern recognition

  39. Sun Y, Wang X, Tang X (2014) Deep learning face representation from predicting 10,000 classes. Proceedings of the IEEE conference on computer vision and pattern recognition

  40. Tran D et al (2016) Deep end2end voxel2voxel prediction. Proceedings of the IEEE conference on computer vision and pattern recognition workshops

  41. Wang Y et al (2011) Large disparity motion layer extraction via topological clustering. IEEE Trans Image Process 20(1):43–52

    Article  MathSciNet  MATH  Google Scholar 

  42. Wang S-J et al (2014) Micro-expression recognition using dynamic textures on tensor independent color space. Pattern Recognition (ICPR), 2014 22nd International Conference on. IEEE

  43. Wang Y et al (2014) Lbp with six intersection points: reducing redundant information in lbp-top for micro-expression recognition. In: Asian Conference on Computer Vision. Springer, Cham

    Google Scholar 

  44. Wang Y et al (2015) Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PLoS One 10(5):e0124674

    Article  Google Scholar 

  45. Wang Y et al (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76(20):21665–21690

    Article  Google Scholar 

  46. Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  47. Yan W-J et al (2013) CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. Automatic face and gesture recognition (FG), 2013 10th IEEE International Conference and Workshops on. IEEE

  48. Yan W-J et al (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):e86041

    Article  Google Scholar 

  49. Yang A et al (2017) Perceptual feature guided rate distortion optimization for high efficiency video coding. Multidim Syst Sign Process 28(4):1249–1266

    Article  MathSciNet  Google Scholar 

  50. Zeng H, Cai C, Ma K-K (2009) Fast mode decision for H. 264/AVC based on macroblock motion activity. IEEE T Circ Syst Vid 19(4):491–499

    Article  Google Scholar 

  51. Zeng H, Ma K-K, Cai C (2010) Hierarchical intra mode decision for H. 264/AVC. IEEE T Circ Syst Vid 20(6):907–912

    Article  Google Scholar 

  52. Zeng H, Ma K-K, Cai C (2011) Fast mode decision for multiview video coding using mode correlation. IEEE T Circ Syst Vid 21(11):1659–1666

    Article  Google Scholar 

  53. Zeng H et al (2014) Fast multiview video coding using adaptive prediction structure and hierarchical mode decision. IEEE T Circ Syst Vid 24(9):1566–1578

    Article  MathSciNet  Google Scholar 

  54. Zeng H et al (2015) Sift-flow-based color correction for multi-view video. Signal Process Image Commun 36:53–62

    Article  Google Scholar 

  55. Zeng H et al (2016) Quad binary pattern and its application in mean-shift tracking. Neurocomputing 217:3–10

    Article  Google Scholar 

  56. Zhang Z et al (2014) Facial landmark detection by deep multi-task learning. In: European Conference on Computer Vision. Springer, Cham

    Google Scholar 

  57. Zhang S et al (2017) Micro-expression recognition by aggregating local spatio-temporal patterns. In: International Conference on Multimedia Modeling. Springer, Cham

    Google Scholar 

  58. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  59. Zhu Z et al (2013) Deep learning identity-preserving face space. Proceedings of the IEEE international conference on computer vision

  60. Zhu J et al (2017) Deep hybrid similarity learning for person re-identification. IEEE T Circ Syst Vid

Download references

Acknowledgments

The authors would like to thank the anonymous reviews for their helpful and constructive comments and suggestions regarding this manuscript.

Funding

This work was supported in part by National Nature Science Foundation of China Grand No:61371156.

Author information

Authors and Affiliations

  1. School of Computer and Information, Hefei University of Technology, Hefei, Anhui, China

    Qiuyu Li, Shu Zhan, Liangfeng Xu & Congzhong Wu

Authors
  1. Qiuyu Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Shu Zhan
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Liangfeng Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Congzhong Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Shu Zhan.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Q., Zhan, S., Xu, L. et al. Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow. Multimed Tools Appl 78, 29307–29322 (2019). https://doi.org/10.1007/s11042-018-6857-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6857-9

Keywords