Wenqiang Jia
ABSTRACT
Micro-expressions are brief, subtle and involuntary facial gestures, which usually hide real emotions of people and are difficult to capture. Considering that micro-expressions data samples contain distinctive features of specific categories and common features of different categories, a micro-expressions recognition network based on bimodal contrastive learning is proposed in this paper. The network mainly includes bimodal feature extraction module, bimodal contrastive learning fusion module and classification and recognition module. First, the micro-expressions sequence is divided into RGB sequence and optical flow sequence, and the loss between them is constructed by contrastive learning. The network extracts bimodal common features. Second, in order to extract distinctive features, bimodal features are fused and label data is used to optimize the network. The network extracts bimodal features, while extracting distinctive features of different categories. The results show the superiority of the proposed method over other state-of-the-art methods for 5 categories of micro-expressions on CASME II, SAMM and 6 categories of micro-expressions on MMEW.
- Ben X, Ren Y, Zhang J, Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms[J]. IEEE transactions on pattern analysis and machine intelligence, 2021, DOI: 10.1109/TPAMI.2021.3067464.Google Scholar
- Gourdin K N. Protecting Airline Passengers in the Age of Terrorism[J]. Defense Transportation Journal, 2009, 65(6): 64-72.Google Scholar
- O'Sullivan M, Frank M G, Tiwana H J. Police Lie Detection Accuracy: The Effect of Lie Scenario[J]. Law and Human Behavior, 2009, 33(6): 530-538, 542-543.Google ScholarCross Ref
- T. A. Russell, E. Chu, and M. L. Phillips. A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. Brit J Clin Psychol, 45(4):579–583, 2006.Google ScholarCross Ref
- Zhao G, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE transactions on pattern analysis and machine intelligence, 2007, 29(6): 915-928.Google ScholarDigital Library
- Liu Y J, Zhang J K, Yan W J, A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2015, 7(4): 299-310.Google ScholarDigital Library
- Klaser A, Marszałek M, Schmid C. A spatio-temporal descriptor based on 3d-gradients[C]//BMVC 2008-19th British Machine Vision Conference. British Machine Vision Association, 2008: 275: 1-10.Google Scholar
- Khor H Q, See J, Phan R C W, Enriched long-term recurrent convolutional network for facial micro-expression recognition[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018: 667-674.Google Scholar
- Li J, Wang Y, See J, Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern Analysis and Applications, 2019, 22(4): 1331-1339.Google ScholarDigital Library
- Xia Z, Hong X, Gao X, Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions[J]. IEEE Transactions on Multimedia, 2019, 22(3): 626-640.Google ScholarCross Ref
- Sohn K. Improved deep metric learning with multi-class n-pair loss objective[C] //Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016: 1857-1865.Google Scholar
- Devlin J, Chang M W, Lee K, Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.Google Scholar
- Zhou Z, Zhao G, Guo Y, An image-based visual speech animation system[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(10): 1420-1432.Google ScholarDigital Library
- Xu F, Zhang J, Wang J Z. Micro-expression identification and categorization using a facial dynamics map[J]. IEEE Transactions on Affective Computing, 2017, 8(2): 254-267.Google ScholarDigital Library
- Liu Y J, Li B J, Lai Y K. Sparse MDMO: Learning a discriminative feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2018.Google ScholarCross Ref
- Peng M, Wu Z, Zhang Z, From macro to micro expression recognition: Deep learning on small datasets using transfer learning[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018: 657-661.Google Scholar
- Kim D H, Baddar W J, Ro Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations[C]// Proceedings of the 24th ACM international conference on Multimedia. 2016: 382-386.Google Scholar
- Hu C, Jiang D, Zou H, Multi-task micro-expression recognition combining deep and handcrafted features[C]//2018 24th International Conference on Pattern Recognition (ICPR). IEEE, 2018: 946-951.Google Scholar
- Ali H B, Powers D. Fusion Based FastICA Method: Facial Expression Recognition[J]. Journal of Image and Graphics, 2014:1-7.Google Scholar
- Y Ye, Song Z, Zhao J. Facial Micro-expression Analysis via a High Speed Structured Light Sensing System[J]. Journal of Image and Graphics, 2021, 9(1):15-19.Google ScholarCross Ref
- Youssif A A, Fathy W E, Zarif S, Benchmarking of Pre-Processing Methods Employed in Facial Image Analysis[J]. International Journal of Image and Graphics, 2018, 6(1):1-9.Google ScholarCross Ref
Recommendations
Micro-expression recognition with supervised contrastive learning
Highlights- Approach to extract credible representations under noise distribution.
- ...
AbstractFacial micro-expressions are involuntary movements of facial muscles that expose individual underlying emotions. Because of the subtle and diverse facial muscles change, extracting effective features to recognize micro-expressions is ...
Learning two groups of discriminative features for micro-expression recognition
Highlights- This work is the first one to learn discriminative features from two groups of features, and two groups of features have some differences, such as ...
AbstractAs a branch of affective computing and machine learning, recognizing micro-expressions is more difficult than recognizing macro-expressions because micro-expression has a small motion and short duration. A large number of features and ...
A micro-expression recognition algorithm based on feature enhancement and attention mechanisms
AbstractIn recent years, facial micro-expression recognition based on deep learning technology has become an important research hotspot. These techniques have been used by researchers in psychology, computer vision, and security to make breakthroughs. ...
Comments