Shiqi Xu
ABSTRACT
Micro-expression recognition is a difficult task in computer vision. Most existing micro-expression recognition methods extract facial features globally, leading to the inclusion of many irrelevant features and affecting the recognition accuracy in a negative way. In this paper, Long Short-Term Memory (LSTM) neural networks with spatial and temporal attention mechanisms are designed and employed to extract features selectively from the input sequences. Key frames are identified from the original micro-expression sequences at first. Then the VGG-Face model is used to extract the spatial features of those key frames. The spatial features of the micro-expression sequences are then fed into attention-enhanced long short-term memory neural networks, using a softmax function for the final classification. Our experiments with CASME II show that the attention-enhanced LSTM models improve the accuracy of micro-expression recognition significantly, compared to the results of several other leading methods.
- Frank M G, Herbasz M, Sinuk K, Keller A, Nolan C. I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: Proceedings of the 2009 Annual Meeting of the International Communication Association. New York, 2009. http://www.allacademic.com/meta/p15018 index.htm.Google Scholar
- G Zhao, M Pietikainen. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915-928.Google ScholarDigital Library
- Zhang P, Ben X Y, Yan R, Wu C, Guo C. Micro-expression recognition system. Optik — International Journal for Light and Electron Optics, 2016, 127(3): 1395−1400.Google Scholar
- Liu Y J, Zhang J K, Yan W J, A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2015, 7(4): 299-310.Google ScholarDigital Library
- Wu W, Shen X B, Fu X L. The machine knows what you are hiding: an automatic micro-expression recognition system. In: Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction. Memphis,TN, USA: Springer-Verlag, 2011. 152−162.Google ScholarCross Ref
- Happy S L, A Routray. Fuzzy Histogram of Optical Flow Orientations for Micro-expression Recognition[J]. IEEE Transactions on Affective Computing, 2017: 394-406.Google Scholar
- Reddy S, Karri S T, Dubey S R, Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks[J]. IEEE, 2019.Google Scholar
- Khor H Q, See J, Phan R, Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[C]// IEEE International Conference on Automatic Face & Gesture Recognition. arXiv, 2018.Google Scholar
- Peng M, Wang C, Chen T, Dual temporal scale convolutional neural network for micro-expression recognition[J]. Frontiers in psychology, 2017, 8: 1745.Google Scholar
- Kim, D. H., W. J. Baddar, and Y. M. Ro. "Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations." Acm on Multimedia Conference ACM, 2016:382-386.Google Scholar
- Li J, Wang Y, See J , Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern Analysis and Applications, 2018.Google Scholar
- Khor H Q, See J , Liong S T , Dual-stream Shallow Networks for Facial Micro-expression Recognition[C]// 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019.Google Scholar
- O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in British Machine Vision Conference, 2015.Google ScholarCross Ref
- Zhou Z, Zhao G, M Pietikäinen. Towards a practical lipreading system[C]// The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011. IEEE, 2011.Google Scholar
- G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Technical Report 07-49, University of Massachusetts, Amherst, Tech. Rep.,2007.Google Scholar
- Yan W J, Li X, Wang S J, CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation[J]. Plos One, 2014, 9(1): e86041.Google ScholarCross Ref
- Huang X, Zhao G, Hong X, Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns[J]. Neurocomputing, 2016, 175(JAN.29PT.A):564-578.Google ScholarDigital Library
- Wang Y, See J, Phan W, LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition[J]. ACCV2014, 2015.Google Scholar
- Liong S T, Gan Y S, Zheng D, Evaluation of the Spatio-Temporal features and GAN for Micro-expression Recognition System[J]. ar Xiv preprint ar Xiv:1904.01748, 2019.Google Scholar
- Min P, Zhan W, Zhang Z, From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning[C]// 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018.Google Scholar
- Wang, Su-Jing, Li, Micro-expression recognition with small sample size by transferring long-term convolutional neural network[J]. Neurocomputing, 2018.Google Scholar
- Liong S T, See J, Wong K S, Less is More: Micro-expression Recognition from Video using Apex Frame[J]. Signal Processing: Image Communication, 2018: 82-92.Google Scholar
- Liong S T, Gan Y S, See J, Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition[C]//Automatic Face and Gesture Recognition (FG 2019), 2019: 1-5.Google Scholar
- He K M, Zhang X Y, Ren S Q, Deep residual learning for image recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770.Google Scholar
- Patel D, Hong X, Zhao G. Selective deep features for micro-expression recognition[C]//International Conference on Pattern Recognition (ICPR), 2017: 2258-2263.Google Scholar
- Xiao T, Xu Y, Yang K, The application of two-level attention models in deep convolutional neural network for fine-grained image classification[C]// Computer Vision and Pattern Recognition (CVPR), 2015: 842-850.Google Scholar
- Sharma S, Kiros R, Salakhutdinov R. Action Recognition using Visual Attention [C]//Neural Information Processing Systems (NIPS) Time Series Workshop, 2017: 1-11.Google Scholar
- Wang C, Peng M, Bi T, Micro-Attention for Micro-Expression Recognition[J]. Neurocomputing, 2020: 354-362Google Scholar
- X Ben, X Jia, R Yan, Learning effective binary descriptors for micro-expression recognition transferred by macroinformation[J]. Pattern Recognition Letters, 2018, 107: 50-58.Google ScholarCross Ref
- Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. PSANet: Point-wise spatial attention network for scene parsing. In ECCV, 2018.Google Scholar
- Y Li, X Huang, G Zhao. Can micro-expression be recognized based on single apex frame?[C]//IEEE International Conference on Image Processing (ICIP), 2018: 3094-3098.Google Scholar
- Shreve M, Godavarthy S, Goldgof D, Macro- and micro-expression spotting in long videos using spatio-temporal strain[J]. IEEE Computer Society, 2011:51-56.Google ScholarCross Ref
- Wang F, Jiang M, Qian C, Residual Attention Network for Image Classification[C] //Computer Vision and Pattern Recognition (CVPR), 2017: 6450-6458.Google Scholar
- Peng M, Wang C, Bi T, A Novel Apex-Time Network for Cross-Dataset Micro-Expression Recognition[C]//Affective Computing and Intelligent Interaction (ACII), 2019: 1-6.Google Scholar
- Li G H, Yuan Y F, Ben X Y and Zhang J P.2020.Spatiotemporal attention network for microexpression recognition.Journal of Image and Graphics, 25( 11) : 2380-2390.Google Scholar
- XU Wei, ZHENG Hao, YANG Zhongxue. Apex frame microexpression recognition based on dual attention model and transfer learning[J]. CAAI transactions on intelligent systems, 2021, 16(6): 1015–1020.Google Scholar
- XIA Z, HONG X, GAO X, Corrections to "Spatio-temporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions"[J]. IEEE Transactions on Multemporaldia, 2020, 22(4):1111-1111.Google Scholar
- Xia Z, Peng W, Khor H Q, Revealing the Invisible with Model and Data Shrinking for Composite-database Micro-expression Recognition[J]. IEEE Transactions on Image Processing, 2020, 29:8590-8605.Google ScholarCross Ref
- SU Yuting, WANG Mengmeng, LIU Jing, Micro-expression recognition algorithm based on multiple motive feature fusion[J]. Laser & Optoelectronics Progress, 2020, 57(14): 353-358.Google ScholarCross Ref
- NIU Ruihua, YANG Jun, XING Lanxin, WU Renbiao. Micro-expression recognition algorithm based on convolutional block attention module and dual path networks[J]. Journal of Computer Applications, 2021, 41(9): 2552 – 2559.Google Scholar
Recommendations
Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations
MM '16: Proceedings of the 24th ACM international conference on MultimediaRecognizing spontaneous micro-expression in video sequences is a challenging problem. In this paper, we propose a new method of small scale spatio-temporal feature learning. The proposed learning method consists of two parts. First, the spatial features ...
AU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition
MM '20: Proceedings of the 28th ACM International Conference on MultimediaMicro-expressions (MEs) are important clues for reflecting the real feelings of humans, and micro-expression recognition (MER) can thus be applied in various real-world applications. However, it is difficult to perceive and interpret MEs correctly. With ...
A micro-expression recognition algorithm based on feature enhancement and attention mechanisms
AbstractIn recent years, facial micro-expression recognition based on deep learning technology has become an important research hotspot. These techniques have been used by researchers in psychology, computer vision, and security to make breakthroughs. ...
Comments