Micro-expression Recognition Based on Attention-enhanced LSTM Neural Networks
Authors: 
Shiqi Xu, Fen XuAuthors Info & Claims
Shiqi Xu, Fen XuAuthors Info & ClaimsPages 394 - 400
Abstract
Micro-expression recognition is a difficult task in computer vision. Most existing micro-expression recognition methods extract facial features globally, leading to the inclusion of many irrelevant features and affecting the recognition accuracy in a negative way. In this paper, Long Short-Term Memory (LSTM) neural networks with spatial and temporal attention mechanisms are designed and employed to extract features selectively from the input sequences. Key frames are identified from the original micro-expression sequences at first. Then the VGG-Face model is used to extract the spatial features of those key frames. The spatial features of the micro-expression sequences are then fed into attention-enhanced long short-term memory neural networks, using a softmax function for the final classification. Our experiments with CASME II show that the attention-enhanced LSTM models improve the accuracy of micro-expression recognition significantly, compared to the results of several other leading methods.
References
[1]
Frank M G, Herbasz M, Sinuk K, Keller A, Nolan C. I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: Proceedings of the 2009 Annual Meeting of the International Communication Association. New York, 2009. http://www.allacademic.com/meta/p15018 index.htm.
[2]
G Zhao, M Pietikainen. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915-928.
[3]
Zhang P, Ben X Y, Yan R, Wu C, Guo C. Micro-expression recognition system. Optik — International Journal for Light and Electron Optics, 2016, 127(3): 1395−1400.
[4]
Liu Y J, Zhang J K, Yan W J, A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2015, 7(4): 299-310.
[5]
Wu W, Shen X B, Fu X L. The machine knows what you are hiding: an automatic micro-expression recognition system. In: Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction. Memphis,TN, USA: Springer-Verlag, 2011. 152−162.
[6]
Happy S L, A Routray. Fuzzy Histogram of Optical Flow Orientations for Micro-expression Recognition[J]. IEEE Transactions on Affective Computing, 2017: 394-406.
[7]
Reddy S, Karri S T, Dubey S R, Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks[J]. IEEE, 2019.
[8]
Khor H Q, See J, Phan R, Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[C]// IEEE International Conference on Automatic Face & Gesture Recognition. arXiv, 2018.
[9]
Peng M, Wang C, Chen T, Dual temporal scale convolutional neural network for micro-expression recognition[J]. Frontiers in psychology, 2017, 8: 1745.
[10]
Kim, D. H., W. J. Baddar, and Y. M. Ro. "Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations." Acm on Multimedia Conference ACM, 2016:382-386.
[11]
Li J, Wang Y, See J, Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern Analysis and Applications, 2018.
[12]
Khor H Q, See J, Liong S T, Dual-stream Shallow Networks for Facial Micro-expression Recognition[C]// 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019.
[13]
O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in British Machine Vision Conference, 2015.
[14]
Zhou Z, Zhao G, M Pietikäinen. Towards a practical lipreading system[C]// The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011. IEEE, 2011.
[15]
G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Technical Report 07-49, University of Massachusetts, Amherst, Tech. Rep.,2007.
[16]
Yan W J, Li X, Wang S J, CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation[J]. Plos One, 2014, 9(1): e86041.
[17]
Huang X, Zhao G, Hong X, Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns[J]. Neurocomputing, 2016, 175(JAN.29PT.A):564-578.
[18]
Wang Y, See J, Phan W, LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition[J]. ACCV2014, 2015.
[19]
Liong S T, Gan Y S, Zheng D, Evaluation of the Spatio-Temporal features and GAN for Micro-expression Recognition System[J]. ar Xiv preprint ar Xiv:1904.01748, 2019.
[20]
Min P, Zhan W, Zhang Z, From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning[C]// 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018.
[21]
Wang, Su-Jing, Li, Micro-expression recognition with small sample size by transferring long-term convolutional neural network[J]. Neurocomputing, 2018.
[22]
Liong S T, See J, Wong K S, Less is More: Micro-expression Recognition from Video using Apex Frame[J]. Signal Processing: Image Communication, 2018: 82-92.
[23]
Liong S T, Gan Y S, See J, Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition[C]//Automatic Face and Gesture Recognition (FG 2019), 2019: 1-5.
[24]
He K M, Zhang X Y, Ren S Q, Deep residual learning for image recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770.
[25]
Patel D, Hong X, Zhao G. Selective deep features for micro-expression recognition[C]//International Conference on Pattern Recognition (ICPR), 2017: 2258-2263.
[26]
Xiao T, Xu Y, Yang K, The application of two-level attention models in deep convolutional neural network for fine-grained image classification[C]// Computer Vision and Pattern Recognition (CVPR), 2015: 842-850.
[27]
Sharma S, Kiros R, Salakhutdinov R. Action Recognition using Visual Attention [C]//Neural Information Processing Systems (NIPS) Time Series Workshop, 2017: 1-11.
[28]
Wang C, Peng M, Bi T, Micro-Attention for Micro-Expression Recognition[J]. Neurocomputing, 2020: 354-362
[29]
X Ben, X Jia, R Yan, Learning effective binary descriptors for micro-expression recognition transferred by macroinformation[J]. Pattern Recognition Letters, 2018, 107: 50-58.
[30]
Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. PSANet: Point-wise spatial attention network for scene parsing. In ECCV, 2018.
[31]
Y Li, X Huang, G Zhao. Can micro-expression be recognized based on single apex frame?[C]//IEEE International Conference on Image Processing (ICIP), 2018: 3094-3098.
[32]
Shreve M, Godavarthy S, Goldgof D, Macro- and micro-expression spotting in long videos using spatio-temporal strain[J]. IEEE Computer Society, 2011:51-56.
[33]
Wang F, Jiang M, Qian C, Residual Attention Network for Image Classification[C] //Computer Vision and Pattern Recognition (CVPR), 2017: 6450-6458.
[34]
Peng M, Wang C, Bi T, A Novel Apex-Time Network for Cross-Dataset Micro-Expression Recognition[C]//Affective Computing and Intelligent Interaction (ACII), 2019: 1-6.
[35]
Li G H, Yuan Y F, Ben X Y and Zhang J P.2020.Spatiotemporal attention network for microexpression recognition.Journal of Image and Graphics, 25( 11) : 2380-2390.
[36]
XU Wei, ZHENG Hao, YANG Zhongxue. Apex frame microexpression recognition based on dual attention model and transfer learning[J]. CAAI transactions on intelligent systems, 2021, 16(6): 1015–1020.
[37]
XIA Z, HONG X, GAO X, Corrections to "Spatio-temporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions"[J]. IEEE Transactions on Multemporaldia, 2020, 22(4):1111-1111.
[38]
Xia Z, Peng W, Khor H Q, Revealing the Invisible with Model and Data Shrinking for Composite-database Micro-expression Recognition[J]. IEEE Transactions on Image Processing, 2020, 29:8590-8605.
[39]
SU Yuting, WANG Mengmeng, LIU Jing, Micro-expression recognition algorithm based on multiple motive feature fusion[J]. Laser & Optoelectronics Progress, 2020, 57(14): 353-358.
[40]
NIU Ruihua, YANG Jun, XING Lanxin, WU Renbiao. Micro-expression recognition algorithm based on convolutional block attention module and dual path networks[J]. Journal of Computer Applications, 2021, 41(9): 2552 – 2559.
Recommendations
Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations
MM '16: Proceedings of the 24th ACM international conference on MultimediaRecognizing spontaneous micro-expression in video sequences is a challenging problem. In this paper, we propose a new method of small scale spatio-temporal feature learning. The proposed learning method consists of two parts. First, the spatial features ...
AU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition
MM '20: Proceedings of the 28th ACM International Conference on MultimediaMicro-expressions (MEs) are important clues for reflecting the real feelings of humans, and micro-expression recognition (MER) can thus be applied in various real-world applications. However, it is difficult to perceive and interpret MEs correctly. With ...
A micro-expression recognition algorithm based on feature enhancement and attention mechanisms
AbstractIn recent years, facial micro-expression recognition based on deep learning technology has become an important research hotspot. These techniques have been used by researchers in psychology, computer vision, and security to make breakthroughs. ...
Comments
Information & Contributors
Information
Published In
February 2022
570 pages
ISBN:9781450395700
DOI:10.1145/3529836
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 June 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICMLC 2022
ICMLC 2022: 2022 14th International Conference on Machine Learning and Computing
February 18 - 21, 2022
Guangzhou, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 58Total Downloads
- Downloads (Last 12 months)7
- Downloads (Last 6 weeks)0
Reflects downloads up to 08 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format