Abstract
Micro-expressions are momentary involuntary facial expressions which may expose a person’s true emotions. Previous work in micro-expression detection mainly focus on finding the peak frame from a video sequence that has been determined to have a micro-expression, and the amount of computation is usually very large. In this paper, we propose a real-time micro-expression detection method based on optical flow and Long Short-term Memory (LSTM) to detect the appearance of micro-expression. This method takes only one step of data preprocessing which is less than previous work. Specifically, we use a sliding window with fixed-length to split a long video into several short videos, then a new and improved optical flow algorithm with low computational complexity was developed to extract feature curves based on the Facial Action Coding System (FACS). Finally, the feature curves were passed to a LSTM model to predict whether micro-expression occurs. We evaluate our method on CASMEll and SAMM databases, and it achieves a new state-of-the-art accuracy (89.87%) on CASMEll database (4.54% improvement). Meanwhile our method only takes 1.48 s to detect the micro-expression in a video sequence with 41 frames (the frame rate is about 28fps). The experimental results show that the proposed method can achieve better comprehensive performances.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ekman, P.: Micro Expression Training Tool (METT) and Subtle Expression Training Tool (SETT). Paul Ekman Company, San Francisco, CA (2003)
Matsumoto, D., Hwang, H.S.: Evidence for training the ability to read micro-expressions of emotion. Motiv. Emot. 35(2), 181–191 (2011)
Brinke, P.L.T.: Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychol. Sci. 19(5), 508–514 (2008)
Liong, S.T., See, J., Wong, K., Le Ngo, A.C., Oh, Y.H., Phan, R.: Automatic apex frame spotting in micro-expression database. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp. 665–669. IEEE (2015)
Li, X., et al.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. 9(4), 563–577 (2018)
Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S.: Towards macro-and micro-expression spotting in video using strain patterns. In: 2009 Workshop on Applications of Computer Vision (WACV), pp. 1–6. IEEE (2009)
Friesen, E., Ekman, P.: Facial action coding system: a technique for the measurement of facial movement, Palo Alto, 3 (1978)
Yan, W.-J., Wang, S.-J., Chen, Y.-H., Zhao, G., Fu, X.: Quantifying micro-expressions with constraint local model and local binary pattern. In: Agapito, L., Bronstein, M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8925, pp. 296–305. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16178-5_20
Patel, D., Zhao, G., Pietikäinen, M.: Spatiotemporal integration of optical flow vectors for micro-expression detection. In: Battiato, S., Blanc-Talon, J., Gallo, G., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2015. LNCS, vol. 9386, pp. 369–380. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25903-1_32
Li, X., Yu, J., Zhan, S.: Spontaneous facial micro-expression detection based on deep learning. In: 2016 IEEE 13th International Conference on Signal Processing (ICSP), pp. 1130–1134. IEEE (2016)
Zhang, Z., Chen, T., Meng, H., Liu, G., Fu, X.: SMEConvNet: a convolutional neural network for spotting spontaneous facial micro-expression from long videos. IEEE Access 6, 71143–71151 (2018)
Verburg, M., Vlado M.: Micro-expression detection in long videos using optical flow and recurrent neural networks. arXiv preprint arXiv:1903.10765 (2019)
Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)
Bouguet, J.Y.: Pyramidal implementation of the affine Lucas Kanade feature tracker description of the algorithm. Intel Corp. 5(1–10), 4 (2001)
Valstar, M.F., Pantic, M.: Fully automatic recognition of the temporal phases of facial actions. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 42(1), 28–43 (2012)
Hamm, J., Kohler, C.G., Gur, R.C., Verma, R.: Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. J. Neurosci. Methods 200(2), 237–256 (2011)
Yan, W.J., et al.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)
Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: SAMM: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2018)
Wu, H.Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.: Eulerian video magnification for revealing subtle changes in the world (2012)
Acknowledgement
This research is supported in part by The Director Foundation Project of National Engineering Laboratory for Public Safety Risk Perception and Control by Big Data (PSRPC), in part by funding from Beijing Key Laboratory for Mental Disorders, and in part by China Postdoctoral Science Foundation (No. 2018M641437).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ding, J., Tian, Z., Lyu, X., Wang, Q., Zou, B., Xie, H. (2019). Real-Time Micro-expression Detection in Unlabeled Long Videos Using Optical Flow and LSTM Neural Network. In: Vento, M., Percannella, G. (eds) Computer Analysis of Images and Patterns. CAIP 2019. Lecture Notes in Computer Science(), vol 11678. Springer, Cham. https://doi.org/10.1007/978-3-030-29888-3_51
Download citation
DOI: https://doi.org/10.1007/978-3-030-29888-3_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29887-6
Online ISBN: 978-3-030-29888-3
eBook Packages: Computer ScienceComputer Science (R0)