Abstract
Recently, micro-expression recognition has seen an increase of interest from psychological and computer vision communities. As micro-expressions are generated involuntarily on a person’s face, and are usually a manifestation of repressed feelings of the person. Most existing works pay attention to either the detection or spotting of micro-expression frames or the categorization of type of micro-expression present in a short video shot. In this paper, we introduced a novel automatic approach to micro-expression recognition from long video that combines both spotting and recognition mechanisms. To achieve this, the apex frame, which provides the instant when the highest intensity of facial movement occurs, is first spotted from the entire video sequence. An automatic eye masking technique is also presented to improve the robustness of apex frame spotting. With the single apex, we describe the spotted micro-expression instant using a state-of-the-art feature extractor before proceeding to classification. This is the first known work that recognizes micro-expressions from a long video sequence without the knowledge of onset and offset frames, which are typically used to determine a cropped sub-sequence containing the micro-expression. We evaluated the spotting and recognition tasks on four spontaneous micro-expression databases comprising only of raw long videos – CASME II-RAW, SMIC-E-HS, SMIC-E-VIS and SMIC-E-NIR. We obtained compelling results that show the effectiveness of the proposed approach, which outperform most methods that rely on human annotated sub-sequences.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Porter, S., ten Brinke, L.: Reading between the lies identifying concealed and falsified emotions in universal facial expressions. Psychol. Sci. 19(5), 508–514 (2008)
Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. J. Interpers. Process. 32, 88–106 (1969)
Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S.: Towards macro-and micro-expression spotting in video using strain patterns. In: Applications of Computer Vision (WACV), pp. 1–6 (2009)
Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Personal. Soc. Psychol. 17, 124 (1971)
Ekman, P.: Lie catching and microexpressions. The philosophy of deception, pp. 118–133 (2009)
Valstar, M.F., Pantic, M.: Combined support vector machines and hidden Markov models for modeling facial action temporal dynamics. Human Computer Interaction, pp. 118–127 (2007)
Yan, W.J., Wang, S.J., Zhao, G., Li, X., Liu, Y.J., Chen, Y.H., Fu, X.: CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9, e86041 (2014)
Li, X., Pfister, T., Huang, X., Zhao, G., Pietikainen, M.: A spontaneous micro-expression database: inducement, collection and baseline. In: Automatic Face and Gesture Recognition, pp. 1–6 (2013)
Yan, W.-J., Wang, S.-J., Chen, Y.-H., Zhao, G., Fu, X.: Quantifying micro-expressions with constraint local model and local binary pattern. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8925, pp. 296–305. Springer, Cham (2015). doi:10.1007/978-3-319-16178-5_20
Cristinacce, D., Cootes, T.: Automatic feature localisation with constrained local models. Pattern Recogn. 41(10), 3054–3067 (2008)
Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recogn. 29(1), 51–59 (1996)
Liong, S.T., See, J., Wong, K., Le Ngo, A.C., Oh, Y.H., Phan, R.C.W.: Automatic apex frame spotting in micro-expression database. In: Asian Conference on Pattern Recognition (ACPR) (2015)
Liong, S.T., See, J., Wong, K., Phan, R.C.W.: Less is more: micro-expression recognition from video using apex frame. arXiv (2016)
Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikäinen, M.: Reading hidden emotions: spontaneous micro-expression spotting and recognition. arXiv preprint arXiv:1511.00423 (2015)
Dalal, N., Triggs, B., Schmid, C.: Human detection using oriented histograms of flow and appearance. In: European conference on computer vision, pp. 428–441 (2006)
Moilanen, A., Zhao, G., Pietikainen, M.: Spotting rapid facial movements from videos using appearance-based feature difference analysis. In: ICPR, pp. 1722–1727 (2014)
Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: Computer Vision and Pattern Recognition, pp. 3444–3451 (2013)
Saragih, J.M., Lucey, S., Cohn, J.F.: Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vis. 91(2), 200–215 (2011)
Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: Computer Vision and Pattern Recognition, pp. 2879–2886 (2012)
Liong, S.-T., See, J., Phan, R.C.-W., Ngo, A.C., Oh, Y.-H., Wong, K.S.: Subtle expression recognition using optical strain weighted features. In: Jawahar, C.V., Shan, S. (eds.) ACCV 2014. LNCS, vol. 9009, pp. 644–657. Springer, Heidelberg (2015). doi:10.1007/978-3-319-16631-5_47
Pérez, J.S., Meinhardt-Llopis, E., Facciolo, G.: Tv-l1 optical flow estimation. Image Process. Line 3, 137–150 (2013)
Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)
Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: IEEE CVPR, vol. 1, pp. I–511 (2001)
Ngo, A.C., Phan, R.C.-W., See, J.: Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 33–48. Springer, Heidelberg (2015). doi:10.1007/978-3-319-16817-3_3
Le Ngo, A.C., Liong, S.T., See, J., Phan, R.C.W.: Are subtle expressions too sparse to recognize? In: Digital Signal Processing (DSP), pp. 1246–1250 (2015)
Le Ngo, A.C., See, J., Phan, R.C.W.: Sparsity in dynamics of spontaneous subtle emotions: analysis and application. Trans. Affect. Comput. (2016)
Wang, Y., See, J., Phan, R.C.-W., Oh, Y.-H.: LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9003, pp. 525–537. Springer, Cham (2015). doi:10.1007/978-3-319-16865-4_34
Liong, S.T., Phan, R.C.W., See, J., Oh, Y.H., Wong, K.: Optical strain based recognition of subtle emotions. In: International Symposium on Intelligent Signal Processing and Communication Systems, pp. 180–184 (2014)
Oh, Y.H., Le Ngo, A.C., See, J., Liong, S.T., Phan, R.C.W., Ling, H.C.: Monogenic Riesz wavelet representation for micro-expression recognition. In: Digital Signal Processing, pp. 1237–1241. IEEE (2015)
Huang, X., Wang, S.J., Zhao, G., Pietikainen, M.: Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: International Conference on Computer Vision Workshops, pp. 1–9 (2015)
Xu, F., Zhang, J., Wang, J.: Microexpression identification and categorization using a facial dynamics map. Trans. Affect. Comput. (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Liong, ST., See, J., Wong, K., Phan, R.CW. (2017). Automatic Micro-expression Recognition from Long Video Using a Single Spotted Apex. In: Chen, CS., Lu, J., Ma, KK. (eds) Computer Vision – ACCV 2016 Workshops. ACCV 2016. Lecture Notes in Computer Science(), vol 10117. Springer, Cham. https://doi.org/10.1007/978-3-319-54427-4_26
Download citation
DOI: https://doi.org/10.1007/978-3-319-54427-4_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-54426-7
Online ISBN: 978-3-319-54427-4
eBook Packages: Computer ScienceComputer Science (R0)