skip to main content
10.1145/2964284.2967247acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
short-paper

Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations

Published: 01 October 2016 Publication History

Abstract

Recognizing spontaneous micro-expression in video sequences is a challenging problem. In this paper, we propose a new method of small scale spatio-temporal feature learning. The proposed learning method consists of two parts. First, the spatial features of micro-expressions at different expression-states (i.e., onset, onset to apex transition, apex, apex to offset transition and offset) are encoded using convolutional neural networks (CNN). The expression-states are taken into account in the objective functions, to improve the expression class separability of the learned feature representation. Next, the learned spatial features with expression-state constraints are transferred to learn temporal features of micro-expression. The temporal feature learning encodes the temporal characteristics of the different states of the micro-expression using long short-term memory (LSTM) recurrent neural networks. Extensive and comprehensive experiments have been conducted on the publically available CASME II micro-expression dataset. The experimental results showed that the proposed method outperformed state-of-the-art micro-expression recognition methods in terms of recognition accuracy.

References

[1]
Bartlett, M.S., Movellan, J.R., and Sejnowski, T.J., 2002. Face recognition by independent component analysis. Neural Networks, IEEE Transactions on 13, 6, 1450--1464.
[2]
Chherawala, Y., Roy, P.P., and Cheriet, M., 2013. Feature design for offline Arabic handwriting recognition: handcrafted vs automated? In Document Analysis and Recognition (ICDAR), 2013 12th International Conference on IEEE, 290--294.
[3]
Chollet, F., 2015. Keras. Available: https://github.com/fchollet/keras
[4]
Demyanov, S., ConvNet. Available: https://github.com/sdemyanov/ConvNet
[5]
Gers, F.A., Schraudolph, N.N., and Schmidhuber, J., 2003. Learning precise timing with LSTM recurrent networks. The Journal of Machine Learning Research 3, 115--143.
[6]
Huang, X., Wang, S.J., Zhao, G., Piteik, M., and Inen, 2015. Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection. In 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), 1--9.
[7]
Huang, X., Zhao, G., Hong, X., Zheng, W., and Pietikäinen, M., 2016. Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing 175, Part A(1/29/), 564--578.
[8]
Khorrami, P., Paine, T., and Huang, T., 2015. Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition? In Proceedings of the IEEE International Conference on Computer Vision Workshops, 19--27.
[9]
Le Ngo, A.C., Liong, S.-T., See, J., and Phan, R.C.-W., 2015. Are subtle expressions too sparse to recognize? In Digital Signal Processing (DSP), 2015 IEEE International Conference on IEEE, 1246--1250.
[10]
Mignon, A. and Jurie, F., 2012. Pcca: A new approach for distance learning from sparse pairwise constraints. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on IEEE, 2666--2672.
[11]
Nair, V. and Hinton, G.E., 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), 807--814.
[12]
Oh, Y.-H., Le Ngo, A.C., See, J., Liong, S.-T., Phan, R.C.-W., and Ling, H.-C., 2015. Monogenic Riesz wavelet representation for micro-expression recognition. In Digital Signal Processing (DSP), 2015 IEEE International Conference on IEEE, 1237--1241.
[13]
Park, S.Y., Lee, S.H., and Ro, Y.M., 2015. Subtle Facial Expression Recognition Using Adaptive Magnification of Discriminative Facial Motion. In Proceedings of the 23rd Annual ACM Conference on Multimedia Conference ACM, 911--914.
[14]
Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., and Fergus, R., 2013. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), 1058--1066.
[15]
Wang, Y., See, J., Phan, R.C.-W., and Oh, Y.-H., 2014. LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition. In Computer Vision-ACCV 2014 Springer, 525--537.
[16]
Wang, Y., See, J., Phan, R.C.-W., and Oh, Y.-H., 2015. Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition. PloS One 10, 5, e0124674.
[17]
Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., and Fu, X., 2014. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PloS One 9, 1, e86041.
[18]
Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., and Fu, X., 2013. How fast are the leaked facial expressions: The duration of micro-expressions. Journal of Nonverbal Behavior 37, 4, 217--230.
[19]
Zhao, G. and Pietikainen, M., 2007. Dynamic texture recognition using local binary patterns with an application to facial expressions. Pattern Analysis and Machine Intelligence, IEEE Transactions on 29, 6, 915--928.

Cited By

View all
  • (2025)AUMEs: AU Detection-Based Dual-Stream Multi-task 3DCNN for Micro-expression RecognitionNeural Processing Letters10.1007/s11063-025-11726-057:1Online publication date: 7-Feb-2025
  • (2025)Micro-Expression Recognition via CNN and Multi-path Vision Transformer Integrated with Spatiotemporal Separated Self-attentionBiometric Recognition10.1007/978-981-96-1071-6_4(35-46)Online publication date: 8-Feb-2025
  • (2024)Microexpression Recognition Method Based on ADP-DSTN Feature Fusion and Convolutional Block Attention ModuleElectronics10.3390/electronics1320401213:20(4012)Online publication date: 12-Oct-2024
  • Show More Cited By

Index Terms

  1. Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        MM '16: Proceedings of the 24th ACM international conference on Multimedia
        October 2016
        1542 pages
        ISBN:9781450336031
        DOI:10.1145/2964284
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 01 October 2016

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. long short term memory
        2. micro-expression recognition
        3. recurrent neural networks

        Qualifiers

        • Short-paper

        Funding Sources

        Conference

        MM '16
        Sponsor:
        MM '16: ACM Multimedia Conference
        October 15 - 19, 2016
        Amsterdam, The Netherlands

        Acceptance Rates

        MM '16 Paper Acceptance Rate 52 of 237 submissions, 22%;
        Overall Acceptance Rate 1,291 of 5,076 submissions, 25%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)59
        • Downloads (Last 6 weeks)3
        Reflects downloads up to 15 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2025)AUMEs: AU Detection-Based Dual-Stream Multi-task 3DCNN for Micro-expression RecognitionNeural Processing Letters10.1007/s11063-025-11726-057:1Online publication date: 7-Feb-2025
        • (2025)Micro-Expression Recognition via CNN and Multi-path Vision Transformer Integrated with Spatiotemporal Separated Self-attentionBiometric Recognition10.1007/978-981-96-1071-6_4(35-46)Online publication date: 8-Feb-2025
        • (2024)Microexpression Recognition Method Based on ADP-DSTN Feature Fusion and Convolutional Block Attention ModuleElectronics10.3390/electronics1320401213:20(4012)Online publication date: 12-Oct-2024
        • (2024)Cross-dataset micro-expression identification based on facial ROIs contribution quantificationJournal of Intelligent Systems10.1515/jisys-2024-021333:1Online publication date: 9-Dec-2024
        • (2024)Boosting Micro-Expression Recognition via Self-Expression Reconstruction and Memory Contrastive LearningIEEE Transactions on Affective Computing10.1109/TAFFC.2024.339770115:4(2083-2096)Online publication date: Oct-2024
        • (2024)DFME: A New Benchmark for Dynamic Facial Micro-Expression RecognitionIEEE Transactions on Affective Computing10.1109/TAFFC.2023.334191815:3(1371-1386)Online publication date: Jul-2024
        • (2024)Micro-Expression Spotting and Recognition Network based on Apex Frame2024 IEEE 5th International Conference on Pattern Recognition and Machine Learning (PRML)10.1109/PRML62565.2024.10779771(490-496)Online publication date: 19-Jul-2024
        • (2024)Motion Consistency Constraint Map for Facial Expression Spotting2024 International Conference on Content-Based Multimedia Indexing (CBMI)10.1109/CBMI62980.2024.10858872(1-7)Online publication date: 18-Sep-2024
        • (2024)An improved HTNet micro-expression recognition method2024 International Conference on Advanced Control Systems and Automation Technologies (ACSAT)10.1109/ACSAT63853.2024.10823960(251-254)Online publication date: 15-Nov-2024
        • (2024)Research on Micro-expression Recognition Based on GAM Attention Mechanism and Transfer Learning2024 4th Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS)10.1109/ACCTCS61748.2024.00068(350-355)Online publication date: 24-Feb-2024
        • Show More Cited By

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media