research-article

Rethinking Optical Flow Methods for Micro-Expression Spotting

Authors:

Jiao LiAuthors Info & Claims

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Pages 7175 - 7179

https://doi.org/10.1145/3503161.3551602

Published: 10 October 2022 Publication History

Abstract

Micro-expressions (MEs) spotting is popular in some fields, for example, criminal investigation and business communication. But it is still a challenging task to spot the onset and offset of MEs accurately in long videos. This paper refines every step of the workflow before feature extraction, which can reduce error propagation. The workflow takes the advantage of high-quality alignment method, more accurate landmark detector, and also more robust optical flow estimation. Besides, Bayesian optimization hybrid with Nash equilibrium is constructed to search for the optimal parameters. It uses two players to optimize two types of parameters, one player is used to control the ME peak spotting, and another for optical flow field extraction. The algorithm can reduce the search space for each player with better generalization. Finally, our spotting method is evaluated on MEGC2022 spotting task, and achieves F1-score 0.3564 on CAS(ME)3-UNSEEN and F1-score 0.3265 on SAMM-UNSEEN.

References

[1]

Wang, S. J., He, Y., Li, J., & Fu, X. (2021). MESNet: A convolutional neural network for spotting multi-scale micro-expression intervals in long videos. IEEE Transactions on Image Processing, 30, 3956--3969.

[2]

Yu, W. W., Jiang, J., & Li, Y. J. (2021, October). LSSNet: A two-stream convolutional neural network for spotting macro-and micro-expression in long videos. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 4745--4749).

Digital Library

[3]

Li, J., Soladie, C., & Seguier, R. (2018, May). Ltp-ml: Micro-expression detection by recognition of local temporal pattern of facial movements. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) (pp. 634--641). IEEE.

Digital Library

[4]

Yuhong, H. (2021, October). Research on micro-expression spotting method based on optical flow features. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 4803--4807).

Digital Library

[5]

ISO/IEC 19794-5:2011. (2011). Information technology biometric data interchange formats part 5: Face image data. Standard ISO/IEC 19794-5:2011, International Organization for Standardization, Geneva, CH.

[6]

Zhao, K., Chu, W. S., & Zhang, H. (2016). Deep region and multi-label learning for facial action unit detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3391--3399).

[7]

Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25.

[8]

Xu, S., & Chen, H. (2018). Nash game based efficient global optimization for large-scale design problems. Journal of Global Optimization, 71(2), 361--381.

Digital Library

[9]

Xu, S., Chen, H., & Zhang, J. (2019). A study of Nash-EGO algorithm for aerodynamic shape design optimizations. Structural and Multidisciplinary Optimization, 59(4), 1241--1254.

Digital Library

[10]

Yap, C. H., Kendrick, C., & Yap, M. H. (2020, November). SAMM long videos: A spontaneous facial micro-and macro-expressions dataset. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) (pp. 771--776). IEEE.

Digital Library

[11]

Qu, F., Wang, S. J., Yan, W. J., Li, H., Wu, S., & Fu, X. (2017). CAS(ME)2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing, 9(4), 424--436.

Digital Library

[12]

Davison, A. K., Lansley, C., Costen, N., Tan, K., & Yap, M. H. (2016). SAMM: A spontaneous micro-facial movement dataset. IEEE transactions on affective computing, 9(1), 116--129.

[13]

Li, J., Dong, Z., Lu, S., Wang, S. J., Yan, W. J., Ma, Y.,... & Fu, X. (2022). CAS(ME)3: A Third Generation Facial Spontaneous Micro-Expression Database with Depth Information and High Ecological Validity. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]

Boyko, N., Basystiuk, O., & Shakhovska, N. (2018, August). Performance evaluation and comparison of software for face recognition, based on dlib and opencv library. In 2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP) (pp. 478--482). IEEE.

[15]

Project webpage. https://github.com/cunjian/pytorch_face_landmark/ issues?q=CPU

[16]

Farnebäck, G. (2003, June). Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis (pp. 363--370). Springer, Berlin, Heidelberg.

[17]

Tao, M., Bai, J., Kohli, P., & Paris, S. (2012, May). SimpleFlow: A Non-iterative, Sublinear Optical Flow Algorithm. In Computer graphics forum (Vol. 31, No. 2pt1, pp. 345--353). Oxford, UK: Blackwell Publishing Ltd.

[18]

Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013). DeepFlow: Large displacement optical flow with deep matching. In Proceedings of the IEEE international conference on computer vision (pp. 1385--1392).

Digital Library

[19]

Kroeger, T., Timofte, R., Dai, D., & Gool, L. V. (2016, October). Fast optical flow using dense inverse search. In European conference on computer vision (pp. 471--488). Springer, Cham.

[20]

Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017). Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2462--2470).

[21]

Hui, T. W., Tang, X., & Loy, C. C. (2020). A lightweight optical flow CNN-Revisiting data fidelity and regularization. IEEE transactions on pattern analysis and machine intelligence, 43(8), 2555--2569.

[22]

Sun, D., Yang, X., Liu, M. Y., & Kautz, J. (2018). Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8934--8943).

[23]

Zhao, S., Sheng, Y., Dong, Y., Chang, E. I., & Xu, Y. (2020). Maskflownet: Asymmetric feature matching with learnable occlusion mask. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6278--6287).

[24]

Ballester, C., Garrido, L., Lazcano, V., & Caselles, V. (2012, August). A TV-L1 optical flow method with occlusion detection. In Joint DAGM (German Association for Pattern Recognition) and OAGM Symposium (pp. 31--40). Springer, Berlin, Heidelberg.

[25]

Yap, C. H., Yap, M. H., Davison, A. K., & Cunningham, R. (2021). 3d-cnn for facial micro-and macro-expression spotting on long video sequences using temporal oriented reference frame. arXiv preprint arXiv:2105.06340.

Cited By

Li YGan WLu KJiang DJain R(2025)AVES: An Audio-Visual Emotion Stream Dataset for Temporal Emotion DetectionIEEE Transactions on Affective Computing10.1109/TAFFC.2024.344092416:1(438-450)Online publication date: Jan-2025
https://doi.org/10.1109/TAFFC.2024.3440924
Yang HHuang SLi M(2025)MSOF: A main and secondary bi-directional optical flow feature method for spotting micro-expressionNeurocomputing10.1016/j.neucom.2025.129676630(129676)Online publication date: May-2025
https://doi.org/10.1016/j.neucom.2025.129676
Zhang ZZhao SMao XLiu SWang HXu TChen ECai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)A Multi-scale Feature Learning Network with Optical Flow Correction for Micro- and Macro-expression SpottingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3689143(11497-11502)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3689143
Show More Cited By

Index Terms

Rethinking Optical Flow Methods for Micro-Expression Spotting
1. Human-centered computing

Recommendations

Facial Expression Spotting Based on Optical Flow Features
MM '22: Proceedings of the 30th ACM International Conference on Multimedia

The purpose of micro expression (ME) and macro expression (MaE) spotting task is to locate the onset and offset frames of MaE and ME clips. Compared with MaEs, MEs are shorter in duration and lower in intensity, which makes MEs harder to be spotted. In ...
Research on Micro-Expression Spotting Method Based on Optical Flow Features
MM '21: Proceedings of the 29th ACM International Conference on Multimedia

This paper aims to propose an automatic micro-expression spotting method of high accuracy and high robustness. Due to the characteristics of small amplitude and short duration, how to accurately capture the subtle movements of micro-expression is a ...
Automatic expression spotting in videos

In this paper, we propose a novel solution for the problem of segmenting macro- and micro-expression frames (or retrieving the expression intervals) in video sequences, which is a prior step for many expression recognition algorithms. The proposed ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

October 2022

7537 pages

ISBN:9781450392037

DOI:10.1145/3503161

General Chairs:
João Magalhães
NOVA University of Lisbon, Portugal
,
Alberto del Bimbo
University of Florence, Italy
,
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Nicu Sebe
University of Trento, Italy
,
Program Chairs:
Xavier Alameda-Pineda
Inria, Grenoble, France
,
Qin Jin
Renmin University of China, China
,
Vincent Oria
New Jersey Institute of Technology, USA
,
Laura Toni
University College London, UK

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10 - 14, 2022

Lisboa, Portugal

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
291
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)10

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li YGan WLu KJiang DJain R(2025)AVES: An Audio-Visual Emotion Stream Dataset for Temporal Emotion DetectionIEEE Transactions on Affective Computing10.1109/TAFFC.2024.344092416:1(438-450)Online publication date: Jan-2025
https://doi.org/10.1109/TAFFC.2024.3440924
Yang HHuang SLi M(2025)MSOF: A main and secondary bi-directional optical flow feature method for spotting micro-expressionNeurocomputing10.1016/j.neucom.2025.129676630(129676)Online publication date: May-2025
https://doi.org/10.1016/j.neucom.2025.129676
Zhang ZZhao SMao XLiu SWang HXu TChen ECai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)A Multi-scale Feature Learning Network with Optical Flow Correction for Micro- and Macro-expression SpottingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3689143(11497-11502)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3689143
Yu JZhang YZhao GHe PZhang ZCai ZLiu QSun JLiang JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Micro-Expression Spotting Based on Optical Flow Feature with Boundary CalibrationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3689142(11490-11496)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3689142
Wang LCai THuang PLiu XCai W(2024)Two-Stage Facial Expression Spotting with Spectrum-Based Post-Processing2024 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME57554.2024.10687531(1-6)Online publication date: 15-Jul-2024
https://doi.org/10.1109/ICME57554.2024.10687531
Liong GLiong SChan CSee J(2024)SFAMNetNeurocomputing10.1016/j.neucom.2023.126998566:COnline publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.neucom.2023.126998
Xian YZhang DWang XLee D(2024)A dual-branch network based on optical flow learning and semantic consistency for macro-expression spottingApplied Intelligence10.1007/s10489-024-05726-154:21(10284-10299)Online publication date: 10-Aug-2024
https://dl.acm.org/doi/10.1007/s10489-024-05726-1
Xie ZYe X(2024)Local and Global Features Interactive Fusion Network for Macro- and Micro-expression Spotting in Long VideosPattern Recognition and Computer Vision10.1007/978-981-97-8795-1_23(336-350)Online publication date: 18-Oct-2024
https://dl.acm.org/doi/10.1007/978-981-97-8795-1_23
He EChen QZhong Q(2023)SL-Swin: A Transformer-Based Deep Learning Approach for Macro- and Micro-Expression Spotting on Small-Size Expression DatasetsElectronics10.3390/electronics1212265612:12(2656)Online publication date: 13-Jun-2023
https://doi.org/10.3390/electronics12122656
Yu JCai ZDu SShen XWang LGao FEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Efficient Micro-Expression Spotting Based on Main Directional Mean Optical Flow FeatureProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612861(9541-9545)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612861
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten