research-article

Micro-Expression Spotting with Face Alignment and Optical Flow

Authors:

Huimin MaAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 9501 - 9505

https://doi.org/10.1145/3581783.3612853

Published: 27 October 2023 Publication History

Abstract

Facial expression spotting holds significant importance as it can signify emotional changes. Particularly, micro-expressions possess the potential to reveal genuine emotions, making them even more valuable in practical domains such as public safety and finance. However, spotting micro-expressions proves challenging due to their subtle movements and brief duration. This paper proposes an expression spotting method based on face alignment and optical flow. We first use a finer crop-align technique to preprocess the facial videos by aligning the face and the nose tip. Then, regions of interest (ROIs) are defined by analyzing the statistics of action units. The optical flow features are then extracted and subjected to low-pass filtering to eliminate high-frequency noise. Furthermore, candidate expression segments are identified based on the magnitude of the processed optical flows. Finally, non-maximum suppression is utilized to remove overlapping segments. The effectiveness of the proposed method is evaluated on the challenge test set, resulting in an overall F1-score of 0.19. Additional results obtained from CAS(ME)2 and SAMM Long videos provide further verification of the method's efficacy. The code is available online.

References

[1]

P. Ekman and W. V Friesen, ?Nonverbal leakage and clues to deception," Psychiatry, vol. 32, no. 1, pp. 88--106, 1969.

[2]

M. O'sullivan, M. G. Frank, C. M. Hurley, and J. Tiwana, ?Police lie detection accuracy: The effect of lie scenario.," Law Hum Behav, vol. 33, no. 6, p. 530, 2009.

[3]

P. A. Stewart, B. M. Waller, and J. N. Schubert, ?Presidential speechmaking style: Emotional response to micro-expressions of facial affect," Motiv Emot, vol. 33, pp. 125--135, 2009.

[4]

J. Endres and A. Laidlaw, ?Micro-expression recognition training in medical students: a pilot study," BMC Med Educ, vol. 9, no. 1, pp. 1--6, 2009.

[5]

H. Xia and Y. Zhan, ?A survey on temporal action localization," IEEE Access, vol. 8, pp. 70477--70487, 2020.

[6]

J. Li et al., ?MEGC2022: ACM Multimedia 2022 Micro-Expression Grand Challenge," in Proceedings of the 30th ACM International Conference on Multimedia, New York, NY, USA: ACM, Oct. 2022, pp. 7170--7174.

Digital Library

[7]

P. Ekman, W. V Friesen, and J. C. Hager, ?Facial action coding system: The manual on CD ROM," A Human Face, Salt Lake City, pp. 77--254, 2002.

[8]

S. J. Wang, Y. He, J. Li, and X. Fu, ?MESNet: A Convolutional Neural Network for Spotting Multi-Scale Micro-Expression Intervals in Long Videos," IEEE Transactions on Image Processing, vol. 30, pp. 3956--3969, 2021.

[9]

B. Yang et al., ?Facial Action Unit-based Deep Learning Framework for Spotting Macro- And Micro-expressions in Long Video Sequences," MM 2021 - Proceedings of the 29th ACM International Conference on Multimedia, no. March 2022, pp. 4794--4798, 2021.

[10]

W. Leng et al., ?Abpn: Apex and boundary perception network for micro-and macro-expression spotting," in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 7160--7164.

Digital Library

[11]

H. Yuhong, ?Research on Micro-Expression Spotting Method Based on Optical Flow Features," MM 2021 - Proceedings of the 29th ACM International Conference on Multimedia, pp. 4803--4807, 2021.

Digital Library

[12]

J. Yu, Z. Cai, Z. Liu, G. Xie, and P. He, ?Facial expression spotting based on optical flow features," in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 7205--7209.

[13]

T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, ?Openface 2.0: Facial behavior analysis toolkit," in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), IEEE, 2018, pp. 59--66.

[14]

G. Farnebäck, ?Two-frame motion estimation based on polynomial expansion," in Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29-July 2, 2003 Proceedings 13, Springer, 2003, pp. 363--370.

[15]

P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid, ?DeepFlow: Large Displacement Optical Flow with Deep Matching," in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 1385--1392.

[16]

J. Wulff and M. J. Black, ?Efficient sparse-to-dense optical flow estimation using a learned basis and layers," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 120--130.

[17]

F. Steinbrücker, T. Pock, and D. Cremers, ?Large displacement optical flow computation without warping," in 2009 IEEE 12th International Conference on Computer Vision, IEEE, 2009, pp. 1609--1614.

[18]

M. Tao, J. Bai, P. Kohli, and S. Paris, ?SimpleFlow: A Non?iterative, Sublinear Optical Flow Algorithm," in Computer graphics forum, Wiley Online Library, 2012, pp. 345--353.

[19]

F. Qu, S. Wang, W. Yan, H. Li, S. Wu, and X. Fu, ?CAS(ME)2: A Database for Spontaneous Macro-expression and Micro-expression Spotting and Recognition," IEEE Trans Affect Comput, vol. 3045, no. c, pp. 1--14, 2017.

[20]

C. H. Yap, C. Kendrick, and M. H. Yap, ?Samm long videos: A spontaneous facial micro-and macro-expressions dataset," in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE, 2020, pp. 771--776.

[21]

J. Li, Z. Dong, S. Lu, S. Wang, and S. Member, ?CAS (ME)3?: A Third Generation Facial Spontaneous Micro-Expression Database with Depth Information and High Ecological Validity," vol. 8828, no. c, pp. 1--18, 2022.

[22]

A. K. Davison, C. Lansley, N. Costen, K. Tan, and M. H. Yap, ?SAMM: A Spontaneous Micro-Facial Movement Dataset," IEEE Trans Affect Comput, vol. 9, no. 1, pp. 116--129, 2018.

Digital Library

[23]

Y. Zhao et al., ?Rethinking Optical Flow Methods for Micro-Expression Spotting," in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 7175--7179.

Digital Library

[24]

G. B. Liong, S.-T. Liong, J. See, and C.-S. Chan, ?MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro-and Micro-Expression with Hard and Soft Pseudo-labels," in Proceedings of the 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis, 2022, pp. 3--10.

[25]

S. Yin, S. Wu, T. Xu, S. Liu, S. Zhao, and E. Chen, ?AU-aware graph convolutional network for Macro-and Micro-expression spotting," arXiv preprint arXiv:2303.09114, 2023.

[26]

L.-W. Zhang et al., ?Spatio-temporal fusion for macro-and micro-expression spotting in long video sequences," in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE, 2020, pp. 734--741.

Cited By

Liu JWang LWang RKe JYe XWu Y(2025)Exposing the Forgery Clues of DeepFakes via Exploring the Inconsistent Expression CuesInternational Journal of Intelligent Systems10.1155/int/79456462025Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1155/int/7945646
Yang HHuang SLi M(2025)MSOF: A main and secondary bi-directional optical flow feature method for spotting micro-expressionNeurocomputing10.1016/j.neucom.2025.129676630(129676)Online publication date: May-2025
https://doi.org/10.1016/j.neucom.2025.129676
Zhang ZZhao SMao XLiu SWang HXu TChen ECai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)A Multi-scale Feature Learning Network with Optical Flow Correction for Micro- and Macro-expression SpottingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3689143(11497-11502)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3689143
Show More Cited By

Index Terms

Micro-Expression Spotting with Face Alignment and Optical Flow
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks

Recommendations

Efficient Micro-Expression Spotting Based on Main Directional Mean Optical Flow Feature
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Human facial expressions can convey a great deal of information in daily life. Spotting macro-expression (MaE) and micro-expression (ME) intervals from long video sequences is a difficult challenge. In this paper, we propose an efficient framework for ...
Integrating VideoMAE based model and Optical Flow for Micro- and Macro-expression Spotting
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

The task of interval localization of macro- and micro-expression in long videos has a wide range of applications in the field of human-computer interaction. Compared with macro-expression, micro-expression has shorter duration, lower intensity, and ...
Facial Expression Spotting Based on Optical Flow Features
MM '22: Proceedings of the 30th ACM International Conference on Multimedia

The purpose of micro expression (ME) and macro expression (MaE) spotting task is to locate the onset and offset frames of MaE and ME clips. Compared with MaEs, MEs are shorter in duration and lower in intensity, which makes MEs harder to be spotted. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Fundamental Research Funds for the Central Universities
National Key Research and Development Program of China
Guangdong Basic and Applied Basic Research Foundation

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
291
Total Downloads

Downloads (Last 12 months)150
Downloads (Last 6 weeks)14

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu JWang LWang RKe JYe XWu Y(2025)Exposing the Forgery Clues of DeepFakes via Exploring the Inconsistent Expression CuesInternational Journal of Intelligent Systems10.1155/int/79456462025Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1155/int/7945646
Yang HHuang SLi M(2025)MSOF: A main and secondary bi-directional optical flow feature method for spotting micro-expressionNeurocomputing10.1016/j.neucom.2025.129676630(129676)Online publication date: May-2025
https://doi.org/10.1016/j.neucom.2025.129676
Zhang ZZhao SMao XLiu SWang HXu TChen ECai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)A Multi-scale Feature Learning Network with Optical Flow Correction for Micro- and Macro-expression SpottingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3689143(11497-11502)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3689143
Yu JZhang YZhao GHe PZhang ZCai ZLiu QSun JLiang JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Micro-Expression Spotting Based on Optical Flow Feature with Boundary CalibrationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3689142(11490-11496)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3689142
Davison ALi JYap MSee JCheng WLi XHong XWang SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)MEGC2023: ACM Multimedia 2023 ME Grand ChallengeProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612833(9625-9629)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612833
Davison ALi JYap MSee JCheng WLi XHong XWang SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)FME '23: 3rd Facial Micro-Expression WorkshopProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3610948(9736-9738)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3610948
Xue R(2023)Face2Emoji: A framework for face micro-expression recognition and emoji-packetization based on convolutional neural networks2023 5th International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI)10.1109/MLBDBI60823.2023.10481910(109-113)Online publication date: 15-Dec-2023
https://doi.org/10.1109/MLBDBI60823.2023.10481910

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten