Micro-Expression Spotting Based on Optical Flow Feature with Boundary Calibration
Authors: 
Jun Yu, Yaohui Zhang, Gongpeng Zhao, Peng He, Zerui Zhang, Zhongpeng Cai, Qingsong Liu, Jianqing Sun, Jiaen LiangAuthors Info & Claims
Jun Yu, Yaohui Zhang, Gongpeng Zhao, Peng He, Zerui Zhang, Zhongpeng Cai, Qingsong Liu, Jianqing Sun, Jiaen LiangAuthors Info & ClaimsPages 11490 - 11496
Abstract
Micro-expressions, as a type of facial expression corresponding to macro-expressions, usually have a short duration and low intensity. Due to these characteristics, micro-expression spotting holds significant value in medical care and public safety. Recent years have witnessed advancements in micro-expression spotting methods; however, spotting micro-expressions remains a challenging task due to their brief duration and low intensity. In this paper, we propose a micro-expression spotting method based on optical flow features with boundary calibration. We first perform face detection, cropping, and alignment on images containing faces. Then, regions of interest (ROIs) are defined, and optical flow features are extracted. Furthermore, candidate expression segments are identified based on the magnitude of the processed optical flows. Finally, a boundary calibration module is utilized to calibrate the boundaries. The effectiveness of the proposed method is evaluated on the MEGC2024 test set, resulting in an overall F1-score of 0.27.
References
[1]
Adrian K Davison, Cliff Lansley, Nicholas Costen, Kevin Tan, and Moi Hoon Yap. 2016. Samm: A spontaneous micro-facial movement dataset. IEEE transactions on affective computing, Vol. 9, 1 (2016), 116--129.
[2]
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5203--5212.
[3]
Paul Ekman and Wallace V Friesen. 1969. Nonverbal leakage and clues to deception. Psychiatry, Vol. 32, 1 (1969), 88--106.
[4]
Paul Ekman and Wallace V Friesen. 1978. Facial action coding system. Environmental Psychology & Nonverbal Behavior (1978).
[5]
Jennifer Endres and Anita Laidlaw. 2009. Micro-expression recognition training in medical students: a pilot study. BMC medical education, Vol. 9 (2009), 1--6.
[6]
Gunnar Farnebäck. 2003. Two-frame motion estimation based on polynomial expansion. In Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29--July 2, 2003 Proceedings 13. Springer, 363--370.
[7]
Norden E Huang, Zheng Shen, Steven R Long, Manli C Wu, Hsing H Shih, Quanan Zheng, Nai-Chyuan Yen, Chi Chao Tung, and Henry H Liu. 1998. The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proceedings of the Royal Society of London. Series A: mathematical, physical and engineering sciences, Vol. 454, 1971 (1998), 903--995.
[8]
Davis E King. 2009. Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research, Vol. 10 (2009), 1755--1758.
[9]
Wenhao Leng, Sirui Zhao, Yiming Zhang, Shiifeng Liu, Xinglong Mao, Hao Wang, Tong Xu, and Enhong Chen. 2022. Abpn: Apex and boundary perception network for micro-and macro-expression spotting. In Proceedings of the 30th ACM International Conference on Multimedia. 7160--7164.
[10]
Jingting Li, Zizhao Dong, Shaoyuan Lu, Su-Jing Wang, Wen-Jing Yan, Yinhuan Ma, Ye Liu, Changbing Huang, and Xiaolan Fu. 2022. CAS (ME) 3: A third generation facial spontaneous micro-expression database with depth information and high ecological validity. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, 3 (2022), 2782--2800.
[11]
Jingting Li, Catherine Soladie, and Renaud Seguier. 2018. Ltp-ml: Micro-expression detection by recognition of local temporal pattern of facial movements. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, 634--641.
[12]
Xiaobai Li, Shiyang Cheng, Yante Li, Muzammil Behzad, Jie Shen, Stefanos Zafeiriou, Maja Pantic, and Guoying Zhao. 2022. 4DME: A spontaneous 4D micro-expression dataset with multimodalities. IEEE Transactions on Affective Computing, Vol. 14, 4 (2022), 3031--3047.
[13]
mindface. 2022. mindface:mindface for face recognition and detection. https://github.com/mindspore-lab/mindface/.
[14]
Antti Moilanen, Guoying Zhao, and Matti Pietikäinen. 2014. Spotting rapid facial movements from videos using appearance-based feature difference analysis. In 2014 22nd international conference on pattern recognition. IEEE, 1722--1727.
[15]
Xuan-Bac Nguyen, Chi Nhan Duong, Xin Li, Susan Gauch, Han-Seok Seo, and Khoa Luu. 2023. Micron-bert: Bert-based facial micro-expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1482--1492.
[16]
Maureen O'sullivan, Mark G Frank, Carolyn M Hurley, and Jaspreet Tiwana. 2009. Police lie detection accuracy: The effect of lie scenario. Law and human behavior, Vol. 33 (2009), 530--538.
[17]
Hang Pan, Lun Xie, and Zhiliang Wang. 2020. Local bilinear convolutional neural network for spotting macro-and micro-expression intervals in long video sequences. In 2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020). IEEE, 749--753.
[18]
Wenfeng Qin, Bochao Zou, Xin Li, Weiping Wang, and HUimin Ma. 2023. Micro-Expression Spotting with Face Alignment and Optical Flow. In Proceedings of the 31st ACM International Conference on Multimedia. 9501--9505.
[19]
Fangbing Qu, Su-Jing Wang, Wen-Jing Yan, He Li, Shuhang Wu, and Xiaolan Fu. 2017. CAS (ME) 2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing, Vol. 9, 4 (2017), 424--436.
[20]
Patrick A Stewart, Bridget M Waller, and James N Schubert. 2009. Presidential speechmaking style: Emotional response to micro-expressions of facial affect. Motivation and Emotion, Vol. 33 (2009), 125--135.
[21]
Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. 2024. HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition. Information Fusion, Vol. 108 (2024), 102382.
[22]
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, Vol. 35 (2022), 10078--10093.
[23]
Thuong-Khanh Tran, Quang-Nhat Vo, Xiaopeng Hong, Xiaobai Li, and Guoying Zhao. 2021. Micro-expression spotting: A new benchmark. Neurocomputing, Vol. 443 (2021), 356--368.
[24]
Qiangchang Wang and Guodong Guo. 2019. LS-CNN: Characterizing local patches at multiple scales for face recognition. IEEE Transactions on Information Forensics and Security, Vol. 15 (2019), 1640--1653.
[25]
Su-Jing Wang, Ying He, Jingting Li, and Xiaolan Fu. 2021. MESNet: A convolutional neural network for spotting multi-scale micro-expression intervals in long videos. IEEE Transactions on Image Processing, Vol. 30 (2021), 3956--3969.
[26]
Ke Xu, Kang Chen, Licai Sun, Zheng Lian, Bin Liu, Gong Chen, Haiyang Sun, Mingyu Xu, and Jianhua Tao. 2023. Integrating VideoMAE based model and Optical Flow for Micro-and Macro-expression Spotting. In Proceedings of the 31st ACM International Conference on Multimedia. 9576--9580.
[27]
Wen-Jing Yan, Qi Wu, Jing Liang, Yu-Hsin Chen, and Xiaolan Fu. 2013. How fast are the leaked facial expressions: The duration of micro-expressions. Journal of Nonverbal Behavior, Vol. 37 (2013), 217--230.
[28]
Bo Yang, Jianming Wu, Zhiguang Zhou, Megumi Komiya, Koki Kishimoto, Jianfeng Xu, Keisuke Nonaka, Toshiharu Horiuchi, Satoshi Komorita, Gen Hattori, et al. 2021. Facial action unit-based deep learning framework for spotting macro-and micro-expressions in long video sequences. In Proceedings of the 29th ACM International Conference on Multimedia. 4794--4798.
[29]
Chuin Hong Yap, Connah Kendrick, and Moi Hoon Yap. 2020. Samm long videos: A spontaneous facial micro-and macro-expressions dataset. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). IEEE, 771--776.
[30]
Chuin Hong Yap, Moi Hoon Yap, Adrian Davison, Connah Kendrick, Jingting Li, Su-Jing Wang, and Ryan Cunningham. 2022. 3d-cnn for facial micro-and macro-expression spotting on long video sequences using temporal oriented reference frame. In Proceedings of the 30th ACM International Conference on Multimedia. 7016--7020.
[31]
Jun Yu, Zhongpeng Cai, Shenshen Du, Xiaxin Shen, Lei Wang, and Fang Gao. 2023. Efficient Micro-Expression Spotting Based on Main Directional Mean Optical Flow Feature. In Proceedings of the 31st ACM International Conference on Multimedia. 9541--9545.
[32]
Jun Yu, Zhongpeng Cai, Zepeng Liu, Guochen Xie, and Peng He. 2022. Facial expression spotting based on optical flow features. In Proceedings of the 30th ACM International Conference on Multimedia. 7205--7209.
[33]
Wang-Wang Yu, Jingwen Jiang, and Yong-Jie Li. 2021. LSSNet: A two-stream convolutional neural network for spotting macro-and micro-expression in long videos. In Proceedings of the 29th ACM International Conference on Multimedia. 4745--4749.
[34]
He Yuhong. 2021. Research on micro-expression spotting method based on optical flow features. In Proceedings of the 29th ACM International Conference on Multimedia. 4803--4807.
[35]
Yuan Zhao, Xin Tong, Zichong Zhu, Jianda Sheng, Lei Dai, Lingling Xu, Xuehai Xia, Yu Jiang, and Jiao Li. 2022. Rethinking optical flow methods for micro-expression spotting. In Proceedings of the 30th ACM International Conference on Multimedia. 7175--7179.
Index Terms
- Micro-Expression Spotting Based on Optical Flow Feature with Boundary Calibration
Recommendations
Micro-Expression Spotting with Face Alignment and Optical Flow
MM '23: Proceedings of the 31st ACM International Conference on MultimediaFacial expression spotting holds significant importance as it can signify emotional changes. Particularly, micro-expressions possess the potential to reveal genuine emotions, making them even more valuable in practical domains such as public safety and ...
Efficient Micro-Expression Spotting Based on Main Directional Mean Optical Flow Feature
MM '23: Proceedings of the 31st ACM International Conference on MultimediaHuman facial expressions can convey a great deal of information in daily life. Spotting macro-expression (MaE) and micro-expression (ME) intervals from long video sequences is a difficult challenge. In this paper, we propose an efficient framework for ...
Facial Expression Spotting Based on Optical Flow Features
MM '22: Proceedings of the 30th ACM International Conference on MultimediaThe purpose of micro expression (ME) and macro expression (MaE) spotting task is to locate the onset and offset frames of MaE and ME clips. Compared with MaEs, MEs are shorter in duration and lower in intensity, which makes MEs harder to be spotted. In ...
Comments
Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Beijing Municipal Science & Technology Commission, Administrative Commission of Zhongguancun Science Park
- Dreams Foundation of Jianghuai Advance Technology Center
- Natural Science Foundation of China
- National Aviation Science Foundation
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 260Total Downloads
- Downloads (Last 12 months)260
- Downloads (Last 6 weeks)147
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in