Wenhao Leng
Enhong Chen
ABSTRACT
Recently, Micro expression~(ME) has achieved remarkable progress in a wide range of applications, since it's an involuntary facial expression that reflects personal psychological state truly. In the procedure of ME analysis, spotting ME is an essential step, and is non trivial to be detected from a long interval video because of the short duration and low intensity issues. To alleviate this problem, in this paper, we propose a novel Micro- and Macro-Expression~(MaE) Spotting framework based on Apex and Boundary Perception Network~(ABPN), which mainly consists of three parts, i.e., video encoding module ~(VEM), probability evaluation module~(PEM), and expression proposal generation module~(EPGM). Firstly, we adopt Main Directional Mean Optical Flow (MDMO) algorithm and calculate optical flow differences to extract facial motion features in VEM, which can alleviate the impact of head movement and other areas of the face on ME spotting. Then, we extract temporal features with one-dimension convolutional layers and introduce PEM to infer the auxiliary probability that each frame belongs to an apex or boundary frame. With these frame-level auxiliary probabilities, the EPGM further combines the frames from different categories to generate expression proposals for the accurate localization. Besides, we conduct comprehensive experiments on MEGC2022 spotting task, and demonstrate that our proposed method achieves significant improvement with the comparison of state-of-the-art baselines on rm CAS(ME)2 and SAMM-LV datasets. The implemented code is also publicly available at https://github.com/wenhaocold/USTC_ME_Spotting.
- Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5203--5212.Google Scholar
Cross Ref
- Xuanyi Dong, Yan Yan, Wanli Ouyang, and Yi Yang. 2018. Style aggregated network for facial landmark detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 379--388.Google ScholarCross Ref
- Paul Ekman. 2003. Darwin, deception, and facial expression. Annals of the new York Academy of sciences 1000, 1 (2003), 205--221.Google ScholarCross Ref
- Paul Ekman. 2009. Telling lies: Clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & Company.Google Scholar
- Paul Ekman and Wallace V Friesen. 1969. Nonverbal leakage and clues to deception. Psychiatry 32, 1 (1969), 88--106.Google ScholarCross Ref
- Rosenberg Ekman. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA.Google Scholar
- Mark Frank, Malgorzata Herbasz, Kang Sinuk, A Keller, and Courtney Nolan. 2009. I see how you feel: Training laypeople and professionals to recognize fleeting emotions. In The Annual Meeting of the International Communication Association. Sheraton New York, New York City. 1--35.Google Scholar
- Ying He, Su-JingWang, Jingting Li, and Moi Hoon Yap. 2020. Spotting macro-and micro-expression intervals in long video sequences. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). IEEE, 742--748.Google ScholarDigital Library
- Huai-Qian Khor, John See, Raphael Chung Wei Phan, and Weiyao Lin. 2018. Enriched long-term recurrent convolutional network for facial micro-expression recognition. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 667--674.Google Scholar
- Jingting Li, Zizhao Dong, Shaoyuan Lu, Su-JingWang,Wen-Jing Yan, Yinhuan Ma, Ye Liu, Changbing Huang, and Xiaolan Fu. 2022. CAS (ME) 3: A Third Generation Facial Spontaneous Micro-Expression Database with Depth Information and High Ecological Validity. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).Google Scholar
- Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. 2018. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the European conference on computer vision (ECCV). 3--19.Google ScholarDigital Library
- Yong-Jin Liu, Jin-Kai Zhang, Wen-Jing Yan, Su-Jing Wang, Guoying Zhao, and Xiaolan Fu. 2015. A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing 7, 4 (2015), 299--310.Google ScholarDigital Library
- Antti Moilanen, Guoying Zhao, and Matti Pietikäinen. 2014. Spotting rapid facial movements from videos using appearance-based feature difference analysis. In 2014 22nd international conference on pattern recognition. IEEE, 1722--1727.Google ScholarDigital Library
- Yee-Hui Oh, John See, Anh Cat Le Ngo, Raphael C-W Phan, and Vishnu M Baskaran. 2018. A survey of automatic facial micro-expression analysis: databases, methods, and challenges. Frontiers in psychology 9 (2018), 1128.Google Scholar
- Stephen Porter and Leanne Ten Brinke. 2008. Reading between the lies: Identifying concealed and falsified emotions in universal facial expressions. Psychological science 19, 5 (2008), 508--514.Google ScholarCross Ref
- Fangbing Qu, Su-Jing Wang, Wen-Jing Yan, He Li, Shuhang Wu, and Xiaolan Fu. 2017. CAS (ME) 2: a database for spontaneous macro-expression and microexpression spotting and recognition. IEEE Transactions on Affective Computing 9, 4 (2017), 424--436.Google ScholarDigital Library
- Javier Sánchez Pérez, Enric Meinhardt-Llopis, and Gabriele Facciolo. 2013. TV-L1 Optical Flow Estimation. Image Processing On Line 3 (2013), 137--150. https://doi.org/10.5201/ipol.2013.26.Google ScholarCross Ref
- Thuong-Khanh Tran, Quang-Nhat Vo, Xiaopeng Hong, Xiaobai Li, and Guoying Zhao. 2021. Micro-expression spotting: A new benchmark. Neurocomputing 443 (2021), 356--368.Google ScholarCross Ref
- Su-Jing Wang, Ying He, Jingting Li, and Xiaolan Fu. 2021. MESNet: A convolutional neural network for spotting multi-scale micro-expression intervals in long videos. IEEE Transactions on Image Processing 30 (2021), 3956--3969.Google ScholarCross Ref
- Sharon Weinberger. 2010. Intent to deceive? Can the science of deception detection help to catch terrorists? Sharon Weinberger takes a close look at the evidence for it. Nature 465, 7297 (2010), 412--416.Google Scholar
- Wanchuang Xia, Wenming Zheng, Yuan Zong, and Xingxun Jiang. 2021. Motion Attention Deep Transfer Network for Cross-database Micro-expression Recognition. In International Conference on Pattern Recognition. Springer, 679--693.Google Scholar
- Bo Yang, Jianming Wu, Zhiguang Zhou, Megumi Komiya, Koki Kishimoto, Jianfeng Xu, Keisuke Nonaka, Toshiharu Horiuchi, Satoshi Komorita, Gen Hattori, et al. 2021. Facial action unit-based deep learning framework for spotting macroand micro-expressions in long video sequences. In Proceedings of the 29th ACM International Conference on Multimedia. 4794--4798.Google ScholarDigital Library
- Chuin Hong Yap, Connah Kendrick, and Moi Hoon Yap. 2020. Samm long videos: A spontaneous facial micro-and macro-expressions dataset. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). IEEE, 771--776.Google ScholarDigital Library
- Chuin Hong Yap, Moi Hoon Yap, Adrian K Davison, and Ryan Cunningham. 2021. 3d-cnn for facial micro-and macro-expression spotting on long video sequences using temporal oriented reference frame. arXiv preprint arXiv:2105.06340 (2021).Google Scholar
- Wang-Wang Yu, Jingwen Jiang, and Yong-Jie Li. 2021. LSSNet: A two-stream convolutional neural network for spotting macro-and micro-expression in long videos. In Proceedings of the 29th ACM International Conference on Multimedia. 4745--4749.Google ScholarDigital Library
- He Yuhong. 2021. Research on micro-expression spotting method based on optical flow features. In Proceedings of the 29th ACM International Conference on Multimedia. 4803--4807.Google ScholarDigital Library
- Li-Wei Zhang, Jingting Li, Su-Jing Wang, Xian-Hua Duan, Wen-Jing Yan, Hai-Yong Xie, and Shu-Cheng Huang. 2020. Spatio-temporal fusion for macro-and micro-expression spotting in long video sequences. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). IEEE, 734--741.Google ScholarDigital Library
- Sirui Zhao, Huaying Tang, Shifeng Liu, Yangsong Zhang, Hao Wang, Tong Xu, Enhong Chen, and Cuntai Guan. 2022. ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition. Neural Networks (2022). https://doi.org/10.1016/j.neunet.2022.06.024Google Scholar
- Sirui Zhao, Hanqing Tao, Yangsong Zhang, Tong Xu, Kun Zhang, Zhongkai Hao, and Enhong Chen. 2021. A two-stage 3D CNN based learning method for spontaneous micro-expression recognition. Neurocomputing 448 (2021), 276--289.Google ScholarCross Ref
Index Terms
- ABPN: Apex and Boundary Perception Network for Micro- and Macro-Expression Spotting
Recommendations
Spontaneous micro-expression spotting via geometric deformation modeling
A probabilistic framework is proposed to detect spontaneous micro-expression clips.The geometric deformation captured by ASM model is utilized as features.The features are robust to subtle head movement and illumination variation.The Adaboost algorithm ...
Needle in a Haystack: Spotting and recognising micro-expressions “in the wild”
AbstractComputational research on facial micro-expressions has long focused on videos captured under constrained laboratory conditions due to the challenging elicitation process and limited samples that are publicly available. Moreover, ...
A main directional maximal difference analysis for spotting facial movements from long-term videos
There is an increasing interests in micro-expression researches. Spotting micro-expressions in long-term videos is very important, not only for providing clues for lie detection, but also for reducing the labor required to collect micro-expression data. ...
Comments