Abstract
Micro-expressions are facial expressions that are revealed unconsciously when suppressing true emotions and are widely used in multiple tasks, such as deception detection. However, at present, the amount of available micro-expression data is small, and there are large differences between different databases, so it is still difficult to accurately perform cross-database micro-expression recognition, which hinders real applications of recognition. To address this issue, this article first presents a meta-learning framework suitable for cross-database micro-expression recognition named Meta-CDMERF, which is trained by combining multiple micro-expression databases. Then, a residual feature-wise linear (RFL) module is proposed to generate more feature distributions and adaptively choose representative features during multi micro-expression database training, thereby reducing the feature distance between samples of the same type in different databases. Next, a new loss function is designed, which combines the cross-entropy loss function with an interclass loss function. Specifically, the inter-class loss is based on the mean value of similar features from the support set and aims to increase the distance between samples of different categories, thereby capturing subtle changes in micro-expression images. Finally, the unweighted average recall (UAR) and unweighted F1 score (UF1) values of the proposed method on the CASME II database are 62.64% and 60.00%, respectively, reaching state-of-the-art performance.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability and access
The dataset generated and analyzed during the current study are available from the corresponding author on reasonable request.
References
Yan W, Wang S, Liu Y, Wu Q, Fu X (2014) For micro-expression recognition: Database and suggestions. Neurocomputing 136:82–87. https://doi.org/10.1016/j.neucom.2014.01.029
Zhao Y, Xu J (2019) A convolutional neural network for compound micro-expression recognition. Sensors 19(24):5553. https://doi.org/10.3390/s19245553
Zhao G, Li X (2019) Automatic micro-expression analysis: open challenges. Front Psychol 10:1833. https://doi.org/10.3389/fpsyg.2019.01833
Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, pp 1449–1456. https://doi.org/10.1109/ICCV.2011.6126401. IEEE
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928. https://doi.org/10.1109/TPAMI.2007.1110
Liu Y, Zhang J, Yan W, Wang S, Zhao G, Fu X (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310. https://doi.org/10.1109/TAFFC.2015.2485205
Zhu Q, Li Z, Kuang W, Ma H (2023) A multichannel location-aware interaction network for visual classification. Appl Intell 53(20):23049–23066. https://doi.org/10.1007/s10489-023-04734-x
Yang X, Li Z, Zhong X, Zhang C, Ma H (2023) Mining graph-based dynamic relationships for object detection. Eng Appl Artif Intell 126:106928. https://doi.org/10.1016/j.engappai.2023.106928
Lan G, Xiao S, Yang J, Zhou Y, Wen J, Lu W, Gao X (2023) Image aesthetics assessment based on hypernetwork of emotion fusion. IEEE Trans Multimedia. https://doi.org/10.1109/TMM.2023.3313507
Peng M, Wu Z, Zhang Z, Chen T (2018) From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 657–661. https://doi.org/10.1109/FG.2018.00103. IEEE
Peng M, Wang C, Chen T (2018) Attention based residual network for micro-gesture recognition. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 790–794. https://doi.org/10.1109/FG.2018.00127. IEEE
Wang C, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing. 410:354–362. https://doi.org/10.1016/j.neucom.2020.06.005
Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90. https://doi.org/10.1145/3065386
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. https://doi.org/10.48550/arXiv.2010.11929
Khor H-Q, See J, Liong S-T, Phan RC, Lin W (2019) Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE International Conference on Image Processing (ICIP), pp 36–40. https://doi.org/10.1109/ICIP.2019.8802965. IEEE
Zhang L, Hong X, Arandjelović O, Zhao G (2022) Short and long range relation based spatio-temporal transformer for micro-expression recognition. IEEE Trans Affect Comput 13(4):1973–1985. https://doi.org/10.1109/TAFFC.2022.3213509
Nguyen X-B, Duong CN, Li X, Gauch S, Seo H-S, Luu K (2023) Micron-bert: Bert-based facial micro-expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1482–1492. https://doi.org/10.48550/arXiv.2304.03195
Yap MH, See J, Hong X, Wang S-J (2018) Facial micro-expressions grand challenge 2018 summary. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 675–678. https://doi.org/10.1109/FG.2018.00106. IEEE
Xia B, Wang W, Wang S, Chen E (2020) Learning from macro-expression: A micro-expression recognition framework. In: Proceedings of the 28th ACM International Conference on Multimedia, pp 2936–2944. https://doi.org/10.1145/3394171.3413774
Bao Y, Wu C, Zhang P, Shan C, Qi Y, Ben X (2024) Boosting micro-expression recognition via self-expression reconstruction and memory contrastive learning. IEEE Trans Affect Comput. https://doi.org/10.1016/j.patrec.2022.09.006
Liong S-T, See J, Wong K, Phan RC-W (2018) Less is more: Micro-expression recognition from video using apex frame. Signal Processing: Image Communication 62:82–92. https://doi.org/10.1016/j.image.2017.11.006
Li Y, Huang X, Zhao G (2018) Can micro-expression be recognized based on single apex frame? In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp 3094–3098. https://doi.org/10.1109/ICIP.2018.8451376. IEEE
Li Y, Huang X, Zhao G (2020) Joint local and global information learning with single apex frame detection for micro-expression recognition. IEEE Trans Image Process 30:249–263. https://doi.org/10.1109/TIP.2020.3035042
Sun M-X, Liong S-T, Liu K-H, Wu Q-Q (2022) The heterogeneous ensemble of deep forest and deep neural networks for micro-expressions recognition. Appl Intell 52(14):16621–16639. https://doi.org/10.1007/s10489-022-03284-y
Zhou L, Mao Q, Xue L (2019) Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp 1–5. https://doi.org/10.1109/FG.2019.8756579. IEEE
Chen B, Liu K-H, Xu Y, Wu Q-Q, Yao J-F (2022) Block division convolutional network with implicit deep features augmentation for micro-expression recognition. IEEE Trans Multimed 25:1345–1358. https://doi.org/10.1109/TMM.2022.3141616
Peng M, Wang C, Bi T, Shi Y, Zhou X, Chen T (2019) A novel apex-time network for cross-dataset micro-expression recognition. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pp 1–6. https://doi.org/10.1109/ACII.2019.8925525. IEEE
Liu J, Zong Y, Zheng W (2022) Cross-database micro-expression recognition based on transfer double sparse learning. Multimed Tools Appl 81(30):43513–43530. https://doi.org/10.1007/s11042-022-12878-0
Zong Y, Zheng W, Huang X, Shi J, Cui Z, Zhao G (2018) Domain regeneration for cross-database micro-expression recognition. IEEE Trans Image Process 27(5):2484–2498. https://doi.org/10.1109/TIP.2018.2797479
Lin W-W, Mak M-W, Li L, Chien J-T (2018) Reducing domain mismatch by maximum mean discrepancy based autoencoders. In: Odyssey, pp 162–167. https://doi.org/10.21437/Odyssey.2018-23
Liu T, Tao D, Song M, Maybank SJ (2016) Algorithm-dependent generalization bounds for multi-task learning. IEEE Trans Pattern Anal Mach Intell 39(2):227–241. https://doi.org/10.1109/TPAMI.2016.2544314
Grefenstette E, Amos B, Yarats D, Htut PM, Molchanov A, Meier F, Kiela D, Cho K, Chintala S (2019) Generalized inner loop meta-learning. arXiv preprint arXiv:1910.01727. https://doi.org/10.48550/arXiv.1910.01727
Finn C, Levine S (2017) Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. arXiv preprint arXiv:1710.11622. https://doi.org/10.48550/arXiv.1710.11622
Hospedales T, Antoniou A, Micaelli P, Storkey A (2021) Meta-learning in neural networks: A survey. IEEE Trans Pattern Anal Mach Intell 44(9):5149–5169. https://doi.org/10.1109/TPAMI.2021.3079209
Guo Y, Codella NC, Karlinsky L, Codella JV, Smith JR, Saenko K, Rosing T, Feris R (2020) A broader study of cross-domain few-shot learning. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16, pp 124–141. https://doi.org/10.1007/978-3-030-58583-9_8. Springer
Hospedales T, Antoniou A, Micaelli P, Storkey A (2021) Meta-learning in neural networks: A survey. IEEE Trans Pattern Anal Mach Intell 44(9):5149–5169. https://doi.org/10.1109/TPAMI.2021.3079209
Wan B, Dang J, Liu X, Wang Q (2022) Micro-expression recognition based on maml meta-learning algorithm. In: 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta), pp 1322–1328. https://doi.org/10.1109/SmartWorld-UIC-ATC-ScalCom-DigitalTwin-PriComp-Metaverse56740.2022.00191. IEEE
Wang J, Tian Y, Yang Y, Chen X, Zheng C, Qiang W (2024) Meta-auxiliary learning for micro-expression recognition. arXiv preprint arXiv:2404.12024. https://doi.org/10.48550/arXiv.2404.12024
Gong W, Zhang Y, Wang W, Cheng P, Gonzalez J (2023) Meta-mmfnet: Meta-learning-based multi-model fusion network for micro-expression recognition. ACM Trans Multimed Comput Commun Appl 20(2):1–20. https://doi.org/10.1145/3539576
Wang Z, Fu W, Zhang Y, Li J, Gong W, Gonzàlez J (2024) Mcnet: meta-clustering learning network for micro-expression recognition. J Electron Imaging 33(2):023014–023014. https://doi.org/10.1117/1.JEI.33.2.023014
Jia D, Wang K, Luo S, Liu T, Liu Y (2021) Braft: Recurrent all-pairs field transforms for optical flow based on correlation blocks. IEEE Signal Process Lett 28:1575–1579. https://doi.org/10.1109/LSP.2021.3099076
Rebuffi S-A, Bilen H, Vedaldi A (2017) Learning multiple visual domains with residual adapters. Adv Neural Inf Process Syst 30. https://doi.org/10.5555/3294771.3294820
Tseng H-Y, Lee H-Y, Huang J-B, Yang M-H (2020) Cross-domain few-shot classification via learned feature-wise transformation. arXiv preprint arXiv:2001.08735. https://doi.org/10.48550/arXiv.2001.08735
Yan W, Wu Q, Liu Y, Wang S, Fu X (2013) Casme database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–7. https://doi.org/10.1109/FG.2013.6553799. IEEE
Yan W, Li X, Wang S, Zhao G, Liu Y, Chen Y-H, Fu X (2014) Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1):86041. https://doi.org/10.1371/journal.pone.0086041
Ben X, Ren Y, Zhang J, Wang S-J, Kpalma K, Meng W, Liu Y-J (2021) Video-based facial micro-expression analysis: A survey of datasets, features and algorithms. IEEE Trans Pattern Anal Mach Intell 44(9):5826–5846. https://doi.org/10.1109/TPAMI.2021.3067464
Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (fg), pp 1–6. https://doi.org/10.1109/FG.2013.6553717. IEEE
Davison AK, Lansley C, Costen N, Tan K, Yap MH (2016) Samm: A spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129. https://doi.org/10.1109/TAFFC.2016.2573832
Yap MH, See J, Hong X, Wang S-J (2018) Facial micro-expressions grand challenge 2018 summary. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 675–678. https://doi.org/10.1109/FG.2018.00106. IEEE
Maaten L, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(11)
Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. https://doi.org/10.1049/ic.2009.0244
Khor H-Q, See J, Phan RCW, Lin W (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 667–674. https://doi.org/10.1109/FG.2018.00105. IEEE
Talluri KK, Fiedler M-A, Al-Hamadi A (2022) Deep 3d convolutional neural network for facial micro-expression analysis from video images. Appl Sci 12(21):11078. https://doi.org/10.3390/app122111078
Zhang M, Huan Z, Shang L (2020) Micro-expression recognition using micro-variation boosted heat areas. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp 531–543. https://doi.org/10.1007/978-3-030-60639-8_44. Springer
Shu X, Li J, Shi L, Huang S (2023) Res-capsnet: an improved capsule network for micro-expression recognition. Multimed Syst 29(3):1593–1601. https://doi.org/10.1007/s00530-023-01068-z
Peng M, Wu Z, Zhang Z, Chen T (2018) From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 657–661. https://doi.org/10.1109/FG.2018.00103. IEEE
Wang C, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing 410:354–362. https://doi.org/10.1016/j.neucom.2020.06.005
Zhang T, Zong Y, Zheng W, Chen CP, Hong X, Tang C, Cui Z, Zhao G (2022) Cross-database micro-expression recognition: A benchmark. IEEE Trans Knowl Data Eng 34(02):544–559. https://doi.org/10.1145/3323873.3326590
Chen Z, Lu C, Zhou F, Zong Y (2023) Tkrm: Learning a transfer kernel regression model for cross-database micro-expression recognition. Mathematics 11(4):918. https://doi.org/10.3390/math11040918
Lan G, Xiao S, Yang J, Wen J, Lu W, Gao X (2024) Active learning inspired method in generative models. Expert Syst Appl 249:123582. https://doi.org/10.1016/j.eswa.2024.123582
Lan G, Xiao S, Yang J, Wen J, Xi M (2023) Generative ai-based data completeness augmentation algorithm for data-driven smart healthcare. IEEE J Biomed Health Inform. https://doi.org/10.1109/JBHI.2023.3327485
Acknowledgements
This research is partly supported by Project of Chongqing Science and Technology Bureau (CSTB2023TIAD-STX0037), Fujian Provincial Science and Technology Plan Project (2022T3016), and Fundamental Research Funds for the Central Universities SWU120083.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. The first draft of the manuscript was written by Hanpu wang and all authors commented on previous versions of the manuscript. Professor Chen Tong refined and revised the manuscript.
Corresponding author
Ethics declarations
Competing Interests
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Ethical and informed consent for data used
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, H., Zhou, J., Liu, X. et al. A cross-database micro-expression recognition framework based on meta-learning. Appl Intell 55, 58 (2025). https://doi.org/10.1007/s10489-024-05896-y
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-05896-y