Skip to main content

Advertisement

A cross-database micro-expression recognition framework based on meta-learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Micro-expressions are facial expressions that are revealed unconsciously when suppressing true emotions and are widely used in multiple tasks, such as deception detection. However, at present, the amount of available micro-expression data is small, and there are large differences between different databases, so it is still difficult to accurately perform cross-database micro-expression recognition, which hinders real applications of recognition. To address this issue, this article first presents a meta-learning framework suitable for cross-database micro-expression recognition named Meta-CDMERF, which is trained by combining multiple micro-expression databases. Then, a residual feature-wise linear (RFL) module is proposed to generate more feature distributions and adaptively choose representative features during multi micro-expression database training, thereby reducing the feature distance between samples of the same type in different databases. Next, a new loss function is designed, which combines the cross-entropy loss function with an interclass loss function. Specifically, the inter-class loss is based on the mean value of similar features from the support set and aims to increase the distance between samples of different categories, thereby capturing subtle changes in micro-expression images. Finally, the unweighted average recall (UAR) and unweighted F1 score (UF1) values of the proposed method on the CASME II database are 62.64% and 60.00%, respectively, reaching state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability and access

The dataset generated and analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Yan W, Wang S, Liu Y, Wu Q, Fu X (2014) For micro-expression recognition: Database and suggestions. Neurocomputing 136:82–87. https://doi.org/10.1016/j.neucom.2014.01.029

    Article  MATH  Google Scholar 

  2. Zhao Y, Xu J (2019) A convolutional neural network for compound micro-expression recognition. Sensors 19(24):5553. https://doi.org/10.3390/s19245553

    Article  MATH  Google Scholar 

  3. Zhao G, Li X (2019) Automatic micro-expression analysis: open challenges. Front Psychol 10:1833. https://doi.org/10.3389/fpsyg.2019.01833

    Article  MATH  Google Scholar 

  4. Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, pp 1449–1456. https://doi.org/10.1109/ICCV.2011.6126401. IEEE

  5. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928. https://doi.org/10.1109/TPAMI.2007.1110

    Article  MATH  Google Scholar 

  6. Liu Y, Zhang J, Yan W, Wang S, Zhao G, Fu X (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310. https://doi.org/10.1109/TAFFC.2015.2485205

    Article  MATH  Google Scholar 

  7. Zhu Q, Li Z, Kuang W, Ma H (2023) A multichannel location-aware interaction network for visual classification. Appl Intell 53(20):23049–23066. https://doi.org/10.1007/s10489-023-04734-x

    Article  MATH  Google Scholar 

  8. Yang X, Li Z, Zhong X, Zhang C, Ma H (2023) Mining graph-based dynamic relationships for object detection. Eng Appl Artif Intell 126:106928. https://doi.org/10.1016/j.engappai.2023.106928

    Article  MATH  Google Scholar 

  9. Lan G, Xiao S, Yang J, Zhou Y, Wen J, Lu W, Gao X (2023) Image aesthetics assessment based on hypernetwork of emotion fusion. IEEE Trans Multimedia. https://doi.org/10.1109/TMM.2023.3313507

    Article  MATH  Google Scholar 

  10. Peng M, Wu Z, Zhang Z, Chen T (2018) From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 657–661. https://doi.org/10.1109/FG.2018.00103. IEEE

  11. Peng M, Wang C, Chen T (2018) Attention based residual network for micro-gesture recognition. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 790–794. https://doi.org/10.1109/FG.2018.00127. IEEE

  12. Wang C, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing. 410:354–362. https://doi.org/10.1016/j.neucom.2020.06.005

    Article  MATH  Google Scholar 

  13. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90. https://doi.org/10.1145/3065386

    Article  MATH  Google Scholar 

  14. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  15. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. https://doi.org/10.48550/arXiv.2010.11929

  16. Khor H-Q, See J, Liong S-T, Phan RC, Lin W (2019) Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE International Conference on Image Processing (ICIP), pp 36–40. https://doi.org/10.1109/ICIP.2019.8802965. IEEE

  17. Zhang L, Hong X, Arandjelović O, Zhao G (2022) Short and long range relation based spatio-temporal transformer for micro-expression recognition. IEEE Trans Affect Comput 13(4):1973–1985. https://doi.org/10.1109/TAFFC.2022.3213509

    Article  MATH  Google Scholar 

  18. Nguyen X-B, Duong CN, Li X, Gauch S, Seo H-S, Luu K (2023) Micron-bert: Bert-based facial micro-expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1482–1492. https://doi.org/10.48550/arXiv.2304.03195

  19. Yap MH, See J, Hong X, Wang S-J (2018) Facial micro-expressions grand challenge 2018 summary. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 675–678. https://doi.org/10.1109/FG.2018.00106. IEEE

  20. Xia B, Wang W, Wang S, Chen E (2020) Learning from macro-expression: A micro-expression recognition framework. In: Proceedings of the 28th ACM International Conference on Multimedia, pp 2936–2944. https://doi.org/10.1145/3394171.3413774

  21. Bao Y, Wu C, Zhang P, Shan C, Qi Y, Ben X (2024) Boosting micro-expression recognition via self-expression reconstruction and memory contrastive learning. IEEE Trans Affect Comput. https://doi.org/10.1016/j.patrec.2022.09.006

    Article  Google Scholar 

  22. Liong S-T, See J, Wong K, Phan RC-W (2018) Less is more: Micro-expression recognition from video using apex frame. Signal Processing: Image Communication 62:82–92. https://doi.org/10.1016/j.image.2017.11.006

    Article  Google Scholar 

  23. Li Y, Huang X, Zhao G (2018) Can micro-expression be recognized based on single apex frame? In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp 3094–3098. https://doi.org/10.1109/ICIP.2018.8451376. IEEE

  24. Li Y, Huang X, Zhao G (2020) Joint local and global information learning with single apex frame detection for micro-expression recognition. IEEE Trans Image Process 30:249–263. https://doi.org/10.1109/TIP.2020.3035042

    Article  MATH  Google Scholar 

  25. Sun M-X, Liong S-T, Liu K-H, Wu Q-Q (2022) The heterogeneous ensemble of deep forest and deep neural networks for micro-expressions recognition. Appl Intell 52(14):16621–16639. https://doi.org/10.1007/s10489-022-03284-y

    Article  Google Scholar 

  26. Zhou L, Mao Q, Xue L (2019) Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp 1–5. https://doi.org/10.1109/FG.2019.8756579. IEEE

  27. Chen B, Liu K-H, Xu Y, Wu Q-Q, Yao J-F (2022) Block division convolutional network with implicit deep features augmentation for micro-expression recognition. IEEE Trans Multimed 25:1345–1358. https://doi.org/10.1109/TMM.2022.3141616

    Article  MATH  Google Scholar 

  28. Peng M, Wang C, Bi T, Shi Y, Zhou X, Chen T (2019) A novel apex-time network for cross-dataset micro-expression recognition. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pp 1–6. https://doi.org/10.1109/ACII.2019.8925525. IEEE

  29. Liu J, Zong Y, Zheng W (2022) Cross-database micro-expression recognition based on transfer double sparse learning. Multimed Tools Appl 81(30):43513–43530. https://doi.org/10.1007/s11042-022-12878-0

    Article  MATH  Google Scholar 

  30. Zong Y, Zheng W, Huang X, Shi J, Cui Z, Zhao G (2018) Domain regeneration for cross-database micro-expression recognition. IEEE Trans Image Process 27(5):2484–2498. https://doi.org/10.1109/TIP.2018.2797479

    Article  MathSciNet  MATH  Google Scholar 

  31. Lin W-W, Mak M-W, Li L, Chien J-T (2018) Reducing domain mismatch by maximum mean discrepancy based autoencoders. In: Odyssey, pp 162–167. https://doi.org/10.21437/Odyssey.2018-23

  32. Liu T, Tao D, Song M, Maybank SJ (2016) Algorithm-dependent generalization bounds for multi-task learning. IEEE Trans Pattern Anal Mach Intell 39(2):227–241. https://doi.org/10.1109/TPAMI.2016.2544314

    Article  MATH  Google Scholar 

  33. Grefenstette E, Amos B, Yarats D, Htut PM, Molchanov A, Meier F, Kiela D, Cho K, Chintala S (2019) Generalized inner loop meta-learning. arXiv preprint arXiv:1910.01727. https://doi.org/10.48550/arXiv.1910.01727

  34. Finn C, Levine S (2017) Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. arXiv preprint arXiv:1710.11622. https://doi.org/10.48550/arXiv.1710.11622

  35. Hospedales T, Antoniou A, Micaelli P, Storkey A (2021) Meta-learning in neural networks: A survey. IEEE Trans Pattern Anal Mach Intell 44(9):5149–5169. https://doi.org/10.1109/TPAMI.2021.3079209

    Article  MATH  Google Scholar 

  36. Guo Y, Codella NC, Karlinsky L, Codella JV, Smith JR, Saenko K, Rosing T, Feris R (2020) A broader study of cross-domain few-shot learning. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16, pp 124–141. https://doi.org/10.1007/978-3-030-58583-9_8. Springer

  37. Hospedales T, Antoniou A, Micaelli P, Storkey A (2021) Meta-learning in neural networks: A survey. IEEE Trans Pattern Anal Mach Intell 44(9):5149–5169. https://doi.org/10.1109/TPAMI.2021.3079209

    Article  MATH  Google Scholar 

  38. Wan B, Dang J, Liu X, Wang Q (2022) Micro-expression recognition based on maml meta-learning algorithm. In: 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta), pp 1322–1328. https://doi.org/10.1109/SmartWorld-UIC-ATC-ScalCom-DigitalTwin-PriComp-Metaverse56740.2022.00191. IEEE

  39. Wang J, Tian Y, Yang Y, Chen X, Zheng C, Qiang W (2024) Meta-auxiliary learning for micro-expression recognition. arXiv preprint arXiv:2404.12024. https://doi.org/10.48550/arXiv.2404.12024

  40. Gong W, Zhang Y, Wang W, Cheng P, Gonzalez J (2023) Meta-mmfnet: Meta-learning-based multi-model fusion network for micro-expression recognition. ACM Trans Multimed Comput Commun Appl 20(2):1–20. https://doi.org/10.1145/3539576

    Article  Google Scholar 

  41. Wang Z, Fu W, Zhang Y, Li J, Gong W, Gonzàlez J (2024) Mcnet: meta-clustering learning network for micro-expression recognition. J Electron Imaging 33(2):023014–023014. https://doi.org/10.1117/1.JEI.33.2.023014

    Article  Google Scholar 

  42. Jia D, Wang K, Luo S, Liu T, Liu Y (2021) Braft: Recurrent all-pairs field transforms for optical flow based on correlation blocks. IEEE Signal Process Lett 28:1575–1579. https://doi.org/10.1109/LSP.2021.3099076

    Article  MATH  Google Scholar 

  43. Rebuffi S-A, Bilen H, Vedaldi A (2017) Learning multiple visual domains with residual adapters. Adv Neural Inf Process Syst 30. https://doi.org/10.5555/3294771.3294820

  44. Tseng H-Y, Lee H-Y, Huang J-B, Yang M-H (2020) Cross-domain few-shot classification via learned feature-wise transformation. arXiv preprint arXiv:2001.08735. https://doi.org/10.48550/arXiv.2001.08735

  45. Yan W, Wu Q, Liu Y, Wang S, Fu X (2013) Casme database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–7. https://doi.org/10.1109/FG.2013.6553799. IEEE

  46. Yan W, Li X, Wang S, Zhao G, Liu Y, Chen Y-H, Fu X (2014) Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1):86041. https://doi.org/10.1371/journal.pone.0086041

    Article  Google Scholar 

  47. Ben X, Ren Y, Zhang J, Wang S-J, Kpalma K, Meng W, Liu Y-J (2021) Video-based facial micro-expression analysis: A survey of datasets, features and algorithms. IEEE Trans Pattern Anal Mach Intell 44(9):5826–5846. https://doi.org/10.1109/TPAMI.2021.3067464

  48. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (fg), pp 1–6. https://doi.org/10.1109/FG.2013.6553717. IEEE

  49. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2016) Samm: A spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129. https://doi.org/10.1109/TAFFC.2016.2573832

    Article  Google Scholar 

  50. Yap MH, See J, Hong X, Wang S-J (2018) Facial micro-expressions grand challenge 2018 summary. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 675–678. https://doi.org/10.1109/FG.2018.00106. IEEE

  51. Maaten L, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(11)

  52. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. https://doi.org/10.1049/ic.2009.0244

  53. Khor H-Q, See J, Phan RCW, Lin W (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 667–674. https://doi.org/10.1109/FG.2018.00105. IEEE

  54. Talluri KK, Fiedler M-A, Al-Hamadi A (2022) Deep 3d convolutional neural network for facial micro-expression analysis from video images. Appl Sci 12(21):11078. https://doi.org/10.3390/app122111078

    Article  Google Scholar 

  55. Zhang M, Huan Z, Shang L (2020) Micro-expression recognition using micro-variation boosted heat areas. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp 531–543. https://doi.org/10.1007/978-3-030-60639-8_44. Springer

  56. Shu X, Li J, Shi L, Huang S (2023) Res-capsnet: an improved capsule network for micro-expression recognition. Multimed Syst 29(3):1593–1601. https://doi.org/10.1007/s00530-023-01068-z

    Article  MATH  Google Scholar 

  57. Peng M, Wu Z, Zhang Z, Chen T (2018) From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 657–661. https://doi.org/10.1109/FG.2018.00103. IEEE

  58. Wang C, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing 410:354–362. https://doi.org/10.1016/j.neucom.2020.06.005

    Article  MATH  Google Scholar 

  59. Zhang T, Zong Y, Zheng W, Chen CP, Hong X, Tang C, Cui Z, Zhao G (2022) Cross-database micro-expression recognition: A benchmark. IEEE Trans Knowl Data Eng 34(02):544–559. https://doi.org/10.1145/3323873.3326590

    Article  MATH  Google Scholar 

  60. Chen Z, Lu C, Zhou F, Zong Y (2023) Tkrm: Learning a transfer kernel regression model for cross-database micro-expression recognition. Mathematics 11(4):918. https://doi.org/10.3390/math11040918

    Article  MATH  Google Scholar 

  61. Lan G, Xiao S, Yang J, Wen J, Lu W, Gao X (2024) Active learning inspired method in generative models. Expert Syst Appl 249:123582. https://doi.org/10.1016/j.eswa.2024.123582

    Article  Google Scholar 

  62. Lan G, Xiao S, Yang J, Wen J, Xi M (2023) Generative ai-based data completeness augmentation algorithm for data-driven smart healthcare. IEEE J Biomed Health Inform. https://doi.org/10.1109/JBHI.2023.3327485

Download references

Acknowledgements

This research is partly supported by Project of Chongqing Science and Technology Bureau (CSTB2023TIAD-STX0037), Fujian Provincial Science and Technology Plan Project (2022T3016), and Fundamental Research Funds for the Central Universities SWU120083.

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the study conception and design. The first draft of the manuscript was written by Hanpu wang and all authors commented on previous versions of the manuscript. Professor Chen Tong refined and revised the manuscript.

Corresponding author

Correspondence to Tong Chen.

Ethics declarations

Competing Interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Ethical and informed consent for data used

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, H., Zhou, J., Liu, X. et al. A cross-database micro-expression recognition framework based on meta-learning. Appl Intell 55, 58 (2025). https://doi.org/10.1007/s10489-024-05896-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05896-y

Keywords