Skip to main content
SpringerLink
Log in

ULME-GAN: a generative adversarial network for micro-expression sequence generation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Currently, the lack of micro-expression datasets is a significant obstacle to micro-expression research and hinders the development of micro-expression supervised data generation. To address this issue, we propose the unsupervised learning micro-expression sequences generative adversarial network (ULME-GAN) approach, which generates micro-expression sequences that can be controlled. By analyzing all action units (AUs) that appear in main micro-expression datasets, a novel method called action unit matrix and re-encoding (AUMR) is proposed to generate micro-expression sequences that appear more natural and seamless by smoothing the AU matrix extracted from the source video. Our experiments demonstrate that the ULME-GAN approach can generate micro-expression videos/images that maintain the input source video/image pattern better than other methods, such as the first order motion model and StyleGAN. Furthermore, the micro-expression recognition task demonstrates that the augmented dataset can lead to a significant improvement in the performance of micro-expression recognition models. Finally, ULME-GAN can generate videos/images with specific micro-expression patterns defined by an input AU matrix, making it suitable for various applications even when there is insufficient source video.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ekman P (2009) Lie catching and microexpressions. The Philosophy of Deception 1(2):5. https://doi.org/10.1093/acprof:oso/9780195327939.003.0008

    Article  Google Scholar 

  2. Josephs L (2005) Emotions revealed: recognizing faces and feelings to improve communication and emotional life, by Paul Ekman. Henry Holt and com. Am J Psychoanal 65(4):409–411. https://doi.org/10.1007/s11231-005-7891-8

  3. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106. https://doi.org/10.1080/00332747.1969.11023575

    Article  Google Scholar 

  4. Wang Y, Guobule N, Li M, Li J (2021) The correlation of facial emotion recognition in patients with drug-naïve depression and suicide ideation. J Affect Disord 295:250–254. https://doi.org/10.1016/j.jad.2021.08.051

    Article  Google Scholar 

  5. Nam B, Kim JY, Bark B, Kim Y, Kim J, So SW, Choi HY, Kim IY (2023) Facialcuenet: unmasking deception-an interpretable model for criminal interrogation using facial expressions. Appl Intell 1–15. https://doi.org/10.1007/s10489-023-04968-9

  6. Jordan S, Brimbal L, Wallace DB, Kassin SM, Hartwig M, Street CN (2019) A test of the micro-expressions training tool: does it improve lie detection? J Investigative Psycho Offender Profiling 16(3):222–235. https://doi.org/10.1002/jip.1532

    Article  Google Scholar 

  7. Picard RW (2000) Affective Computing. MIT press, Cambridge Massachusetts

    Book  Google Scholar 

  8. Yan WJ, Wu Q, Liang J, Chen YH, Fu X (2013) How fast are the leaked facial expressions: the duration of micro expressions. J Nonverbal Behav 37:217–230. https://doi.org/10.1007/s10919-013-0159-8

    Article  Google Scholar 

  9. Yan WJ, Wu Q, Liu YJ, Wang SJ, Fu X (2013) CASME database: a dataset of spontaneous micro–expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–7. https://doi.org/10.1109/FG.2013.6553799

  10. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1):86041. https://doi.org/10.1371/journal.pone.0086041

    Article  Google Scholar 

  11. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (fg), pp 1–6 (2013). https://doi.org/10.1109/FG.2013.6553717

  12. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2016) Samm: a spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129. https://doi.org/10.1109/TAFFC.2016.2573832

    Article  Google Scholar 

  13. Li J, Dong Z, Lu S, Wang SJ, Yan WJ, Ma Y, Liu Y, Huang C, Fu X (2022) CAS (ME) 3: a third generation facial spontaneous micro expression database with depth information and high ecological validity. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2022.3174895

    Article  Google Scholar 

  14. Lin WP, Ge QC, Liong ST, Liu JT, Liu KH, Wu QQ (2023) The design of error-correcting output codes based deep forest for the micro-expression recognition. Appl Intell 53(3):3488–3504. https://doi.org/10.1007/s10489-022-03590-5

    Article  Google Scholar 

  15. Gupta P (2022) Persist: improving micro-expression spotting using better feature encodings and multi-scale gaussian tcn. Appl Intell 1–15. https://doi.org/10.1007/s10489-022-03553-w

  16. Rabiner LR (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286. https://doi.org/10.1109/5.18626

    Article  Google Scholar 

  17. Jiang L, Zhang H, Cai Z (2009) A novel bayes model: hidden naive bayes. IEEE Trans Knowl Data Eng 21(10):1361–1371. https://doi.org/10.1109/TKDE.2008.234

    Article  Google Scholar 

  18. Peng M, Wu Z, Zhang Z, Chen T (2018) From macro to micro expression recognition: deep learning on small datasets using transfer learning. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp 657–661. https://doi.org/10.1109/FG.2018.00103

  19. Verma M, Vipparthi SK, Singh G, Murala S (2019) LEARNet: dynamic imaging network for micro expression recognition. IEEE Trans Image Process 29:1618–1627. https://doi.org/10.1109/TIP.2019.2912358

    Article  MathSciNet  Google Scholar 

  20. Wang C, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing 410:354–362. https://doi.org/10.1016/j.neucom.2020.06.005

    Article  Google Scholar 

  21. Zhou L, Shao X, Mao Q (2021) A survey of micro-expression recognition. Image Vis Comput 105:104043. https://doi.org/10.1016/j.imavis.2020.104043

    Article  Google Scholar 

  22. Peng M, Wang C, Chen T, Liu G, Fu X (2017) Dual temporal scale convolutional neural network for micro-expression recognition. Front Psychol 8:1745–1750. https://doi.org/10.3389/fpsyg.2017.017

    Article  Google Scholar 

  23. Creswell A, White T, Dumoulin V, Arulkumaran K, Sengupta B, Bharath AA (2018) Generative adversarial networks: an overview. IEEE Signal Process Mag 35(1):53–65. https://doi.org/10.1145/3422622

    Article  Google Scholar 

  24. Yang L, Zhang Z, Song Y, Hong S, Xu R, Zhao Y, Zhang W, Cui B, Yang MH (2022) Diffusion models: a comprehensive survey of methods and applications. ACM Comput Surv. https://doi.org/10.1145/3626235

    Article  Google Scholar 

  25. Lugmayr A, Danelljan M, Romero A, Yu F, Timofte R, Van Gool L (2022) Repaint: inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11461–11471. https://doi.org/10.1109/CVPR52688.2022.01117

  26. Choi Y, Choi M, Kim M, Ha JW, Kim S, Choo J (2018) Stargan: unified generative adversarial networks for multi-domain image to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8789–8797. https://doi.org/10.1109/CVPR.2018.00916

  27. Pumarola A, Agudo A, Martinez AM, Sanfeliu A, Moreno-Noguer F (2018) Ganimation: anatomically-aware facial animation from a single image. In: Proceedings of the European conference on computer vision (ECCV), pp 818–833. https://doi.org/10.1007/978-3-030-01249-6_50

  28. Yu J, Zhang C, Song Y, Cai W (2021) ICE-GAN: identity aware and capsule-enhanced gan with graph-based reasoning for micro-expression recognition and synthesis. In: 2021 international joint conference on neural networks (IJCNN), pp 1–8.https://doi.org/10.1109/IJCNN52387.2021.9533988

  29. Liong ST, Gan YS, Zheng D, Li SM, Xu HX, Zhang HZ, Lyu RK, Liu KH (2020) Evaluation of the spatio-temporal features and gan for micro-expression recognition system. J Signal Process Syst 92:705–725. https://doi.org/10.1007/s11

    Article  Google Scholar 

  30. Fan X, Shahid AR, Yan H (2022) Adaptive dual motion model for facial micro-expression generation. In: Proceedings of the 30th ACM international conference on multimedia, pp 7125–7129. https://doi.org/10.1145/3503161.3551592

  31. Kammoun A, Slama R, Tabia H, Ouni T, Abid M (2022) Generative adversarial networks for face generation: a survey. ACM Comput Surv 55(5):1–37. https://doi.org/10.1145/3527850

    Article  Google Scholar 

  32. Yadav NK, Singh SK, Dubey SR (2022) Csa-gan: cyclic synthesized attention guided generative adversarial network for face synthesis. Appl Intell 52(11):12704–12723. https://doi.org/10.1007/s10489-021-03064-0

    Article  Google Scholar 

  33. Baltrušaitis T, Mahmoud M, Robinson P (2015) Cross-dataset learning and person-specific normalisation for automatic action unit detection. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), vol 6, pp 1–6. https://doi.org/10.1109/FG.2015.7284869

  34. Qu F, Wang SJ, Yan WJ, Li H, Wu S, Fu X (2017) CAS (ME) 2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans Affect Comput 9(4):424–436. https://doi.org/10.1109/TAFFC.2017.2654440

    Article  Google Scholar 

  35. Ben X, Ren Y, Zhang J, Wang SJ, Kpalma K, Meng W, Liu YJ (2021) Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans Pattern Anal Mach Intell 44(9):5826–5846. https://doi.org/10.1109/TPAMI.2021.3067464

  36. Ekman P, Friesen WV (2002) Facial action coding system. Environ Psycho Nonverbal Behav 1(1):1–1. https://doi.org/10.1037/t27734-000

  37. Baltrusaitis T, Robinson P, Morency LP (2013) Constrained local neural fields for robust facial landmark detection in the wild. In: Proceedings of the IEEE international conference on computer vision workshops, pp 354–361. https://doi.org/10.1109/ICCVW.2013.54

  38. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC (2017) Improved training of wasserstein gans. In: Advances in neural information processing systems, vol 30, p 1

  39. Fabian Benitez-Quiroz C, Srinivasan R, Martinez AM (2016) Emotionet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5562–5570. https://doi.org/10.1109/CVPR.2016.600

  40. Siarohin A, Lathuilière S, Tulyakov S, Ricci E, Sebe N (2019) First order motion model for image animation. Adv Neural Inf Process Syst 32

  41. Karras T, Laine S, Aittala M, Hellsten J, Lehtinen J, Aila T (2020) Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8110–8119. https://doi.org/10.1109/CVPR42600.2020.00813

  42. Georgescu MI, Ionescu RT, Popescu M (2019) Local learning with deep and handcrafted features for facial expression recognition. IEEE Access 7:64827–64836. https://doi.org/10.1109/ACCESS.2019.2917266

    Article  Google Scholar 

  43. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  44. Tan M, Le Q (2021) Efficientnetv2: smaller models and faster training. In: Proceedings of international conference on machine learning, pp 10096–10106

Download references

Acknowledgements

We appreciate the contribution of the numerous participants and developers of the micro-expression database, who made the database available and public to the research community

Author information

Author notes

  1. Sirui Sun, Haolin Xia, Xinyu Liu and Hanpu Wang are equally contributed to this work.

Authors and Affiliations

  1. College of Electronic and Information Engineering, Southwest University, Chongqing, 400715, China

    Ju Zhou, Sirui Sun, Haolin Xia, Xinyu Liu, Hanpu Wang & Tong Chen

  2. Chongqing Key Laboratory of Non-linear Circuit and Intelligent Information Processing, Southwest University, Chongqing, 400715, China

    Ju Zhou, Sirui Sun, Haolin Xia, Xinyu Liu, Hanpu Wang & Tong Chen

  3. Institute of Legal Psychology and Intelligent Computing, Southwest University, Chongqing, 400715, China

    Ju Zhou, Sirui Sun, Haolin Xia, Xinyu Liu, Hanpu Wang & Tong Chen

Authors
  1. Ju Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sirui Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haolin Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hanpu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tong Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, J., Sun, S., Xia, H. et al. ULME-GAN: a generative adversarial network for micro-expression sequence generation. Appl Intell 54, 490–502 (2024). https://doi.org/10.1007/s10489-023-05213-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05213-z

Keywords

Search

Navigation