Skip to main content
SpringerLink
Log in

Cross-database micro-expression recognition based on transfer double sparse learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, the cross-database micro-expression problem has become a research hotspot in the affective computing and multimedia areas due to its vital role in analyzing human behavior and potential valuable application such as criminal investigation, lie detection and education which are closely associated with multimedia. Unlike common micro-expression recognition problem, cross-database micro-expression conducts micro-expression recognition using a database as training set (source database) while another database as testing set (target database), which is more challenging than common micro-expression recognition issue since it has a serious inconsistency of feature distribution between source database and target database. To handle the crucial cross-database micro-expression issue, a novel transfer double sparse learning method is proposed in this paper. The advantage of the proposed transfer double sparse learning model is that it can select the features and facial regions which have contributions to the cross-database micro-expression problem efficiently while further refining their corresponding features according to the importance of these features in cross-database micro-expression. Extensive experiments on three widely used micro-expression databases show that the proposed transfer double sparse learning model gets the best performance than other state-of-the-art methods. Specially, transfer double sparse learning model achieves which proves that the proposed it can cope with the cross-database micro-expression problem efficiently since it successfully refines the facial features and bridged the emotion gaps between different domains.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Bastanfard A, Takahashi H, Nakajima M (2004) Toward e-appearance of human face and hair by age, expression and rejuvenation. In: 2004 International conference on cyberworlds. IEEE, pp 306–311

  2. Chang CC, Lin CJ (2011) Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2(3):1–27

    Article  Google Scholar 

  3. Chu WS, De la Torre F, Cohn JF (2013) Selective transfer machine for personalized facial action unit detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3515–3522

  4. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106

    Article  Google Scholar 

  5. Fernando B, Habrard A, Sebban M, Tuytelaars T (2013) Unsupervised visual domain adaptation using subspace alignment. In: Proceedings of the IEEE International conference on computer vision, pp. 2960–2967

  6. Frank M, Herbasz M, Sinuk K, Keller A, Nolan C (2009) I see how you feel: Training laypeople and professionals to recognize fleeting emotions. In: The annual meeting of the International communication association. Sheraton new york, new york city

  7. Frank MG, Maccario CJ, Govindaraju V (2009) Behavior and security protecting airline passengers in the age of terrorism

  8. Ganin Y, Lempitsky V (2014) Unsupervised domain adaptation by backpropagation. arXiv:1409.7495

  9. Gers FA, Schraudolph NN, Schmidhuber J (2002) Learning precise timing with lstm recurrent networks. J Mach Learn Res 3(Aug):115–143

    MathSciNet  MATH  Google Scholar 

  10. Gong B, Shi Y, Sha F, Grauman K (2012) Geodesic flow kernel for unsupervised domain adaptation. In: 2012 IEEE Conference on computer vision and pattern recognition. IEEE, pp 2066–2073

  11. Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy. Springer, pp 154–165

  12. Hassan A, Damper R, Niranjan M (2013) On acoustic emotion recognition: compensating for covariate shift. IEEE Transactions on Audio Speech, and Language Processing 21(7):1458–1468

    Article  Google Scholar 

  13. Hong X, Xu Y, Zhao G (2016) Lbp-top: a tensor unfolding revisit. In: Asian conference on computer vision. Springer, pp 513–527

  14. Kim DH, Baddar WJ, Ro YM (2016) Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM International conference on multimedia, pp. 382–386

  15. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105

  16. Le Ngo AC, Oh YH, Phan RCW, See J (2016) Eulerian emotion magnification for subtle expression recognition. In: 2016 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 1243–1247

  17. Le Ngo AC, Phan RCW, See J (2014) Spontaneous subtle expression recognition: Imbalanced databases and solutions. In: Asian conference on computer vision. Springer, pp 33–48

  18. Le Ngo AC, See J, Phan RCW (2016) Sparsity in dynamics of spontaneous subtle emotions: analysis and application. IEEE Trans Affect Comput 8 (3):396–411

    Article  Google Scholar 

  19. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10Th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–6

  20. Lin Z, Chen M, Ma Y (2010) The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv:1009.5055

  21. Liu J, Ji S, Ye J, et al. (2009) Slep: Sparse learning with efficient projections. Arizona State University 6(491):7

    Google Scholar 

  22. Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  23. Long M, Cao Y, Wang J, Jordan MI (2015) Learning transferable features with deep adaptation networks. arXiv:1502.02791

  24. Long M, Wang J, Sun J, Philip SY (2014) Domain invariant transfer kernel learning. IEEE Trans Knowl Data Eng 27(6):1519–1532

    Article  Google Scholar 

  25. Lu P, Zheng W, Wang Z, Li Q, Zong Y, Xin M, Wu L (2016) Micro-expression recognition by regression model and group sparse spatio-temporal feature learning. IEICE Trans Inf Syst 99(6):1694–1697

    Article  Google Scholar 

  26. Modhej N, Bastanfard A, Teshnehlab M, Raiesdana S (2020) Pattern separation network based on the hippocampus activity for handwritten recognition. IEEE Access 8:212803–212817

    Article  Google Scholar 

  27. Oh YH, Le Ngo AC, See J, Liong ST, Phan RCW, Ling HC (2015) Monogenic riesz wavelet representation for micro-expression recognition. In: 2015 IEEE International conference on digital signal processing (DSP). IEEE, pp 1237–1241

  28. O’Sullivan M, Frank MG, Hurley CM, Tiwana J (2009) Police lie detection accuracy: The effect of lie scenario. Law Hum Behav 33(6):530

    Article  Google Scholar 

  29. Pan SJ, Tsang IW, Kwok JT, Yang Q (2010) Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks 22 (2):199–210

    Article  Google Scholar 

  30. Patel D, Hong X, Zhao G (2016) Selective deep features for micro-expression recognition. In: 2016 23Rd international conference on pattern recognition (ICPR). IEEE, pp 2258–2263

  31. Pfister T, Li X, Zhao G, Pietikäinen M. (2011) Recognising spontaneous facial micro-expressions. In: 2011 International conference on computer vision. IEEE, pp 1449–1456

  32. Stickel C, Ebner M, Steinbach-Nordmann S, Searle G, Holzinger A (2009) Emotion detection: application of the valence arousal space for rapid biological usability testing to enhance universal access. In: International conference on universal access in human-computer interaction. Springer, pp 615–624

  33. Wang SJ, Yan WJ, Li X, Zhao G, Zhou CG, Fu X, Yang M, Tao J (2015) Micro-expression recognition using color spaces. IEEE Trans Image Process 24(12):6034–6047

    Article  MathSciNet  MATH  Google Scholar 

  34. Wang Y, See J, Phan RCW, Oh YH (2014) Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision. Springer, pp 525–537

  35. Xia Z, Liang H, Hong X, Feng X (2019) Cross-database micro-expression recognition with deep convolutional networks. In: Proceedings of the 2019 3rd International Conference on Biometric Engineering and Applications, pp. 56–60

  36. Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  37. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) Casme ii: An improved spontaneous micro-expression database and the baseline evaluation PloS one 9(1)

  38. Zhang J, Han Y, Tang J, Hu Q, Jiang J (2016) Semi-supervised image-to-video adaptation for video action recognition. IEEE Transactions on Cybernetics 47(4):960–973

    Article  Google Scholar 

  39. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE transactions on pattern analysis and machine intelligence 29(6):915–928

    Article  Google Scholar 

  40. Zheng W (2014) Multi-view facial expression recognition based on group sparse reduced-rank regression. IEEE Trans Affect Comput 5(1):71–85

    Article  Google Scholar 

  41. Zheng W, Xin M, Wang X, Wang B (2014) A novel speech emotion recognition method via incomplete sparse least square regression. IEEE Sig Process Lett 21(5):569–572

    Article  Google Scholar 

  42. Zhou L, Mao Q, Xue L (2019) Cross-database micro-expression recognition: a style aggregated and attention transfer approach. In: 2019 IEEE International conference on multimedia & expo workshops (ICMEW). IEEE, pp 102–107

  43. Zhou L, Mao Q, Xue L (2019) Dual-inception network for cross-database micro-expression recognition. In: 2019 14Th IEEE international conference on automatic face & gesture recognition (FG 2019). IEEE, pp 1–5

  44. Zhou Z, Zhao G, Pietikäinen M (2011) Towards a practical lipreading system. In: CVPR 2011, pp. 137–144. IEEE

  45. Zong Y, Huang X, Zheng W, Cui Z, Zhao G (2017) Learning a target sample re-generator for cross-database micro-expression recognition. In: Proceedings of the 25th ACM international conference on Multimedia, pp. 872–880

  46. Zong Y, Zheng W, Huang X, Shi J, Cui Z, Zhao G (2018) Domain regeneration for cross-database micro-expression recognition. IEEE Trans Image Process 27(5):2484–2498

    Article  MathSciNet  MATH  Google Scholar 

Download references

Funding

This work was supported in part by the NSFC under grants U2003207, 61902064, and 62076195, in part by the Jiangsu Frontier Technology Basic Research Project under the Grant BK20192004, and in part by the Zhishan Young Scholarship of Southeast University.

Author information

Author notes

    Authors and Affiliations

    1. Key Laboratory of Child Development and Learning Science of Ministry of Education, School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, 210096, China

      Jiateng Liu, Yuan Zong & Wenming Zheng

    Authors
    1. Jiateng Liu
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Yuan Zong
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Wenming Zheng
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Wenming Zheng.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Jiateng Liu and Yuan Zong are contributted equally to this work.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Liu, J., Zong, Y. & Zheng, W. Cross-database micro-expression recognition based on transfer double sparse learning. Multimed Tools Appl 81, 43513–43530 (2022). https://doi.org/10.1007/s11042-022-12878-0

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-022-12878-0

    Keywords

    Search

    Navigation