Skip to main content
SpringerLink
Log in

The design of error-correcting output codes based deep forest for the micro-expression recognition

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

A micro-expression (ME) is hard to fake and reflects people’s true feelings, micro-expression recognition (MER) is useful in a wide range of applications, such as medical research, criminal detection, and national security. Nevertheless, the existing deep learning methods based on deep neural networks (DNNs) fail to handle it, owing to the limited amount of training data and the black-box structure. In order to cope with these issues, this paper proposes a novel deep forest (DF) based on error-correcting output codes (ECOC), named EDF. The ECOC is innovatively deployed to encourage diversity and help to summarize differences among classes from multiple perspectives. Compared to DNN-based models, EDF is a more general solution for the MER since it is not sensitive to the size of data sets and needn’t other approaches such as transfer learning. On the other hand, EDF copies with the interpretability-accuracy trade-off to a certain extent. The experimental results on several data sets confirm that EDF outperforms most state-of-the-art methods. Furthermore, EDF is an interpretable method. It can reconstruct feature values by tracing and fusing the decision paths, from which we can easily determine important features. According to the results of the back-reconstruction process, the difference among classes can be summarized.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Ben X, Jia X, Yan R, Zhang X, Meng W (2018) Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recogn Lett 107:50–58

  2. Zong Y, Zheng W, Cui Z, Zhao G, Hu B (2020) Toward bridging microexpressions from different domains. IEEE Trans Cybernetics 50(12):5047–5060

  3. Rolls E, Ekman P, Perrett D, Ellis H (1992) Facial expressions of emotion: an old controversy and new findings: discussion. Philosophical Trans Royal Soc London Series B 335:69–69

  4. Wu Q, Shen X, Fu X (2011) The machine knows what you are hiding: an automatic micro-expression recognition system. In international conference on affective computing and intelligent interaction. 2011. Springer

  5. Hurley CM, Anker AE, Frank MG, Matsumoto D, Hwang HC (2014) Background factors predicting accuracy and improvement in micro expression recognition. Motiv Emot 38(5):700–714

  6. Zhang P, Ben X, Yan R, Wu C, Guo C (2016) Micro-expression recognition system. Optik-Int J Light Electron Optics 127(3):1395–1400

    Article  Google Scholar 

  7. Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao GY, Fu XL (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4): 299–310

  8. Zhao Y, Xu J (2019) An improved micro-expression recognition method based on necessary morphological patches. Symmetry 11(4):497

    Article  MathSciNet  Google Scholar 

  9. Peng M, Wang C, Chen T, Liu G, Fu X (2017) Dual temporal scale convolutional neural network for micro-expression recognition. Front Psychol 8:1745

    Article  Google Scholar 

  10. Verburg M, Menkovski V (2019) Micro-expression detection in long videos using optical flow and recurrent neural networks. In 2019 14th IEEE International conference on automatic face & gesture recognition (FG 2019)

  11. Min P, Zhan W, Zhang Z, Tong C (2018) From macro to micro expression recognition: deep learning on small datasets using transfer learning. In IEEE international conference on automatic face & gesture recognition

  12. Khor HQ, See J, Phan R CW, Lin WY (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition. In proceedings 2018 13th IEEE international conference on automatic face & gesture recognition. IEEE 667–674

  13. Wang C, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing 410:354–362

  14. Gan YS, Liong ST, Yau WC, Huang YC, Tan LK (2019) OFF-ApexNet on micro-expression recognition system. Signal Process-Image Commun 74:129–139

    Article  Google Scholar 

  15. Wang XD, Zheng ZD, He Y, Yan F, Zeng ZQ, Yang Y (2021) Soft person reidentification network pruning via blockwise adjacent filter decaying. IEEE Trans Cybernetics

  16. Liong S T, Gan Y S, See J, Khor H Q (2019) A shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition system. In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019)

  17. Verma M, Vipparthi SK, Singh G, Murala S (2020) LEARNet: dynamic imaging network for micro expression recognition. IEEE Trans Image Process 29:1618–1627

    Article  MathSciNet  MATH  Google Scholar 

  18. Zhou ZH, Feng J (2019) Deep forest. Natl Sci Rev 6(1):74–86

    Article  MathSciNet  Google Scholar 

  19. Dietterich TG, Bakiri G (1994) Solving multiclass learning problems via error-correcting output codes. J Artif Intell Res 2:263–286

    Article  MATH  Google Scholar 

  20. Sun M, Liong S, Liu K, Wu Q (2022) The heterogeneous ensemble of deep forest and deep neural networks for micro-expressions recognition. Appl Intell

  21. Chen B, Liu KH, Xu Y, Wu QQ, Yao JF (2022) Block division convolutional network with implicit deep features augmentation for micro-expression recognition. IEEE Trans Mult

  22. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  23. Brunet N, Perez F, and Torre FD (2009) Learning Good Features for Active Shape Models, in 2009 IEEE 12th International Conference on Computer Vision Workshops. 206–211

  24. Wathan J, Burrows AM, Waller BM, McComb K (2015) EquiFACS: the equine facial action coding system. PLoS One 10(8):e0131738

    Article  Google Scholar 

  25. Ekman P and Friesen WV (1978) Facial action coding system: A technique for the measurement of facial movement. a technique for the measurement of facial action

  26. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59

    Article  Google Scholar 

  27. Pfister T, Li XB, Zhao GY, Pietikainen M (2011) Recognising spontaneous facial micro-expressions. In 2011 international conference on computer vision. 1449–1456

  28. Wang Y, See J, Phan R C W, Oh Y H (2014) Lbp with six intersection points: reducing redundant information in lbp-top for micro-expression recognition. In Asian conference on computer vision.

  29. Huang X, Wang S, Liu X, Zhao G, Feng X, Pietikainen M (2017) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 10(1):32–47

  30. Polikovsky S, Kameda Y, and Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In 3rd international conference on imaging for crime detection and prevention (ICDP 2009)

  31. Liu YJ, Li BJ, Lai YK (2021) Sparse MDMO: learning a discriminative feature for micro-expression recognition. IEEE Trans Affect Comput 12(1):254–261

    Google Scholar 

  32. Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  33. Zong Y, Huang X, Zheng W, Cui Z, Zhao G (2018) Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Trans Multimedia 20(11):3160–3172

    Article  Google Scholar 

  34. Ding J, Tian Z, Lyu X, Wang Q, Zou B, Xie H (2019) Real-time micro-expression detection in unlabeled long videos using optical flow and LSTM neural network. In international conference on computer analysis of images and patterns. 2019. Springer

  35. Liong S-T, See J, Wong KS, Phan RCW (2018) Less is more: micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92

    Article  Google Scholar 

  36. Liong ST, See J, Phan RCW, Le Ngo AC, Oh YH, Wong K(2014) Subtle expression recognition using optical strain weighted features. Computer vision - ACCV 2014 workshops. PT II 9009:644–657

  37. Happy S, Routray A (2017) Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans Affect Comput 10(3):394–406

    Article  Google Scholar 

  38. Xia Z, Hong X, Gao X, Feng X, Zhao G (2020) Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans Multimedia 22(3):626–640

    Article  Google Scholar 

  39. Wang SJ, Li BJ, Liu YJ, Yan WJ, Ou X, Huang X, Xu F, Fu X (2018) Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing 312:251–262

    Article  Google Scholar 

  40. Xia ZQ, Peng W, Khor HQ, Feng XY, Zhao GY (2020) Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Trans Image Process 29:8590–8605

  41. Quang NV, Chun J, Tokuyama T (2019) CapsuleNet for micro-expression recognition. In 14th IEEE international conference on automatic face and gesture recognition (FG 2019). Lille, FRANCE: IEEE 635–641

  42. Li Y, Huang X, Zhao G (2021) Joint local and global information learning with single apex frame detection for micro-expression recognition. IEEE Trans Image Process 30:249–263

    Article  Google Scholar 

  43. Utkin LV (2020) An imprecise deep forest for classification. Expert Syst Appl 141:112978

    Article  Google Scholar 

  44. Ma C, Liu Z, Cao Z, Song W, Zhang J, Zeng W (2020) Cost-sensitive deep forest for price prediction. Pattern Recogn 107:107499

    Article  Google Scholar 

  45. Wang HY, Tang Y, Jia ZY, Ye F (2020) Dense adaptive cascade forest: a self-adaptive deep ensemble for classification problems. Soft Comput 24(4):2955–2968

  46. Yang L, Wu XZ, Jiang Y, Zhou ZH (2019) Multi-Label Learning with Deep Forest. In 24th European conference on artificial intelligence. 325:1634–1641

  47. Gao J, Lin W, Liu K, Hong Q, Lin G, Wang B (2020) Imprecise deep Forest for partial label learning. IEEE Access 8:218530–218541

    Article  Google Scholar 

  48. Zhou M, Zeng XH, Chen AZ (2019) Deep forest hashing for image retrieval. Pattern Recogn 95:114–127

    Article  Google Scholar 

  49. Liu XO, Wang RL, Cai ZH, Cai YM, Yin X (2019) Deep multigrained Cascade Forest for hyperspectral image classification. IEEE Trans Geosci Remote Sens 57(10):8169–8183

  50. Su R, Liu X, Wei L, Zou Q (2019) Deep-Resp-Forest: a deep forest model to predict anti-cancer drug response. Methods 166:91–102

    Article  Google Scholar 

  51. Wang W, Dai QY, Li F, Xiong Y, Wei DQ (2020) MLCDForest: multi-label classification with deep forest in disease prediction for long non-coding RNAs. Brief Bioinform 22

  52. Liu KH, Ye XN, Guo HZ, Wu QQ, Hong QQ (2021) The design of soft recoding-based strategies for improving error-correcting output codes. Appl Intell

  53. Nazari S, Moin MS, Kanan HR (2018) Securing templates in a face recognition system using error-correcting output code and chaos theory. Comput Electrical Eng 72:644–659

    Article  Google Scholar 

  54. Liu KH, Zhan WP, Liang YF, Zhang YN, Guo HZ, Yao JF,Wu QQ, Hong QQ (2021) The design of error-correcting output codes algorithm for the open-set recognition. Appl Intell

  55. Yu H and Liu K (2017) Classification of multi-class microarray datasets using a minimizing class-overlapping based ECOC algorithm. In proceedings of the 5th international conference on bioinformatics and computational biology

  56. Zhang ML, Fei Y, Tang CZ (2017) Disambiguation-free partial label learning. IEEE Trans Knowledge Data Eng 29(10):2155–2167

  57. Feng J and Zhou ZH (2017) AutoEncoder by Forest. in AAAI-18

  58. Yan W-J, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):e86041

    Article  Google Scholar 

  59. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: inducement, collection and baseline. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (fg). IEEE

  60. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2016) Samm: a spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129

    Article  Google Scholar 

  61. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Patt Analy Mach Intell 6:915–928

    Article  Google Scholar 

  62. Zhou L, Mao Q, and Xue L (2019) Dual-Inception Network for Cross-Database Micro-Expression Recognition. in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)

  63. Krizhevsky A, Sutskever I, and Hinton GE (2012) Imagenet classification with deep convolutional neural networks. in Advances in neural information processing systems

  64. Iandola F N, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. arXiv. 1602.07360

  65. Simonyan K and Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  66. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. (2015) Going deeper with convolutions. In proceedings of the IEEE conference on computer vision and pattern recognition

  67. Demiar J, Schuurmans D (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7(1):1–30

    MathSciNet  Google Scholar 

  68. Friedman M (1939) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Publ Am Stat Assoc 32(200):675–701

    Article  MATH  Google Scholar 

  69. Friedman M (1940) A comparison of alternative tests of significance for the problem of $m$ rankings. Ann Math Stat 11(1):86–92

    Article  MathSciNet  MATH  Google Scholar 

  70. Nemenyi P (1963) Distribution-Eree multiple comparison. Biometrics

  71. Díez-Pastor JF, Rodríguez JJ, García-Osorio CI, Kuncheva LI (2015) Diversity techniques improve the performance of the best imbalance learning ensembles. Inf Sci 325:98–117

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 61772023), National Key Research and Development Program of China (No. 2019QY1803) and Ministry of Science and Technology, Taiwan (MOST108-2221-E-035-066-, MOST108-2218-E-009-054-MY2, MOST 108-2218-E-035 -007-).

Author information

Authors and Affiliations

  1. School of Informatics, Xiamen University, Xiamen, China

    Wei-Ping Lin, Qi-Chao Ge, Jia-Tong Liu, Kun-Hong Liu & Qing-Qiang Wu

  2. Department of Electronic Engineering, Feng Chia University, Taichung, Taiwan

    Sze-Teng Liong

Authors
  1. Wei-Ping Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi-Chao Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sze-Teng Liong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jia-Tong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kun-Hong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qing-Qiang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Kun-Hong Liu or Qing-Qiang Wu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, WP., Ge, QC., Liong, ST. et al. The design of error-correcting output codes based deep forest for the micro-expression recognition. Appl Intell 53, 3488–3504 (2023). https://doi.org/10.1007/s10489-022-03590-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03590-5

Keywords

Search

Navigation