Skip to main content
SpringerLink
Log in

The heterogeneous ensemble of deep forest and deep neural networks for micro-expressions recognition

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Micro-Expressions (MEs) are a kind of short-lived and uncontrollable facial expressions. Therefore, the MEs recognition task poses a great challenge to both the psychological and computer vision research communities. In this study, a new ensemble algorithm is proposed by fusing two different deep learning frameworks: Deep Forest (DF) and Convolutional Neural Networks (CNN) (DFN for short). A modified DF structure is deployed to extract features through the multi-grained scanning technique, along with three different sliding windows to gain diverse motion features. In addition, two shallow CNNs are deployed to extract the features from the three-dimensional optical flow vector and the apex frame. In this way, the fusion of DF and CNNs forms DFN to extract the static and dynamic features for MEs, to generate diverse features with high-level abstraction. Consequently, this heterogeneous ensemble deploys the high diversity in these two models to promote the overall discriminative ability. Comprehensive experiments have confirmed the robustness and effectiveness of the proposed DFN with relatively less computational consumption. Related theoretical analysis has been given to further provide evidence and insights into the proposed method. Our source code is publicly available for non-commercial or research use at https://github.com/MLDMXM2017/DFN_ME.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. H. Wang, "Facial expression decomposition," in Proceedings ninth IEEE international conference on computer vision, 2003, pp. 958–965: IEEE.

  2. Hughes MA (May 2003) “Emotions revealed: Recognizing faces and feelings to improve communication and emotional life,” (in English). Library Journal, Book Review 128(8):140–140

    Google Scholar 

  3. Weiss J and D. Ph., Ekman, P. (2009) Telling Lies : Clues to Deceit in the Marketplace, Politics, and Marriage. New York: Norton. American Journal of Clinical Hypnosis, 2011.

  4. Hurley CM, Anker AE, Frank MG, Matsumoto D, Hwang HCJM, emotion (2014) Background factors predicting accuracy and improvement in micro expression recognition. Motivation and Emotion 38(5):700–714

    Article  Google Scholar 

  5. Wu Q, Shen X, Fu X (2011) The machine knows what you are hiding: an automatic micro-expression recognition system, in international conference on affective computing and intelligent Interaction, pp. 152–162: Springer

  6. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline," in 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6: IEEE

  7. Yan W-J et al (2014) CASME II: An improved spontaneous micro-expression database and the baseline evaluation, PloS one, vol. 9, no. 1,

  8. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2016) SAMM: A Spontaneous Micro-Facial Movement Dataset. IEEE Transactions on Affective Computing, no 99:1–1

    Google Scholar 

  9. Wang Y et al (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimedia Tools and Applications 76(20):21665–21690

    Article  Google Scholar 

  10. Lu Z., Luo Z, Zheng H, Chen J, Li W (2014) A Delaunay-based temporal coding model for micro-expression recognition, in Asian conference on computer vision, pp. 698–711: Springer

  11. Takalkar MA, Xu M (2017) Image based facial micro-expression recognition using deep learning on small datasets, in 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–7: IEEE.

  12. Y. H. a. X. He, "Facial Expression Recognition Based on Multi-Feature Fusion and HOSVD," 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), pp. 638–643, 2019

  13. Zhou Z-H, Feng J (2017) Deep forest: Towards an alternative to deep neural networks, International Joint Conference on Artificial Intelligence (IJCAI), 2017

  14. Marreddy M, Oota SR, Agarwal R, Mamidi R (2019) Evaluating the Combination of Word Embeddings with Mixture of Experts and Cascading gcForest in Identifying Sentiment Polarity

  15. Wang W, Wei D-Q (2020) MLCDForest: Multi-labels Classification with Deep Forest in Disease Prediction for Long Non-coding RNAs, Briefings in Bioinformatics, 2020

  16. Chen ZH, Li LP, He Z, Zhou JR, Li YM, Wong L (2019) An Improved Deep Forest Model for Predicting Self-Interacting Proteins From Protein Sequence Using Wavelet Transformation, (in English). Frontiers in Genetics 10(10):90

    Article  Google Scholar 

  17. Guoying Z, Matti P (2007) Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions. IEEE Transactions on Pattern Analysis & Machine Intelligence 29:915–928

    Article  Google Scholar 

  18. Lim CH, Goh KM (2017) Fuzzy qualitative approach for micro-expression recognition, in 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1669–1674: IEEE

  19. Li X et al (2018) Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods. IEEE Trans Affect Comput 9(4):563–577

    Article  Google Scholar 

  20. Wang Y., See J, Phan RC-W,  Oh Y-H (2014) "Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition," in Asian conference on computer vision, pp. 525–537: Springer

  21. Wang Y, See J, Phan RC-W, Oh Y-H (2015) Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PloS one 10:5

    Google Scholar 

  22. Huang X, Zhao G, Hong X, Zheng W, Pietikäinen M (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578

    Article  Google Scholar 

  23. Huang X, Wang S, Liu X, Zhao G, Feng X, Pietikäinen M (2019) Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition. IEEE Trans Affect Comput 10(1):32–47

    Article  Google Scholar 

  24. Liong S-T, See J, Wong K, Phan RC-W (2018) Less is more: Micro-expression recognition from video using apex frame. Signal Processing: Image Communication 62:82–92

    Google Scholar 

  25. Li Q, Yu J, Kurihara T, Zhan S (2018) Micro-expression analysis by fusing deep convolutional neural network and optical flow, in 2018 5th International Conference on Control, Decision and Information Technologies (CoDIT), 2018, pp. 265–270: IEEE

  26. Liong S-T, Wong K (2017) Micro-expression recognition using apex frame with phase information," in 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 534–537: IEEE

  27. Guo Y, Zhao G, Pietikäinen M (2012) Dynamic facial expression recognition using longitudinal facial expression atlases, in European Conference on Computer Vision, pp. 631–644: Springer

  28. Khor H, See J, Liong S, Phan RCW, Lin W (2019) Dual-stream Shallow Networks for Facial Micro-expression Recognition," in 2019 IEEE International Conference on Image Processing (ICIP), pp. 36–40: IEEE

  29. Gan YS, Liong S (2018) Bi-Directional Vectors from Apex in CNN for Micro-Expression Recognition, IEEE 3rd International Conference on Image, Vision and Computing, pp. 168–172

  30. Oh Yee-Hui, Raphael Phan CW, Sze-Teng Liong, Su-Wei Tan (2016) Spontaneous subtle expression detection and recognition based on facial strain. Signal Processing: Image Communication 47:170–182

    Google Scholar 

  31. Liong ST, See J, Phan CW, Ngo ACL, Oh YH, Wong KS (2014) Subtle expression recognition using optical strain weighted features, in Asian Conference on Computer Vision, pp. 644–657: Springer

  32. Li XHY, Zhao G (2018) Can micro-expression be recognized based on single apex frame?, in 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3094–3098: IEEE

  33. Peng M, Wang C, Chen T, Liu G, Fu X (2017) Dual temporal scale convolutional neural network for micro-expression recognition, Frontiers in psychology, vol. 8

  34. Peng M, Wu Z, Zhang Z, Chen T (2018) From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning., in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 657–661: IEEE

  35. Peng M, Wang T, Bi Y, Shi, X. Zhou, and T. Chen, "A novel apex-time network for cross-dataset micro-expression recognition," in 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), 2019, pp. 1–6: IEEE

  36. Verburg M, Menkovski V (2019) Micro-expression detection in long videos using optical flow and recurrent neural networks, in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–6: IEEE

  37. Gan Y, Liong S-T, Yau W-C, Tan Y-CHL-K (2019) OFF-ApexNet on micro-expression recognition system. Signal Processing: Image Communication 74:129–139

    Google Scholar 

  38. Verma M, Vipparthi S, Singh G, Murala S (2020) LEARNet: Dynamic Imaging Network for Micro Expression Recognition. IEEE Trans Image Process 29:1618–1627

    Article  MathSciNet  MATH  Google Scholar 

  39. Liong S, Gan YS, See J, Khor H, Huang Y (2019) Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition, in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5

  40. Liu K, Jin Q, Xu H, Gan Y, Liong S (2021) Micro-expression recognition using advanced genetic algorithm, Signal Processing Image Communication, vol. 93

  41. Gupta P (2021) MERASTC: Micro-expression Recognition using Effective Feature Encodings and 2D Convolutional Neural network. IEEE Transactions on Affective Computing, pp(99): p. 1-1.

  42. Lessmann S, Baesens B, VonnSeow Hsin, Thomas LC (2015) Benchmarking state-of-the-art classification algorithms for credit scoring: an update of research. European Journal of Operational Research 247(1):124–136

    Article  MATH  Google Scholar 

  43. Y. Zheng, H. Peng, X. Zhang, X. Gao, and J. Li, "Predicting Drug Targets from Heterogeneous Spaces using Anchor Graph Hashing and Ensemble Learning," in 2018 International Joint Conference on Neural Networks (IJCNN), 2018, pp. 1–7: IEEE

  44. Zheng J, Cao X, Zhang B, Zhen X, Su X (2019) Deep Ensemble Machine for Video Classification, in IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 2, pp. 553–565: IEEE

  45. Xu J, Zhang J (2019) An Heterogeneous Ensemble Learning based Method for ECG Classification, in 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), pp. 439–443: IEEE

  46. Ioannis P, Grigorios T, Ioannis V (2009) Pruning an ensemble of classifiers via reinforcement learning. Neurocomputing 72:1900–1909

    Article  Google Scholar 

  47. Suting Y, Ning Z (2020) Construction of Structural Diversity of Ensemble Learning Based on Classification Coding, in 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), pp. 1205–1208: IEEE

  48. Yang F, Xu Q, Li B, Ji Y (2018) Ship detection from thermal remote sensing imagery through region-based deep forest. IEEE Geosci Remote Sens Lett 15(3):449–453

    Article  Google Scholar 

  49. Wang J, Yang Y, Mao J, Huang Z, Huang C, Xu W (2016) Cnn-rnn: A unified framework for multi-label image classification, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2285–2294

  50. Dikbayir HS, Bülbül HÏ (2020) Deep Learning Based Vehicle Detection From Aerial Images," 2020 19th IEEE International Conference on Machine Learning and Applications, pp. 956–960

  51. Gidaris S, Komodakis N (2015) Object detection via a multi-region and semantic segmentation-aware cnn model, in Proceedings of the IEEE international conference on computer vision, pp. 1134–1142

  52. Wang Y, Meng M (2013) 3d facial expression recognition on curvature local binary patterns, in 2013 5th International Conference on Intelligent Human-Machine Systems and Cybernetics, vol. 2, pp. 123–126: IEEE

  53. Fehr J, Burkhardt H (2008) 3D rotation invariant local binary patterns, in 2008 19th International Conference on Pattern Recognition, pp. 1–4: IEEE

  54. Pfister T,  Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions, in 2011 international conference on computer vision, pp. 1449–1456: IEEE

  55. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition, in ICLR

  56. Howard AG et al (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint

  57. Zoph B, Vasudevan V, Shlens J, Le QV (2018) Learning transferable architectures for scalable image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697–8710

  58. Xia Z, Peng W, Khor H-Q, Feng X, Zhao G (2020) Revealing the Invisible With Model and Data Shrinking for Composite-Database Micro-Expression Recognition. IEEE Trans Image Process 29:8590–8605

    Article  MATH  Google Scholar 

  59. Li Y, Huang X, Zhao G (2021) Joint Local and Global Information Learning With Single Apex Frame Detection for Micro-Expression Recognition. IEEE Trans Image Process 30:249–263

    Article  Google Scholar 

  60. Quang NV, Chun J, Tokuyama T (2019) CapsuleNet for Micro-Expression Recognition, FG 2019, pp. 1–7

  61. Su Y, Zhang J, Liu J, Zhai G (2021) Key Facial Components Guided Micro-Expression Recognition Based on First & Second-Order Motion, International Conference on Multimedia and Expo, pp. 1–6

  62. Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207

    Article  MATH  Google Scholar 

  63. Fleiss JL, Levin B, Paik MC (1981) The measurement of interrater agreement. Statistical methods for rates and proportions 2(212–236):22–23

    Google Scholar 

  64. Shipp CA, Kuncheva LI (2002) Relationships between combination methods and measures of diversity in combining classifiers. Information fusion 3(2):135–148

    Article  Google Scholar 

  65. Iman RL, Davenport JM (1980) Approximations of the critical region of the fbietkan statistic. Communications in Statistics-Theory and Methods 9(6):571–595

    Article  MATH  Google Scholar 

  66. Pohlert TJRp (2017) The pairwise multiple comparison of mean ranks package (PMCMR), 27(2019), p. 9

  67. Radiuk PM (2017) Impact of training set batch size on the performance of convolutional neural networks for diverse datasets. Information Technology and Management Science 20(1):20–24

    Article  Google Scholar 

  68. Peng C et al (2018) Megdet: A large mini-batch object detector, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6181–6189

  69. You Y, Gitman I, Ginsburg B(2017) Large batch training of convolutional networks, arXiv preprint , 2017

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (No. 61772023), National Key Research and Development Program of China (No. 2019QY1803), Fujian Science and Technology Plan Industry-University-Research Cooperation Project (No.2021H6015) and Ministry of Science and Technology, Taiwan (MOST108-2221-E-035-066-, MOST 108-2218-E-009 -054 -MY2, MOST 108-2218-E-035 -007-).

Author information

Authors and Affiliations

  1. School of Informatics, Xiamen University, Xiamen, Fujian Province, China

    Meng-Xin Sun, Kun-Hong Liu & Qing-Qiang Wu

  2. Department of Electronic Engineering, Feng Chia University, Taichung, Taiwan

    Sze-Teng Liong

Authors
  1. Meng-Xin Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sze-Teng Liong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kun-Hong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qing-Qiang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kun-Hong Liu.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, MX., Liong, ST., Liu, KH. et al. The heterogeneous ensemble of deep forest and deep neural networks for micro-expressions recognition. Appl Intell 52, 16621–16639 (2022). https://doi.org/10.1007/s10489-022-03284-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03284-y

Keywords

Search

Navigation