Skip to main content
SpringerLink
Log in

Pairwise dependency-based robust ensemble pruning for facial expression recognition

  • 1230: Sentient Multimedia Systems and Visual Intelligence
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

A Correction to this article was published on 08 February 2024

This article has been updated

Abstract

Facial expression recognition is crucial in analyzing an individual’s emotional state. Ensemble pruning becomes essential to enhance the effectiveness of this recognition by selecting appropriate classifiers from a pool of base classifiers. However, outliers and noise within the classifiers can adversely affect the final recognition results and the generalization performance of the selected subset of classifiers. Additionally, effectively combining accuracy and low redundancy of base classifiers remains a challenging problem that requires further investigation. In this paper, we propose a novel algorithm called Pairwise Dependency-based Robust Ensemble Pruning (PDREP) to address these issues. The PDREP algorithm treats the predicted results of classifiers for sample instances as features of the classifier and evaluates their dependencies between pairs of classifiers using mutual information. By incorporating this dependency measure into the regression-based objective equation, we can assess the redundancy of a subset of base classifiers and prune redundant classifiers. We use the \(\varvec{l}_{\textbf{2},\textbf{1}}\)-norm in PDREP’s objective equation to perform robust classifier pruning while considering the base classifiers’ dependencies and accuracy. Furthermore, we introduce weight control parameters to balance accuracy and dependencies, facilitating the elimination of ineffective and redundant classifiers. The results show significant improvements when evaluating the proposed method’s recognition accuracy on five public face sentiment datasets (FER2013, JAFFE, CK+, RaFD, and KDEF). Specifically, the proposed method achieves 3.15%, 9.39%, 1.72%, 3.70%, and 4.70% higher accuracy than integrating all base classifiers, respectively. Extensive experiments conducted on five facial expression datasets demonstrate that the proposed method consistently outperforms existing state-of-the-art ensemble pruning algorithms in most cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data Availability

The datasets employed and analyzed throughout this study are openly accessible through the following repositories:

FER2013 dataset: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data

JAFFE dataset: https://zenodo.org/record/3451524

CK+ dataset: http://vasc.ri.cmu.edu/idb/html/face/facial_expression/

RaFD dataset: http://www.socsci.ru.nl:8180/RaFD2/RaFD

KDEF dataset: https://www.kdef.se/home/aboutKDEF

Change history

References

  1. Lyons MJ, Kamachi M, Gyoba J (2020) Coding Facial Expressions with Gabor Wavelets (IVC Special Issue). https://doi.org/10.5281/zenodo.4029680

  2. Li X et al (2023) Facial expression recognition network with slow convolution and zero-parameter attention mechanism. Optik 283:170892. https://doi.org/10.1016/j.ijleo.2023.170892

    Article  Google Scholar 

  3. Lyons MJ (2021) “Excavating AI” Revisited: Discrediting Misleading Accounts of the JAFFE Dataset. zenodo. https://doi.org/10.5281/zenodo.5147170

  4. Li D, Wen G, Li X, Cai X (2019) Graph-based dynamic ensemble pruning for facial expression recognition. Appl Intell 49:3188–3206. https://doi.org/10.1007/s10489-019-01435-2

    Article  Google Scholar 

  5. Li D, Wen G (2018) Mrmr-based ensemble pruning for facial expression recognition. Multimed Tools Appl. https://doi.org/10.1007/s11042-017-5105-z

  6. Li D et al (2018) Rtcrelief-f: an effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition. Knowl Inf Syst 59. https://doi.org/10.1007/s10115-018-1176-z

  7. Zhenlei W et al (2021) Ensemble selection with joint spectral clustering and structural sparsity. Pattern Recognit 119:108061. https://doi.org/10.1016/j.patcog.2021.108061

    Article  Google Scholar 

  8. Huang S et al (2023) Cslsep: an ensemble pruning algorithm based on clustering soft label and sorting for facial expression recognition. Multimed Syst 29:1463–1479. https://doi.org/10.1007/s00530-023-01062-5

    Article  Google Scholar 

  9. Dai Q, Ye R, Liu Z (2017) Considering diversity and accuracy simultaneously for ensemble pruning. Appl Soft Comput 58:75–91. https://doi.org/10.1016/j.asoc.2017.04.058

    Article  Google Scholar 

  10. Fletcher S, Verma B, Zhang M (2020) A non-specialized ensemble classifier using multi-objective optimization. Neurocomputing 409:93–102. https://doi.org/10.1016/j.neucom.2020.05.029

    Article  Google Scholar 

  11. Lim H, Kim D-W (2021) Pairwise dependence-based unsupervised feature selection. Pattern Recognit 111:107663. https://doi.org/10.1016/j.patcog.2020.107663

    Article  Google Scholar 

  12. Du S, Ma Y, Li S, Ma Y (2017) Robust unsupervised feature selection via matrix factorization. Neurocomputing 241:115–127. https://doi.org/10.1016/j.neucom.2017.02.034

    Article  Google Scholar 

  13. Xia X, Jiang D (2023) Hit-mst: Dynamic facial expression recognition with hierarchical transformers and multi-scale spatiotemporal aggregation. Inform Sci 644:119301. https://doi.org/10.1016/j.ins.2023.119301

    Article  Google Scholar 

  14. Wu X et al (2023) Fer-chc: Facial expression recognition with cross-hierarchy contrast. Appl Soft Comp 110530. https://doi.org/10.1016/j.asoc.2023.110530

  15. Nan Y, Ju J, Hua Q, Zhang H, Wang B (2022) A-mobilenet: An approach of facial expression recognition. Alex Eng J 61:4435–4444. https://doi.org/10.1016/j.aej.2021.09.066

    Article  Google Scholar 

  16. Guo Y et al (2022) Facial expressions recognition with multi-region divided attention networks for smart education cloud applications. Neurocomputing 493:119–128. https://doi.org/10.1016/j.neucom.2022.04.052

    Article  Google Scholar 

  17. Ganaie MA, Hu M, Malik A, Tanveer M, Suganthan P (2022) Ensemble deep learning: A review. Eng Appl Artif Intell 115:105151. https://doi.org/10.1016/j.engappai.2022.105151

    Article  Google Scholar 

  18. Chirra VRR, Uyyala SR, Kolli VKK (2021) Virtual facial expression recognition using deep cnn with ensemble learning. J Ambient Intell Humaniz Comput 1–19. https://doi.org/10.1109/10.1007/s12652-020-02866-3

  19. Rajagopal SD, Ramachandran B (2022) 3d face expression recognition with ensemble deep learning exploring congruent features among expressions. Comput Intell 38:345–365. https://doi.org/10.1111/coin.12498

    Article  Google Scholar 

  20. Tsai K-Y, Tsai Y-W, Lee Y-C, Ding J-J, Chang RY (2021) Frontalization and adaptive exponential ensemble rule for deep-learning-based facial expression recognition system. Signal Process Image Commun 96:116321. https://doi.org/10.1016/j.image.2021.116321

    Article  Google Scholar 

  21. Jia X, Zhang Y, Powers D, Ali HB (2014) Multi-classifier fusion based facial expression recognition approach. KSII Trans Internet Inform Syst 8. https://doi.org/10.1007/s11042-020-08746-4

  22. Li D, Zhang Z, Wen G (2023) Classifier subset selection based on classifier representation and clustering ensemble. Appl Intell 1–23. https://doi.org/10.1007/s10489-023-04572-x

  23. Roshan E (2021) Development of ensemble learning classification with density peak decomposition-based evolutionary multi-objective optimization. Int J Mach Learn Cybernet 12:1737–1751. https://doi.org/10.1007/s13042-020-01271-8

    Article  Google Scholar 

  24. Ji J, Li J (2023) Tri-objective optimization-based cascade ensemble pruning for deep forest. Pattern Recognit 143:109744. https://doi.org/10.1016/j.patcog.2023.109744

    Article  Google Scholar 

  25. Yang AY, Lv BH, Chen CN, Wu DY, Zheng EZ (2020) Ftbme: feature transferring based multi-model ensemble. Multimed Tools Appl 79:18767–18799. https://doi.org/10.1007/s11042-020-08746-4

    Article  Google Scholar 

  26. Shen X, Dai Q, Zhu G, Zhang J (2023) Dynamic ensemble pruning algorithms fusing meta-learning with heuristic parameter optimization for time series prediction. Exp Syst Appl 225:120148. https://doi.org/10.1016/j.eswa.2023.120148

    Article  Google Scholar 

  27. Martínez-Muñoz G, Suárez A (2006) Pruning in ordered bagging ensembles https://doi.org/10.1145/1143844.1143921

  28. Guo H et al (2018) Margin & diversity based ordering ensemble pruning. Neurocomputing 275:237–246. https://doi.org/10.1016/j.neucom.2017.06.052

    Article  Google Scholar 

  29. Zhang H, Wu S, Zhang X, Han L, Zhang Z (2022) Slope stability prediction method based on the margin distance minimization selective ensemble. CATENA 212:106055. https://doi.org/10.1016/j.catena.2022.106055

    Article  Google Scholar 

  30. Lv S, Peng L, Hu H, Wang L (2022) Effective machine learning model combination based on selective ensemble strategy for time series forecasting. Inform Sci 612:994–1023. https://doi.org/10.1016/j.ins.2022.09.002

    Article  Google Scholar 

  31. Bakkouri I, Afdel K (2023) Mlca2f: Multi-level context attentional feature fusion for covid-19 lesion segmentation from ct scans. Sig Image Video Process 17:1181–1188. https://doi.org/10.3837/tiis.2014.01.012

    Article  Google Scholar 

  32. Bakkouri I, Karim A (2020) Computer-aided diagnosis (cad) system based on multi-layer feature fusion network for skin lesion recognition in dermoscopy images. Multimed Tools Appl 79:20483–20518. https://doi.org/10.1007/s11042-019-07988-1

    Article  Google Scholar 

  33. Afonso MV, Bioucas-Dias JM, Figueiredo MAT (2011) An augmented lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE Trans Image Process 20:681–695. https://doi.org/10.1109/TIP.2010.2076294

    Article  MathSciNet  Google Scholar 

  34. Goodfellow IJ, Erhan D, Carrier PL et al. (2015) Challenges in representation learning: a report on three machine learning contests. Neural Networks 64:59–63. https://doi.org/10.1016/j.neunet.2014.09.005

  35. Langner O, Dotsch R, Bijlstra G, Wigboldus DHJ, Hawk ST, van Knippenberg A (2010) Presentation and validation of the Radboud Faces Database. Cognition and Emotion 24(8)1377–1388. https://doi.org/10.1080/02699930903485076

  36. Goeleven E, Raedt RD, Leyman L, Verschuere B (2008) The karolinska directed emotional faces: A validation study. Cogn Emot 22:1094–1118. https://doi.org/10.1080/02699930701626582

    Article  Google Scholar 

  37. Dai Q (2013) A novel ensemble pruning algorithm based on randomized greedy selective strategy and ballot. Neurocomputing 122:258–265. https://doi.org/10.1016/j.neucom.2013.06.026

    Article  Google Scholar 

  38. Yin X-C, Huang K, Yang C, Hao H-W (2014) Convex ensemble learning with sparsity and diversity. Inf Fusion 20:49–59. https://doi.org/10.1016/j.inffus.2013.11.003

    Article  Google Scholar 

  39. Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51:181–207. https://doi.org/10.1023/A:1022859003006

    Article  Google Scholar 

  40. Hou C, Xia Y, Xu Z, Sun J (2016). Learning classifier competence based on graph for dynamic classifier selection. https://doi.org/10.1109/FSKD.2016.7603343

    Article  Google Scholar 

  41. Woods K, Kegelmeyer W, Bowyer K (1997) Combination of multiple classifiers using local accuracy estimates. IEEE Trans Pattern Anal Mach Intell 19:405–410. https://doi.org/10.1109/34.588027

    Article  Google Scholar 

  42. Didaci L, Giacinto G, Roli F, Marcialis GL (2005) A study on the performances of dynamic classifier selection based on local accuracy estimation. Pattern Recogn 38:2188–2191. https://doi.org/10.1016/j.patcog.2005.02.010

    Article  Google Scholar 

  43. Markatopoulou F, Tsoumakas G, Vlahavas I (2014) Dynamic ensemble pruning based on multi-label classification. Neurocomputing 150:501–512. https://doi.org/10.1016/j.neucom.2014.07.063

    Article  Google Scholar 

  44. Yang Z, Lu H, Yu Q (2022) Critical independent sets of könig-egerváry graphs. Discret Appl Math 318:1–5. https://doi.org/10.1016/j.dam.2022.04.014

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Science and Technology Plan Project of Guizhou Province (Qiankehe Platform Talents [2018] 5781) and was supported by the Guizhou Provincial Science and Technology Plan Project (Qiankehe Support [2023] General 251).

Author information

Authors and Affiliations

  1. College of Big Data and Information Engineering, Guizhou University, Jiaxiu South Road, Guiyang City, Guizhou Province, 550025, China

    Xing Chen, Danyang Li, Yumei Tang, Shisong Huang, Yiqing Wu & Yating Wu

Authors
  1. Xing Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danyang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yumei Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shisong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yiqing Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yating Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Danyang Li.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: The original publication of this article contains incorrect details of references [1], [3], [34], and [35] and incorrect Data availability statement.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, X., Li, D., Tang, Y. et al. Pairwise dependency-based robust ensemble pruning for facial expression recognition. Multimed Tools Appl 83, 37089–37117 (2024). https://doi.org/10.1007/s11042-023-16756-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16756-1

Keywords

Search

Navigation