Abstract
Applying ensemble learning to facial expression recognition is an important research field nowadays, but all may not be better than many, the redundant learners in the classifier pool may hinder the ensemble system’s performance, so ensemble pruning is needed. Ensemble pruning selects the most suitable subset of classifiers to classify test samples according to the classifier competence. However, the noisy and redundant samples in the validation set will often adversely affect the evaluation of the classifier, making it impossible to select the most suitable classifier. In this paper, a novel ensemble pruning algorithm based on clustering soft label optimization and sorting for facial expression recognition is proposed. First, to increase classifier evaluation objectivity, the novel method uses the clustering optimization model to perform prototype selection and classifier clustering simultaneously. Then the accuracy-based ordering is employed to remove the redundant or poor quality learners, and keep a balance between diversity and accuracy of the ensemble system. Experimental results show that the proposed method outperforms or competes with some state-of-the-art methods on several typical facial expression datasets.
Similar content being viewed by others
Availability of data and material
The datasets used during the current study are reasonably available from the corresponding authors.
Code availability
Some codes for this experiment are available from the corresponding author.
References
Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020). https://doi.org/10.1109/taffc.2020.2981446
Wu, S.H., Wang, B.: Facial expression recognition based on computer deep learning algorithm: taking cognitive acceptance of college students as an example. J. Ambient Intell. Humaniz. Comput. (2021). https://doi.org/10.1007/s12652-021-03113-z
Zhang, H.F., Su, W., Yu, J., Wang, Z.F.: Identity-expression dual branch network for facial expression recognition. IEEE Trans. Cogn. Dev. Syst. 13(4), 898–911 (2021). https://doi.org/10.1109/tcds.2020.3034807
Ge, H., Zhu, Z., Dai, Y., Wang, B., Wu, X.: Facial expression recognition based on deep learning. Comput. Methods Prog. Biomed. 215, 106621 (2022)
Zou, W., Zhang, D., Lee, D.-J.: A new multi-feature fusion based convolutional neural network for facial expression recognition. Appl. Intell. 52(3), 2918–2929 (2021). https://doi.org/10.1007/s10489-021-02575-0
Sun, Z., Chiong, R., Hu, Z.-p: An extended dictionary representation approach with deep subspace learning for facial expression recognition. Neurocomputing 316, 1–9 (2018). https://doi.org/10.1016/j.neucom.2018.07.045
Li, D., Wen, G., Hou, Z., Huan, E., Hu, Y., Li, H.: RTCRelief-F: an effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition. Knowl. Inf. Syst. 59(1), 219–250 (2018). https://doi.org/10.1007/s10115-018-1176-z
Liang, Z.F., Wang, H., Yang, K.X., Shi, Y.F.: Adaptive fusion based method for imbalanced data classification. Front. Neurorobotics 16, 8 (2022). https://doi.org/10.3389/fnbot.2022.827913
Martinez, W.G.: Ensemble pruning via quadratic margin maximization. IEEE Access 9, 48931–48951 (2021). https://doi.org/10.1109/access.2021.3062867
Zhang, C., Wu, Y., Zhu, M.: Pruning variable selection ensembles. Stat. Analy. Data Min. mASA Data Sci. J. 12(3), 168–184 (2019). https://doi.org/10.1002/sam.11410
Zyblewski, P., Woźniak, M.: Novel clustering-based pruning algorithms. Patt. Anal. Appl. 23(3), 1049–1058 (2020). https://doi.org/10.1007/s10044-020-00867-8
Diao, R., Chao, F., Peng, T.X., Snooke, N., Shen, Q.: Feature selection inspired classifier ensemble reduction. IEEE T. Cybern. 44(8), 1259–1268 (2014). https://doi.org/10.1109/tcyb.2013.2281820
Alberghini, G., Barbon Junior, S., Cano, A.: Adaptive ensemble of self-adjusting nearest neighbor subspaces for multi-label drifting data streams. Neurocomputing 481, 228–248 (2022). https://doi.org/10.1016/j.neucom.2022.01.075
Wang, Z., Wang, K., Wang, X., Pan, S., Qiao, X.: Dynamic ensemble selection of convolutional neural networks and its application in flower classification. Int. J. Agri. Biol. Eng. 14(6), 216–223 (2021). https://doi.org/10.25165/j.ijabe.20221501.6313
Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C.: Prototype selection for dynamic classifier and ensemble selection. Neural Comput. Appl. 29(2), 447–457 (2016). https://doi.org/10.1007/s00521-016-2458-6
Ben, X., Ren, Y., Zhang, J., Wang, S.J., Kpalma, K., Meng, W., Liu, Y.J.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Patt. Anal. Mach. Intell. 44(9), 5826–5846 (2022). https://doi.org/10.1109/TPAMI.2021.3067464
Cheng, J., Liu, G.Y.: Affective detection based on an imbalanced fuzzy support vector machine. Biomed. Sig. Process. Control 18, 118–126 (2015). https://doi.org/10.1016/j.bspc.2014.12.006
Ben, X., Gong, C., Huang, T., Li, C., Yan, R., Li, Y.: Tackling micro-expression data shortage via dataset alignment and active learning. IEEE Trans. Multimed. (2022). https://doi.org/10.1109/TMM.2022.3192727
Xu, C.E., Cui, Y., Zhang, Y.H., Gao, P., Xu, J.Y.: Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware. Multimed. Syst. 26(1), 53–61 (2020). https://doi.org/10.1007/s00530-019-00628-6
Kumar, Y., Verma, S.K., Sharma, S.: Multi-pose facial expression recognition using hybrid deep learning model with improved variant of gravitational search algorithm. Int. Arab. J Inf. Technol. 19(2), 281–287 (2022). https://doi.org/10.3402/iajit/19/2/15
Zhu, X.L., Ye, S.H., Zhao, L., Dai, Z.C.: Hybrid attention cascade network for facial expression recognition. Sensors 21(6), 15 (2021). https://doi.org/10.3390/s21062003
Li, B., Zhou, Y., Xiao, R., Wang, J., Ben, X., Kpalma, K., Zhou, H.: Unsupervised cross-database micro-expression recognition based on distribution adaptation. Multimed. Syst. 28(3), 1099–1116 (2022). https://doi.org/10.1007/s00530-022-00896-9
Xie, S., Hu, H.: Facial expression recognition with FRR-CNN. Electron. Lett. 53(4), 235–237 (2017). https://doi.org/10.1049/el.2016.4328
Agrawal, A., Mittal, N.: Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis. Comput. 36(2), 405–412 (2019). https://doi.org/10.1007/s00371-019-01630-9
Wen, G.H., Li, H.H., Li, D.Y.: Ieee, An Ensemble Convolutional Echo State Networks for Facial Expression Recognition, 6th AAAC Affective Computing and Intelligent Interaction International Conference (ACII), pp. 873–878. Xian, Peoples R China, Ieee (2015)
Yu Z,Zhang C, Image based Static Facial Expression Recognition with Multiple Deep Network Learning, Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015, pp. 435–442.
Pons, G., Masip, D.: Supervised committee of convolutional neural networks in automated facial expression analysis. Affect. Comput. IEEE Trans. 9(3), 343–350 (2018)
Sultan Zia, M., Hussain, M., Arfan Jaffar, M.: A novel spontaneous facial expression recognition using dynamically weighted majority voting based ensemble classifier. Multimed. Tools Appl. 77(19), 25537–25567 (2018). https://doi.org/10.1007/s11042-018-5806-y
Zavaschi T, Koerich AL,Oliveira L, Facial expression recognition using ensemble of classifiers, IEEE International Conference on Acoustics, 2011.
Li, D., Wen, G., Li, X., Cai, X.: Graph-based dynamic ensemble pruning for facial expression recognition. Appl. Intell. 49(9), 3188–3206 (2019). https://doi.org/10.1007/s10489-019-01435-2
Zhang, H., Cao, L.: A spectral clustering based ensemble pruning approach. Neurocomputing 139, 289–297 (2014). https://doi.org/10.1016/j.neucom.2014.02.030
Guo, L., Boukir, S.: Margin-based ordered aggregation for ensemble pruning. Pattern Recogn. Lett. 34(6), 603–609 (2013). https://doi.org/10.1016/j.patrec.2013.01.003
Dai, Q., Han, X.: An efficient ordering-based ensemble pruning algorithm via dynamic programming. Appl. Intell. 44(4), 816–830 (2015). https://doi.org/10.1007/s10489-015-0729-z
Nan L, Yang Y,Zhou ZH, Diversity Regularized Ensemble Pruning, Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2012.
Liu, Z., Dai, Q., Liu, N.: Ensemble selection by GRASP. Appl. Intell. 41(1), 128–144 (2014). https://doi.org/10.1007/s10489-013-0510-0
Fatemifar S, Awais M, Akbari A,Kittler J, A Stacking Ensemble for Anomaly Based Client-Specific Face Spoofing Detection, 2020 IEEE International Conference on Image Processing (ICIP), 2020.
Fatemifar, S., Asadi, S., Awais, M., Akbari, A., Kittler, J.: Face spoofing detection ensemble via multistage optimisation and pruning. Patt. Recogn. Lett. 158, 1–8 (2022). https://doi.org/10.1016/j.patrec.2022.04.006
Luong, A.V., Nguyen, T.T., Liew, A.W.C., Wang, S.L.: Heterogeneous ensemble selection for evolving data streams. Patt. Recogn. 112, 16 (2021). https://doi.org/10.1016/j.patcog.2020.107743
Elmi, J., Eftekhari, M.: Multi-Layer Selector(MLS): dynamic selection based on filtering some competence measures. Appl. Soft Comput. (2021). https://doi.org/10.1016/j.asoc.2021.107257
Li, D., Wen, G.: MRMR-based ensemble pruning for facial expression recognition. Multimed. Tools Appl. 77(12), 15251–15272 (2018). https://doi.org/10.1007/s11042-017-5105-z
Markatopoulou, F., Tsoumakas, G., Vlahavas, I.: Dynamic ensemble pruning based on multi-label classification. Neurocomputing 150, 501–512 (2015). https://doi.org/10.1016/j.neucom.2014.07.063
Lin, X., Mao, S., Jiao, L.: Selective ensemble based on transformation of classifiers used SPCA. Int. J. Patt. Recogn. Artif. Intell. 29(1), 1550005 (2015)
Jan, M.Z., Verma, B.: A novel diversity measure and classifier selection approach for generating ensemble classifiers. Ieee Access 7, 156360–156373 (2019). https://doi.org/10.1109/access.2019.2949059
Baek, B., Euh, S., Baek, D., Kim, D., Hwang, D.: Histogram entropy representation and prototype based machine learning approach for malware family classification. Ieee Access 9, 152098–152114 (2021). https://doi.org/10.1109/access.2021.3127195
Zhang, X.X., Zhu, Z.F., Zhao, Y., Chang, D.X., Liu, J.: Seeing all from a few: l(1)-norm-induced discriminative prototype selection. IEEE Trans. Neural Netw. Learn. Syst. 30(7), 1954–1966 (2019). https://doi.org/10.1109/tnnls.2018.2875347
Cruz, R.M.O., Oliveira, D.V.R., Cavalcanti, G.D.C., Sabourin, R.: FIRE-DES plus plus : Enhanced online pruning of base classifiers for dynamic ensemble selection. Patt. Recogn. 85, 149–160 (2019). https://doi.org/10.1016/j.patcog.2018.07.037
Sajjanhar A, Wu ZQ,Wen Q, Deep Learning Models for Facial Expression Recognition, 2018 Digital Image Computing: Techniques and Applications (DICTA), 2018.
Kumari, N., Bhatia, R.: Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter. Soft. Comput. (2022). https://doi.org/10.1007/s00500-022-06804-7
Conlin, R., Erickson, K., Abbate, J., Kolemen, E.: Keras2c: A library for converting Keras neural networks to real-time compatible C. Eng. Appl. Artif. Intell. (2021). https://doi.org/10.1016/j.engappai.2021.104182
Gan YJ,Acm, Facial Expression Recognition Using Convolutional Neural Network, 2nd International Conference on Vision, Image and Signal Processing (ICVISP), Assoc Computing Machinery, Las Vegas, NV, 2018.
Shafira SS, Ulfa N, Wibawa HA, Rismiyati,Ieee, Facial Expression Recognition Using Extreme Learning Machine, 3rd International Conference on Informatics and Computational Sciences (ICICoS), Ieee, Univ Diponegoro, Dept Informat, Semarang, INDONESIA, 2019.
Videla LS, Kumar PMA,Ieee, Facial Expression Classification Using Vanilla Convolution Neural Network, 7th IEEE International Conference on Smart Structures and Systems (ICSSS), Ieee, Saveetha Engn Coll, Chennai, INDIA, 2020, pp. 107–111.
Langner O, Dotsch R, Bijlstra G, Wigboldus DH, Hawk ST,Knippenberg AV (2010) Presentation and validation of the Radboud Faces Database.
Kuncheva, L.I.: A bound on kappa-error diagrams for analysis of classifier ensembles. IEEE Trans. Knowl. Data Eng. 25(3), 494–501 (2013). https://doi.org/10.1109/tkde.2011.234
Hou C, Xia Y, Xu Z,Sun J, Learning classifier competence based on graph for dynamic classifier selection, 2016 12th International Conference on Natural Computation and 13th Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), 2016.
Ko, A.H.R., Sabourin, R., Britto, J.A.S.: From dynamic classifier selection to dynamic ensemble selection. Patt. Recogn. 41(5), 1718–1731 (2008). https://doi.org/10.1016/j.patcog.2007.10.015
Funding
No funding was received to assist with the preparation of this manuscript.
Author information
Authors and Affiliations
Contributions
Shisong Huang carried out the experimental design and completed the writing of the paper. Danyang Li instructed the proposal of the method and reviewed and revised the article. Zhuhong Zhang put forward guiding opinions. Yumei Tang, Xing Chen and Yiqing Wu collected and processed the experimental data.All authors reviewed the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Huang, S., Li, D., Zhang, Z. et al. CSLSEP: an ensemble pruning algorithm based on clustering soft label and sorting for facial expression recognition. Multimedia Systems 29, 1463–1479 (2023). https://doi.org/10.1007/s00530-023-01062-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00530-023-01062-5