Skip to main content
Log in

An adaptive multi-class imbalanced classification framework based on ensemble methods and deep network

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Data imbalance is one of the most difficult problems in machine learning. The improved ensemble learning model is a promising solution to mitigate this challenge. In this paper, an improved multi-class imbalanced data classification framework is proposed by combining the Focal Loss with Boosting model (FL-Boosting). By addressing the confusion of the second-order derivation of Focal Loss in the traditional ensemble learning model, the proposed model achieves a more efficient and accurate classification of the imbalanced data. More specifically, a Highly Adaptive Focal Loss (HAFL) is proposed to ensure that the model maintains lasting attention to the minority samples, which could be combined with boosting model to build HAFL-Boosting to achieve better performance. The framework has the scalability to adapt to different situations according to typical ensemble learning algorithms such as LightGBM, XGBoost and CatBoost. In addition, to implement the application of the proposed framework on deep models, a two-stage classification method combining ConvNeXt with the improved boosting model is proposed, which could improve the recognition ability to high-dimensional imbalanced data. We evaluate the HAFL-Boosting and the two-stage class imbalance classification method by ablation experiments and benchmark experiments, which demonstrated that the proposed methods obviously improved the scores on several evaluation indexes. The comparative experiments with the latest classification models show that the proposed methods could achieve leading performance from multiple perspectives.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Ye Q, Liu TY (2017) Lightgbm: a highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst 30:3146–3154

    Google Scholar 

  2. Chen T, Guestrin C (2016) XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA, pp 785-794. https://doi.org/10.1145/2939672.2939785

  3. Dorogush AV, Ershov V, Gulin A (2018) CatBoost: gradient boosting with categorical features support. http://arxiv.org/abs/1810.11363

  4. Tyralis H, Papacharalampous G (2021) Boosting algorithms in energy research: a systematic review. Neural Comput Appl 33(21):14101–14117. https://doi.org/10.1007/s00521-021-05995-8

    Article  Google Scholar 

  5. Wang J, Jiang X, Meng Q, Saada M, Cai H (2022) Walking motion real-time detection method based on walking stick, IoT, COPOD and improved LightGBM. Appl Intell 45:1–19. https://doi.org/10.1007/s10489-022-03264-2

    Article  Google Scholar 

  6. Li S, Zhang X (2020) Research on orthopedic auxiliary classification and prediction model based on XGBoost algorithm. Neural Comput Appl 32(7):1971–1979. https://doi.org/10.1007/s00521-019-04378-4

    Article  Google Scholar 

  7. Jabeur SB, Gharib C, Mefteh-Wali S, Arfi WB (2021) CatBoost model and artificial intelligence techniques for corporate failure prediction. Technol Forecast Soc Change 166:120658. https://doi.org/10.1016/j.techfore.2021.120658

    Article  Google Scholar 

  8. Taherkhani A, Cosma G, McGinnity TM (2020) AdaBoost-CNN: An adaptive boosting algorithm for convolutional neural networks to classify multi-class imbalanced datasets using transfer learning. Neurocomputing 404:351–366

    Article  Google Scholar 

  9. Yu L, Zhou R, Tang L, Chen R (2018) A DBN-based resampling SVM ensemble learning paradigm for credit classification with imbalanced data. Appl Soft Comput 69:192–202. https://doi.org/10.1016/j.asoc.2018.04.049

    Article  Google Scholar 

  10. Niu K, Zhang Z, Liu Y, Li R (2020) Resampling ensemble model based on data distribution for imbalanced credit risk evaluation in P2P lending. Inf Sci 536:120–134. https://doi.org/10.1016/j.ins.2020.05.040

    Article  MathSciNet  Google Scholar 

  11. Koziarski M, Woźniak M, Krawczyk B (2020) Combined cleaning and resampling algorithm for multi-class imbalanced data with label noise. Knowl-Based Syst 204:106223. https://doi.org/10.1016/j.knosys.2020.106223

    Article  Google Scholar 

  12. Lin TY, Goyal P, Girshick R, He K, Dollár P (2020) Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 42(2):318–327. https://doi.org/10.1109/tpami.2018.2858826

    Article  Google Scholar 

  13. Wang C, Deng C, Wang S (2020) Imbalance-XGBoost: leveraging weighted and focal losses for binary label-imbalanced classification with XGBoost. Pattern Recognit Lett 136:190–197. https://doi.org/10.1016/j.patrec.2020.05.035

    Article  Google Scholar 

  14. Cao J, Wang D, Qu Z, Cui M, Xu P, Xue K, Hu K (2020) A novel false data injection attack detection model of the cyber-physical power system. IEEE Access 8:95109–95125. https://doi.org/10.1109/access.2020.2995772

    Article  Google Scholar 

  15. Buda M, Maki A, Mazurowski MA (2018) A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw 106:249–259. https://doi.org/10.1016/j.neunet.2018.07.011

    Article  Google Scholar 

  16. Lin E, Chen Q, Qi X (2020) Deep reinforcement learning for imbalanced classification. Appl Intell 50(8):2488–2502. https://doi.org/10.1007/s10489-020-01637-z

    Article  Google Scholar 

  17. Sun C, Cui H, Zhou W, Nie W, Wang X, Yuan Q (2019) Epileptic seizure detection with EEG textural features and imbalanced classification based on EasyEnsemble learning. Int J Neural Syst 29(10):1950021. https://doi.org/10.1142/s0129065719500217

    Article  Google Scholar 

  18. Liu Z, Mao H, Wu CY, Feichtenhofer C, Darrell T, Xie S (2022) A ConvNet for the 2020s. http://arxiv.org/abs/2201.03545

  19. Bader-El-Den M, Teitei E, Perry T (2019) Biased random forest for dealing with the class imbalance problem. IEEE Trans Neural Netw Learn Syst 30(7):2163–2172. https://doi.org/10.1109/tnnls.2018.2878400

    Article  Google Scholar 

  20. Yuan X, Chen S, Sun C, Yuwen L (2022) A novel early diagnostic framework for chronic diseases with class imbalance. Sci Rep 12(1):1–16. https://doi.org/10.1038/s41598-022-12574-x

    Article  Google Scholar 

  21. Siddiqui MK, Huang X, Morales-Menendez R, Hussain N, Khatoon K (2020) Machine learning based novel cost-sensitive seizure detection classifier for imbalanced EEG data sets. Int J Interactive Des Manuf (IJIDeM) 14(4):1491–1509. https://doi.org/10.1007/s12008-020-00715-3

    Article  Google Scholar 

  22. Guzmán-Ponce A, Sánchez JS, Valdovinos RM, Marcial-Romero JR (2021) DBIG-US: a two-stage under-sampling algorithm to face the class imbalance problem. Expert Syst Appl 168:114301. https://doi.org/10.1016/j.eswa.2020.114301

    Article  Google Scholar 

  23. Lei K, Xie Y, Zhong S, Dai J, Yang M, Shen Y (2020) Generative adversarial fusion network for class imbalance credit scoring. Neural Comput Appl 32(12):8451–8462. https://doi.org/10.1007/s00521-019-04335-1

    Article  Google Scholar 

  24. Temraz M, Keane MT (2022) Solving the class imbalance problem using a counterfactual method for data augmentation. Mach Learn Appl 9:100375. https://doi.org/10.1016/j.mlwa.2022.100375

    Article  Google Scholar 

  25. Kumar S, Biswas SK, Devi D (2019) TLUSBoost algorithm: a boosting solution for class imbalance problem. Soft Comput 23(21):10755–10767. https://doi.org/10.1007/s00500-018-3629-4

    Article  Google Scholar 

  26. Jiang M, Yang Y, Qiu H (2022) Fuzzy entropy and fuzzy support-based boosting random forests for imbalanced data. Appl Intell 52(4):4126–4143. https://doi.org/10.1007/s10489-021-02620-y

    Article  Google Scholar 

  27. Shen F, Zhao X, Kou G, Alsaadi FE (2021) A new deep learning ensemble credit risk evaluation model with an improved synthetic minority oversampling technique. Appl Soft Comput 98:106852. https://doi.org/10.1016/j.asoc.2020.106852

    Article  Google Scholar 

  28. Zhang W, Ramezani R, Naeim A (2019) WOTBoost: Weighted Oversampling Technique in Boosting for imbalanced learning. In: 2019 IEEE International Conference on Big Data (Big Data). pp 2523-2531. https://doi.org/10.1109/bigdata47090.2019.9006091.

  29. Wang W, Sun D (2021) The improved AdaBoost algorithms for imbalanced data classification. Inf Sci 563:358–374. https://doi.org/10.1016/j.ins.2021.03.042

    Article  MathSciNet  Google Scholar 

  30. Jiang X, Xu Y, Ke W, Zhang Y, Zhu QX, He YL (2022) An imbalanced multi-fault diagnosis method based on bias weights AdaBoost. IEEE Trans Instrumentation Measure 5:45. https://doi.org/10.1109/TIM.2022.3149097

    Article  Google Scholar 

  31. Hsu CC, Wang KS, Chung HY, Chang SH (2019) Equation of SVM-rebalancing: the point-normal form of a plane for class imbalance problem. Neural Comput Appl 31(10):6013–6025. https://doi.org/10.1007/s00521-018-3419-z

    Article  Google Scholar 

  32. Ganaie MA, Tanveer M (2021) Fuzzy least squares projection twin support vector machines for class imbalance learning. Appl Soft Comput 113:107933. https://doi.org/10.1016/j.asoc.2021.107933

    Article  Google Scholar 

  33. Hossain MS, Betts JM, Paplinski AP (2021) Dual Focal Loss to address class imbalance in semantic segmentation. Neurocomputing 462:69–87. https://doi.org/10.1016/j.neucom.2021.07.055

    Article  Google Scholar 

  34. Yeung M, Sala E, Schönlieb CB, Rundo L (2022) Unified focal loss: Generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Computerized Med Imag Gr 95:102026. https://doi.org/10.1016/j.compmedimag.2021.102026

    Article  Google Scholar 

  35. Zhang X, Wan F, Liu C, Ji X, Ye Q (2021) Learning to match anchors for visual object detection. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/tpami.2021.3050494

    Article  Google Scholar 

  36. Chen J, Wu Z, Zhang J (2019) Driving safety risk prediction using cost-sensitive with nonnegativity-constrained autoencoders based on imbalanced naturalistic driving data. IEEE Trans Intell Transp Syst 20(12):4450–4465. https://doi.org/10.1109/tits.2018.2886280

    Article  Google Scholar 

  37. Shon HS, Batbaatar E, Kim KO, Cha EJ, Kim K-A (2020) Classification of kidney cancer data using cost-sensitive hybrid deep learning approach. Symmetry 12(1):154. https://doi.org/10.3390/sym12010154

    Article  Google Scholar 

  38. Mulyanto M, Faisal M, Prakosa S, Leu J (2020) Effectiveness of focal loss for minority classification in network intrusion detection systems. Symmetry 13(1):4. https://doi.org/10.3390/sym13010004

    Article  Google Scholar 

  39. Zhao Y, Shen C, Wang H, Chen S (2019) Structural analysis of attributes for vehicle re-identification and retrieval. IEEE Trans Intell Transp Syst 21(2):723–734. https://doi.org/10.1109/tits.2019.2896273

    Article  Google Scholar 

  40. Xu X, Duan H, Guo Y, Deng Y (2020) A cascade adaboost and CNN algorithm for drogue detection in UAV autonomous aerial refueling. Neurocomputing 408:121–134. https://doi.org/10.1016/j.neucom.2019.10.115

    Article  Google Scholar 

  41. Guo Y, Li Z, Li Z, Xiong G, Jiang M, Gou G (2020) FLAGB: focal loss based adaptive gradient boosting for imbalanced traffic classification. In: 2020 International Joint Conference on Neural Networks (IJCNN). Glasgow, UK, pp 1-8. https://doi.org/10.1109/ijcnn48605.2020.9207336

  42. Guo H, Li Y, Jennifer S, Gu M, Huang Y, Gong B (2017) Learning from class-imbalanced data: review of methods and applications. Expert Syst Appl 73:220–239. https://doi.org/10.1016/j.eswa.2016.12.035

    Article  Google Scholar 

  43. Gao L, Zhang L, Liu C, Wu S (2020) Handling imbalanced medical image data: a deep-learning-based one-class classification approach. Artif Intell Med 108:101935. https://doi.org/10.1016/j.artmed.2020.101935

    Article  Google Scholar 

  44. Li J, Du Q, Li Y, Li W (2018) Hyperspectral image classification with imbalanced data based on orthogonal complement subspace projection. IEEE Trans Geosci Remote Sens 56(7):3838–3851. https://doi.org/10.1109/tgrs.2018.2813366

    Article  Google Scholar 

  45. Duan A, Guo L, Gao H, Wu X, Dong X (2020) Deep focus parallel convolutional neural network for imbalanced classification of machinery fault diagnostics. IEEE Trans Instrumentation Measure 69(11):8680–8689. https://doi.org/10.1109/tim.2020.2998233

    Article  Google Scholar 

  46. Chen J, Huang H, Cohn AG, Zhou M, Zhang D, Man J (2022) A hierarchical DCNN-based approach for classifying imbalanced water inflow in rock tunnel faces. Tunnell Underground Space Technol 122:104399. https://doi.org/10.1016/j.tust.2022.104399

    Article  Google Scholar 

  47. Liu J, Guo F, Gao H, Huang Z, Zhang Y, Zhou H (2021) Image classification method on class imbalance datasets using multi-scale CNN and two-stage transfer learning. Neural Comput Appl 33(21):14179–14197. https://doi.org/10.1007/s00521-021-06066-8

    Article  Google Scholar 

  48. Nie Y, Zamzam AS, Brandt A (2021) Resampling and data augmentation for short-term PV output prediction based on an imbalanced sky images dataset using convolutional neural networks. Solar Energy 224:341–354. https://doi.org/10.1016/j.solener.2021.05.095

    Article  Google Scholar 

  49. Shamsolmoali P, Zareapoor M, Shen L, Sadka AH, Yang J (2021) Imbalanced data learning by minority class augmentation using capsule adversarial networks. Neurocomputing 459:481–493. https://doi.org/10.1016/j.neucom.2020.01.119

    Article  Google Scholar 

  50. Guo R, Liu H, Xie G, Zhang Y (2021) Weld defect detection from imbalanced radiographic images based on contrast enhancement conditional generative adversarial network and transfer learning. IEEE Sensors J 21(9):10844–10853. https://doi.org/10.1109/jsen.2021.3059860

    Article  Google Scholar 

  51. Dablain D, Krawczyk B, Chawla NV (2022) DeepSMOTE: fusing deep learning and SMOTE for imbalanced data. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/tnnls.2021.3136503

    Article  Google Scholar 

  52. Zhang C, Tan KC, Li H, Hong GS (2019) A cost-sensitive deep belief network for imbalanced classification. IEEE Trans Neural Netw Learn Syst 30(1):109–122. https://doi.org/10.1109/tnnls.2018.2832648

    Article  Google Scholar 

  53. Huang C, Li Y, Loy CC, Tang X (2020) Deep imbalanced learning for face recognition and attribute prediction. IEEE Trans Pattern Anal Mach Intell 42(11):2781–2794. https://doi.org/10.1109/tpami.2019.2914680

    Article  Google Scholar 

  54. Fuqua D, Razzaghi T (2020) A cost-sensitive convolution neural network learning for control chart pattern recognition. Expert Syst Appl 150:113275. https://doi.org/10.1016/j.eswa.2020.113275

    Article  Google Scholar 

  55. Zhang J, Xing M, Sun GC, Chen J, Li M, Hu Y, Bao Z (2021) Water body detection in high-resolution SAR images with cascaded fully-convolutional network and variable focal loss. IEEE Trans Geosci Remote Sens 59(1):316–332. https://doi.org/10.1109/tgrs.2020.2999405

    Article  Google Scholar 

  56. Ribeiro VHA, Reynoso-Meza G (2020) Ensemble learning by means of a multi-objective optimization design approach for dealing with imbalanced data sets. Expert Syst Appl 147:113232. https://doi.org/10.1016/j.eswa.2020.113232

    Article  Google Scholar 

  57. Carniato L (2022) Multi-Class classification using Focal Loss and LightGBM. Towards Data Science. https://towardsdatascience.com/multi-class-classification-using-focal-loss-and-lightgbm-a6a6dec28872. Accessed 13 Sep 2022

  58. Max Halford (2020). Focal loss implementation for LightGBM. https://maxhalford.github.io/blog/lightgbm-focal-loss/#first-order-derivative

  59. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  60. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Las Vegas, NV, USA, pp 770–778

  61. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp 10012-10022

  62. Xie S, Girshick R, Dollar P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, Hawaii, USA, pp 1492–1500

  63. Dua D, Graff C (2017) UCI machine learning repository.http://archive.ics.uci.edu/ml

  64. Vanschoren J, Van RJN, Bischl B, Torgo L (2014) OpenML: networked science in machine learning. ACM SIGKDD Explor Newslett 15(2):49–60. https://doi.org/10.1145/2641190.2641198

    Article  Google Scholar 

  65. Jurman G, Riccadonna S, Furlanello C (2012) A comparison of MCC and CEN error measures in multi-class prediction. PLoS ONE 7(8):e41882–e41882. https://doi.org/10.1371/journal.pone.0041882

    Article  Google Scholar 

  66. Bergstra J, Komer B, Eliasmith C, Yamins D, Cox DD (2015) Hyperopt: a python library for model selection and hyperparameter optimization. Comput Sci Discov 8(1):014008. https://doi.org/10.1088/1749-4699/8/1/014008

    Article  Google Scholar 

  67. Mourtada J, Gaïffas S, Scornet E (2021) AMF: aggregated Mondrian forests for online learning. J R Stat Soc: Ser B (Stat Methodol) 83(3):505–533. https://doi.org/10.1111/rssb.12425

    Article  MATH  MathSciNet  Google Scholar 

  68. Reis I, Baron D, Shahaf S (2019) Probabilistic random forest: a machine learning algorithm for noisy data sets. Astron J 157:16. https://doi.org/10.3847/1538-3881/aaf101

    Article  Google Scholar 

  69. Sprangers O, Schelter S, de Rijke M (2021) Probabilistic gradient boosting machines for large-scale probabilistic regression. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. Association for Computing Machinery, New York, NY, USA, pp 1510-1520. https://doi.org/10.1145/3447548.3467278

Download references

Acknowledgements

The authors would like to acknowledge financial support from the National Natural Science Foundation of China (61903075), and the Guangdong Basic and Applied Basic Research Foundation (2022A1515140126, 2023A1515011172), and the Young and Middle-aged Science and Technology Innovation Talent of Shenyang (RC220485).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Junyi Wang.

Ethics declarations

Conflicts of interest

The authors declare no conflict of interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, X., Wang, J., Meng, Q. et al. An adaptive multi-class imbalanced classification framework based on ensemble methods and deep network. Neural Comput & Applic 35, 11141–11159 (2023). https://doi.org/10.1007/s00521-023-08290-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08290-w

Keywords

Navigation