Abstract
Data imbalance is one of the most difficult problems in machine learning. The improved ensemble learning model is a promising solution to mitigate this challenge. In this paper, an improved multi-class imbalanced data classification framework is proposed by combining the Focal Loss with Boosting model (FL-Boosting). By addressing the confusion of the second-order derivation of Focal Loss in the traditional ensemble learning model, the proposed model achieves a more efficient and accurate classification of the imbalanced data. More specifically, a Highly Adaptive Focal Loss (HAFL) is proposed to ensure that the model maintains lasting attention to the minority samples, which could be combined with boosting model to build HAFL-Boosting to achieve better performance. The framework has the scalability to adapt to different situations according to typical ensemble learning algorithms such as LightGBM, XGBoost and CatBoost. In addition, to implement the application of the proposed framework on deep models, a two-stage classification method combining ConvNeXt with the improved boosting model is proposed, which could improve the recognition ability to high-dimensional imbalanced data. We evaluate the HAFL-Boosting and the two-stage class imbalance classification method by ablation experiments and benchmark experiments, which demonstrated that the proposed methods obviously improved the scores on several evaluation indexes. The comparative experiments with the latest classification models show that the proposed methods could achieve leading performance from multiple perspectives.
Similar content being viewed by others
Availability of data and materials
Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.
References
Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Ye Q, Liu TY (2017) Lightgbm: a highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst 30:3146–3154
Chen T, Guestrin C (2016) XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA, pp 785-794. https://doi.org/10.1145/2939672.2939785
Dorogush AV, Ershov V, Gulin A (2018) CatBoost: gradient boosting with categorical features support. http://arxiv.org/abs/1810.11363
Tyralis H, Papacharalampous G (2021) Boosting algorithms in energy research: a systematic review. Neural Comput Appl 33(21):14101–14117. https://doi.org/10.1007/s00521-021-05995-8
Wang J, Jiang X, Meng Q, Saada M, Cai H (2022) Walking motion real-time detection method based on walking stick, IoT, COPOD and improved LightGBM. Appl Intell 45:1–19. https://doi.org/10.1007/s10489-022-03264-2
Li S, Zhang X (2020) Research on orthopedic auxiliary classification and prediction model based on XGBoost algorithm. Neural Comput Appl 32(7):1971–1979. https://doi.org/10.1007/s00521-019-04378-4
Jabeur SB, Gharib C, Mefteh-Wali S, Arfi WB (2021) CatBoost model and artificial intelligence techniques for corporate failure prediction. Technol Forecast Soc Change 166:120658. https://doi.org/10.1016/j.techfore.2021.120658
Taherkhani A, Cosma G, McGinnity TM (2020) AdaBoost-CNN: An adaptive boosting algorithm for convolutional neural networks to classify multi-class imbalanced datasets using transfer learning. Neurocomputing 404:351–366
Yu L, Zhou R, Tang L, Chen R (2018) A DBN-based resampling SVM ensemble learning paradigm for credit classification with imbalanced data. Appl Soft Comput 69:192–202. https://doi.org/10.1016/j.asoc.2018.04.049
Niu K, Zhang Z, Liu Y, Li R (2020) Resampling ensemble model based on data distribution for imbalanced credit risk evaluation in P2P lending. Inf Sci 536:120–134. https://doi.org/10.1016/j.ins.2020.05.040
Koziarski M, Woźniak M, Krawczyk B (2020) Combined cleaning and resampling algorithm for multi-class imbalanced data with label noise. Knowl-Based Syst 204:106223. https://doi.org/10.1016/j.knosys.2020.106223
Lin TY, Goyal P, Girshick R, He K, Dollár P (2020) Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 42(2):318–327. https://doi.org/10.1109/tpami.2018.2858826
Wang C, Deng C, Wang S (2020) Imbalance-XGBoost: leveraging weighted and focal losses for binary label-imbalanced classification with XGBoost. Pattern Recognit Lett 136:190–197. https://doi.org/10.1016/j.patrec.2020.05.035
Cao J, Wang D, Qu Z, Cui M, Xu P, Xue K, Hu K (2020) A novel false data injection attack detection model of the cyber-physical power system. IEEE Access 8:95109–95125. https://doi.org/10.1109/access.2020.2995772
Buda M, Maki A, Mazurowski MA (2018) A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw 106:249–259. https://doi.org/10.1016/j.neunet.2018.07.011
Lin E, Chen Q, Qi X (2020) Deep reinforcement learning for imbalanced classification. Appl Intell 50(8):2488–2502. https://doi.org/10.1007/s10489-020-01637-z
Sun C, Cui H, Zhou W, Nie W, Wang X, Yuan Q (2019) Epileptic seizure detection with EEG textural features and imbalanced classification based on EasyEnsemble learning. Int J Neural Syst 29(10):1950021. https://doi.org/10.1142/s0129065719500217
Liu Z, Mao H, Wu CY, Feichtenhofer C, Darrell T, Xie S (2022) A ConvNet for the 2020s. http://arxiv.org/abs/2201.03545
Bader-El-Den M, Teitei E, Perry T (2019) Biased random forest for dealing with the class imbalance problem. IEEE Trans Neural Netw Learn Syst 30(7):2163–2172. https://doi.org/10.1109/tnnls.2018.2878400
Yuan X, Chen S, Sun C, Yuwen L (2022) A novel early diagnostic framework for chronic diseases with class imbalance. Sci Rep 12(1):1–16. https://doi.org/10.1038/s41598-022-12574-x
Siddiqui MK, Huang X, Morales-Menendez R, Hussain N, Khatoon K (2020) Machine learning based novel cost-sensitive seizure detection classifier for imbalanced EEG data sets. Int J Interactive Des Manuf (IJIDeM) 14(4):1491–1509. https://doi.org/10.1007/s12008-020-00715-3
Guzmán-Ponce A, Sánchez JS, Valdovinos RM, Marcial-Romero JR (2021) DBIG-US: a two-stage under-sampling algorithm to face the class imbalance problem. Expert Syst Appl 168:114301. https://doi.org/10.1016/j.eswa.2020.114301
Lei K, Xie Y, Zhong S, Dai J, Yang M, Shen Y (2020) Generative adversarial fusion network for class imbalance credit scoring. Neural Comput Appl 32(12):8451–8462. https://doi.org/10.1007/s00521-019-04335-1
Temraz M, Keane MT (2022) Solving the class imbalance problem using a counterfactual method for data augmentation. Mach Learn Appl 9:100375. https://doi.org/10.1016/j.mlwa.2022.100375
Kumar S, Biswas SK, Devi D (2019) TLUSBoost algorithm: a boosting solution for class imbalance problem. Soft Comput 23(21):10755–10767. https://doi.org/10.1007/s00500-018-3629-4
Jiang M, Yang Y, Qiu H (2022) Fuzzy entropy and fuzzy support-based boosting random forests for imbalanced data. Appl Intell 52(4):4126–4143. https://doi.org/10.1007/s10489-021-02620-y
Shen F, Zhao X, Kou G, Alsaadi FE (2021) A new deep learning ensemble credit risk evaluation model with an improved synthetic minority oversampling technique. Appl Soft Comput 98:106852. https://doi.org/10.1016/j.asoc.2020.106852
Zhang W, Ramezani R, Naeim A (2019) WOTBoost: Weighted Oversampling Technique in Boosting for imbalanced learning. In: 2019 IEEE International Conference on Big Data (Big Data). pp 2523-2531. https://doi.org/10.1109/bigdata47090.2019.9006091.
Wang W, Sun D (2021) The improved AdaBoost algorithms for imbalanced data classification. Inf Sci 563:358–374. https://doi.org/10.1016/j.ins.2021.03.042
Jiang X, Xu Y, Ke W, Zhang Y, Zhu QX, He YL (2022) An imbalanced multi-fault diagnosis method based on bias weights AdaBoost. IEEE Trans Instrumentation Measure 5:45. https://doi.org/10.1109/TIM.2022.3149097
Hsu CC, Wang KS, Chung HY, Chang SH (2019) Equation of SVM-rebalancing: the point-normal form of a plane for class imbalance problem. Neural Comput Appl 31(10):6013–6025. https://doi.org/10.1007/s00521-018-3419-z
Ganaie MA, Tanveer M (2021) Fuzzy least squares projection twin support vector machines for class imbalance learning. Appl Soft Comput 113:107933. https://doi.org/10.1016/j.asoc.2021.107933
Hossain MS, Betts JM, Paplinski AP (2021) Dual Focal Loss to address class imbalance in semantic segmentation. Neurocomputing 462:69–87. https://doi.org/10.1016/j.neucom.2021.07.055
Yeung M, Sala E, Schönlieb CB, Rundo L (2022) Unified focal loss: Generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Computerized Med Imag Gr 95:102026. https://doi.org/10.1016/j.compmedimag.2021.102026
Zhang X, Wan F, Liu C, Ji X, Ye Q (2021) Learning to match anchors for visual object detection. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/tpami.2021.3050494
Chen J, Wu Z, Zhang J (2019) Driving safety risk prediction using cost-sensitive with nonnegativity-constrained autoencoders based on imbalanced naturalistic driving data. IEEE Trans Intell Transp Syst 20(12):4450–4465. https://doi.org/10.1109/tits.2018.2886280
Shon HS, Batbaatar E, Kim KO, Cha EJ, Kim K-A (2020) Classification of kidney cancer data using cost-sensitive hybrid deep learning approach. Symmetry 12(1):154. https://doi.org/10.3390/sym12010154
Mulyanto M, Faisal M, Prakosa S, Leu J (2020) Effectiveness of focal loss for minority classification in network intrusion detection systems. Symmetry 13(1):4. https://doi.org/10.3390/sym13010004
Zhao Y, Shen C, Wang H, Chen S (2019) Structural analysis of attributes for vehicle re-identification and retrieval. IEEE Trans Intell Transp Syst 21(2):723–734. https://doi.org/10.1109/tits.2019.2896273
Xu X, Duan H, Guo Y, Deng Y (2020) A cascade adaboost and CNN algorithm for drogue detection in UAV autonomous aerial refueling. Neurocomputing 408:121–134. https://doi.org/10.1016/j.neucom.2019.10.115
Guo Y, Li Z, Li Z, Xiong G, Jiang M, Gou G (2020) FLAGB: focal loss based adaptive gradient boosting for imbalanced traffic classification. In: 2020 International Joint Conference on Neural Networks (IJCNN). Glasgow, UK, pp 1-8. https://doi.org/10.1109/ijcnn48605.2020.9207336
Guo H, Li Y, Jennifer S, Gu M, Huang Y, Gong B (2017) Learning from class-imbalanced data: review of methods and applications. Expert Syst Appl 73:220–239. https://doi.org/10.1016/j.eswa.2016.12.035
Gao L, Zhang L, Liu C, Wu S (2020) Handling imbalanced medical image data: a deep-learning-based one-class classification approach. Artif Intell Med 108:101935. https://doi.org/10.1016/j.artmed.2020.101935
Li J, Du Q, Li Y, Li W (2018) Hyperspectral image classification with imbalanced data based on orthogonal complement subspace projection. IEEE Trans Geosci Remote Sens 56(7):3838–3851. https://doi.org/10.1109/tgrs.2018.2813366
Duan A, Guo L, Gao H, Wu X, Dong X (2020) Deep focus parallel convolutional neural network for imbalanced classification of machinery fault diagnostics. IEEE Trans Instrumentation Measure 69(11):8680–8689. https://doi.org/10.1109/tim.2020.2998233
Chen J, Huang H, Cohn AG, Zhou M, Zhang D, Man J (2022) A hierarchical DCNN-based approach for classifying imbalanced water inflow in rock tunnel faces. Tunnell Underground Space Technol 122:104399. https://doi.org/10.1016/j.tust.2022.104399
Liu J, Guo F, Gao H, Huang Z, Zhang Y, Zhou H (2021) Image classification method on class imbalance datasets using multi-scale CNN and two-stage transfer learning. Neural Comput Appl 33(21):14179–14197. https://doi.org/10.1007/s00521-021-06066-8
Nie Y, Zamzam AS, Brandt A (2021) Resampling and data augmentation for short-term PV output prediction based on an imbalanced sky images dataset using convolutional neural networks. Solar Energy 224:341–354. https://doi.org/10.1016/j.solener.2021.05.095
Shamsolmoali P, Zareapoor M, Shen L, Sadka AH, Yang J (2021) Imbalanced data learning by minority class augmentation using capsule adversarial networks. Neurocomputing 459:481–493. https://doi.org/10.1016/j.neucom.2020.01.119
Guo R, Liu H, Xie G, Zhang Y (2021) Weld defect detection from imbalanced radiographic images based on contrast enhancement conditional generative adversarial network and transfer learning. IEEE Sensors J 21(9):10844–10853. https://doi.org/10.1109/jsen.2021.3059860
Dablain D, Krawczyk B, Chawla NV (2022) DeepSMOTE: fusing deep learning and SMOTE for imbalanced data. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/tnnls.2021.3136503
Zhang C, Tan KC, Li H, Hong GS (2019) A cost-sensitive deep belief network for imbalanced classification. IEEE Trans Neural Netw Learn Syst 30(1):109–122. https://doi.org/10.1109/tnnls.2018.2832648
Huang C, Li Y, Loy CC, Tang X (2020) Deep imbalanced learning for face recognition and attribute prediction. IEEE Trans Pattern Anal Mach Intell 42(11):2781–2794. https://doi.org/10.1109/tpami.2019.2914680
Fuqua D, Razzaghi T (2020) A cost-sensitive convolution neural network learning for control chart pattern recognition. Expert Syst Appl 150:113275. https://doi.org/10.1016/j.eswa.2020.113275
Zhang J, Xing M, Sun GC, Chen J, Li M, Hu Y, Bao Z (2021) Water body detection in high-resolution SAR images with cascaded fully-convolutional network and variable focal loss. IEEE Trans Geosci Remote Sens 59(1):316–332. https://doi.org/10.1109/tgrs.2020.2999405
Ribeiro VHA, Reynoso-Meza G (2020) Ensemble learning by means of a multi-objective optimization design approach for dealing with imbalanced data sets. Expert Syst Appl 147:113232. https://doi.org/10.1016/j.eswa.2020.113232
Carniato L (2022) Multi-Class classification using Focal Loss and LightGBM. Towards Data Science. https://towardsdatascience.com/multi-class-classification-using-focal-loss-and-lightgbm-a6a6dec28872. Accessed 13 Sep 2022
Max Halford (2020). Focal loss implementation for LightGBM. https://maxhalford.github.io/blog/lightgbm-focal-loss/#first-order-derivative
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Las Vegas, NV, USA, pp 770–778
Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp 10012-10022
Xie S, Girshick R, Dollar P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, Hawaii, USA, pp 1492–1500
Dua D, Graff C (2017) UCI machine learning repository.http://archive.ics.uci.edu/ml
Vanschoren J, Van RJN, Bischl B, Torgo L (2014) OpenML: networked science in machine learning. ACM SIGKDD Explor Newslett 15(2):49–60. https://doi.org/10.1145/2641190.2641198
Jurman G, Riccadonna S, Furlanello C (2012) A comparison of MCC and CEN error measures in multi-class prediction. PLoS ONE 7(8):e41882–e41882. https://doi.org/10.1371/journal.pone.0041882
Bergstra J, Komer B, Eliasmith C, Yamins D, Cox DD (2015) Hyperopt: a python library for model selection and hyperparameter optimization. Comput Sci Discov 8(1):014008. https://doi.org/10.1088/1749-4699/8/1/014008
Mourtada J, Gaïffas S, Scornet E (2021) AMF: aggregated Mondrian forests for online learning. J R Stat Soc: Ser B (Stat Methodol) 83(3):505–533. https://doi.org/10.1111/rssb.12425
Reis I, Baron D, Shahaf S (2019) Probabilistic random forest: a machine learning algorithm for noisy data sets. Astron J 157:16. https://doi.org/10.3847/1538-3881/aaf101
Sprangers O, Schelter S, de Rijke M (2021) Probabilistic gradient boosting machines for large-scale probabilistic regression. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. Association for Computing Machinery, New York, NY, USA, pp 1510-1520. https://doi.org/10.1145/3447548.3467278
Acknowledgements
The authors would like to acknowledge financial support from the National Natural Science Foundation of China (61903075), and the Guangdong Basic and Applied Basic Research Foundation (2022A1515140126, 2023A1515011172), and the Young and Middle-aged Science and Technology Innovation Talent of Shenyang (RC220485).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare no conflict of interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Jiang, X., Wang, J., Meng, Q. et al. An adaptive multi-class imbalanced classification framework based on ensemble methods and deep network. Neural Comput & Applic 35, 11141–11159 (2023). https://doi.org/10.1007/s00521-023-08290-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-023-08290-w