Skip to main content
SpringerLink
Log in

The effect of varying levels of class distribution on bagging for different algorithms: An empirical study

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Many real world applications involve highly imbalanced class distribution. Research into learning from imbalanced class distribution is considered to be one of ten challenging problems in data mining research, and it has increasingly captured the attention of both academia and industry. In this work, we study the effects of different levels of imbalanced class distribution on bagging predictors by using under-sampling techniques. Despite the popularity of bagging in many real-world applications, some questions have not been clearly answered in the existing research, such as the effect of varying the levels of class distribution on different bagging predictors, e.g., whether bagging is superior to single learners when the levels of class distribution change. Most classification learning algorithms are designed to maximize the overall accuracy rate and assume that training instances are uniformly distributed; however, the overall accuracy does not represent correct prediction on the minority class, which is the class of interest to users. The overall accuracy metric is therefore ineffective for evaluating the performance of classifiers in extremely imbalanced data. This study investigates the effect of varying levels of class distribution on different bagging predictors based on the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC) as a performance metric, using an under-sampling technique on 14 data-sets with imbalanced class distributions. Our experimental results indicate that Decision Table (DTable) and RepTree are the learning algorithms with the best bagging AUC performance. The AUC performances of bagging predictors are statistically superior to single learners, with the exception of Support Vector Machines (SVM) and Decision Stump (DStump).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Batista GE, Prati RC, Monard MC (2004) A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explor Newsl 6(1):20–29

    Article  Google Scholar 

  2. Boehm O, Hardoon DR, Manevitz LM (2011) Classifying cognitive states of brain activity via one-class neural networks with feature selection by genetic algorithms. Int J Mach Learn Cybern 2:1–10

    Google Scholar 

  3. Bradley AP (1997) The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recogn 30(7):1145–1159

    Article  Google Scholar 

  4. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    MATH  MathSciNet  Google Scholar 

  5. Bunkhumpornpat C, Sinapiromsaran K, Lursinsap C (2009) Safe-level-SMOTE: Safe-Level-Synthetic Minority Over-Sampling Technique for handling the class imbalanced problem. In: Advances in Knowledge Discovery and Data Mining, vol 5476, pp 475–482

  6. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16(1):321–357

    MATH  Google Scholar 

  7. Chawla NV, Lazarevic A, Hall LO, Bowyer KW (2003) SMOTEBoost: improving prediction of the minority class in boosting. Paper presented at the PKDD 2003:107–119

    Google Scholar 

  8. Chawla NV, Japkowicz N, Kotcz A (2004) Editorial: special issue on learning from imbalanced data sets. ACM SIGKDD Explor Newsl 6(1):1–6

    Article  Google Scholar 

  9. Cieslak D, Chawla N (2008) Learning decision trees for unbalanced data. Paper presented at the ECML PKDD 2008, pp 241–256

  10. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MATH  MathSciNet  Google Scholar 

  11. Estabrooks A, Jo T, Japkowicz N (2004) A multiple resampling method for learning from imbalanced data sets. Comput Intell 20(1):18–36

    Article  MathSciNet  Google Scholar 

  12. Fawcett T (2004) ROC graphs: Notes and practical considerations for researchers. Mach Learn 31 (HPL-2003-4):1–38

  13. Fawcett T (2006) An introduction to ROC analysis. Pattern Recogn Lett 27(8):861–874

    Article  MathSciNet  Google Scholar 

  14. Guoxun H, Hui H, Wenyuan W (2005) An Over-sampling expert system for learning from imbalanced data sets. Paper presented at the International Conference on Neural Networks and Brain, 13–15 Oct, 537–541

  15. Han H, Wang WY, Mao BH (2005) Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning. In: Advances in Intelligent Computing, vol 3644. Lecture notes in Computer Science, pp 878–887

  16. He H, Garcia AE (2009) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21(9):1263–1284

    Article  Google Scholar 

  17. Hido S, Kashima H, Takahashi Y (2009) Roughly balanced bagging for imbalanced data. Stat Anal Data Min 2(5–6):412–426

    Article  MathSciNet  Google Scholar 

  18. Jun W, Shitong W, Chung F (2011) Positive and negative fuzzy rule system, extreme learning machine and image classification. Int J Mach Learn Cybern: 1–11

  19. Kang P, Cho S (2006) EUS SVMs: Ensemble of under-sampled SVMs for data imbalance problems. Paper presented at the Neural Information Processing, pp 837–846

  20. Koknar-Tezel S, Latecki LJ (2009) Improving SVM classification on imbalanced data sets in distance spaces. Paper presented at the 9th IEEE International Conference on Data Mining, 6–9 Dec, pp 259–267

  21. Kotsiantis S, Kanellopoulos D, Pintelas P (2006) Handling imbalanced datasets: a review. Int Trans Comput Sci Eng 30(1):25–36

    Google Scholar 

  22. Liang G (2012) An investigation of sensitivity on bagging predictors: An empirical approach. Paper presented at the 26th AAAI Conference on Artificial Intelligence, AAAI 2012, Toronto, 22–26 July, 2439–2440

  23. Liang G, Zhang C (2011) Empirical study of bagging predictors on medical data. Paper presented at the 9th Australian Data Mining Conference, AusDM 2011, Ballarat, Australia, pp 31–40

  24. Liang G, Zhang C (2011) An empirical evaluation of bagging with different learning algorithms on imbalanced data. In: Proceedings of the 7th International Conference on Advanced Data Mining and Applications, ADMA 2011. 339–352

  25. Liang G, Zhang C (2012) An efficient and simple under-sampling technique for imbalanced time series classification. Paper presented at the ACM 21st Conference on Information and Knowledge Management, CIKM 2012, Maui Hawaii, 29th October – 2nd November

  26. Liang G, Zhang C (2012) A comparative study of sampling methods and algorithms for imbalanced time series classification. Paper presented at the 25th Australasian Joint Conference on Artificial Intelligence, AI 2012, Sydney, 4th–7th December

  27. Liang G, Zhu X, Zhang C (2011a) An empirical study of bagging predictors for different learning algorithms. Paper presented at the 25th AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, 7–11 August, 1802–1803

  28. Liang G, Zhu X, Zhang C (2011b) An empirical study of bagging predictors for imbalanced data with different levels of class distribution. Paper presented at the 24th Australasian Joint Conference on Artificial Intelligence, AI 2011, Perth, 5th–8th December, 213–222

  29. Ling C, Huang J, Zhang H (2003) AUC: A better measure than accuracy in comparing learning algorithms. Paper presented at the AI 2003, pp 329–341

  30. Liu XY, Wu J, Zhou ZH (2009) Exploratory undersampling for class imbalance learning. IEEE Trans Syst Man Cybern B Cybern 39(2):539–550

    Article  Google Scholar 

  31. Liu W, Chawla S, Cieslak DA, Chawla NV (2010) A robust decision tree algorithm for imbalanced data sets. Paper presented at the SIAM International Conference on Data Mining, SDM 2010, Columbus, Ohio, USA, April 29–May 1, pp 766–777

  32. Maloof M (2003) Learning when data sets are imbalanced and when costs are unequal and unknown. In: Proceedings of the ICML-2003 Workshop on Learning from Imbalanced Data Sets II, Washington

  33. Mazurowski MA, Habas PA, Zurada JM, Lo JY, Baker JA, Tourassi GD (2008) Training neural network classifiers for medical decision making: the effects of imbalanced datasets on classification performance. Neural Netw 21(2–3):427–436

    Article  Google Scholar 

  34. Melville P, Mooney RJ (2005) Creating diversity in ensembles using artificial data. Inform Fusion 6(1):99–111

    Article  Google Scholar 

  35. Mena L, Gonzalez J (2006) Machine learning for imbalanced datasets: Application in medical diagnostic. Paper presented at the Proceedings of the 19th International FLAIRS Conference, pp 574–579

  36. Merz C, Murphy P (2006) UCI Repository of Machine Learning Databases. http://archive.ics.uci.edu/ml/

  37. Phua C, Alahakoon D, Lee V (2004) Minority report in fraud detection: classification of skewed data. ACM SIGKDD Explor Newsl 6(1):50–59

    Article  Google Scholar 

  38. Provost F, Fawcett T, Kohavi R (1998) The case against accuracy estimation for comparing induction algorithms. Paper presented at the 15th International Conference on Machine Learning, pp 445–453

  39. Rao RB, Krishnan S, Niculescu RS (2006) Data mining for improved cardiac care. ACM SIGKDD Explor Newsl 8(1):3–10

    Article  Google Scholar 

  40. Su CT, Hsiao YH (2007) An evaluation of the robustness of MTS for imbalanced data. IEEE Trans Knowl Data Eng:1321–1332

  41. Sun Y, Kamel M, Wong A, Wang Y (2007) Cost-sensitive boosting for classification of imbalanced data. Pattern Recogn 40(12):3358–3378

    Article  MATH  Google Scholar 

  42. Wang X, Wang Y, Xu X, Ling W, Yeung DS (2001) A new approach to fuzzy rule generation: fuzzy extension matrix. Fuzzy Sets Syst 123(3):291–306

    Article  MATH  MathSciNet  Google Scholar 

  43. Wang XZ, He Q, Chen DG, Yeung D (2005) A genetic algorithm for solving the inverse problem of support vector machines. Neurocomputing 68:225–238

    Article  Google Scholar 

  44. Weiss GM (2004) Mining with rarity: a unifying framework. ACM SIGKDD Explor Newsl 6(1):7–19

    Article  Google Scholar 

  45. Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco

    Google Scholar 

  46. Wu X, Kumar V, Ross Quinlan J, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Yu PS (2008) Top 10 algorithms in data mining. Knowl Inf Syst 14(1):1–37

    Article  Google Scholar 

  47. Yang Q, Wu X (2006) 10 challenging problems in data mining research. Int J Inform Technol Decis Mak 5(4):597–604

    Article  Google Scholar 

  48. Zeng-Chang Q (2005) ROC analysis for predictions made by probabilistic classifiers. Paper presented at the International Conference on Machine Learning and Cybernetics 18–21 Aug. 2005, pp 3119–3124

Download references

Author information

Authors and Affiliations

  1. Centre for Quantum Computation & Intelligent Systems, FEIT, University of Technology, Sydney, NSW, 2007, Australia

    Guohua Liang, Xingquan Zhu & Chengqi Zhang

  2. Centre for Cardiovascular and Chronic Care, Faculty of Health, University of Technology, Sydney, NSW, 2007, Australia

    Guohua Liang

Authors
  1. Guohua Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingquan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengqi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guohua Liang.

Additional information

This paper was presented in part at the Proceedings of the 24th Australian Joint Conference on Artificial Intelligence (AI 2011), Western Australia, December 2011 [28].

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liang, G., Zhu, X. & Zhang, C. The effect of varying levels of class distribution on bagging for different algorithms: An empirical study. Int. J. Mach. Learn. & Cyber. 5, 63–71 (2014). https://doi.org/10.1007/s13042-012-0125-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-012-0125-5

Keywords

Search

Navigation