Addressing Local Class Imbalance in Balanced Datasets with Dynamic Impurity Decision Trees

Abstract

Decision trees are among the most popular machine learning algorithms, due to their simplicity, versatility, and interpretability. Their underlying principle revolves around the recursive partitioning of the feature space into disjoint subsets, each of which should ideally contain only a single class. This is achieved by selecting features and conditions that allow for the most effective split of the tree structure. Traditionally, impurity metrics are used to measure the effectiveness of a split, as ideally in a given subset only instances from a single class should be present. In this paper, we discuss the underlying shortcoming of such an assumption and introduce the notion of local class imbalance. We show that traditional splitting criteria induce the emergence of increasing class imbalances as the tree structure grows. Therefore, even when dealing with initially balanced datasets, class imbalance will become a problem during decision tree induction. At the same time, we show that existing skew-insensitive split criteria return inferior performance when data is roughly balanced. To address this, we propose a simple, yet effective hybrid decision tree architecture that is capable of dynamically switching between standard and skew-insensitive splitting criterion during decision tree induction. Our experimental study depicts that local class imbalance is embedded in most standard classification problems and that the proposed hybrid approach is capable of alleviating its influence.