Classification and regression using augmented trees

Abstract

In this work, we present an algorithm for regression and classification tasks on big datasets using augmented tree models. Partitioning a big dataset using a tree model permits us to apply a divide and conquer strategy to classification and regression tasks. Experiments conducted as part of this study illustrate that such an approach has an important benefit. Methods associated with good accuracies on learning tasks on big datasets such as ensemble tree methods or neural networks produce models that are not interpretable. The models produced by the proposed algorithm are interpretable while being as accurate as ensemble methods such as random forests or gradient boosted trees. Model interpretation can be performed at coarse and fine granularity. This permits us to extract insights that characterize the entire dataset or a particular subset of the data. Models that are accurate and interpretable are highly desirable in many application settings. The partitions created by the algorithm also permit a divide and conquer approach to model analysis. Analysis of performance by partition helped identify problems such as possible data errors and model overfitting.

Appendix

1.1 Optimality of Bayes classifier

Consider a binary classifier (From [31] and [32]). A classifier is associated with a decision function, $f(X): X \mapsto \{0,1\}$. $X\ \in \mathbb {R}^d$, denotes the input space and $Y \in \{ 0, 1\}$, denotes the output space.

Definition 1

The Bayes classifier $f^*(X)$ is the following mapping:

$$\begin{aligned} f^*(X) = {\left\{ \begin{array}{ll} 1 &{} \quad \eta (X) \ge \frac{1}{2}, \\ 0 &{} \quad \text {otherwise}. \end{array}\right. } \end{aligned}$$

(3)

Here $\eta (x) = \mathbf {P}[Y = 1 | X = x]$.

Theorem 1

The Bayes classifier, $f^*(X): X \mapsto \{0,1\}$ is one that minimizes the probability of misclassification, i.e.,

$$\begin{aligned} \mathbf {P}[f^*(X) \ne Y] \le \mathbf {P}[f(X) \ne Y] \end{aligned}$$

(4)

Proof

$$\begin{aligned} \mathbf {P}[f(X) \ne Y | X = x]&= (1 - \mathbf {P}[Y = f(X)| X = x])\nonumber \\&= (1 - (\mathbf {P}[Y = 1, f(X) = 1 | X = x]) \nonumber \\&\quad +\mathbf {P}[Y = 0, f(X) = 0 | X = x]))\nonumber \\&= ( 1 - (\mathbf {I}_{(f(X) = 1)} \mathbf {P}[Y = 1 | X = x] \nonumber \\&\quad + \mathbf {I}_{(f(X) = 0)} \mathbf {P}[Y = 0 | X = x]))\nonumber \\&= 1 - (\mathbf {I}_{(f(X) = 1)]} \eta (x) \nonumber \\&\quad + \mathbf {I}_{(f(X) = 0)]}(1 -\eta (x))). \end{aligned}$$

(5)

Here $\mathbf {I}_A$ is the indicator function associated with A. Along similar lines, we can express the probability of misclassification associated with the Bayes classifier ($f^*$) as:

$$\begin{aligned} \mathbf {P}[f^*(X) \ne Y | X = x]&= 1 - (\mathbf {I}_{(f*(X) = 1)]} \eta (x) \nonumber \\&\quad + \mathbf {I}_{(f^*(X) = 0)]}(1 -\eta (x))). \end{aligned}$$

(6)

Subtracting Eqs. (6) from (5) and using the relations:

$$\begin{aligned} \mathbf {I}_{(f(X) = 0)} = (1 - \mathbf {I}_{(f(X) = 1)}) \end{aligned}$$

and

$$\begin{aligned} \mathbf {I}_{(f^*(X) = 0)} = (1 - \mathbf {I}_{(f^*(X) = 1)}), \end{aligned}$$

we have:

$$\begin{aligned}&\mathbf {P}[f(X) \ne Y | X = x] - \mathbf {P}[f^*(X) \ne Y | X = x] \nonumber \\&\quad =2 (\eta (x) - 1) ( \mathbf {I}_{(f^*(X) = 1)} - \mathbf {I}_{(f(X) = 1)}) \end{aligned}$$

(7)

Let us examine the right-hand side of Eq. 7:

$$\begin{aligned} 2 (\eta (x) - 1) ( \mathbf {I}_{(f^*(X) = 1)} - \mathbf {I}_{(f(X) = 1)}) \end{aligned}$$

$\eta (X)$ takes values in [0, 1]. There are two cases to consider to evaluate the right-hand side of Eq. (7). These are :

• 1. $0\le \eta (X) < \frac{1}{2}$

• 2. $\frac{1}{2} \le \eta (X) \le 1$.

When we have $0\le \eta (X) < \frac{1}{2}$:

• 1. $2 (\eta (x) - 1) < 0$,

• 2. Applying Eq. (3), $( \mathbf {I}_{(f^*(X) = 1)} - \mathbf {I}_{(f(X) = 1)}) \le 0$.

Therefore, $2 (\eta (x) - 1) ( \mathbf {I}_{(f^*(X) = 1)} - \mathbf {I}_{(f(X) = 1)}) \ge 0$.

When we have $\frac{1}{2} \le \eta (X) \le 1$:

• 1. $2 (\eta (x) - 1) \ge 0$,

• 2. Applying Eq. (3), $( \mathbf {I}_{(f^*(X) = 1)} - \mathbf {I}_{(f(X) = 1)}) \ge 0$.

Therefore, $2 (\eta (x) - 1) ( \mathbf {I}_{(f^*(X) = 1)} - \mathbf {I}_{(f(X) = 1)}) \ge 0$. In other words, the difference in probability of misclassification between any classifier f and the Bayes classifier $f^*$ is always a positive quantity:

$$\begin{aligned} \mathbf {P}[f(X) \ne Y | X = x] - \mathbf {P}[f^*(X) \ne Y | X = x] \ge 0. \end{aligned}$$

This implies that the Bayes classifier, $f^*$, is the optimal classifier. $\square$