Approximation trees: statistical reproducibility in model distillation

Abstract

This paper examines the reproducibility of learned explanations for black-box predictions via model distillation using classification trees. We find that common tree distillation methods fail to reproduce a single stable explanation when applied to the same teacher model due the randomness of the distillation process. We study this issue of reliable interpretation and propose a standardized framework for tree distillation to achieve reproducibility. The proposed framework consists of (1) a statistical test to stabilize tree splits, and (2) a stopping rule for tree building when using a teacher that provides an estimate of the uncertainty of its predictions, e.g. random forests. We demonstrate the empirical performance of the proposed distillation method on a variety of synthetic and real-world datasets.

Appendices

Appendix A: Limiting distribution of Gini index

Following the discussion, we now move to analyze a pseudo sample. Using similar notation, we denote the numbers of sample points and the probabilities over class labels in each child by, for $p \in \{1,2\}, j \in \{1,\dots ,k\},$

$$\begin{aligned} n^p_0 = \sum _{i=1}^n \mathbb {1}_{\{G_p(X_i)=0\}}, \quad {\hat{\theta }}^p_{0,j} = \frac{1}{n^p_0} \sum _{i=1}^n Y_i^j\cdot \mathbb {1}_{\{G_p(X_i)=0\}}, \\ n^p_1 = \sum _{i=1}^n \mathbb {1}_{\{G_p(X_i)=1\}}, \quad {\hat{\theta }}^p_{1,j} = \frac{1}{n^p_1} \sum _{i=1}^n Y_i^j\cdot \mathbb {1}_{\{G_p(X_i)=1\}}. \end{aligned}$$

For simplicity we write, for $p \in \{1,2\},q \in \{0,1\},$ $N^p_q = ( n^p_q{\hat{\theta }}^p_{q,1}, \dots , n^p_q{\hat{\theta }}^p_{q,k} )^T.$ Employing a multivariate CLT we obtain

$$\begin{aligned} \sqrt{n} \left( \frac{1}{n}\begin{bmatrix} N^1_0 \\ N^1_1 \\ N^2_0 \\ N^2_1 \end{bmatrix} - \begin{bmatrix} \varTheta ^1_0 \\ \varTheta ^1_1 \\ \varTheta ^2_0 \\ \varTheta ^2_1 \end{bmatrix} \right) \longrightarrow N(0, \varSigma ). \end{aligned}$$

To relate this limiting distribution to the difference of Gini indices we shall employ the $\delta$-method. Consider the analytic function $f: {\mathbb {R}}^{4k} \rightarrow {\mathbb {R}}$ s.t.

$$\begin{aligned} f(x_1, \dots , x_{4k}) =&-\frac{1}{\pi ^1_0}\sum _{i=1}^k x_i^2 -\frac{1}{\pi ^1_1} \sum _{i=k+1}^{2k} x_i^2 \\&+\frac{1}{\pi ^2_0} \sum _{i=2k+1}^{3k} x_i^2 +\frac{1}{\pi ^2_1} \sum _{i=3k+1}^{4k} x_i^2. \end{aligned}$$

The $\delta$-method implies that

$$\begin{aligned} \sqrt{n} \left( f\left( \frac{1}{n}\begin{bmatrix} N^1_0 \\ N^1_1 \\ N^2_0 \\ N^2_1 \end{bmatrix}\right) - f\left( \begin{bmatrix} \varTheta ^1_0 \\ \varTheta ^1_1 \\ \varTheta ^2_0 \\ \varTheta ^2_1 \end{bmatrix}\right) \right) \longrightarrow N(0, \varTheta ^T\varSigma \varTheta ). \end{aligned}$$

(9)

We should point out that while (9) provides us with the CLT we need to assess the difference between two Gini indices. After expanding (9),

$$\begin{aligned} \sqrt{n} \left( ({\hat{g}}_{1,n}-{\hat{g}}_{2,n}) - (g_1 - g_2)\right) \longrightarrow N(0, \varTheta ^T \varSigma \varTheta ). \end{aligned}$$

or approximately,

$$\begin{aligned} ({\hat{g}}_{1,n} - {\hat{g}}_{2,n}) - (g_1 - g_2) \sim N \left( 0, \frac{\varTheta ^T \varSigma \varTheta }{n}\right) . \end{aligned}$$

Appendix B: Additional results on AppTree

1.1 Real datasets

In this section we will show the results of our method on eight datasets. Seven of them are available on the UCI repository (Lichman 2013) and one is the CAD-MDD data used by Gibbons et al. (2013). We manually split each dataset into train and test for cross validation. In the main body, Table 1 shows the number of covariates, training and testing sample size and the levels of responses for each dataset.

To decide the generative distribution of covariates before running our algorithm, we perturb the empirical distribution by Gaussian noise whose variances are approximately 1/50 of the ranges of corresponding covariates. The probability of jumping to a neighboring category for discrete covariates is set to be 1/7.

We compare across four methods here: classification trees (CART), random forests (RF), our proposed approximation tree (AppTree), and a baseline method (BASE). Previous work (Johansson and Niklasson 2009; Johansson et al. 2010) fixes the number of pseudo sample from the oracle during the coaching procedure. Analogously, we set BASE to be a non-adaptive version of our AppTree which requests a pseudo sample from the RF only once at the root node and uses it all the way down. The pseudo sample size is set to be 9 times the size of the training data; this is larger than the samples used in Johansson and Niklasson 2009; Johansson et al. 2010 and is designed as a reasonable blind decision without any prior information.

We use the same setting for all datasets for consistency. For each dataset, we train a RF containing 200 trees, a CART tree, then 100 AppTrees and 100 BASE trees approximating the RF. $N_{ps} = 500,000$, which means each node of AppTree can generate at most $5\times 10^5$ pseudo examples to decide its split. CART, BASE and AppTree all grow to the 6th layer including the root. Confidence level $\alpha$ is set to be 0.1.

1.2 Binary classification

Fig. 12

Performance evaluation on binary classification datasets. From top to bottom: CAD-MDD, BreastCancer, Car, ClimateModel. From left to right: ROC curves, fidelity to RF on testing set, fidelity to RF on new data points

Figure 12 shows the evaluation of methods on binary classification datasets. Johansson et al. (2011) has pointed out that a single decision tree is already capable of mimicking an oracle predictor to make highly accurate predictions given the oracle is not overly complicated. Our simulation shows similar results, as all three methods CART, BASE and AppTree tightly follow the ROC curves of the RF and there is no significance difference among them. Fidelity is measured as the frequency of a model agreeing with the RF when predicting on same input covariates. We use a moving threshold as the classification boundary, and evaluate the fidelity on both the testing data and the new data (as marked “test" and “new" in the plot). First, all three 6-layer trees can approximate the RF with over 80% probability for almost any given classification threshold, and there is no significant difference among them. Further, the behaviors of both “test” and “new” plot seem similar, which lends support to our generative covariate distribution. While the overall 80% fidelity may seem not powerful enough to make those trees aligned with the oracle RF, we can build the trees deeper to better “overfit” the RF.

1.3 Multiclass classification

Fig. 13

Performance evaluation on multiclass (3-class) classification dataset Cardiotocography. ROC curves are plotted in a one versus all fashion. Fidelity is only checked on testing data

Fig. 14

Performance evaluation on multiclass (3-class) classification datasets WineRed (top two rows) and WineWhite (bottom two rows). ROC curves are plotted in a one versus all fashion. Fidelity is only checked on testing data

Figures 13 and 14 demonstrate evaluate performance on the three multiclass classification datasets. We observe similar patterns as we did in binary cases that all three tree methods work similarly. ROC curves of RF show poorer performance than in binary classification, and ROC curves of the three tree methods are further from random forests, suggesting that deeper trees might be necessary. Nonetheless, all tree methods again agree with the RF on about 80% of the predictions made by RF on the test data. We have therefore shown that our reproducibility requirement of AppTree does not undermine its predictive power and fidelity with the black box when compared with other tree methods.

1.4 Impact of $N_{ps}$ on distillation accuracy and fidelity

Table 3 Student accuracy and fidelity metrics on the real world datasets with varying $N_{ps}$

Table 3 reflects the student’s accuracy and fidelity remains unchanged with respect to a changing $N_{ps}$. This behavior is expected and guaranteed by the distillation practice. We anticipate $N_{ps}$ in this case solely plays the role of a reproducibility hyperparameter.

1.5 Smoothing the sampler

Throughout we have assumed that $F_X$—the distribution for generating covariates during approximation tree construction—is fixed by the practitioner. This is to allow differing modeling goals to influence its choice; we might either want to ensure coverage of the whole space of possible feature values, or they may want to localize to a particular patient of interest, an approach not dissimilar to LIME (Ribeiro et al. 2016).

In most of our empirical studies, we have employed a Gaussian kernel smoother of the training data as a means of focusing our approximation near the feature values of interest while providing more coverage of the teacher’s behavior. This choice is subject to two potential critiques: the training data that we use to produce $F_X$ is itself a source of uncertainty, or that simply specifying $F_X$ to be the empirical distribution of the training data we can remove all variability associated with it.

Here we conduct an experiment that shows that using the Gaussian kernel smoothing both produces greater distillation fidelity to the teacher, and reduces the variability inherited from the training data distribution relative to simply using the training data. In particular, we use the following response functions

• Setup 1: $X \sim \text{ Unif }[-1, 1]^3$,

  $$\begin{aligned} \text{ logit }P(y=1|x) = -0.5\cdot \mathbb {1}_{\{x_1 < 0\}} + 0.5 \cdot \sin (\pi x_2 / 2)\cdot \mathbb {1}_{\{x_1 > 0\}}. \end{aligned}$$

• Setup 2: $X \sim \text{ Unif }[-1, 1]^5$,

  $$\begin{aligned}{} & {} \text{ logit }P(y=1|x) = -0.5(1+x_3x_4)\cdot \mathbb {1}_{\{x_1 < 0\}} + 0.5 \cdot (\sin (\pi x_2 / 2)\\ {}{} & {} \quad + \sin (\pi x_5 / 2)\cdot \mathbb {1}_{\{x_1 > 0\}}. \end{aligned}$$

We repeat 100 experiments in each of the two setting, where for each experiment we bootstrap a new sample of 1000 data points from a uniform distribution and either build a standard CART using the new sample and the teacher’s responses or apply the approximation tree algorithm with a kernel smoother using bandwidth $\sigma = 0.2$. Figure 15 confirms that employing the smoother contributes to both reproducibility and faithfulness to the teacher model.

Fig. 15

Comparison between bootstrap sampler and Gaussian kernel smoother in approximation trees when the distillation is given a bootstrap of the original sample distribution. LHS: unique tree structures on two simple synthetic settings. Kernel smoother outputs are much more stable. In each column, a single black bar represents a unique structure of the tree, while the height of the bar represents the number of occurrence of that structure out of 100 repetitions. RHS: prediction accuracy with respect to the teacher model

1.6 Reproducibility of approximation tree for neural networks

Most of our experiments are conducted with random forests, which comes with statistical estimates of sample variability to help guide the stopping rule. But the approximation tree can be used for any model types if we only focus on the first objective of ensuring that the distillation process reliably produces the same splits. In this part we present some results when the teacher model is a neural network.

We use Breast Cancer Data for demonstration purposes, see Sect. 5.4.2 for details. Here we build a neural network of one hidden layer of size 5. Figure 16 shows the reproducibility of AppTree from its top 4 layers with different $N_{ps}$ values, which is the cap on the maximal number of pseudo examples AppTree can generate.

We observe a similar pattern as in Fig. 3: increasing Nps can improve reproducibility, but larger values than are often employed may be needed to make this satisfactory. Compared to random forests as teacher models (Fig. 9), neural networks produce slightly more stable BASE. This is because Breast Cancer Data is a relatively simple classification task for a neural network, which is able to produce class probabilities close to 0 or 1. Our approach can still improve reproducibility of the approximation, resulting in fewer distinct structures.

Fig. 16

Reproducibility of AppTree with different $N_{ps}$ values and a neural network teacher. The top 4 layers with $N_{ps}=5{,}000$ and 50,000 of the trees are summarized respectively. In each column, a single black bar represents a unique structure of the tree, while the height of the bar represents the number of occurrence of that structure out of 100 repetitions

Appendix C: Approximation tree for CAD-MDD data

See Table 4 for full details of the approximation tree for CAD-MDD data.

Table 4 Details of the approximation tree for the CAD-MDD data

Appendix D: Approximation tree for Breast Cancer Data

See Tables 5 and 6 for full details of the approximation tree for Breast Cancer Data. The data set used has 8 attributes and 699 samples.²

Table 5 Feature names correspondence for Breast Cancer Data

Table 6 Details of the approximation tree for Breast Cancer Data