Balancing the trade-off between accuracy and interpretability in software defect prediction

Abstract

Context

Classification techniques of supervised machine learning have been successfully applied to various domains of practice. When building a predictive model, there are two important criteria: predictive accuracy and interpretability, which generally have a trade-off relationship. In particular, interpretability should be accorded greater emphasis in the domains where the incorporation of expert knowledge into a predictive model is required.

Objective

The aim of this research is to propose a new classification model, called superposed naive Bayes (SNB), which transforms a naive Bayes ensemble into a simple naive Bayes model by linear approximation.

Method

In order to evaluate the predictive accuracy and interpretability of the proposed method, we conducted a comparative study using well-known classification techniques such as rule-based learners, decision trees, regression models, support vector machines, neural networks, Bayesian learners, and ensemble learners, over 13 real-world public datasets.

Results

A trade-off analysis between the accuracy and interpretability of different classification techniques was performed with a scatter plot comparing relative ranks of accuracy with those of interpretability. The experiment results show that the proposed method (SNB) can produce a balanced output that satisfies both accuracy and interpretability criteria.

Conclusions

SNB offers a comprehensible predictive model based on a simple and transparent model structure, which can provide an effective way for balancing the trade-off between accuracy and interpretability.

Appendices

Appendix 1

The AIC for the contingency table of Table 1 is derived as follows (Sakamoto and Akaike 1978):

$$ {AIC}_1^{(i)}=\left(-2\right)\sum \limits_{1\le y\le 2}\sum \limits_{1\le x\le k}\left\{{n}_{\left(i,y,x\right)}\log \left({n}_{\left(i,y,x\right)}/{n}_{\left(i,\ast, \ast \right)}\right)\right\}+2\left(2k-1\right) $$

(11)

If mutual independence between Y and X_i is assumed, the AIC changes like the following:

$$ {AIC}_0^{(i)}=\left(-2\right)\sum \limits_{1\le y\le 2}\sum \limits_{1\le x\le k}\left\{{n}_{\left(i,y,x\right)}\log \left({n}_{\left(i,y,\ast \right)}{n}_{\left(i,\ast, x\right)}/{n}_{\left(i,\ast, \ast \right)}^2\right)\right\}+2k $$

(12)

The difference between the AIC of the dependence model (${AIC}_1^{(i)}$) and that of the independence model (${AIC}_0^{(i)}$) is given by

$$ {\displaystyle \begin{array}{l}\ \varDelta {AIC}^{(i)}={AIC}_1^{(i)}-{AIC}_0^{(i)}\\ {}=\left(-2\right)\sum \limits_{1\le y\le 2}\sum \limits_{1\le x\le k}\left\{{n}_{\left(i,y,x\right)}\log \frac{n_{\left(i,\ast, \ast \right)}{n}_{\left(i,y,x\right)}}{n_{\left(i,y,\ast \right)}{n}_{\left(i,\ast, x\right)}}\right\}+2\left(k-1\right)\ \end{array}} $$

(13)

Similarly to the MDL criterion (Fayyad and Irani 1993), the discretization method using AIC starts with a single interval containing all data, and recursively splits intervals. If ΔAIC⁽ⁱ⁾ is greater than a given (negative) threshold, the independence model will be selected as a model with a smaller AIC, so that no more splitting is needed. The total sum of ΔAIC⁽ⁱ⁾ for all the splittings is AIC⁽ⁱ⁾, the reversal of which indicates the importance of the variable X_i. The variables that have positive AIC can be ignored, because the AIC of a single interval variable is equal to zero, as derived from Eq. 11.

The discretization process using BIC is almost same as that based on AIC. ΔBIC⁽ⁱ⁾, i.e., the difference between the BIC of a dependence model (${BIC}_1^{(i)}$) and that of a independence model (${BIC}_0^{(i)}$), is obtained by simply replacing the last term 2(k − 1) in Eq. 13 with logn(k − 1), where n is the number of instances.

$$ {\displaystyle \begin{array}{l}\ \varDelta {BIC}^{(i)}={BIC}_1^{(i)}-{BIC}_0^{(i)}\\ {}=\left(-2\right)\sum \limits_{1\le y\le 2}\sum \limits_{1\le x\le k}\left\{{n}_{\left(i,y,x\right)}\log \frac{n_{\left(i,\ast, \ast \right)}{n}_{\left(i,y,x\right)}}{n_{\left(i,y,\ast \right)}{n}_{\left(i,\ast, x\right)}}\right\}+\log n\left(k-1\right)\end{array}} $$

(14)

The total sum of ΔBIC⁽ⁱ⁾ for all the splittings is BIC⁽ⁱ⁾, the reversal of which indicates the importance of the variable X_i.

Appendix 2

Abbr.	Parameter Settings in Weka machine learning toolkit
OneR	OneR -B 6
JRip	JRip -F 3 -N 2.0 -O 2 -S 1
J48	J48 -C 0.25 -M 2
NBTree	NBTree
LR	Logistic -R 1.0E-8 -M -1
SVM	SMO -C 1.0 -L 0.001 -P 1.0E-12 -N 0 -M -V -1 -W 1 -K “weka.classifiers.functions.supportVector.PolyKernel -C 250007 -E 1.0”
MLP	MultilayerPerceptron -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H a
NBc	NaiveBayes
NBd	NaiveBayes –D
TAN	BayesNet -D -Q weka.classifiers.bayes.net.search.local.TAN -- -S BAYES -E weka.classifiers.bayes.net.estimate SimpleEstimator -- -A 0.5
AODE	FilteredClassifier -F “weka.filter.supervised.attribute.Discretize -R first-last” -W weka.classifiers.bayes.AODE -- -F 1
HNB	FilteredClassifier -F “weka.filter.supervised.attribute.Discretize -R first-last” -W weka.classifiers.bayes.HNB
AdaBst	AdaBoostM1 -P 100 -S 1 -I 100 -W weka.classifiers.trees.DecisionStump
RF	RandomForest -I 100 -K 0 -S 1

Appendix 3

Dataset	Random Seeds used in Weka
	1st CV	2nd CV	3rd CV	4th CV	5th CV
MC2	869	280	820	326	961
KC3	309	433	717	708	321
MW1	154	509	564	850	493
CM1	325	629	602	391	467
PC1	111	501	663	151	200
PC2	474	303	537	936	614
PC3	298	566	445	44	381
PC4	711	652	811	516	832
PC5	543	569	505	684	978
MC1	292	604	61	154	746
JM1	172	283	92	966	398
Bugzilla	559	776	599	493	383
Columba	782	155	941	517	882