Techniques for evaluating fault prediction models

Abstract

Many statistical techniques have been proposed to predict fault-proneness of program modules in software engineering. Choosing the “best” candidate among many available models involves performance assessment and detailed comparison, but these comparisons are not simple due to the applicability of varying performance measures. Classifying a software module as fault-prone implies the application of some verification activities, thus adding to the development cost. Misclassifying a module as fault free carries the risk of system failure, also associated with cost implications. Methodologies for precise evaluation of fault prediction models should be at the core of empirical software engineering research, but have attracted sporadic attention. In this paper, we overview model evaluation techniques. In addition to many techniques that have been used in software engineering studies before, we introduce and discuss the merits of cost curves. Using the data from a public repository, our study demonstrates the strengths and weaknesses of performance evaluation techniques and points to a conclusion that the selection of the “best” model cannot be made without considering project cost characteristics, which are specific in each development environment.

Appendices

Appendix A: Characteristics of NASA MDP dataset

Accurate prediction of fault-proneness in the software development process enables effective identification of modules which are likely to hide faults. Corrective and remedial steps can be adopted early in the development lifecycle before the project encounters costly redevelopment efforts in the later phases. Software projects vary in size and complexity, programming languages, development processes, etc. When reporting a fault prediction modeling experiment, it is important to share the characteristics of the datasets. Here, we outline the characteristics of NASA Metrics Data Prediction (MDP) datasets, which inevitably have a significant impact on fault prediction. Some of our observations seem to be general, i.e., valid across many reported modeling results, while others are specific to this dataset.

First, only a small proportion of software modules are faulty (Menzies et al. 2007)

This is one of the most consistent characteristics of software defect databases. Faulty modules constitute only a small portion of the software product base. Table 1 (Section 1.1) provides a sample of projects, released as a part of NASA Software Metrics Data Program (Metrics Data Program NASA IV&V facility, http://mdp.ivv.nasa.gov/). Module faults were detected either during the development or in the deployment. For project PC1, only 7% of the modules are faulty and 93% are fault-free; KC4 has the largest percentage of faulty modules among all five projects considered here, 48%, but it is also the smallest one (only 125 modules) and the only one that uses a scripting language (Perl). In all other datasets, there are at least three times as many fault free modules than faulty ones. Such a significantly skewed distribution of faulty and non-faulty modules in sample datasets presents a problem for supervised learning algorithms typically utilized in software engineering studies. The reason is that most supervised learning techniques aim to maximize the overall classification accuracy and ignore the class distribution. For example, even if we declare all modules in PC1 as fault-free and misclassify all faulty modules the overall accuracy would achieve 93%. By all means, in the field of predictive models of software fault-proneness, this would be a fantastic result. However, such a model would be useless, as the sole purpose of predictive software quality studies is predicting where faults hide (Menzies et al. 2007).

Second, software metrics used to build predictive models exhibit high correlation (Menzies et al. 2007)

Typically, models use software metrics as prediction variables. These metrics tend to correlate with each other. Take the projects listed in Table 1, for example. Each data set contains twenty-one software metrics, which describe the product size, complexity and some structural properties (Metrics Data Program NASA IV&V facility, http://mdp.ivv.nasa.gov/). Tables 8 and 9 display the Pearson’s correlation coefficients among the predictive variables for projects PC1 and KC2, respectively. Due to space constraint, we only tabulate the correlation between the lines of code (LOC) and five other randomly chosen variables for each project. In the Table 8, the five randomly selected variables and their metric types are: total operands (TOpnd)—Basic Halstead, volume (V)—Derived Halstead, effort estimate (B)—Derived Halstead, lines of code and comment (LCC)—Line Count, and length (N)—Derived Halstead. The variables selected for KC2 (Table 9) are: unique operators (UOp)—Basic Halstead, volume (V)—Derived Halstead, design complexity (IV.G)—McCabe, total operators (TOp)—Basic Halstead, and lines of blank (LOB)—Line Count. For both projects, the listed variables are either moderately or strongly correlated; the majority of the correlation coefficients are above 0.90.

Table 8 Correlation coefficients among six predictive variables in project PC1

Table 9 Correlation coefficients among six predictive variables in project KC2

The phenomenon of high correlation among predictive variables is termed multicolinearity in regression analysis. As Fenton and Neil (1999) state in (Fenton and Neil 1999), multicolinearity produces unacceptable uncertainty in regression coefficient estimates. Specifically, the coefficients can change drastically depending on which terms (that is, metrics) are present in the model and also depending on the order in which they are placed in the model.

Third, many fault-free modules tend to be small in size

The lines of code (LOC) metric is commonly used to measure the size of a module. We calculated the 90th percentile of LOC for faulty and fault-free modules for five MDP projects, and summarized the findings in Table 10. A 90th percentile is a score such that 90% of the scores are below it. For example, for project PC1, about 90% of the fault-prone models have at most 114 lines of codes, while 90% of the non-fault modules have at most 47 lines of codes. Except in KC4, fault-free modules tend to be shorter than faulty modules. But many software metrics depend on size and measurements of relatively small modules tend to be “close” to each other. This may confuse machine learning algorithms and is likely to cause poor prediction results. Koru and Liu (2005) made the same observation and stated that “small modules show little variation, which would make it difficult for a machine-learning algorithm to distinguish between small—defective and small—nondefective modules”. Dealing with such small components appears to be specific for the MDP dataset studies, limiting the generality of conclusions reached by studying projects included in it.

Table 10 The 90th percentile of lines of code (LOC) for the collection of faulty modules and fault-free modules in each project

Fourth, a significant portion of the minority class instances (i.e., fault-prone modules) are “close neighbors” with the majority class instances

Table 11 provides the evidence. For each module in a training set, we find the nearest (in Euclidean distance) training set vector (the module with the most similar metrics) and count the proportion of faulty modules whose nearest neighbor belongs to the majority class (i.e., fault-free modules). Besides, we find three nearest neighbors of each faulty module, and compute the percentage of faulty modules that has at least two neighbors (among three) from the majority class. Table 11 shows that a significant number of the minority class cases are close to the majority class instances in the feature space. This phenomenon is the consequence of the previously mentioned characteristics. With the majority of the learning set representing fault-free modules, faulty modules are likely to have their measurements similar to many fault-free ones. With the second and third properties, small modules, faulty or not, are likely similar to each other. This poses a problem for all machine learning techniques (Boetticher 2005), especially for the instance-based learning methods. Project KC4 is an exception and this may be part of the reason why in some studies classification algorithms tend to achieve better overall classification results on this dataset (Menzies et al. 2007) than on others analyzed here.

Table 11 Class membership of the neighborhood instance for the faulty modules

These four observations are not necessarily the only interesting and/or unique aspects of the MDP datasets. But good understanding of the dataset is necessary for the selection of adequate classification techniques and play important role in model selection and comparison.

Appendix B: A Brief Description of the Six Classification Algorithms

Six machine learning algorithms are used in illustrative examples throughout the paper: Random Forest, Naïve Bayes, Bagging, J48, Logistic Regression and IBk.

Random forest (rf) is a decision tree-based classifier demonstrated to have good performance in software engineering studies by Guo et al. (2004). As implied from its name, it builds a “forest” of decision trees. The trees are constructed using the following strategy: The root node of each tree contains a bootstrap sample data of the same size as the original data. Each tree has a different bootstrap sample. At each node, a subset of variables is randomly selected from all the input variables to split the node and the best split is adopted. Each tree is grown to the largest extent possible without pruning. When all trees in the forest are built, new instance(s) is fitted to all the trees and a voting process is taken place. The forest selects the classification with the most votes as the prediction of new instance(s).

Naïve Bayes (nb) “naively” assumes data independence. This assumption may be considered overly simplistic in many application scenarios. However, in software engineering data sets its performance is surprisingly good. Naive Bayes classifier has been used extensively in fault-proneness prediction by (Menzies et al. 2007).

Bagging (bag) stands for bootstrap aggregating. It relies on an ensemble of different models. The training data is resampled from the original data set. According to Witten and Frank (2005), bagging typically performs better than single method models and almost never significantly worse.

J48 is a Weka (Witten and Frank 2005) implementation of Quinlan’s C4.5 (Guo et al. 2004) decision tree algorithm. A decision tree is a tree structure where non-terminal nodes represent tests on one or more attributes and terminal nodes reflect decision outcomes. It is a popular and classic machine learning algorithm (Witten and Frank 2005).

Logistic regression (log) is a classification scheme which uses mathematical logistic regression function. The most popular models are generalized linear models.

IBk is the Weka (Witten and Frank 2005) tool implementation of k-nearest-neighbor classifier. With k = 1 the default value, IBk is in fact IB1. This is the basic nearest-neighbor instance based learner that searches for the training instance closest in Euclidean distance to the given test instance and uses the result of the search for classification (Witten and Frank 2005).