Robust estimations of fault content with capture–recapture and detection profile estimators

Abstract

Inspections are widely used in the software engineering community as efficient contributors to reduced fault content and improved product understanding. In order to measure and control the effect and use of inspections, the fault content after an inspection must be estimated. The capture–recapture method, with its origin in biological sciences, is a promising approach for estimation of the remaining fault content in software artefacts. However, a number of empirical studies show that the estimates are neither accurate nor robust. In order to find robust estimates, i.e., estimates with small bias and variations, the adherence to the prerequisites for different estimation models is investigated. The basic hypothesis is that a model should provide better estimates the closer the actual sample distribution is to the model's theoretical distribution. Firstly, a distance measure is evaluated and secondly a χ²-based procedure is applied. Thirdly, smoothing algorithms are tried out, e.g., mean and median values of the estimates from a number of estimation models. Based on two different inspection experiments, we conclude that it is not possible to show a correlation between adherence to the models’ theoretical distributions and the prediction capabilities of the models. This indicates that there are other factors that affect the estimation capabilities more than the prerequisites. Neither does the investigation point out any specific model to be superior. On the contrary, the Mh–JK model, which has been shown as the best alternative in a prior study, is inferior in this study. The most robust estimations are achieved by the smoothing algorithms.

Introduction

In the software engineering community, inspections are widely accepted as efficient contributors to improved software quality and reduced costs. However, there is a need to control the inspection process in order to know whether a software artefact is of sufficient quality or if it needs improvement. This process control requires data on how many residual faults are there in a software artefact. The data generally available after an inspection are the number of faults detected and removed. However, these faults do not cause any problems after removal. It is more important to know how many faults remain in the software artefact, but this information is generally not known.

The capture–recapture method is a simple and useful approach to estimate the number of faults remaining after an inspection. The method is derived from biological applications, where it is applied to estimate wildlife populations. Individual animals are counted by different observers or at different trapping occasions. If the overlap among the animals found by observers is large, most of the population is assumed to be found and the remainder is estimated to be small. On the other hand, if the overlap among the animals found by observers is small, there are probably many remaining animals in the population. This method can be applied to software inspections. The inspectors correspond to the observers, and the faults in the inspected document correspond to the animals to be counted. If many inspectors find the same faults, they have probably covered most of the faults, while if they find disjunct sets of faults, there are probably many faults yet to be discovered.

The application of capture–recapture to software inspections is proposed by Eick et al. (1992); further studies and improvements are presented in Eick et al., 1993, Vander Wiel and Votta, 1993, Wohlin et al., 1995, Briand et al., 1997, Runeson and Wohlin, 1998. A related method, the detection profile method, is proposed by Wohlin and Runeson (1998) and evaluated by Briand et al. (1998). Although the principles behind the methods are intuitively right, the published results show a large variation in the estimation capabilities. Large over and underestimates are sometimes produced and the estimators are sensitive to the underlying data. This is not satisfactory, since we need robust methods that deliver trustworthy estimates.

The estimation models have different underlying assumptions on, e.g., the statistical distribution of the fault detection probability. In this paper an empirical study of fault estimation methods is presented, which aims to investigate whether the estimates depend on how the data fit into the statistical prerequisites of the estimation models or the varying estimates have random causes.

Two alternative approaches to evaluating the dependence of underlying assumptions are presented:

• a distance measure which is based on a graphical representation of data and prerequisites;

• a χ² selection algorithm which tests statistically whether prerequisites are fulfilled or not.

The reason for investigating both methods is that the former is a graphical approach to be used as descriptive statistics while the latter is a statistical approach using hypothesis testing. The study indicates no clear pattern in the dependence on underlying data distributions, which implies that there are other factors that affect the outcome more than the prerequisites. As an alternative to the prerequisites approach various smoothing algorithms are evaluated, e.g., mean or median of 2, 3 or 4 different estimates or a simple random selection from the 4 estimates. It is concluded that no single model is superior to the others. The most robust results are achieved by the smoothing algorithms, which combine the estimate of different models and the detection profile method.

In this paper three different notations are used, estimators, models and algorithms. An estimator is a formula used to predict the number of faults remaining in a document. A model is the umbrella term for a number of estimators with the same prerequisites. Algorithms are different methods for combining two or more estimators (smoothing algorithms) or selecting (selection algorithms) one out of a set of estimators.

The paper is structured as follows. In Section 2, the prerequisites for the estimation models are introduced. In Section 3, distance measures for the models and the analysis with these measures are presented. In Section 4, the χ² method is elaborated and in Section 5, the smoothing algorithms are presented. The analysis results are given in 6 Evaluation of the algorithms, 7 Discussion contains a discussion about the results. Finally, the conclusions from the study are presented in Section 8.