A comprehensive empirical evaluation of missing value imputation in noisy software measurement data

Abstract

The handling of missing values is a topic of growing interest in the software quality modeling domain. Data values may be absent from a dataset for numerous reasons, for example, the inability to measure certain attributes. As software engineering datasets are sometimes small in size, discarding observations (or program modules) with incomplete data is usually not desirable. Deleting data from a dataset can result in a significant loss of potentially valuable information. This is especially true when the missing data is located in an attribute that measures the quality of the program module, such as the number of faults observed in the program module during testing and after release. We present a comprehensive experimental analysis of five commonly used imputation techniques. This work also considers three different mechanisms governing the distribution of missing values in a dataset, and examines the impact of noise on the imputation process. To our knowledge, this is the first study to thoroughly evaluate the relationship between data quality and imputation. Further, our work is unique in that it employs a software engineering expert to oversee the evaluation of all of the procedures and to ensure that the results are not inadvertently influenced by poor quality data. Based on a comprehensive set of carefully controlled experiments, we conclude that Bayesian multiple imputation and regression imputation are the most effective techniques, while mean imputation performs extremely poorly. Although a preliminary evaluation has been conducted using Bayesian multiple imputation in the empirical software engineering domain, this is the first work to provide a thorough and detailed analysis of this technique. Our studies also demonstrate conclusively that the presence of noisy data has a dramatic impact on the effectiveness of imputation techniques.

Introduction

In the domain of software quality, estimation models and methods are widely recognized as a useful methodology for optimizing scarce resources. By constructing a model using data from past projects to capture the relationship between software metrics (independent variables) and the number of faults (dependent variable) observed in the program modules (instances), the quality of current software development projects can be estimated. This allows for the optimal allocation of additional resources to perform extra inspection of at-risk program modules. Various estimation procedures, such as instance-based learning (Khoshgoftaar and Seliya, 2003a) and linear regression, have been applied to perform such tasks (Khoshgoftaar and Seliya, 2003b).

One of the most important issues that needs to be addressed before data analysis can be completed is missing data. Linear regression techniques, for example, are unable to directly handle missing data. Many software implementations of linear regression delete any instances with missing data (listwise deletion) before the model is constructed. Such an approach leads to significant information loss, especially as the amount of missing data increases, and may result in severely biased models if the missing data is not distributed completely randomly (Allison, 2001, Little and Rubin, 2002). In response to these issues, significant effort has been devoted to developing and evaluating various imputation methodologies. Imputation techniques ‘fill-in’ (or impute) the missing values with one or more plausible values.

At the present time, very little emphasis in the software quality estimation domain has been given to imputation techniques. Therefore, we present in this work a comprehensive evaluation of five imputation procedures using a real-world software measurement dataset in numerous carefully controlled experiments. As the experimental datasets do not have missing values, missingness is injected at various levels according to three common missingness mechanisms (explained in Section 3.2.2). We only consider missingness in the dependent variable, which is extremely important because the dependent variable is often the most valuable attribute in the dataset. Since information loss in the dependent variable is often intolerable, the successful treatment of missing data in the dependent variable in software measurement data is especially critical. Since software measurement datasets are often small in size, it may not be practical to eliminate instances with missing data from any analysis. To our knowledge, no research in the software engineering (SE) domain has focused specifically on the important occurrence of missing data in the dependent variable. We believe it is critical to understand and evaluate imputation techniques with data missing from a single variable before multi-attribute missingness is explored. The scenario of having missing values in the dependent variable (which relates to the fault-proneness of the software modules) is realistic and commonly-encountered in real-world software quality modeling applications. Code metrics are often recorded by software tools, while process metrics, which include information about the fault-proneness of the modules, may be self-reported, and hence have a higher likelihood to be missing. As we discuss in more detail in Section 3.2.2, all three types of missing data mechanisms (MCAR, MAR, and NI) can reasonably occur in measurement data, and hence our simulations are consistent with what is encountered in real-world SE applications. Considering missingness in only the dependent variable has precedence in literature on missing values. Little and Rubin (2002) discuss missing values in controlled experiments and contend that ‘since the levels of the design factor in an experiment are fixed by the experimenter, missing values, if they do occur, do so far more frequently in the outcome variable…’. Even though the domain is different, our consideration of missing values only in the dependent variable is well-founded.

Our study is the first to evaluate the impact of noise in software measurement data on the imputation process. To the best of our knowledge, our research group was the first to address the important subject of data quality in SE measurement data (Khoshgoftaar and Seliya, 2004, Khoshgoftaar et al., 2005). Data quality has been demonstrated to be a key component of the analytical and knowledge discovery process (Khoshgoftaar and Seliya, 2004), and we have proposed some procedures to cope with this critical issue (Khoshgoftaar and Van Hulse, 2006a, Van Hulse et al., 2007). Finally, some preliminary work on Bayesian multiple imputation (MI) in SE has been recently presented (Twala and Cartwright, 2005), however, our work is the first to provide an in-depth analysis of MI for handling missing data in empirical SE applications. Our experimental results strongly support the use of MI and further demonstrate that mean imputation is a very poor imputation technique. In particular, we expand upon our initial results demonstrating the strong imputation performance of MI (Khoshgoftaar and Van Hulse, 2006b).

The following topics are considered in this empirical study, and represent the main contributions of this work:

• What is the impact of poor data quality (i.e., the presence of noise in a dataset) on the analysis of the effectiveness of imputation techniques?

  • The accuracy of an imputation technique should be measured by comparing the imputed value to what the value should be (i.e., the clean value), as opposed to what the value is (i.e., the noisy value). If a particular attribute is noisy, these two values will not be the same. Given a real-world dataset, the difficulty is that it is not typically known which attribute values are noisy or what the ‘correct’ value should be. As noise is a common occurrence in real-world datasets, any analysis of the effectiveness of imputation techniques that does not also consider data quality is fundamentally flawed. In order to assess the quality of the underlying data, we employ an expert in the SE domain who has worked specifically with the dataset used in our experiments for many years. With respect to considering the impact of poor data quality on the imputation process, our study is unique and unprecedented. The precise method of our evaluation is explained more fully in Section 4.1. From this study, we conclude that the quality of the underlying data cannot be ignored when examining the efficacy of imputation procedures.

  • Independent of the above, does noise in a dataset affect the imputation results? In other words, given a dataset with a subset of relatively clean instances and a subset of relatively noisy instances, does the presence of noise dramatically affect the imputed values of the relatively clean instances? Our experiments demonstrate conclusively that data quality does play an important role in the effectiveness of imputation techniques (Section 4.2). We conclude that the imputation results obtained using a noisy dataset will be worse than those obtained using a clean dataset, and that noise will adversely impact the imputation process itself.

• Which techniques provide the best imputation performance?

  • Mean imputation (MEI) performs very poorly. In relation to (1a) above, when using the noisy value to measure the quality of the imputation, MEI appears to show good results (Section 4.1). When the imputed value is measured against what the value should be (i.e., the clean value), however, MEI has very poor imputation accuracy. MEI is, therefore, not a reliable imputation technique.

  • Both Bayesian multiple imputation and regression imputation are very effective for imputing missing values in the dependent variable.

• Is the missingness mechanism, either MCAR, MAR, or NI, or the percentage of missing data (5%, 10%, 15%, and 20%) an important factor in the imputation process? Based on our experiments, all imputation techniques perform significantly better when the data is missing completely at random (MCAR). At the four levels considered in this study, the percentage of missing data was not deemed significant.

Listwise deletion (LD), which consists of removing incomplete cases from the dataset, is not considered in this work for numerous reasons. As previously mentioned, numerous studies (Allison, 2001, Little and Rubin, 2002) have remarked that LD results in a loss of precision and introduces a bias when the missing data mechanism is not MCAR. As the objective of this work is to understand the relationship between imputation and data quality, performing LD provides no insight into this relationship. Finally, it may not be possible to use LD in certain circumstances. For example, suppose that a software quality regression model has been constructed using historical fault data, and the number of faults for each program module in a test dataset are to be estimated using the model. Suppose further that some of the instances in the test dataset contain missing values in one of the attributes used in the regression model. Clearly it is infeasible to discard the modules with missing values from the test dataset using LD, so the common practice is to utilize MEI before applying the regression model. Due to these reasons, we have only considered imputation procedures in this work.

The remainder of this work is organized as follows. Some remarks on related work for imputation in the software engineering domain are presented in Section 2. The design of our experiments is provided in Section 3, with the results presented in Section 4. Conclusions and future work are provided in Section 5.