Imputation techniques for multivariate missingness in software measurement data

Abstract

The problem of missing values in software measurement data used in empirical analysis has led to the proposal of numerous potential solutions. Imputation procedures, for example, have been proposed to ‘fill-in’ the missing values with plausible alternatives. We present a comprehensive study of imputation techniques using real-world software measurement datasets. Two different datasets with dramatically different properties were utilized in this study, with the injection of missing values according to three different missingness mechanisms (MCAR, MAR, and NI). We consider the occurrence of missing values in multiple attributes, and compare three procedures, Bayesian multiple imputation, k Nearest Neighbor imputation, and Mean imputation. We also examine the relationship between noise in the dataset and the performance of the imputation techniques, which has not been addressed previously. Our comprehensive experiments demonstrate conclusively that Bayesian multiple imputation is an extremely effective imputation technique.

Appendix: Attribute noise ranking procedure

In related work, we have proposed a procedure for ranking attributes in a dataset from most to least noisy. Information about the relative quality of each of the attributes in the datasets used in this work will be utilized in our experiments. We present a brief overview of our procedure for completeness; additional information can be obtained in the references (Khoshgoftaar and Van Hulse 2005b).

Our attribute noise ranking technique utilizes a procedure called PANDA (Fig. 2), which provides a ranking of instances from most to least noisy based on the Noise Factor S _i . For each observation in the dataset, PANDA examines each pair of attributes and computes the deviation of the second attribute from its mean value given the partitioned value of the first attribute. For a given instance, if these deviations occur often and severely enough when compared to the remainder of the dataset, that instance will appear more noisy.

Fig. 2

Pairwise attribute noise detection algorithm (PANDA)

The procedure to rank the attributes from most to least noisy (Khoshgoftaar and Van Hulse 2005b) is presented in Fig. 3. Suppose there are a total of n instances and m attributes. PANDA is executed with all m attributes and the instance noise ranking is created. Denote this ordering by rank and the rank of instance i as rank(i), where 1 ≤ i ≤ n (Line 1 of Fig. 3). Each of the m attributes is removed and PANDA is executed using only the remaining m − 1 attributes (Line 3). The output is another instance ordering from most likely to least likely noise. If the jth attribute is removed, denote the output of PANDA by rank _j and the rank of instance i as rank _j (i). For example, if the second attribute is removed and instance 100 is ranked by PANDA as most noisy, then rank ₂(100) = 1.

Fig. 3

Noisy attribute ranking methodology

Kendall’s Tau rank correlation (Conover 1971), a non-parametric measure of the association between two attributes, is calculated between rank _j and rank for each attribute j (Line 4). Attributes are ordered from most noisy to least noisy based on their correlation with the ranking rank when the attribute is removed from the dataset. The attribute that creates the ranking with the lowest correlation to rank after it is removed has the most noise, while the attribute with the highest correlation to rank is considered to be the cleanest attribute.