Re-estimating software effort using prior phase efforts and data mining techniques

Abstract

Software effort estimation has played an important role in software project management. An accurate estimation helps reduce cost overrun and the eventual project failure. Unfortunately, many existing estimation techniques rely on the total project effort which is often determined from the project life cycle. As the project moves on, the course of action deviates from what originally has planned, despite close monitoring and control. This leads to re-estimating software effort so as to improve project operating costs and budgeting. Recent research endeavors attempt to explore phase level estimation that uses known information from prior development phases to predict effort of the next phase by using different learning techniques. This study aims to investigate the influence of preprocessing in prior phases on learning techniques to re-estimate the effort of next phase. The proposed re-estimation approach preprocesses prior phase effort by means of statistical techniques to select a set of input features for learning which in turn are exploited to generate the estimation models. These models are then used to re-estimate next phase effort by using four processing steps, namely data transformation, outlier detection, feature selection, and learning. An empirical study is conducted on 440 estimation models being generated from combinations of techniques on 5 data transformation, 5 outlier detection, 5 feature selection, and 5 learning techniques. The experimental results show that suitable preprocessing is significantly useful for building proper learning techniques to boosting re-estimation accuracy. However, there is no one learning technique that can outperform other techniques over all phases. The proposed re-estimation approach yields more accurate estimation than proportion-based estimation approach. It is envisioned that the proposed re-estimation approach can facilitate researchers and project managers on re-estimating software effort so as to finish the project on time and within the allotted budget.

Appendices

A data set

This section provides details on all 38 software projects collected from two software development organizations. The project data being collected in compliance with COCOMO II data collection guidelines [14] are shown in Table 6. Only KSLOC represented quantitative data in ratio scale which could be directly used as a feature for the estimation. In contrast, all remaining features were represented as qualitative data in ordinal scale or 6-likert scale, i.e., very low (VL), low (L), nominal (N), high (H), very high (VH), and extra high (XH). These features would be transformed according to COCOMO II rules [14] before participating in the estimation. For example, the rule for CPLX is 0.73, 0.87, 1.00, 1.17, 1.34, and 1.74 for VL, L, N, H, VH, and XH, respectively. As a result, when project 1–5 (see Table 6) contain ordinal scales as L, L, L, N, and N, a transformed value of those projects is 0.87, 0.87, 0.87, 1, and 1, respectively. Even though there was one software project less than 1 KSLOC (0.27 KSLOC) that claimed to be not appropriate to calibrate COCOMO II model, we decided to keep it since GLOBAL COCOMO II and LOCAL calibration COCOMO II were not involved in our study. Instead, we built proportion-based estimation model with the help of learning techniques, where all collected projects were used. However, when an outlier detection was performed, the small project was often considered as an outlier and excluded from the training set. Software efforts of individual phase and total project measured in man-day are summarized in Table 7, which will be used to building re-estimation models. Characteristics of individual project are shown in Table 8 which subsequently are used to drill down in project category for better estimation accuracy of both re-estimation and proportion models.

Table 6 Software project data collected according to COCOMO II

Table 7 Software effort of different phases and total project measured in man-day

B parameter set up

The 10-fold cross-validation is applied to determine optimal parameters for the five learning techniques as discussed below.

Regression analysis (RA): RA is a statistical technique to estimate the relationships among variables. OLS is a traditional regression analysis to approximate the target values in a linear regression model. It is carried out under an assumption that the data are normally distributed to provide favorable estimation results. To find the optimal parameters, the first step is to generate multiple choices by the combination of parameters. In this case, there was only one parameter, i.e., regression constant. Therefore, there were two choices, i.e., applying regression constant and not applying regression constant. The second step is to select the best choice by applying 10-fold cross-validation to the training set. The choice yielding the lowest sum of MdBRE, MBRE, MIBRE, and MdIBRE is selected to be the optimal parameter.

Table 8 Characteristics of software projects

Support vector regression (SVR) SVR employs the ideas of SVM for regression task [21]. SVR defines $\epsilon$-intensive loss function to establish a band around the true outputs [20]. Three parameters are considered, namely kernel functions composing of linear and radial basis, $\epsilon$ value in loss function being 0.0001, 0.001, 0.01, and 0.1, regularization parameter in loss function ranging from 1 to 10 with 2 step increment, and gamma value or width of radial basis function ranging from 0.1 to 1 with 0.2 step increment. The choices for the linear function are 20 (4 $\epsilon$ values $\times$ 5 regularization parameters). The choices for radial basis function are 100 (4 $\epsilon$ values $\times$ 5 regularization parameters $\times$ 5 gamma values). The total becomes 120 choices (20 for linear function and 100 for radial basis function.

Radial basis function (RBF) RBF is one of feed-forward neural networks that generally uses radial basis function as the activation function. RBF takes a sequence of two mappings consisting of nonlinear mapping of the input data via the basis function and a linear mapping of the basis function to output. There are two parameters to be considered, namely number of basis functions ranging from 1 to the number of projects with s step increment, where s is the number of projects/10. The width of the basis function ranging from 0.1 to 1 with 0.2 step increment. The total becomes 50 choices (10 neurons $\times$ 50 widths).

Classification and regression tree (CART) CART builds a decision tree for the prediction that works for both classification and regression problems [9]. In this experiment is to apply CART in the regression problem. There are two parameters to be focused, namely pruning tree and stopping criterion. The pruning is to reduce a tree by removing some leaf nodes from the original branch. For example, a tree would be terminated if the number of projects in a leaf node was less than threshold value, where threshold ranged from 1 to 10 with 1 step increment. Hence, the combinations of the two parameters generate 20 choices (2 pruning $\times$ 10 projects).

$\mathbf{K}$-nearest neighbor (KNN) KNN uses local neighborhood data points to obtain the prediction in analogy-based estimation. KNN finds the most similar projects from the training set by measuring the distance between test and training data points, where each point denotes a project in the computation space. KNN selects the first k points from the training set having the smallest distance from the given test point. These efforts can be weighted in proportion to the above-measured distance. Compute the average of these efforts to yield the predicted effort of the test project. Thus, there are three parameters to be chosen, namely distance measurements (Euclidean and Minkowski), number of K projects (ranging from 1 to 10), and the effort (total weighted effort average of k nearest neighbor projects). The total becomes 40 choices (2 distances $\times$ 10 neighbors $\times$ 2 efforts).