Investigating the use of Support Vector Regression for web effort estimation

Abstract

Support Vector Regression (SVR) is a new generation of Machine Learning algorithms, suitable for predictive data modeling problems. The objective of this paper is twofold: first, to investigate the effectiveness of SVR for Web effort estimation using a cross-company dataset; second, to compare different SVR configurations looking at the one that presents the best performance. In particular, we took into account three variables’ preprocessing strategies (no-preprocessing, normalization, and logarithmic), in combination with two different dependent variables (effort and inverse effort). As a result, SVR was applied using six different data configurations. Moreover, to understand the suitability of kernel functions to handle non-linear problems, SVR was applied without a kernel, and in combination with the Radial Basis Function (RBF) and the Polynomial kernels, thus obtaining 18 different SVR configurations. To identify, for each configuration, which were the best values for each of the parameters we defined a procedure based on a leave-one-out cross-validation approach. The dataset employed was the Tukutuku database, which has been adopted in many previous Web effort estimation studies. Three different training and test set splits were used, including respectively 130 and 65 projects. The SVR-based predictions were also benchmarked against predictions obtained using Manual StepWise Regression and Case-Based Reasoning. Our results showed that the configuration corresponding to the logarithmic features’ preprocessing, in combination with the RBF kernel provided the best results for all three data splits. In addition, SVR provided significantly superior prediction accuracy than all the considered benchmarking techniques.

Appendix

1.1 Manual Stepwise Regression

Stepwise Regression (Maxwell 2002) is a statistical technique whereby a prediction model (Equation) is built and represents the relationship between independent (e.g. number of Web pages) and dependent variables (e.g. total Effort). This technique builds the model by adding, at each stage, the independent variable with the highest association to the dependent variable, taking into account all variables currently in the model. It aims to find the set of independent variables (predictors) that best explains the variation in the dependent variable (response). In particular, we applied a Manual Stepwise Regression using the technique proposed by Kitchenham (Kitchenham 1998). Basically the idea is to use this technique to select the important independent variables, and then to use linear regression to obtain the final model.

In our study we employed the variables shown in Table 1 with Manual Stepwise Regression in order to select the most important size measures. Once selected they were the ones used for cross-validation on each split, i.e. we did not perform a separate Manual Stepwise Regression for each split; we simply performed a regression using the variables previously selected using the manual stepwise procedure.

Whenever variables were highly skewed they were transformed before being used in the forward stepwise procedure. This was done in order to comply with the assumptions underlying stepwise regression (Maxwell 2002) (i.e. residuals should be independent and normally distributed; relationship between dependent and independent variables should be linear). The transformation employed was to take the natural log (Ln), which makes larger values smaller and brings the data values closer to each other (Maxwell 2002). A new variable containing the transformed values was created for each original variable that needed to be transformed.

In addition, whenever a variable needed to be transformed but had zero values, the natural logarithmic transformation was applied to the variable’s value after adding 1.

To verify the stability of each effort model built using forward stepwise regression, the following steps were employed (Kitchenham and Mendes 2004):

• Use of a residual plot showing residuals vs. fitted values to investigate if the residuals are randomly and normally distributed.

• Calculate Cook’s distance values (Cook 1977) for all projects to identify influential data points. Any projects with distances higher than 3 × (4/n), where n represents the total number of projects, are immediately removed from the data analysis (Maxwell 2002). Those with distances higher than 4/n but smaller than (3 × (4/n)) are removed in order to test the model stability, by observing the effect of their removal on the model. If the model coefficients remain stable and the adjusted R² (goodness of fit) improves, the highly influential projects are retained in the data analysis.

1.2 Case-Based Reasoning

Case-Based Reasoning (CBR) is a branch of Artificial Intelligence where knowledge of similar past cases is used to solve new cases (Shepperd & Kadoda 2001). Within the context of our investigation, the idea behind the use of the CBR technique is to predict the effort of a new project by considering similar projects previously developed. In particular, the completed projects are characterized in terms of a set of p features and form the case base. The new project is also characterized in terms of the same p features and it is referred as the target case. Then, the similarity between the target case and the other cases in the p-dimensional feature space is measured, and the most similar cases are used, possibly with adaptations to obtain a prediction for the target case. To apply the method, we have to select the relevant project features, the appropriate similarity function, the number of analogies to choose the similar projects to consider for estimation, and the analogy adaptation strategy for generating the estimation. The selection of the similarity function and the number of analogies are crucial decisions. The similarity measure used in this study is the Euclidean distance as this has been the measure used in the literature with the best results (Mendes et al. 2003c). In addition, all the project attributes considered by the similarity function had equal influence upon the selection of the most similar project(s).

Estimates were based on the average effort of the most similar projects in the case base, with no different weights for attributes or adaptation of the estimated effort.