Predictive accuracy comparison of fuzzy models for software development effort of small programs

Abstract

Regression analysis to generate predictive equations for software development effort estimation has recently been complemented by analyses using less common methods such as fuzzy logic models. On the other hand, unless engineers have the capabilities provided by personal training, they cannot properly support their teams or consistently and reliably produce quality products. In this paper, an investigation aimed to compare personal Fuzzy Logic Models (FLM) with a Linear Regression Model (LRM) is presented. The evaluation criteria were based mainly upon the magnitude of error relative to the estimate (MER) as well as to the mean of MER (MMER). One hundred five small programs were developed by thirty programmers. From these programs, three FLM were generated to estimate the effort in the development of twenty programs by seven programmers. Both the verification and validation of the models were made. Results show a slightly better predictive accuracy amongst FLM and LRM for estimating the development effort at personal level when small programs are developed.

Introduction

Software development effort estimates are the basis for project bidding and planning (both are critical practices in the software industry). The consequences of poor budgets and plans can be dramatic: if they are too pessimistic, business opportunities can be lost, while over optimism may be followed by significant losses (Grimstad et al., 2005). Software effort estimation has even been identified as one of the three great challenges in Computer Science (Brooks, 2003).

Software development estimation techniques can be classified into three general categories (Mendes et al., 2002):

• Expert judgment: It aims to derive estimates based on an experience of experts on similar projects. The means of deriving an estimate are not explicit and therefore not repeatable (Mendes et al., 2002). Studies on expert estimation rate it to be the preferred method among professional software developers (Molokken and Jorgensen, 2004). The term expert estimation is not clearly defined and covers a wide range of estimation approaches. A common characteristic is, however, that intuitive processes constitute a determinant part of the estimation (Jorgensen et al., 2000).

• Algorithmic models: To date the is the most popular in literature (Briand and Wieczorek, 2001). It attempts to represent the relationship between effort and one or more characteristic of a project; the main cost driver in such a model is usually taken to be some notion of software size (e.g. the number of lines of source code). Its general form is a linear regression equation, as that used by Kok et al. (1990), or a group of non-linear regression equations as used by Boehm (1981) in COCOMO 81 and COCOMO II (Boehm et al., 2000).

• Machine learning: Machine learning techniques have in recent years been used as a complement or alternative to the previous two techniques (Pedrycz, 2002). Fuzzy logic models are included in this category (MacDonell and Gray, 1996) as well as neural networks (Idri et al., 2002a), genetic programming (Burguess and Lefley, 2001), regression trees (Srinivasan and Fisher, 1995), and case-based reasoning (Kadoda et al., 2000).

Given the fact that no single software development estimation technique is best for all situations, a careful comparison of the results of several approaches is most likely to produce realistic estimates (Boehm et al., 1998).

In this paper, the result of using a Linear Regression Model (LRM), is compared with those of three Fuzzy Logic Models (FLM). This comparison is based upon the two following main stages when using an estimation model: (1) the model adequacy checking (or model verification) must be determined, that is, whether the model is adequate to describe the observed (actual) data; if so then (2) the estimation model is validated using new data.

On the other hand, considering that unless engineers have the capabilities provided by personal training, they will not be able to properly support their teams or consistently and reliably produce quality products (Humphrey, 2000), this paper suggests that the software estimation activity could start using a personal level approach, starting with the development of small size programs.

The Capability Maturity Model (CMM) gives an available description of the goals, methods, and practices needed in software engineering industrial practice, while Personal Software Process (PSP) allows its instrumentation at a personal level (Humphrey, 1995). Twelve of the eighteen key process areas of the CMM are at least partially considered in PSP. This paper is based upon some PSP practices and takes account of the guidelines suggested in Kitchenham et al. (2002).

In this study, lines of code and development time were gathered from 105 small programs developed by 30 programmers. From this set of programs three FLM and a LRM were generated and their adequacy was checked (verification). Then, the results of the FLM and LRM equations, were validated in the effort estimation of 20 programs developed by other group integrated by seven programmers.

In spite of the availability of a wide range of software product size measures, source lines of code (LOC) remains in favor of many models (MacDonell, 2003). There are two measures of source code size: physical source lines and logical source statements. The count of physical lines gives the size in terms of the physical length of the code as it appears when printed (Park, 1992).

To generate the FLM and LRM, New and Changed (N&C) (Humphrey, 1995) physical lines of code (LOC) were used. N&C is composed of added and modified code. The added code is the LOC written during the current programming process, while the modified code is the LOC changed in the base program when modifying a previously developed program. The base program is the total LOC of the previous program. Reused code (the LOC of previously developed programs that are used without modification) was analyzed and considered when establishing the LRM and FLM; however it did not showed a statistical significance (Section 3.1).

A coding standard should establish a consistent set of coding practices that is used as a criterion when judging the quality of the produced code (Humphrey, 1995). Hence, it is necessary to always use the same coding and counting standards. The programs developed within this study followed these guidelines.

The solution adopted by cost estimation researches for selecting the relevant attributes is to test the correlation (r) between efforts and attributes (Idri and Khoshgoftaar, 2001). In this study, lines of code are correlated to the development efforts.

A fuzzy model is a modeling construct featuring two main properties (Pedrycz, 1998): (1) It operates at a level of linguistic terms (fuzzy sets), and (2) it represents and processes uncertainty.

All estimation techniques has an important limitation, which arises when software projects are described using categorical data (nominal or ordinal scale) such as small, medium, average, or high (linguistic terms or values). A more comprehensive approach to deal with linguistic values is using the fuzzy set theory (Idri et al., 2002b). Specifically, fuzzy logic offers a particularly convenient way to generate a keen mapping between input and output spaces thanks to the natural expression of fuzzy rules (Zadeh, 1999).

In software development effort estimation, two considerations justify the decision of implementing a fuzzy model: first, it is impossible to develop a precise mathematical model of the domain (Lewis, 2001); second, metrics only produce estimations of the real complexity. Thus, according to the previous assertions, formulating a tiny set of natural rules describing underlying interactions between the software metrics and the effort estimation could effortlessly reveal their intrinsic and wider correlations; hence, this paper presents a comparative study designed to evaluate this proposition.

Disadvantages of a fuzzy model could be that (1) it requires a lot of data, (2) the estimators must be familiar with the historically developed programs, and (3) it is not useful for programs much larger or smaller than the historical data (Humphrey, 1995). In this research, only the third one represents a disadvantage since the sample was conformed by more than a hundred programs and those were well known to the estimators.

There are several ways in data fuzzification that could potentially be applied to the effort estimation problem (Schofield, 1998). One of them is used in this study: to construct a rule induction system replacing the crisp facts with fuzzy inputs. An inference engine uses a base of rules to map inputs to a fuzzy output which can either be translated back to a crisp value or left as a fuzzy value.

A previous research (Foss et al., 2003) has demonstrated that the Magnitude of Relative Error (MRE) (a common criterion for the evaluation of cost estimation models (Briand et al., 1998)) does not identify the best prediction model. In accordance with Foss et al. (2003), the implications of this finding are that the results and conclusions on prediction models over the past 15–25 years are unreliable and may have misled the entire software engineering discipline. Therefore, it is strongly recommended not using MMRE to evaluate and compare prediction models (Foss et al., 2003), but the MMER that was proposed by Kitchenham et al. (2001).

The MER is defined as follows:

$${\text{MER}}_i=\frac{|{\text{Actual Effort}}_i-{\text{Predicted Effort}}_i|}{{\text{Predicted Effort}}_i}$$

The MER value is calculated for each observation i whose effort is predicted. The aggregation of MER over multiple observations (N) can be achieved through the Mean MER (MMER) as follows:

$$\text{MMER}=\frac{1}{N}\sum _i^N{\text{MER}}_i$$

Intuitively, MER seems preferable to MRE since MER measures the error relative to the estimate.

In Foss et al. (2003) MMER had better results than MMRE. This fact is the reason for using MMER in this study.

However, the MMER are sensitive to individual predictions with excessively large MERs. Therefore, an aggregate measure less sensitive to extreme values is also considered, namely the median of MER values for the N observations (MdMER) (Briand et al., 2000).

A complementary criterion is the prediction at level l, Pred(l) = k/N, where k is the number of observations where MER is less than or equal to l, and N is the total number of observations. Thus, Pred(25) gives a percentage of projects which were predicted with a MER less or equal than 0.25.

In general, the accuracy of an estimation technique is proportional to the Pred(l) and inversely proportional to the MMER. As reference, for effort prediction models, a MMRE ⩽ 0.25 is considered as acceptable (Conte et al., 1986). A reference for an acceptable value of MMER was not found.

A group of papers regarding aspects related to previous research on software development effort estimation based on FLM were reviewed.

Four different aspects were analyzed: (1) What has been achieved by empirical research studies on software effort estimation based on fuzzy logic? (2) From where has the dataset been gathered? (3) What metrics have been used? and (4) What were the results? A brief description of each paper found is given:

• Ahmed et al. (2004), present a FLM based upon triangular membership functions. The dataset for validating the FLM was (a) generated randomly and (b) that of COCOMO 81 was used. Results showed that the FLM was slightly better than COCOMO equations. In addition, they reported promising experimental summary results in spite of the little background knowledge of the rule base and training data. It does signify that there are potentials for improvements when the framework is deployed in practice, since experienced experts could augment the dataset value with their knowledge.

• Crespo et al. (2004), explore fuzzy regression techniques based upon fuzzification of input values. Project database of COCOMO-81 are used. Their conclusions are that fuzzy regression is able to obtain estimation models with similar predictive properties than existing basic estimation models.

• Gray and MacDonell (1997), compare a FLM with linear regression equations as well as neural networks. The FLM is based upon triangular membership functions. The neural network was better than FLM and regression equation. The dataset was obtained from a Canadian thesis. They concluded that, compared with other techniques, the fuzzy logic model shows good performance. Moreover, they suggest that there is a place in the field of software metrics for fuzzy logic models.

• Host and Wohlin (1997), report an experiment based upon personal practices, but it only uses expert judgment for estimating the effort.

• Huang et al. (2003), propose a model combining fuzzy logic and neural networks. The dataset was obtained from the original COCOMO (1981). The results of the fuzzy logic model were better than those of the COCOMO equations. The FLM was based upon triangular membership functions. The main benefit of this model is its good interpretability by using the fuzzy rules. Another great advantage of this research is that they could put together expert knowledge (fuzzy rules), project data and the traditional algorithmic model into one general framework that may have a wide range of applicability in software cost estimation.

• Idri et al. (2000), apply fuzzy logic to the fifteen cost factors of COCOMO 81. The FLM is based upon trapezoidal membership functions. The dataset is randomly generated and compared with actual data of COCOMO 81. Results of the FLM were similar to those of COCOMO 81 (but in some cases worse). Researches suggest using other kind of membership functions as triangular.

• Idri et al. (2002b), propose an approach based on fuzzy logic named Fuzzy Analogy. Its dataset is that of COCOMO 81. Taking into account their results, they suggest the following ranking of the four techniques in terms of accuracy and adequacy to deal with linguistic values: 1. Fuzzy Logic, 2. Fuzzy intermediate COCOMO’81, 3. Classical intermediate COCOMO’81, and 4. Classical Analogy.

• Musflek et al. (2000), propose a fuzzy model for COCOMO 81 named f-COCOMO that describes the relationship between size fuzzy sets and effort fuzzy sets. Triangular membership functions are used in this study. Moreover, they concluded that (a) fuzzy sets help articulate the estimates and their essence (by exploiting fuzzy numbers described by asymmetric membership functions) and (b) they generate a feedback as to the given uncertainty (granularity) of the results.

• Reformat et al. (2004), propose an estimation model based upon fuzzy neural networks for development effort. This model is applied in a medical information system. The dataset is divided in three subsets, one of them is used for validating the model.

• Zu and Khoshgoftaar (2004), presented a fuzzy identification cost estimation modeling technique to deal with linguistic data, and automatically generate fuzzy membership functions and rules. A case of study based on the COCOMO’81 database compared the proposed model with all three COCOMO’81 models (basic, intermediate and detailed). It was observed that the fuzzy identification model provided significantly better cost estimations than the three COCOMO’81 models.

Two of the last ten references propose a hybrid system combining fuzzy logic with neural networks and analogy. These papers use only fuzzy logic in the estimation without combining it.

Seven of the ten last references used COCOMO’81 as dataset. In this paper, the dataset was gathered from 105 programs generated by thirty developers; while the dataset for validating the models was gathered from 20 programs generated by other seven developers.

References used lines of code, function points (FP) or a variant of FP as metrics. In this paper lines of code (N&C) were used as the input variable.

Not one paper proposing and comparing a fuzzy logic model in estimating the development effort in small programs based upon personal practices, was found.