Elsevier

Applied Soft Computing

Volume 11, Issue 1, January 2011, Pages 724-732
Applied Soft Computing

A fuzzy logic model for predicting the development effort of short scale programs based upon two independent variables

https://doi.org/10.1016/j.asoc.2009.12.034Get rights and content

Abstract

Fuzzy models have been recently used for estimating the development effort of software projects and this practice could start with short scale programs. In this paper, new and changed (N&C) as well as reused code were gathered from small programs developed by 74 programmers using practices of the Personal Software Process; these data were used as input for a fuzzy model for estimating the development effort. Accuracy of this fuzzy model was compared with the accuracy of a statistical regression model. Two samples of 163 and 68 programs were used for verifying and validating respectively the models; the comparison criterion was the Mean Magnitude of Error Relative to the estimate (MMER). In verification and validation stages, fuzzy model kept a MMER lower or equal than that regression model and an accuracies comparison of the models based on ANOVA, did not show a statistically significant difference amongst their means. This result suggests that fuzzy logic could be used for predicting the effort of small programs based upon these two kinds of lines of code.

Introduction

Development effort estimates are required by project managers for planning and for controlling the process of software development [18].

Software estimation has been identified as one of the three great challenges for half-century-old computer science [7] and several effort estimation techniques have been proposed and researched at least from 1966 [30]. The aims of researchers have been (1) to determine which technique has the greatest effort prediction accuracy, or (2) propose new or combined techniques that could provide better estimates. Most of the research has focused on the construction of formal models and the early models were typically statistical regression-based [18].

Software development estimation techniques can be classified into three general categories [28]:

  • (1)

    Expert judgment: The term expert estimation is not clearly defined and covers a wide range of estimation approaches; a common characteristic is, however, that intuitive processes constitute a determinant part of the estimation [16]. This technique implies a lack of analytical argumentation and by the frequent use of phrases such as “I think that …” and “I feel that …” [18] and it aims to derive estimates based on an experience of experts on similar projects. The means of deriving an estimate are not explicit and therefore not repeatable [28].

  • (2)

    Algorithmic models: To date is the most popular technique in the literature [6], [28]. It attempts to represent the relationship between effort and one or more characteristic of a project; the main cost driver in a model is usually taken to be some notion of software size (e.g. the number of lines of source code). Its general form is a linear regression equation, as that used by Kok et al. [22], or by a group of non-linear regression equations as those used by Boehm [2], [4].

  • (3)

    Machine learning: In recent years machine learning techniques have been used as a complement or alternative to the previous two techniques [32], [35]. Fuzzy logic models are included in this category [27] as well as neural networks [9], genetic programming [8], regression trees [18] and case-based reasoning [19].

In accordance with a primary conclusion of a previous research [3], no single technique is best for all situations, and that a careful comparison of the results of several approaches is most likely to produce realistic estimates. Based upon this conclusion, in this paper are compared two techniques: fuzzy model with a statistical regression model. These models are generated from data of programs developed with practices of Personal Software Process (PSP). This kind of process was selected because the levels of software engineering education could be classified in the small as well as in the large, and when the approach is related to small programs, the PSP whose practices and methods are used for delivering quality products on predictable schedules, can be useful. The PSP assumes that unless engineers have the capabilities provided by personal training, they cannot properly support their teams or consistently and reliably produce quality products [12].

In this study, two kinds of lines of code as well as development time (effort) were gathered from 163 small programs developed by 53 programmers (data gathered through 2005, 2006 and 2007 years). From this set of programs both a multiple regression equation and a fuzzy logic model were generated, then their adequacy was checked (verification). Afterwards, these two models were validated with 68 programs developed by other group integrated by 21 programmers (data gathered through 2008 year); the estimation accuracies comparison amongst models was made using the Magnitude of Error Relative to the estimate (MER, with its mean MMER) as criterion. Names of the 74 programmers are depicted in Appendix C.

This paper considers guidelines for software engineering research suggested in [21].

A fuzzy model has two main properties [33]: (1) it operates at a level of linguistic terms (fuzzy sets), and (2) it represents and processes uncertainty.

Estimation techniques have an important limitation, which arises when software projects are described using categorical data (nominal or ordinal scale) such as small, medium, average, or high (linguistic terms or values). A more comprehensive approach to deal with linguistic values is using the fuzzy set theory [15]. Specifically, fuzzy logic offers a particularly convenient way to generate a keen mapping between input and output spaces thanks to the natural expression of fuzzy rules [38].

Two considerations could justify the decision of implementing a fuzzy model in software development effort estimation: first, it is impossible to develop a precise mathematical model of the domain [23]; second, metrics only produce estimations of the real complexity. In accordance with the previous assertions, formulating a tiny set of natural rules describing underlying interactions between the software metrics and the effort estimation could effortlessly reveal their intrinsic and wider correlations. Disadvantages of a fuzzy model could be that (1) it requires a lot of data, (2) the estimators must be familiar with the historically developed programs, and (3) it is not useful for programs much larger or smaller than the historical data [12]. In this research, only the third one represents a disadvantage since the sample for generating the models was conformed by 163 programs and those ones were well known to the estimator.

In this paper, data defuzzification is constructed based on a rule induction system replacing the crisp facts with fuzzy inputs, then an inference engine uses a base of rules to map inputs to a fuzzy output which can either be translated back to a crisp value or left as a fuzzy value [34]. The software tool used was MATLAB 6.1.

Before defuzzification phase, the following four stages are implemented through the fuzzy inference process: fuzzification of the input variables, application of the fuzzy operator (AND or OR) in the antecedent (independent variables), implication from the antecedent to the consequent (effort), and aggregation of the consequents across the rules.

Fuzzification stage is related to take the inputs and determine the degree to which they belong to each of the appropriate fuzzy sets via membership functions (depicted in Section 3.2), whereas the application of the fuzzy operators (AND or OR) in the antecedent are based on the truth tables depicted in Table 1.

In accordance with operations of fuzzy sets, the intersection (∩), union (∪) and complement operations are represented by the minimum, maximum and one-complement for all x  X as following (where A and B are sets defined in a universe X):μAB(x)=min(μA(x),μB(x))=A(x)B(x)μAB(x)=max(μA(x),μB(x))=A(x)B(x)A(x)=1A(x)

Once rules have been defined (depicted in Section 3.2), the implication method is implemented. A consequent is a fuzzy set represented by a membership function, which weights appropriately the linguistic characteristics that are attributed to it. The consequent is reshaped using a function associated with the antecedent. The input for the implication process is a single number given by the antecedent, and the output is a fuzzy set. Implication is implemented for each rule. The AND (min) implication method was used in this study, which truncates the output fuzzy set.

Aggregation is the process by which the fuzzy sets that represent the outputs of each rule are combined into a single fuzzy set. The input of the aggregation process is the list of truncated output functions returned by the implication process for each rule. The output of the aggregation process is one fuzzy set for each output variable. The OR (max) aggregation method was used in this study.

The type of fuzzy inference system that was used in this study was Mamdani, which expects the output membership functions to be fuzzy sets (whereas a Sugeno-type system can be used to model any inference system in which the output are either linear or constant). After the aggregation process, there is a fuzzy set for each output variable that needs defuzzification.

The input for the defuzzification process is a fuzzy set (the aggregate output fuzzy set) and the output is a single number. The aggregate of a fuzzy set encompasses a range of output values, and so must be defuzzified in order to resolve a single output value from the set. The defuzzification method used in this study is the centroid calculation, which returns the center of area under the curve.

Source lines of code (LOC) remains in favor of many models [26], [28] and researches used to utilize them to correlate effort [14]. There are two measures of source code size: physical source lines and logical source statements. The count of physical lines gives the size in terms of the physical length of the code as it appears when printed [31].

In this study, the independent variables of the models are new and changed (N&C) as well as reused code [12] and all of them are considered as physical lines of code (LOC). N&C is composed of added and modified code. The added code is the LOC written during the current programming process, while the modified code is the LOC changed in the base program when modifying a previously developed program. The base program is the total LOC of the previous program while the reused code is the LOC of previously developed programs that are used without any modification.

A coding standard should establish a consistent set of coding practices that is used as a criterion when judging the quality of the produced code [12]. Hence, it is necessary to always use the same coding and counting standards. The programs developed of this study followed these guidelines.

With the precedent of that the magnitude of relative error or MRE (a common criterion for the evaluation of cost estimation models) does not identify the best prediction model [11], it implies that the results and conclusions on prediction models over the past 15–25 years are unreliable and may have misled the entire software engineering discipline; therefore, in [11] they strongly recommend not using MRE to evaluate and compare prediction models, but the Magnitude of Error Relative to the estimate or MER proposed in [20] and defined as follows:MERi=|actualeffortiestimatedefforti|estimatedefforti

The MER value is calculated for each observation i whose effort is predicted. The aggregation of MER over multiple observations (N) can be achieved through the mean (MMER) as follows:MMER=1Ni=1NMERi

Intuitively, MER seems preferable to MRE since MER measures the error relative to the estimate. Results of MMER in [11] had better results than MMRE; this fact is the reason for using MMER in this study.

The accuracy of an estimation technique is inversely proportional to the MMER. In several papers, a MMRE  0.25 has been considered as acceptable, however, who has proposed this value neither present any reference to studies nor any argumentation providing evidence [17]. On the other hand, a reference for an acceptable value of MMER has not been found.

Fuzzy logic has been used for predicting the development effort of large programs [1], [5], [10], [29], [37], however, not any paper which predicts the effort of small programs based upon both N&C and reused code has been found having all the following characteristics (as this paper propose it):

  • Data gathered from programs developed with practices of the PSP.

  • Samples of 163 and 68 small programs for verifying and validating respectively the models.

  • Estimation accuracy comparison amongst multiple regression equation with a fuzzy logic model.

  • Magnitude of Error Relative to the estimate (MER) and its mean (MMER) used as accuracy evaluation criterion.

A paper [25] deals with small programs, however the reused code was not statistically significant because of its sample data size.

Section snippets

Materials and methods

The experiment was done inside a controlled environment having the following characteristics:

  • Each programmer developed seven small programs. Seven was a number established because of the availability of developers (the course duration is a principal concern for industrial organizations [13]).

  • Ten sessions were carried out by programmer. The first session was spent for the introduction to PSP and for studying and making the coding and counting standards. From the second to the eighth training

Calculation

The number of programs developed by the 53 programmers was 371. 159 of the 371 were excluded because they corresponded to the first, second or third program. From those 212 programs, in 175 of them lines of code were reused. In addition, 12 programs of those 175 were excluded because they presented errors in their time recording. Hence 163 was the final number that represents the sample for verifying both the fuzzy model and the regression model. In Appendix A, actual data by developer is

Verification of models

The ANOVA for MER of the programs from Appendix A shows that there is not a statistically significant difference amongst the accuracy of prediction for the two models (p-value of Table 4 is greater than 0.05).

The validity of MER ANOVA is based on the analysis of the following three assumptions of residuals:

  • (1)

    Independent samples: The group of developers was made up of separate programmers and each of them developed its own programs, hence the data are independent.

  • (2)

    Equal standard deviations: In a

Results, discussion and future research

This paper compared a machine learning technique with a regression statistical: to date the most popular technique in the literature. This comparison was motivated by the following primary conclusion of a previous research: no single technique is best for all situations, and a careful comparison of the results of several approaches is most likely to produce realistic estimates.

The samples of 163 and 68 programs for verifying and validating respectively the models were developed by 53 and 21

Acknowledgements

The author of this paper would like to thank CUCEA of Guadalajara University, Jalisco, México, Consejo Nacional de Ciencia y Tecnología (Conacyt), as well as to Programa de Mejoramiento del Profesorado (PROMEP).

Cuauhtémoc López-Martín Ph.D. in Computer Science in the Center for Computing Research of the National Polytechnic Institute of Mexico (2007). He is member of the Mexican National System of Researches. He has been reviewer from 2007 of five journals included in the ISI Thompson's Science Citation Index: Journal of Fuzzy Sets and Systems (Elsevier), the Journal of Systems and Software (Elsevier), Multiple-Valued Logic and Soft Computing, Supercomputing (Springer) and International Journal of

References (38)

  • T.I.F. De Barcelos et al.

    An investigation of artificial neural networks based prediction systems in software project management

    Journal of Systems and Software

    (2008)
  • M. Elanhi, M.W. Nisar, Y. Wang, I.A. Khan, Software Development Effort Estimation Using Fuzzy Logic, Information...
  • T. Foss et al.

    A simulation study of the model evaluation criterion MMRE

    IEEE Transactions on Software Engineering

    (2003)
  • W. Humphrey

    A Discipline for Software Engineering

    (1995)
  • W.S. Humphrey

    PSP, A Self-improvement Process for Software Engineers, SEI Series in Software Engineering

    (2005)
  • A. Idri et al.

    Fuzzy analogy: a new approach for software cost estimation

  • A. Idri et al.

    Estimating software project effort by analogy based on linguistic values

  • M. Jorgensen et al.

    Human judgment in effort estimation of software projects

  • M. Jørgensen

    A critique of how we measure and interpret the accuracy of software development effort estimation

  • Cited by (34)

    • The application of artificial intelligence in software engineering: A review challenging conventional wisdom

      2020, Data Democracy: At the Nexus of Artificial Intelligence, Software Development, and Knowledge Engineering
    • Satin bowerbird optimizer: A new optimization algorithm to optimize ANFIS for software development effort estimation

      2017, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      The first group tries to increase the accuracy of COCOMO model through fuzzification of its cost drivers (Ziauddin et al., 2013; Vishal and Kumar Verma, 2010; Abeer, 2012; Sheenu et al., 2014; Kumar and Chopra, 2013; Xua and Khoshgoftaar, 2004). The second group of research studies are concentrated on classification of software projects to remedy the problem of inconsistency and facilitate the estimation process through assigning fuzzy rules to each class of projects (Agrawal and Shrivastava, 2015; Cuauhtemoc Lopez-Martin, 2010). The similarity function of ABE is improved by the fuzzy models proposed in the third group of studies (Idri et al., 2016; Azzeh et al., 2009, 2011).

    • Predicting recycling behaviour: Comparison of a linear regression model and a fuzzy logic model

      2016, Waste Management
      Citation Excerpt :

      In Shahabi et al. (2015), logistic regression predicted landslides somewhat better than fuzzy logic (for a related study see Pradhan, 2011). Lopez-Martin (2011) did not find statistically significant differences in the accuracy of prediction of software projects’ development time when a simple fuzzy logic model was benchmarked against a linear regression model. Mahabir et al. (2006) compare two fuzzy logic models of river ice breakup to a linear regression model.

    • Towards an early software estimation using log-linear regression and a multilayer perceptron model

      2013, Journal of Systems and Software
      Citation Excerpt :

      References (Ahmed et al., 2005; Papatheocharous et al., 2010) used fuzzy logic and fuzzy decision tree, respectively for software effort estimation. Other works such as (Heiat, 2002; Lopez-Martín, 2011a,b; Lopez-Martín et al., 2008, 2011) developed neural network models and compared their works with regression models. Regression models such as linear, non-linear, stepwise and ridge have been used to predict software effort as shown in (Jiang et al., 2007; Xia et al., 2008; Tan et al., 2009; Li et al., 2010).

    View all citing articles on Scopus

    Cuauhtémoc López-Martín Ph.D. in Computer Science in the Center for Computing Research of the National Polytechnic Institute of Mexico (2007). He is member of the Mexican National System of Researches. He has been reviewer from 2007 of five journals included in the ISI Thompson's Science Citation Index: Journal of Fuzzy Sets and Systems (Elsevier), the Journal of Systems and Software (Elsevier), Multiple-Valued Logic and Soft Computing, Supercomputing (Springer) and International Journal of Software Engineering and Knowledge Engineering. He is author of two books related to estimation models and statistics applied to software engineering. His research interests are on software engineering, specifically on verification and validation of models for estimating the effort, schedule and costs of software projects. These models are based upon statistics, fuzzy logic, neural networks, genetic algorithms, and classifiers. Today he is researcher of Guadalajara University at Jalisco, Mexico.

    View full text