An Overview of Web Effort Estimation

Abstract

A cornerstone of Web project management is sound effort estimation, the process by which effort is predicted and used to determine costs and allocate resources effectively, thus enabling projects to be delivered on time and within budget. Effort estimation is a complex domain where the causal relationship among factors is nondeterministic with an inherently uncertain nature. For example, assuming there is a relationship between development effort and developers’ experience using the development environment, it is not necessarily true that higher experience will lead to decreased effort. However, as experience increases so does the probability of decreased effort. The objective of this chapter is to provide an introduction to the process of estimating effort, discuss existing techniques used for effort estimation, and explain how a Web company can take into account the uncertainty inherent to effort estimation when preparing a quote. Therefore, this chapter is aimed to provide Web companies, researchers, and students with an introduction to the topic of Web effort estimation.

Introduction

The Web is used as a delivery platform for numerous types of Web applications, ranging from complex e-commerce solutions with back-end databases using content management systems to online personal static Web pages and blogs [1]. With the great diversity of types of Web applications and technologies employed to develop these applications, there is an ever growing number of Web companies bidding for as many Web projects as they can accommodate. Under such circumstances it is usual that companies, in order to win the bid, estimate unrealistic schedules, leading to applications that are rarely developed on time and within budget.

Note that within the context of this chapter, cost and effort are used interchangeably because effort is taken as the main component of project costs. However, given that project costs also take into account other factors such as contingency and profit [2] we will use most often the word “effort” and not “cost” throughout.

The reason to estimate effort is to predict the total amount of time it will take one person or a group of people to accomplish a given task/activity/process; this estimate is generally obtained taking into account the characteristics of the new project for which an estimate is needed, and also the characteristics of previous “similar” projects for which actual effort is known. The project characteristics employed herein are only those assumed to be relevant in determining effort.

Figure 1 details an effort estimation process.

The input to this process comprises the following:

• Data on past finished projects, for which actual effort is known, represented by project characteristics (independent variables) believed to have an effect upon the amount of effort needed to accomplish a task/activity/process.

• Estimated data relating to the new project for which effort is to be estimated. Such data also uses the same project characteristics employed in (1) above.

The estimation process itself includes two subprocesses, detailed below:

• Effort model building—This subprocess represents the construction of a tangible representation of the association between project characteristics and effort using data/knowledge from past finished projects for which actual effort is known. Such representation can take several forms, for example, an equation, a binary tree, an acyclic graph. This subprocess is shown using a dashed line because in some instances no concrete model representation exists, as for example, when an effort estimate is obtained from a domain expert based solely on their previous experience.

• Deriving an effort estimate—This subprocess represents either the use of the estimated characteristics of a new project as input to a concrete effort model that provides an effort estimate, or the use of the estimated characteristics of a new project as input to [the] subprocess [from step 1 above] that derives an effort estimate using previous knowledge from past projects.

The output to the estimation process outlined in Fig. 1 is an effort estimate (dependent variable) for a new project.

We will use an example to illustrate this process. Suppose a Web company employs the following project characteristics as input to predict the effort necessary to implement a new Web application:

• Estimated number of new Web pages.

• The number of functions/features (e.g., shopping cart) to be offered by the new Web application.

• Total number of developers who will help develop the new Web application.

• Developers’ average number of years of experience with the development tools employed.

• The number of different technologies used to develop the new Web application (e.g., relational database, HTML, Javascript, Graphics software).

Of these variables, first two inputs are project characteristics used to estimate the size the problem to be solved (Web application). They are called size measures. The other three also represent project characteristics that are associated with effort; however, these three project characteristics do not measure an application's size, and hence are cojointly named “cost drivers.”

The task/activity/process to be estimated can be as simple as developing a single function (e.g., creating a Web form with 10 fields) or as complex as developing a large e-commerce application that retrieves data from a Relational database system. Regardless of the application type, in general, the one consistent input (independent variable) believed to have the strongest influence on effort is size (i.e., the total number of Web pages), with cost drivers also playing an influential role.

In most cases, even when effort estimation is mostly based on the developer(s) and project manager(s) current expertise, data and/or knowledge from past finished projects can be used to help estimate effort for new projects yet to start.

Several techniques for effort estimation have been proposed over the past 30 years in software engineering. These fall into three broad categories [3]: expert-based effort estimation, algorithmic models, and artificial intelligence (AI) techniques. Each category is described in the following sections. These techniques are presented herein for two reasons: the first one is to present the reader with a wide view regarding the types of effort estimation techniques used in the literature; the second reason is most of the techniques presented herein have also been used for Web effort estimation, as detailed in Section 4.

Expert-based effort estimation is the process whereby effort is estimated by subjective means, often based on previous experience with developing and/or managing similar projects [4]. This is by far the most commonly used technique for Web effort estimation, with the attainment of accurate effort estimates being directly proportional to the competence and experience of the individuals involved (e.g., project manager, developer). Within the context of Web development, our experience suggests that expert-based effort estimates are obtained using one of the following mechanisms:

• An estimate that is based on a detailed effort breakdown that takes into account all of the lowest level parts of an application and the functional tasks necessary to develop this application. Each task attributed with effort estimates, is repeatedly combined into higher level estimates until we finally obtain one estimate that is considered as the sum of all lower level estimate parts. This type of estimation is called bottom-up [5]. Each estimate can be an educated guess or based on sound previous experience with similar projects.

• An estimate representing an overall process to be used to develop an application, as well as knowledge about the application to be developed, that is, the product. A total estimate is suggested and used to calculate estimates for the component parts (tasks), relative portions of the whole. This type of estimation is called top-down [5].

Estimates can be suggested by a project manager, or by a group of people mixing project manager(s) and developers, usually by means of one or more brainstorming sessions.

A survey of 32 Web companies in New Zealand conducted in 2004 [6], showed that 32% prepared effort estimates during the requirements gathering phase, 62% prepared estimates during their design phase, while 6% did not have to provide any effort estimates to their customers since these were happy to pay for the development costs without the need for a quote.

Of the 32 companies surveyed, 38% did not refine their effort estimate, and 62% did refine their estimates but not often. Therefore, these results suggest that for the majority of the companies we surveyed, the initial effort estimate was used as their “final” estimate, and work was adjusted to fit this initial quote. These results corroborated those published by Jørgensen and Sjøberg [7].

Sometimes Web companies gather only effort data for past Web projects believing it to be sufficient to help obtain accurate estimates for new projects. However, without even gathering data on the project characteristics (factors) that influence effort within the context of a specific company, effort data alone is unlikely to be sufficient to warrant the attainment of accurate effort estimates for new projects.

The drawbacks of expert-based estimation are as follows:

• It is very difficult to quantify and to clearly determine the factors that have been used to derive an estimate, making it difficult to apply the same reasoning to other projects (repeatability);

• When a company finally builds up its expertise with developing Web applications using a given set of technologies, other technologies may appear and be rapidly adopted (mostly due to hype), thus leaving behind valuable knowledge that had been accumulated in the past.

• Obtaining an effort estimate based on experience with past similar projects can be misleading when projects vary in their characteristics. For example, knowing that a Web application W1 containing 20 new static HTML Web pages and 20 new images, with a development time of 80 person hours, does not mean that a very similar application (W2) will also consume 80 person hours. Let us assume that W1 was developed by a single person and that W2 will be developed by two people. Two people may need additional time to communicate, and may also have different experiences with using HTML, which may affect total effort. In addition, another application eight times the size of W1 is unlikely to take exactly eight times longer to complete. This suggests that experts should be fully aware of most, if not all, project characteristics that impact effort in order to make an informed decision; however, it is our experience that often this is not the case.

• Developers and project managers are known for providing optimistic effort estimates for tasks that they have to carry out themselves [8]. Optimistic estimates lead to underestimated effort with the direct consequence of projects being over budget and over time.

To cope with underestimation, it is suggested that experts provide three different estimates [9]: an optimistic estimate, o; a realistic estimate, r; and a pessimistic estimate, p. Based on a beta distribution, the estimated effort E is then calculated as:

$$E=(o+4r+p)/6$$

This measure is likely to be better than a simple average of o and p; however, caution is still necessary.

Although there are problems related to using expert-based estimations, some studies have reported that when used in combination with other less subjective techniques (e.g., algorithmic models, Bayesian networks) expert-based effort estimation can be an effective estimating tool [[10], [11], [12]].

Expert-based effort estimation is a process that has not been objectively detailed, where only tacit knowledge is employed; however, it can still be represented in terms of the diagram presented in Fig. 1, where the order of steps that take place to obtain an expert-based effort estimate are as follows:

• An expert/group of developers implicitly look(s) at the estimated characteristics of the new project for which effort needs to be predicted.

• Based on the data obtained in (a) they remember or retrieve data/knowledge on past finished projects for which actual effort is known.

• Based on the data from (a) and (b) they subjectively estimate effort for the new project.

Therefore, data characterizing the new project for which effort is to be estimated is needed in order to retrieve, from memory and/or a database, data/knowledge on finished similar projects. Once this data/knowledge is retrieved, effort can be estimated.

It is important to stress that within a context where estimates are obtained via expert-based opinion, deriving a good effort estimate is much more likely to occur when the previous knowledge/data about completed projects relates to projects that are very similar to the one having its effort estimated. Here, we use the principle “similar problems have similar solutions.” Note that for this assumption to be correct we also need to guarantee that the productivity of the team working on the new project is similar to the team productivity for the past similar projects.

The problems related to expert-based effort estimation aforementioned led to the proposal of other techniques for effort estimation. Such techniques are presented in the following sections.

Algorithmic techniques are to date the most popular techniques described in the Web and software effort estimation literature. Such techniques attempt to build tangible models that represent the relationship between effort and one or more project characteristics via the use of algorithmic models. Such models assume that application size is the main contributor to effort thus in any algorithmic model the central project characteristic used is usually taken to be some notion of application size (e.g., the number of lines of source code, function points, number of Web pages, number of new images). The relationship between size and effort is often translated into an equation. An example of such type of equation is given as Equation 2, where a and b are constants, S is the estimated size of an application, and E is the estimated effort required to develop an application of size S.

$$E={\mathit{aS}}^b$$

In Equation 2, when b < 1 we have economies of scale, that is, larger projects use comparatively less effort than smaller projects. The opposite situation (b > 1) gives diseconomies of scale, that is, larger projects use comparatively more effort than smaller projects. When b is either > or < 1, the relationship between S and E is nonlinear. Conversely, when b = 1 the relationship is linear.

However, as previously discussed, size alone is most unlikely to be the only contributor to effort. Other project characteristics (cost drivers), such as developer's programming experience, tools used to implement an application, maximum/average team size, are also believed to influence the amount of effort required to develop an application. Therefore, an algorithmic model should include as input not only application size but also the cost drivers believed to influence effort, such that estimated effort can be obtained by also taking into account these cost drivers (see Equation 3).

$$E={\mathit{aS}}^b\text{CostDrivers}$$

Different proposals have been made in an attempt to define the exact form such algorithmic model should take. The most popular are presented below.

COCOMO

One of the first algorithmic models to be proposed in the literature was the Constructive COst MOdel (COCOMO) [13]. COCOMO aimed to be a generic algorithmic model that could be applied by any organization to estimate effort at three different stages in a software project's development's life cycle: early on in the development life cycle, when requirements have not yet been fully specified (Basic COCOMO); once detailed requirements have been specified (Intermediate COCOMO); and when the application's design has been finalized (Advanced COCOMO). Each stage corresponds to a different model, and all three models take the same form (see Equation 4):

$$\text{EstimatedEffort}=a\mathit{EstSizeNew}\text{Pr}{\mathit{oj}}^b\mathit{EAF}$$

where:

• EstimatedEffort is the estimated effort, measured in person months, to develop an application;

• EstSizeNewProj is the size of an application measured in thousands of delivered source instructions (KDSI);

• a and b are constants which are determined by the class of project to be developed. The three possible classes are:

  • Organic: The organic class incorporates small, noncomplicated software projects, developed by teams that have a great amount of experience with similar projects, and where software requirements are not strict.

  • Semidetached: The semidetached class incorporates software projects that are half-way between “small to easy” and “large to complex.” Development teams show a mix of experiences, and requirements also present a mix of strict and slightly vague requirements.

  • Embedded: The embedded class incorporates projects that must be developed within a context where there are rigid hardware, software, and operational restrictions.

• EAF is an effort adjustment factor, calculated from cost drivers (e.g., developers, experience, tools).

The COCOMO model makes it clear that size is the main component of an effort estimate. Constants a and b, and the adjustment factor EAF all vary depending on the model used, and in the following ways:

The Basic COCOMO uses an value EAF of 1; a and b differ depending on a project's class (see Table I).

The Intermediate COCOMO calculates EAF based on 15 cost drivers, grouped into four categories: product, computer, personnel, and project (see Table II). Each cost driver is rated on a 6-point ordinal scale ranging from “very low importance” to “extra high importance.” Each scale rating determines an effort multiplier, and the product of all 15 effort multipliers is taken as the EAF.

The Advanced COCOMO uses the same 15 cost drivers as the Intermediate COCOMO; however, they are all weighted according to each phase of the development lifecycle, that is, each cost driver is broken down by development's phase (see example in Table III). This model, therefore, enables the same cost driver to be rated differently depending on development's phase. In addition, it views a software application as a composition of modules and subsystems, to which the Intermediate COCOMO model is applied to.

The four development phases used in the Advanced COCOMO model are requirements planning and product design (RPD), detailed design (DD), coding and unit test (CUT), and integration and test (IT). An overall project estimate is obtained by aggregating the estimates obtained for each of the subsystems, which themselves were obtained by combining estimates calculated for each module.

The original COCOMO model was radically improved 15 years later, and renamed as COCOMO II, and incorporates changes that have occurred in software development environments and practices over the previous 15 years [14]. COCOMO II is not detailed in this chapter; however, interested readers are referred to [14], [15].

The COCOMO model is an example of a general purpose algorithmic model, where it is assumed it is not compulsory for ratings and parameters to be adjusted (calibrated) to specific companies in order for the model to be used effectively. We have presented this model herein because it is the best known general-purpose effort estimation algorithmic model proposed to date. However, the original COCOMO model proposed in 1981 was created focusing at contractor developed mostly military projects on remote batch processing mainframe computer systems. This means that the environment assumptions used as basis for the cost drivers proposed in that model are likely not to be applicable to Web development projects.

Despite the existence of general purpose models, such as COCOMO, the effort estimation literature has numerous examples of specialized algorithmic models that were built using applied regression analysis techniques on datasets of past completed projects (e.g., [4]). Specialized and regression-based algorithmic models are most suitable to local circumstances, such as “in-house” analysis, as they are derived from past data that often represent projects from the company itself. Regression analysis, used to generate regression-based algorithmic models, provides a procedure for determining the “best” straight-line fit (see Fig. 2) to a set of project data that represents the relationship between effort (response or dependent variable) and size and cost drivers (predictor or independent variables) [4].

Using real data on Web projects, Fig. 2 shows an example of a regression line that describes the relationship between log(Effort) and log(totalWebPages). It should be noted that the original variables Effort and totalWebPages have been transformed using the natural logarithmic scale to comply more closely with the assumptions of the regression analysis techniques. Details on these assumptions and how to identify variables that need transformation described in more depth in Ref. [1] and Chapter 5. Further details on regression analysis techniques are provided in Ref. [1] and Chapter 10.

The equation represented by the regression line in Fig. 2 is as follows:

$$log(\text{Effort})=a+blog(\text{totalWebPages})$$

where a is the point in which the regression line intercepts the Y-axis, known simply as the intercept and b represents the slope of the regression line, that is, its inclination.

Equation 5 shows a linear relationship between log(Effort) and log(totalWebPages). However, since the original variables have been transformed before the regression technique was employed, this equation needs to be transformed back such that it uses the original variables. This is done by taking the long of both sides in order to provide a linear fit to the data. The resultant equation is:

$$\text{Effort}=a{\text{totalWebPages}}^b$$

This equation presents the same relationship as COCOMO's Effort Equation (see Equation 2).

Regarding the regression analysis itself, two of the most widely used techniques in the software and Web effort estimation literature are multiple regression (MR) and stepwise regression (SWR). The difference between these two techniques is that MR obtains a regression line using all the independent variables at the same time, whereas SWR is a technique that examines different combinations of independent variables, looking for the best grouping to explain the greatest amount of variation in effort. Both use least squares regression, where the regression line selected is the one that reflects the minimum values of the sum of the squared errors. Errors are calculated as the difference between actual and estimated effort and are known as “residuals” [4].

In terms of the diagram presented in Fig. 1, an algorithmic model uses constant scalar values based on past project data; however, for anyone wishing to use this model, the steps to use are as follows:

• Data on past finished projects, for which actual effort is known, is used to build an effort estimation model using multivariate regression analysis.

• The estimated characteristics of the new project for which effort needs to be predicted are entered as input to the Effort Equation built in (a).

• An effort estimate for the new Web application is obtained from (b)

How often step (a) is to be carried out varies from one company to another. If a general-purpose algorithmic model is in use, step (a) may have taken place only once, given that it is likely that such models are be used by companies without recalibration of values for the constants. Within the context of a specialized algorithmic model, step (a) is used whenever it is necessary to recalibrate the model. This can occur after several new projects are finished and incorporated to the company's database of data on past finished projects. However, a company may also decide to recalibrate a model after every new project is finished, or to use the initial model for a longer time period. If the development team remains unchanged (and assumes that the team does not have an excessive learning curve for each new project) and new projects are similar to past projects there is no pressing need to recalibrate an algorithmic model too often.

AI techniques have also been used in the field of effort estimation as a complement to, or as an alternative to, the two previous categories. Examples include fuzzy logic [16], classification and regression trees (CART) [17], neural networks [18], case-based reasoning (CBR) [3], and Bayesian networks (BN) [[19], [20], [21], [22], [23]]. Within the context of this chapter, we will cover in detail CBR, CART, and BNs models as these are currently the most popular machine learning techniques employed for Web effort estimation. A useful summary of numerous machine learning techniques can also be found in [24], [25].

CBR uses the assumption that “similar problems provide similar solutions” [18]. It provides effort estimates by comparing the characteristics of the current project for which effort is to be estimated, against a library of historical information from completed projects with known actual effort (case base).

Using CBR involves [26]:

• Characterizing a new project p, for which an estimate is required, with variables (features) common to those completed projects stored in the case base. In terms of Web and software effort estimation, features represent size measures and cost drivers that have a bearing on effort. This means that if a Web company has stored data on past projects where the, for example, data represents the features effort, size, development team size, and tools used, then the data used as input to obtaining an effort estimate will also need to include these same features.

• Use of this characterization as a basis for finding similar (analogous) completed projects, for which effort is known. This process can be achieved by measuring the “distance” between two projects at a time (project p and one finished project), based on the features’ values, for all features (k) characterizing these projects. Each finished project is compared to project p, and the finished project presenting the shortest distance overall is the “most similar project” to project p. Although numerous techniques can be used to measure similarity, nearest neighbor algorithms using the unweighted Euclidean distance measure have been the most widely used to date in Web engineering.

• Generation of an effort estimate for project p based on the effort of those completed projects that are similar to p. The number of similar projects taken into account to obtain an effort estimate will depend on the size of the case base. For small case bases (e.g., up to 90 cases), typical values use the most similar finished project, or the two or three most similar finished projects (1, 2, and 3 closest analogues) [26]. For larger case bases no conclusions have been reached regarding the best number of similar projects to use. The calculation of estimated effort is obtained using the same effort value as the closest neighbor, or the mean effort for two or more closest neighbors. This is the common choice in Web engineering.

With reference to Fig. 1, the sequence of steps used with CBR is as follows:

• The estimated size and cost drivers relating to a new project p are used as input to retrieve similar projects from the case base, for which actual effort is known.

• Using the data from (a) a suitable CBR tool retrieves similar projects to project p, and ranks these similar projects in ascending order of similarity, that is, from “most similar” to “least similar.”

• A suitable CBR tool provides an effort estimate for project p.

The sequence just described is similar to that employed with expert-based effort prediction, that is, the characteristics of a new project must be known in order to retrieve finished similar projects. Once similar projects are retrieved, effort can then be estimated.

When using CBR there are six parameters that need to be considered, which are as follows [27]:

Feature subset selection involves determining the optimum subset of features that yield the most accurate estimations. Some existing CBR tools, for example, ANGEL [3] optionally offer this functionality using a brute-force algorithm, searching for all possible feature subsets. Other CBR tools (e.g., CBR-Works from tec:inno) have no such functionality, and therefore to obtain estimated effort, we must use all of the known features of a new project to retrieve the most similar finished projects.

Case similarity relates to measuring the level of similarity between different cases. Several case similarity measures have been proposed to date in the literature, where the three most popular measures used in both the Web and software engineering literature have been the unweighted Euclidean distance, the weighted Euclidean distance, and the maximum distance [[26], [27], [28]]. However, there are also other similarity measures available, and presented in Ref. [26]. The three case similarity measures aforementioned are described below.

Unweighted Euclidean distance: The unweighted Euclidean distance measures the Euclidean (straight-line) distance d between two cases, where each case has n features. This measure has a geometrical meaning as the shortest distance between two points in an n-dimensional Euclidean space [26]. The equation used to calculate the distance between two cases x and y is the following:

$$d(x,y)=\sqrt{{|x_0-y_0|}^2+{|x_1-y_1|}^2+\cdots +{|x_{n-1}-y_{n-1}|}^2+{|x_n-y_n|}^2}$$

where x₀ to x_n represent features 0 to n of case x; y₀ to y_n represent features 0 to n of case y.

Given the following example 1:

The unweighted Euclidean distance between the new project 1 and finished project 2 would be calculated using the following equation:

$$d=\sqrt{{|100-300|}^2+{|20-15|}^2}=200.0625$$

The unweighted Euclidean distance between the new project 1 and finished project 3 would be calculated using the following equation:

$$d=\sqrt{{|100-560|}^2+{|20-45|}^2}=460.6788$$

Using the weighted Euclidean distance, the distance between projects 1 and 2 is smaller than the distance between projects 1 and 3, thus project 2 is more similar to project 1 than project 3.

Weighted Euclidean distance: The difference between the unweighted and the weighted Euclidean distance is that the latter enables the allocation of weights to features, thus quantifying their relative importance. Herein the equation used to calculate the distance between two cases x and y is the following:

$$d(x,y)=\sqrt{w_0{|x_0-y_0|}^2+w_1{|x_1-y_1|}^2+\cdots +w_{n-1}{|x_{n-1}-y_{n-1}|}^2+w_n{|x_n-y_n|}^2}$$

where x₀ to x_n represent features 0 to n of case x; y₀ to y_n represent features 0 to n of case y; w₀ to w_n are the weights for features 0 to n.

Based on Example 1 above, if we were now to attribute weight = 5 to feature newImages, and weight = 1 to feature newWebPages, the distances would be as follows:

Distance between the new project 1 and finished project 2:

$$d=\sqrt{1{|100-300|}^2+5{|20-15|}^2}=200.3123$$

The unweighted Euclidean distance between the new project 1 and finished project 3 would be calculated using the following equation:

$$d=\sqrt{1{|100-560|}^2+5{|20-45|}^2}=463.3843$$

The pattern observed herein would be similar to that previously found: the distance between projects 1 and 2 is smaller than the distance between projects 1 and 3, making project 2 more similar to project 1 than project 3.

Maximum distance: The maximum distance computes the highest feature similarity, that is, the one to define the closest analogy. For two points (x₀, y₀) and (x₁, y₁), the maximum measure d is equivalent to the equation:

$$d=\sqrt{max({(x_0-y_0)}^2,{(x_1-y_1)}^2)}$$

This effectively reduces the similarity measure down to a single feature, where the feature that is chosen may differ for each retrieval episode. So, for a given “new” project P_new, the closest project in the case base will be the one that has at least one feature with the most similar value to the same feature in project P_new.

Scaling (also known as standardization) represents the transformation of a feature's values according to a defined rule, such that all features present values within the same range; as a consequence all features have the same degree of influence upon the results [26]. A common method of scaling is to assign “zero” to the minimum observed value and “one” to the maximum observed value [29]. This is the strategy used by ANGEL and CBR-Works, that is, for each feature, its original values are normalized (between 0 and 1) by CBR tools to guarantee that they all influence the results in a similar fashion.

The number of analogies refers to the number of most similar analogues (cases) that will be used to generate an effort estimate. With small sets of data, it is reasonable to consider only a small number of most the similar analogues [26]. Several studies in Web and software engineering have used only the closest analogue (CA) (k = 1) to obtain an estimated effort for a new project [[30], [31]], while others have also used the two closest and the three closest analogues [[5], [26], [28], [32], [33], [34], [35], [36]].

Once the most similar analogues have been selected the next step is to identify how to adapt an effort estimate for project P_new. Choices of adaptation techniques presented in the literature vary from the nearest neighbor [[30], [33]], the mean of the CAs [3], the median of the CAs [26], the inverse distance weighted mean and inverse rank weighted mean [29], to illustrate just a few. The adaptations used to date for Web engineering are the nearest neighbor, mean of the CAs [[5], [28]], and the inverse rank weighted mean [[31], [34], [35], [37]].

Each adaptation is explained below:

Nearest neighbour: For the estimated effort P_new, this type of adaptation uses the same effort of its CA.

Mean effort: For the estimated effort P_new, this type of adaptation uses the average of its closest k analogues, when k > 1. This is a typical measure of central tendency, often used in the Web and software engineering literature. It treats all analogs as being equally important towards the outcome—the estimated effort.

Median effort: For the estimated effort P_new, this type of adaptation uses the median of the closest k analogs, when k > 2. This is also a measure of central tendency, and has been used in the literature when the number of selected closest projects is > 2 [26].

Inverse rank weighted mean: This type of adaptation allows higher ranked analogues to have more influence over the outcome than lower ones. For example, if we use three analog, then the CA would have weight = 3, the second closest analogue (SC) would have weight = 2, and the third closest analogue (LA) would have weight = 1. The estimated effort would then be calculated as:

$$\text{Inverse Rank Weighed Mean}=\frac{3\mathit{CA}+2\mathit{SC}+\mathit{LA}}{6}$$

Adaptation rules represent the adaptation of the estimated effort, according to a given criterion, such that it reflects the characteristics of the target project (new project) more closely. For example, in the context of effort prediction, the estimated effort to develop an application a would be adapted such that it would also take into consideration the size value of application a. The adaptation rule that has been employed to date in Web engineering is based on the linear size adjustment to the estimated effort [[34], [35]], obtained as follows:

• Once the most similar analogue in the case base has been retrieved, its effort value is adjusted and used as the effort estimate for the target project (new project).

• A linear extrapolation is performed along the dimension of a single measure, which is a size measure strongly correlated with effort. The linear size adjustment is calculated using the equation presented below.

$${\text{Effort}}_{\text{newProject}}=\frac{{\text{Effort}}_{\text{finishedProject}}}{{\text{Size}}_{\text{finishedProject}}}{\text{Size}}_{\text{newProject}}$$

Given the following example:

The estimated effort for the target project will be calculated as:

$${\text{Effort}}_{\text{newProject}}=\frac{80}{450}100=17.777$$

When we use more than one size measure as feature, the equation changes to:

$$E_{\text{est}.P}=\frac{1}{q}\left(\sum _{q=1}^{q=x}\frac{E_{\text{act}}{S}_{\text{est}.q}}{{S_{\text{act}.q}|}_{>0}}\right)$$

where

q is the number of size measures used as features.

E_est.P is the Total Effort estimated for the new Web project P.

E_act is the Total Effort for the CA obtained from the case base.

S_est.q is the estimated value for the size measure q, which is obtained from the client.

S_act.q is the actual value for the size measure q, for the CA obtained from the case base.

This type of adaptation assumes that all projects present similar productivity; however, this may not be an adequate assumption for numerous Web development companies worldwide.

CART [38] are techniques whereby binary trees are constructed using data from finished projects for which effort is known, such that each leaf node represents either a category to which an estimate belongs, or a value for an estimate.

The data used to build a CART model is called a “learning sample,” and once a tree has been built it can be used to estimate effort for new projects.

In order to obtain an effort estimate for a new project p one has to traverse the tree from root to leaf by selecting the node values/categories that are the closest match to the estimated characteristics of p.

For example, assume we wish to obtain an effort estimate for a new Web project using as its basis the simple binary tree presented in Fig. 3.

This binary tree was built from data obtained from past completed Web applications, taking into account their existing values of effort and project characteristics (independent variables) believed to be influential upon effort (e.g., total number of new Web pages (newWebPages), total number of new Images (newImages), and the number of Web developers (numDevelopers)).

Assuming that the estimated values for newWebPages, newImages, and numDevelopers for a new Web project are 38, 15, and 3, respectively, we would obtain an estimated effort of 75 person hours after traversing the tree from its root down to leaf “Effort = 75.”

If we now assume that the estimated values for newWebPages, newImages, and numDevelopers are 26, 34, and 7, respectively, we would obtain an estimated effort of 65 person hours after navigating the tree from its root down to leaf “Effort = 65.”

In this particular example all the variables that characterize a Web project are numerical. Whenever this is the case the binary tree is called a regression tree.

Now let us look at the binary tree shown in Fig. 4. It uses the same variable names as that shown in Fig. 3; however, these variables are now all categorical, where possible categories (classes) are “Yes” and “No.” This binary tree is therefore called a Classification tree.

A CART model constructs a binary tree by recursively partitioning the predictor space (set of all values or categories for the independent variables judged relevant) into subsets where the distribution of values or categories for the dependent variable (e.g., effort) is successively more uniform. The partition (split) of a subset S1 is decided on the basis that the data in each of the descendant subsets should be “purer” than the data in S1. Thus node “impurity” is directly related to the amount of different values or classes in a node, that is, the greater the mix of classes or values, the higher the node “impurity.” A “pure” node means that all the cases (e.g., Web projects) belong to the same class, or have the same value. The partition of subsets continues until a node contains only one class or value. Note that it is not always the case that all the independent variables are used to build a CART model, rather only those variables that contribute effectively to effort are selected by the model. This means that in this instance a CART model can be used not only for effort prediction, but also to obtain insight and understanding of the variables that are relevant to estimate effort. A detailed description of CART models is given in Ref. [1] and Chapter 7.

In terms of the diagram presented in Fig. 1, an effort estimate is obtained according to the following steps:

• Data on past finished projects, for which actual effort is known, is used to build an effort estimation model using a binary tree (CART).

• The estimated characteristics of the new project for which effort needs to be predicted are used to traverse the tree built in (a).

• An effort estimate for the new Web application is obtained from (b).

The sequence of steps described above, corresponds to the same sequence used with the algorithmic techniques. In both situations, data is used to either build an equation that represents an effort model, or to build a binary tree, which is later used to obtain effort estimates for new projects. This sequence of steps contrasts to the different sequence of steps used for expert opinion and CBR, where knowledge about a new project is used to select similar projects.

A BN is a model that supports reasoning with uncertainty due to the way in which it incorporates existing knowledge of a complex domain [39]. This knowledge is characterized using two parts. The first, the qualitative part, represents the structure of a BN as depicted by a directed acyclic graph (digraph) (see Fig. 5). The digraph's nodes represent the relevant variables (factors) in the domain being modeled, which can be of different types (e.g., observable or latent, categorical). The digraph's arcs represent the causal relationships between variables, where relationships are quantified probabilistically [39].

The second, the quantitative part, associates a conditional probability table (CPT) to each node, its probability distribution. A parent node's CPT describes the relative probability of each state (value) (Fig. 5, nodes “Pages complexity” and “Functionality complexity”); a child node's CPT describes the relative probability of each state conditional on every combination of states of its parents (Fig. 5, node “Total Effort”). So, for example, the relative probability of “Total Effort” being “Low” conditional on “Pages complexity” and “Functionality complexity” being both “Low” is 0.7.

Each row in a CPT represents a conditional probability distribution and therefore its values sum up to 1 [39].

Formally, the posterior distribution of a BN is based on Bayes’ rule [39]:

$$p(X|E)=\frac{p(E|X)p(X)}{p(E)}$$

where

• p(X|E) is called the posterior distribution and represents the probability of X given evidence E;

• p(X) is called the prior distribution and represents the probability of X before evidence E is given;

• p(E|X) is called the likelihood function and denotes the probability of E assuming X is true.

Once a BN is specified, evidence (e.g., values) can be entered into any node, and probabilities for the remaining nodes are automatically calculated using Bayes’ rule [39]. Therefore, BNs can be used for different types of reasoning, such as predictive, diagnostic, and “what-if” analyses to investigate the impact that changes on some nodes have on others [40].

BNs can be built from different sources—data on past finished projects, expert opinion, or a combination of these. In any case, the process used to build a BN is depicted in Fig. 6 and briefly explained below.

This process is called the Knowledge Engineering of Bayesian Networks (KEBN) process, which was initially proposed in Ref. [41] (see Fig. 6). In Fig. 6, arrows represent flows through the different processes, depicted by rectangles. Such processes are executed either by people—the Knowledge Engineer (KE) and the Domain Experts (DEs) (white rectangles), or by automatic algorithms (dark gray rectangles).

The three main steps within the KEBN process are the Structural development, parameter estimation, and model validation. This process iterates over these steps until a complete BN is built and validated. Each of these three steps is detailed below:

Structural development: This step represents the qualitative component of a BN, which results in a graphical structure comprised of, in our case, the factors (nodes, variables) and causal relationships identified as fundamental for effort estimation of Web projects. In addition to identifying variables, their types (e.g., query variable, evidence variable) and causal relationships, this step also comprises the identification of the states (values) that each variable should take, and if they are discrete or continuous. In practice, currently available BN tools require that continuous variables be discretized by converting them into multinomial variables [39]. The BN's structure is refined through an iterative process, and uses the principles of problem solving employed in data modeling and software development. In order to elicit a BN structure, a combination of existing literature in the related knowledge domain, existing data and knowledge from DEs can be used. Throughout this step the KE(s) also evaluate(s) the BN's structure in two stages. The first entails checking whether variables and their values have a clear meaning; all relevant variables have been included; variables are named conveniently; all states are appropriate (exhaustive and exclusive); a check for any states that can be combined. The second stage entails reviewing the BN's graph structure (causal structure) to ensure that any identified d-separation dependencies comply with the types of variables used and causality assumptions. D-separation dependencies are used to identify variables influenced by evidence coming from other variables in the BN [[39], [42]]. Once the BN structure is assumed to be close to final, KEs may still need to optimize this structure to reduce the number of probabilities that need to be elicited or learnt for the network. If optimization is needed then techniques that change the causal structure (e.g., divorcing) and the use of parametric probability distributions (e.g., noisy-OR gates) are employed [[39], [42]].

Parameter estimation: This step represents the quantitative component of a BN, where conditional probabilities corresponding to the quantification of the relationships between variables are obtained [[39], [42]]. Such probabilities can be attained via Expert Elicitation, automatically from data, from existing literature, or using a combination of these. When probabilities are elicited from scratch, or even if they only need to be revisited, this step can be very time consuming.

Model validation: This step validates the BN that results from the two previous steps, and determines whether it is necessary to revisit any of those steps. Two different validation methods are generally used—Model Walkthrough and Predictive Accuracy.

Model Walkthrough represents the use of real case scenarios that are prepared and used by DEs to assess if the predictions provided by a BN correspond to the predictions experts would have chosen based on their own expertise. Success is measured as the frequency with which the BN's predicted value for a target variable (e.g., quality, effort) that has the highest probability corresponds to the experts’ own assessment.

Predictive Accuracy uses past data (e.g., past project data), rather than scenarios, to obtain predictions. Data (evidence) is entered on the BN model, and success is measured as the frequency with which the BN's predicted value for a target variable (e.g., quality, effort) that has the highest probability corresponds to the actual past data.

Within the context of Web effort estimation and to some extent software effort estimation, the challenge using Predictive Accuracy is the lack of reliable effort data gathered by Web and Software companies. Most companies, who claim to collect effort data, use manually entered electronic timesheets (or even paper!) which is unreliable when staff rely on their memory and complete their timesheets at the end of the day. Collecting manually entered timesheets every 5 min (assume 1 min/entry) in a bid to improve data accuracy increases data collection cost by as much as 10-fold. The problem here is that “effort accuracy” is inversely related to productivity, that is, the longer one takes filling out timesheets the less time one has to do the real work!

In terms of the diagram presented in Fig. 1, an effort estimate is obtained according to the following steps:

• Data on past finished projects/knowledge from DEs/both is/are used to build an effort estimation BN model.

• The estimated characteristics of the new project for which effort needs to be predicted are used as evidence, and applied to the BN model built in (a).

• An effort estimate for the new Web application is obtained from (b).

The sequence of steps described above, corresponds to the same sequence used with the algorithmic techniques and CART. In all situations, data/knowledge is used to either build an equation, a binary tree, or an acyclic graph that represents an effort model, which is later used to obtain effort estimates for new projects. As previously stated, this sequence of steps contrasts to the different sequence of steps used for expert opinion and CBR, where knowledge about a new project is used to select similar projects.