1 Introduction

Software reliability is the most important quality metric used to measure the quality of commercial software during the testing phase and at the release time. It can provide vital information for the software release decision. Software reliability growth models (SRGMs) are the tools used to evaluate software quantitatively, develop test cases, schedule status, faults remaining in the software and estimate and predict the reliability of software during testing and operation profile. SRGMs describe the failure occurrence and\or failure removal and consequently enhancement in reliability with respect to time (CPU time, calendar time or execution time) and testing effort. Several SRGMs have been discussed and validated in literature by many authors under different set of assumptions (Pham 2006; Kapur et al. 2011). Some SRGMs describe the failure phenomenon by an exponential (G–O Model Goel and Okumoto 1979) or a S-shaped (Yamada delayed S-shaped, Yamada et al. 1983 model). While some of the SRGMs are flexible in the sense that they can depict, depending upon parameter values, both exponential and S-shaped model (Ohba 1984; Bittanti et al. 1988; Kapur and Garg 1992, etc.).

In most of the models discussed above it is assumed that whenever an attempt is made to remove a fault, it is removed with certainty i.e. a case of perfect debugging. But the debugging activity is not always perfect because of number of factors like tester’s skill/expertise etc. In practical software development scenario, the number of failures observed/detected may not be necessarily same as the number of errors removed/corrected. Kapur and Garg (1992) have discussed in their error removal phenomenon model that as testing grows and testing team gains experience, additional numbers of faults are removed without them causing any failure.

The testing team, however, may not be able to remove/correct fault perfectly on observation/detection of a failure and the original fault may remain leading to a phenomenon known as imperfect debugging, or replaced by another fault resulting in error generation. In case of imperfect debugging the fault content of the software is not changed, but because of incomplete understanding of the software, the original detected fault is not removed perfectly. But in case of error generation, the total fault content increases as the testing progresses because new faults are introduced in the system while removing the old original faults.

It was Goel (1985) who first introduced the concept of imperfect debugging. Model due to Obha and Chou (1989) is an error generation model applied on G-O model and has been also named as Imperfect debugging model. Kapur et al. (2011) introduced the imperfect debugging in Goel and Okumoto (1985). They assumed that the fault detection rate per remaining faults is reduced due to imperfect debugging. Thus the number of failures observed/detected by time infinity is more than the initial fault content. Although these two models describe the imperfect debugging phenomenon yet the software reliability growth curve of these models is always exponential. Moreover, they assume that the probability of imperfect debugging is independent of the testing time. Thus, they ignore the role of the learning process during the testing phase by not accounting for the experience gained with the progress of software testing. Pham (2006) developed an SRGM for multiple failure types incorporating error generation. Zhang et al. (2003) proposed a testing efficiency model which includes both imperfect debugging and error generation, modeling it on the number of failures experienced/observed/detected however both imperfect debugging and error generation are actually seen during fault removal/correction. Recently, Kapur et al. (2006) proposed a flexible SRGM with imperfect debugging and error generation using a logistic function for fault detection rate which reflects the efficiency of the testing/removal team.

Testing is an efficient way to remove faults in the software products. However, exhaustive testing of all possible executable paths in a general program is, in fact, impractical. Moreover, after reaching a certain level of software refinement, any effort to increase reliability will require exponential increase in cost. Also accurate software pricing is critical to both developers and customers. It can be used for generating request for proposals, contract negotiations, scheduling, monitoring and control. Underestimating the costs may result in management approving proposed systems that then exceed their budgets, with underdeveloped functions and poor quality, and failure to complete on time. Overestimating may result in too many resources committed to the project, or, during contract bidding, result in not winning the contract, which can lead to loss of jobs. Therefore, it is important to determine the time to stop testing or when to release the software system to the user and is known as “Release Time Problem”. This decision depends on the model used for describing the failure phenomenon and the criterion used for determining system readiness. The optimization problem of determining the optimal time of software release can be formulated based on goals set by the management. Firstly the management may wish to determine the optimal release time such that total expected cost of testing in the testing and operation phase is minimum. Secondly they may set a reliability level to be achieved by the release time. Thirdly they may wish to determine the release time such that the total expected cost of the software is minimum and reliability of the software is achieved to a certain desired level. Such a problem is known as a Bi-criteria release time problem. For Bi-criteria release time problem release time is determined by carrying a tradeoff between cost and reliability so that the total expected software cost under a given warranty is minimum and customers get the desired quality of software at the best price. Many researchers in literature have studied various release time problems for different SRGMs (Pham 2006; Kapur and Garg 1990; Yamada 1994; Kapur et al. 2007, 2011).

Software cost estimation historically has been a major difficulty in software development. In the last three decades, many quantitative software cost estimation models have been developed. They range from empirical models such as Boehm’s COCOMO models (1981) to analytical models such as those in (Boehm et al. 1996; Cantone et al. 1986). An empirical model uses data from previous projects to evaluate the current project and derives the basic formulae from analysis of the particular database available. An analytical model, on the other hand, uses formulae based on global assumptions, such as the rate at which developer solves problems and the number of problems available.

1.1 Cost factors

The most comprehensive set of cost factors is proposed and used by Boehm et al. (1996) in the COCOMO II model. These cost factors can be divided into four types:

Product factors required reliability; product complexity; database size used; required reusability; documentation match to life-cycle needs;

Computer factors execution time constraint; main storage constraint; computer turnaround constraints; platform volatility;

Personnel factors analyst capability; application experience; programming capability; platform experience; language and tool experience; personnel continuity;

Project factors multisite development; use of software tool; required development schedule.

The above factors are not necessarily independent, and most of them are hard to quantify. In many models, some of the factors appear in combined form and some are simply ignored. Also, some factors take discrete values, resulting in an estimation function with a piece-wise form.

1.2 Warranty

Warranty is seller assurance to a buyer that a product will carry out as stated and this promise works as a confidence for the buyer in purchasing the product. It is not uncommon for software companies to provide a warranty period during which they are responsible for fixing software errors. Whether a firm is selling commercial products or marketing to the Government, warranties is an imperative ingredient to competitive victory. Warranties are also an effective means of promoting a product in the market when the company or product is not well known. Or in other words warranty serves as a source for spread of an innovation in the market. But servicing a warranty engages additional costs to the manufacturer and this has an effect on the profit levels. A warranty is the developer’s promise to the user that the software will meet user’s expectations. It is a contractual agreement between the developer and the user prepared at the time of sale.

Nowadays high technology products are shrinking in the size of their appearance. But this contraction is expanding their technological complexities. Hence, today customers analyze the performances and functioning of these products with an eagle eye before purchasing. The willingness of a consumer to shell out a high price for the product is governed by its quality. And in turn this assurance from seller’s side is promised by the product’s warranty. Product warranty is a major factor that influences the buyer’s decision to purchase the product in a high competitive marketing environment. It is used as a instrument to catch the attention of the customer and construct faith relationship between the buyer and seller in product. Many marketers employ warranty as improved service to its basic product offerings. It is used as a promotional tool by various well established companies for selling the product.

Warranty is an important marketing tool which is viewed as an indicator of the product reliability and the quality. Servicing a warranty involves additional cost to the developer which in turn affects his profit level. It can be reduced by prolonged testing of the software which in turn raises the software testing cost. Thus a software development process need to be closely monitored under the consideration of testing cost, release time, warranty cost and its reliability. If no proper planning is done in these directions then sales price of the software may increase. Higher prices may act as deterrent to customer purchases thereby reducing sales and huge dent in the developer’s profit. Clearly the warranty period, warranty cost, testing cost, sales price, product adoption and reliability are inter-related parameters. Out of all these three namely warranty period, sales price and reliability are set by the developer. As warranty period, sales price and reliability are inter-related so there is a need to determine these three parameters simultaneously so as to maximize the profit to the developer.

The proposed model helps in quality planning with respect to determination of optimal testing time and pricing policy under reliability and sales constraint with given warranty period under two types of imperfect debugging. For this we have used NHPP flexible SRGM. This paper is organized as follows: In Sect. 2, we review Kapur and Garg model for fault removal process. In Sect. 3, we develop sales function and the cost function to formulate an optimal release problem for determining optimal testing time, and price of the product with given warranty under two types of imperfect debugging. In Sect. 4, the results are illustrated with a numerical example. Finally conclusions are drawn in Sect. 5.

2 A flexible SRGM for fault removal process (Kapur and Garg model)

During the testing and debugging process the test cases are executed and the results are obtained. This model is based upon the assumption that if there is a failure, then the underlined fault is removed. The testing team may remove some faults in the software, without these faults causing any failure; although this may involve some additional effort. Fault, which is removed consequent to a failure, is known as a leading fault. While removing the leading faults, some other faults are removed which may have caused failure in future. These faults are known as dependent faults.

Kapur and Garg (KG) (1992) have described the above error removal phenomenon in their SRGM based on the non homogeneous poisson process (NHPP). Here we discuss KG model in detail.

This model is based upon the following assumptions:

  1. 1.

    Fault removal phenomenon is modeled by NHPP.

  2. 2.

    Software is subject to failures during execution caused by faults remaining in the software.

  3. 3.

    Each time a failure is observed, an immediate effort takes place to decide the cause of the failure in order to remove it.

  4. 4.

    The time between failure observation and subsequent fault removal is negligible.

  5. 5.

    Failure rate is equally affected by all the faults remaining in the software.

  6. 6.

    When a software failure occurs, an instantaneous repair effort starts and the following may occur:

    1. (a)

      Fault content is reduced by one with probability p.

    2. (b)

      Fault content remains unchanged with probability 1-p.

  7. 7.

    During the fault removal process, whether the fault is removed successfully or not, new faults are generated with a constant probabilityα.

Assumption 6 and 7 captures the effect of imperfect debugging and error generation respectively.

Based on the assumptions the fault removal intensity per unit time can be written as

$$\frac{d}{dt}m_{r} (t) = p_{1} [a(t) - m_{r} (t)] + q_{1} \frac{{m_{r} (t)}}{a}[a(t) - m_{r} (t)]$$
(1)

Here p 1 and q 1 are fault detection rate for independent and dependent fault. Let us define b = (p 1 + q 1) and β = q 1/p 1, then KG model can be derived alternatively if we assume logistic learning fault detection rate given by

$$b(t) = {\raise0.7ex\hbox{$b$} \!\mathord{\left/ {\vphantom {b {1 + \beta e^{ - bt} }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${1 + \beta e^{ - bt} }$}}$$

using above (1) can be rewritten as:

$$\frac{d}{dt}m_{r} (t) = \frac{\beta }{{1 + \beta \cdot e^{ - b.t} }}[a(t) - m_{r} (t)]$$
(2)

We assume that faults can be introduced during the debugging phase with a constant fault introduction rate ɛ. Therefore, the fault content rate function, a(t), is a linear function of the expected number of faults detected by time t. That is,

$$a\left( t \right) = a + \varepsilon m_{r} \left( t \right)$$
(3)

Now incorporating the imperfect debugging and error generation in modeling, Eq. (2) becomes:

$$\frac{d}{dt}m_{r} (t) = \frac{\beta p}{{1 + \beta .e^{ - b.t} }}[a + \varepsilon m_{r} (t) - m_{r} (t)]$$
(4)

Solving Eq. (4) with the initial condition m r (0) = 0, the expected number of faults removed in (0, t] is given as

$$m_{r} (t) = \frac{a}{(1 - \varepsilon )}\,\,\left[ {1 - \left( {\frac{{(1 + \beta )e^{ - b\,\,t} }}{{1 + \beta \,\,e^{ - b\,\,t} }}} \right)^{p(1 - \varepsilon )} } \right]$$
(5)

Note that m r (t) = pm f (t)

In the next section we develop a software cost model by taking into consideration set-up cost, cost of testing, warranty cost and cost of removing faults during testing and operational phase.

2.1 Proposed optimization model

2.1.1 Assumptions

  • The cost to perform testing is proportional to the testing time.

  • The cost to remove errors during the testing phase is proportional to the total time of removing all errors detected by the end of the testing phase.

  • The fault removal process is modeled by KG model.

  • There is a set-up cost at the beginning of the software development process.

  • The software sales increases as the software reliability and warranty period increases and it decreases as the selling price increases.

Software can neither be tested for an infinite period, since it increases its testing cost exponentially during the later stages of testing nor it can be released prematurely as the cost of fixing the faults in operational phase increases. On the contrary for safety critical systems developing highly reliable software is the prime objective of the management.

The optimization problem of determining the optimal time of software release can be formulated based on goals set by the management in terms of cost, reliability and failure intensity etc. subject to the system constraints. Such a problem is known as “Optimal Software Release time Problem”. It is important to determine when to stop testing, or when to release the software to customers, so that the expected total software cost subject to warranty and risk costs is minimum. Although lot of research has been done in the area of release time optimization during last three decades and several new cost models have been developed, the foremost among all is to minimize the total software development cost since the development of a software system is time-consuming and costly. Software users demand faster deliveries, cheaper software and quality product whereas software developers aim at minimizing their development cost, maximizing the profit margins and meeting the competitive requirements. Now we present software pricing model by considering release planning, the expected software sales and expected software development cost.

3 Software cost

A major concern in software development is the cost. It is well known that the development of a software system is time-consuming and costly process. Testing is performed under controlled environment. Cost involved in testing and removing of faults and documentation during testing can be estimated but real difficulties arise in quantifying the cost of a failure at the user end. In our model the cost of failure and removal of a fault during a fixed warranty period after the release of the software is included. Thus the total expected software-testing cost comprises of the set-up cost, the cost to do testing, the cost incurred in removing errors during the testing phase and during the warranty period. Hence the expected total software system cost can be expressed as follows:

3.1 Setup cost

The development cost is the sum of the requirement study and design cost (to improve reliability) and coding cost. The development cost is incurred before the testing of the software starts. It usually depends on the development time and work effort spent during the initial phase of software development life cycle viz. feasibility study, requirement analysis, design and coding. We assume that the research and development cost is a function of unreliability parameter λ, and is modeled as follows:

$$E_{0} (T) = C_{0} = S_{1} + S_{2} \cdot \lambda^{ - m}$$

where S1 is the fixed (setup) cost, λ is the product unreliability parameter and S1, S2, m > 0. This implies that as λ decreases (the product is more reliable) the development cost increases. It may be noted that higher is the reliability less is the value of λ and C0 is the development cost. λ is given by \(\lambda = \frac{{\left( {a - m_{r} (T)} \right)}}{a}\) Lyu (1996)

3.2 Faults removal cost during testing period

The expected cost to remove errors detected by time T in the testing phase is given by \(E_{1} \left( {T,p} \right)\; = \;\left( {C_{1} p\; + \;C_{1}^{\prime} \left( {1\; - \;p} \right)} \right)\cdot m_{f} \left( T \right)\)

3.3 Faults removal cost during warranty period

Similarly the expected cost to remove errors during warranty period \(\left[ {T,\;T\; + \;W} \right]\) is given by E 2(T, p) = (C 2 p + C 2′(1 − p)) · {m f (T + W) − m f (T)} · X δ This cost is multiplied by the factor X δ, as there is lesser number of customers who come to the company for error removal during the warranty period due to number of reasons such as some may not find any error during warranty period or some may find error in the software after the warranty of the software expires or unwillingness to report the bug to the developer (Here it may be noted that 0 < δ < 1). For simplicity we have taken δ = 0.5.

3.4 Cost of testing

Testing cost parameter C 3 depends on the testing team composition and testing strategy used. If the probability of perfect debugging is to be increased and probability of error generation to be decreased, it is expected that extra financial resources will be needed to engage more experienced testing personnel, and this will result in an increase of C 3, and the expected cost is given by

$$E_{3} \left( {T,p,\varepsilon } \right) = \frac{{C_{3} \cdot T}}{{\left( {1 - p(1 - \varepsilon )} \right)}}$$

3.5 Total expected software cost

All these costs lead to the following form of the expected total software development cost function:

$$E(T,W,p,\varepsilon) = \left( \begin{array}{l} C_{0} + \left( {C_{1} p + C_{1} '\left( {1 - p} \right)} \right)\cdot m_{f} (T) \\ + \left( {C_{2} p + C_{2} '\left( {1 - p} \right)} \right).\{ m_{f} (T + W) - m_{f} (T)\} \cdot \sqrt X \\ + \frac{{C_{3} \cdot T}}{{\left( {1 - p(1 - \varepsilon )} \right)}} \\ \end{array} \right)$$

3.6 Sales model

We assume that the expected software sales depends:

  1. (1)

    Inversely upon price of the software with the parameter β(>0) which reflects the influence of price variable. It may be noted that the price has negative effect on the sales as the sales decreases with increase in price of the software.

  2. (2)

    Directly upon the warranty period (W) with elasticity parameter α(>0). Here length of the warranty period has a positive effect on sales. As the increased warranty provides a greater assurance to the buyer about the quality of the product.

  3. (3)

    Inversely upon λ, which denotes the unreliability of the software.

Including all the above factors the expected sales is given by

$$X = \theta e^{{ - \beta_{1} P}} W^{\alpha } \lambda^{{ - \gamma_{1} }}$$

The factor θ, which represents the competitor and the other environmental influences, such as number of potential consumers, the consumer purchasing power, technology up gradation etc. The relationship between sales and the factors mentioned above has been captured by Cobb–Douglus type sales function Ladany and Shore (2007).

Revenue from the expected sales is given by

$$U(T,P,W,p,\varepsilon ) = P.X$$

3.7 Expected profit

The expected profit over the life cycle of the product depends upon (1) sales price (2) sales volume (3) development cost (4) expected warranty cost. This is given by

$$\Pr ofit = Z(T,P,W,p,\varepsilon ) = U(T,P,W,p,\varepsilon ) - E(T,P,W,p,\varepsilon )$$

Our objective is to determine the optimal value of testing time and price per unit for given period of warranty under the constraints on reliability and sales of the software product. Therefore the mathematical model is

$$\begin{aligned} & Max\,Z\,\,(T,P,W)\,\, = P.X - \,\left( \begin{array}{l} C_{0} + \left( {C_{1} p + C_{1} '\left( {1 - p} \right)} \right)\cdot m_{f} (T) \\ + \left( {C_{2} p + C_{2} '\left( {1 - p} \right)} \right)\cdot \{ m_{f} (T + W) - m_{f} (T)\} \cdot \sqrt{X}l \\ + \frac{{C_{3} \cdot T}}{{\left( {1 - p(1 - \varepsilon )} \right)}} \\ \end{array} \right) \\ & \begin{array}{ll} subject\,to & \\ X \ge X_{0} & \left( {p1} \right) \\ \lambda \le \lambda_{0} & \\ P,W,T \ge 0 & \\ \end{array} \\ \end{aligned}$$

Here \(X_{0} \,{\text{and}}\,\lambda_{0}\) are the aspiration level on the total sales and unreliability of the software.

4 Numerical example and data set

To verify the proposed model that considers both testing and operational phases simultaneously, we used software data collected from Tandem computers (Wood 1996).This dataset includes software faults removed during 20 weeks of testing. In our study, we use Method of Least Squares for estimation of parameters. The parameters of the release are estimated and the related mean value functions are obtained. Here we estimated the parameters a, b and β for testing phase using SPSS. Estimated values are a = 110.83, b = 0.172, β = 1.205.

Let us consider the following coefficients in the cost model:

S 1 = 20000, S 2 = 10000, m = 2, C 1 = 200, C 1′ = 225, C 2 = 250, C 2′ = 350, C 3 = 50, α = 0.15, γ 1 = 0.07, θ = 100000, β1 = 0.0006, p = .96, ɛ = .0230. Let the sales and unreliability target be X 0 = 75000, λ 0 = 0.05. On substituting the above values in problem (p) we get

$$\begin{gathered} Max\,Z\,(T,P,W) = P \cdot (10^{5} Exp( - 0.0006P)W^{0.15} \cdot \left( {1 - \frac{{m_{r} (T)}}{110.83}} \right)^{ - 0.06} \hfill \\ - 2 \times 10^{4} + 10^{4} \times \,\,\left( {1 - \frac{{m_{r} (T)}}{110.83}} \right)^{ - 2} \hfill \\ \,\, + (200 \times .96 + 225(1 - .96))\cdot m_{f} (T) \hfill \\ + (250 \times .96 + 350(1 - .96))\left( {m_{f} (T + W) - m_{f} (T)} \right) \cdot \left( {(10^{5} Exp( - 0.0006P)W^{0.15} \cdot \left( {1 - \frac{{m_{r} (T)}}{110.83}} \right)^{ - 0.06} } \right)^{{{\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}}} \hfill \\ + 50\frac{T}{(1 - .96(1 - .0230))}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \hfill \\ subject\,to \hfill \\ X \ge 75000 \hfill \\ \lambda \le 0.05 \hfill \\ P,W,T \ge 0 \hfill \\ \end{gathered}$$

As per the industry norms the warranty is provided for different time periods varying from few months to few years in the case of software industry (Warranty is usually for 6 months or 1 year i.e. 24 or 52 weeks). In our problem we assumed W = 52. Solving the above problem using maple software we get Profit = 1.322 × 108, P = 1669.73 and T = 24.73.

4.1 Sensitivity analysis

From the discussion given in the preceding section, it is good to know that the optimal decision-making depends on various parameters that may not be precise. The use of sensitivity analysis will help the analyst to understand how changing the parameters of the model will affect the decision outcome. The decision model is then rerun by holding all other parameters constant. Sensitivity analysis is generally done by changing one parameter and setting the other parameters at their fixed values. We have conducted sensitivity analysis by calculating relative change of optimal time (T) for different values of λ o for X o  = 75000 and W = 52, P = 1300 in Table 1.

Table 1 (For fixed p = 1300)
Full size table

4.2 Effect of change of λ o on testing time

From the Table 1 it may be observed as the reliability requirements become more stringent (i.e. lower levels of unreliability) there is an increase in testing time of the software. It may be attributed to the fact that if the developer wants to offer a more reliable product then more bugs need to be removed from the software which in turn will increase the testing time. It shows that increase in reliability increases the sales of software which in turn in increases the increases the profit of company.

5 Conclusions and future scope

In the present paper we have formulated an optimization problem which integrates engineering and marketing decision of price and warranty to the testing phase of the software product. We have developed a profit model for a software which is sold under warranty and two types of imperfect debugging with the consideration of expected sales, time to remove faults, cost of removing faults during testing phase and warranty phase. The model involves generating optimally the testing time and price of the product. The benefits of using our model are that it provides

  1. (1)

    Assurance that the software has achieved the aspiration level of reliability and

  2. (2)

    Optimal testing time and price of the software for the given warranty period under imperfect debugging.

With this type of information, a software manager can determine how much testing is required to achieve a certain level of reliability and what should be the price of the software for providing warranty. Such predictions provide a quantitative basis for achieving optimal reliability and sales target. In this paper we have considered constant price but in future we can extend our model to dynamic price changing over the software life cycle. The model can be further extended to incorporate the effect of promotional efforts and exogenous variables like competitors strategy in the profit function.