Elsevier

Performance Evaluation

Volume 40, Issue 4, April 2000, Pages 199-222
Performance Evaluation

Software dependability evaluation based on Markov usage models

https://doi.org/10.1016/S0166-5316(99)00052-8Get rights and content

Abstract

A general technique for computing optimal state transition probabilities for software tests, based on a Markov usage model, is developed. The optimization criterion is maximum precision of unbiased dependability estimates derived from the test results. Three different dependability measures are considered: (i) risk, (ii) safety, and (iii) reliability. As input, pre-information on failure probabilities and losses in case of failure related with single operations is used. The optimization itself is done by means of a numerical procedure which is fast because of the convexity of the underlying stochastic optimization problem. The procedure can be improved by the construction of a distribution with a common lower bound on state transition probabilities; this distribution may also be used in the more general context of structural statistical testing of software.

Introduction

In the last years, there is a growing number of publications on Markov-type software usage models (see, e.g., [1], [18], [22], [24], [25], [26], [27]). The reason seems to be that models of this type have turned out as flexible, easily understandable descriptions of the operational profiles of given programs or software systems, providing both a wide range of applications and a potential for deep formal analysis.

A software usage model characterizes the (estimated) distribution of possible uses of a given program or software system in its intended environment. Such a model is required whenever unbiased estimates of quantities like reliability or risk are to be derived from the results of tests. The testing method allowing such estimates is random testing with suitably chosen input distributions, and the most direct way to achieve the goal of unbiased estimation consists in choosing the operational profile, as described by the usage model, as the input distribution (cf. [14], [22] or [2]).

In its simplest form, a (partial) usage model is already specified by a table assigning to each program function the corresponding probability of a call. (To complete the model, it has to be supplied with a component that describes the selection of inputs for each program function, a problem which is usually solved in a pragmatic way.) However, the usage structure of many programs includes time dependencies of function calls. Then, from a probabilistic point of view, the usage model will rather be based on a stochastic process than on a single discrete random variable. Markov chains are, as stated above, very convenient model processes for a large spectrum of applications.

In this paper, we consider three different quantities to be estimated from test results, namely (i) risk, (ii) safety, and (iii) reliability. These quantities are partial aspects of the more general notion of dependability (see [11]), which additionally includes other features like availability and security. We think that the presented approach can also be extended to the last-mentioned properties, but exclude these topics for the sake of brevity.

We have mentioned that most usage-model-based attempts to predict dependability properties of a given program rely on testing with the operational profile, which is (hopefully) specified adequately by the usage model. However, as it will be argued, this approach has severe drawbacks. It will be shown that a software usage model can be applied for generating non-operational test cases as well, and which is of vital interest, the result of executing these tests can nevertheless be used to derive unbiased estimates of the interesting quantities in a statistically sound way.

Let us start by recapitulating the limitations of testing with the operational profile. In the context of the Markov usage model literature, these limitations have been formulated very clearly by Walton et al. [22, pp. 105–106]. They state that it is often important to ensure the exercise of specific operations of the software irrespective of their usage probabilities. For example, there may be operations of high criticality (i.e., high loss in case of failure); nevertheless, if these operations are infrequently executed in real applications, they are possibly not tested at all during operational-profile testing. Another example are newly implemented operations which have a high prior failure probability and should therefore be given a good chance of being tested, no matter whether they are frequently or infrequently used. Walton et al. suggest modifications and extensions to operational-profile testing in order to overcome the mentioned drawbacks, but, as they admit, the suggested measures are not able to allow unbiased estimates of, say, the reliability of the software under test.

The aim of the present paper is to develop a general method of sampling test cases from a Markov software usage model, such that

  • 1.

    test results can still be used to compute unbiased estimates of a dependability measure under consideration (risk, safety or reliability), although

  • 2.

    the distribution of test cases is shifted from the operational distribution to a test profile designed for exploiting prior information on possibly error-prone or critical operations.

Moreover, a technique shall be presented for computing optimal shifts of operational transition probabilities in the considered Markov usage model, where the criterion of optimality is minimal imprecision (statistically speaking: minimal variance) of the resulting estimate.

For some special cases, the suggested approach has already been indicated in other recent publications: the simple, non-Markovian case of risk estimation is treated in [6]; a generalization to Markov models under specific model assumptions (again for risk) is shortly outlined in [7], and some aspects of reliability estimation are discussed in [8]. The present paper develops the approach within a unified framework in full generality (2 Formal pre-requisites, 3 Changing the test profile), specifies it, for two different dependability models, to the three particular dependability measures, including safety (Section 4), and elaborates an efficient procedure for the iterative computation of optimal transition probabilities (Section 5). In Section 6, we develop an algorithm for computing a test distribution guaranteeing for each operation a certain minimal probability of being executed; this algorithm supports the procedure of Section 5, but is also of interest for structural statistical testing [19], [20] outside the context of our approach. Finally, Section 7 contains a short discussion of the results.

Section snippets

Formal description of a Markov model

First of all, a definition of a Markov usage model will be given (cf. also [22], [26]). By a Markov usage model, we understand a Markov chain with a unique initial state, symbolizing program invocation, a unique final state, symbolizing program termination, and other states symbolizing intermediate usage or processing states of the program under consideration. As customary, the Markov chain will be represented by a directed graph G=(V,A) and a function p:V×V→[0,1] with the following properties:

  • V

Changing the test profile

The simplest way of obtaining unbiased estimates for risk(π), safety(π) and reliability(π) is operational profile testing. Let c(x,π) be the cost function under consideration. Then, an unbiased estimate of EP(c(X,π)) is obtained by selecting X according to the operational profile P, performing a test of π with path X, and determining c(X,π) from the test result. In practical applications, this process is always iterated (cf., e.g., [22]): Draw N test paths X(1),…,X(N) according to P, perform

Optimal probability shifts for specific measures

In this section, the problem of computing optimal shifts of the transition probabilities of the Markov chain is reduced to the solution of specific stochastic optimization problems by deriving explicit representations of the function H(x) defined by (10). We shall do this separately for the three considered measures of interest (risk, safety and reliability). The distribution Q will be taken into account by two alternative models, the parameters of which have to be estimated, e.g., by educated

A numerical solution algorithm

In this section, we turn to the question how to solve the stochastic optimization problem from Lemma 3.1,MinimizeG(T)=EP(L(X)H(X))forT∈TPnumerically, where TP denotes the set of all transition matrices (t(i,j))(1≤i,j≤n) with the propertiest(i,j)≥0,j=1nt(i,j)=1(i=1,…,n),t(n,n)=1,t(i,j)=0ifandonlyifp(i,j)=0.In order to avoid technical complications, we further restrict ourselves to the set TP0TP∩{T∣G(T)<∞} instead of TP, which obviously does not change the optimizers of (17).

The crucial

Constructing a distribution with arc traversal probability guarantees

In this section, we show how a distribution P0, defined by transition probabilities p0(i,j), can be found that guarantees each fixed arc a probability of at least 1/(n+#(arcs)) of being traversed. Let us emphasize that an algorithm for computing such a distribution is not only applicable within the context of our approach, but also useful in a broader range of application: Thévenod-Fosse has introduced the concept of structural statistical testing (see, e.g., [19], [20]), whereby the

Example

As an illustration for the application of our technique, we re-investigate a (relatively small) Markov usage model described in [7] (see Fig. 4).

The model reflects part of a train schedule program, and the nodes 1–12 correspond to the following program states respectively screens: (1) start, (2) initial screen, (3) main menu, (4) information request, (5) departure information, (6) arrival information, (7) board service information, (8) booking places, (9) preferences, (10) place reservation,

Conclusion

We have developed a general technique for computing optimal test transition probabilities in a Markov software usage model, based on pre-information on failure probabilities and losses in case of failure of single operations. The optimization criterion is maximum precision of dependability estimates obtained from the test results. Two different basic model (a static and a dynamic model) for the dependability aspects of the given program have been used. The optimization itself is done by means

Acknowledgements

The author would like to thank the anonymous referees for their helpful comments to a previous version of the paper.

Walter J. Gutjahr received his M.Sc. and Ph.D. degrees in mathematics from the University of Vienna, Austria, in 1980 and 1985, respectively. From 1980 to 1988 he was with Siemens Corporate, working in technical and in management positions on diverse software development projects. Since 1988, he is at the University of Vienna, currently as an Associate Professor of computer science and applied mathematics. His research interests include analysis of algorithms, optimization, software

References (27)

  • K.S. Trivedi et al.

    Composite performance and dependability analysis

    Performance Evaluation

    (1992)
  • A. Avritzer et al.

    The automatic generation of load test suites and the assessment of the resulting software

    IEEE Trans. Software Engrg.

    (1995)
  • A. Bertolino, L. Strigini, Acceptance criteria for critical software based on testability estimates and test results,...
  • D.E. Eckhardt et al.

    A theoretical basis for the analysis of multiversion software subject to coincident errors

    IEEE Trans. Software Engrg.

    (1985)
  • W.D. Ehrenberger, Combining probabilistic and deterministic verification efforts, in: H. Frey (Ed.), Proceedings of the...
  • P.W. Glynn et al.

    Importance sampling for stochastic simulations

    Management Sci.

    (1989)
  • W.J. Gutjahr

    Optimal test distributions for software failure cost estimation

    IEEE Trans. Software Engrg.

    (1995)
  • W.J. Gutjahr, Failure risk estimation via Markov software usage models, in: E. Schoitsch (Ed.), SAFECOMP’96,...
  • W.J. Gutjahr

    Importance sampling of test cases in Markovian software usage models

    Prob. Eng. Informat. Sci.

    (1997)
  • R. Hamlet, R. Taylor, Partition testing does not inspire confidence, Proceedings of the Second Workshop on Software...
  • P. Heidelberger

    Fast simulation of rare events in queuing and reliability models

    ACM Trans. Modeling Comput. Simul.

    (1995)
  • J.-C. Laprie, Dependability — its attributes, impairments and means, in: B. Randell, J.-C. Laprie, H. Kopetz, B....
  • P. Ljung,G. Pflug, H. Walk, Stochastic Approximation and Optimization of Stochastic Systems, Birkhauser, Basel,...
  • Cited by (44)

    • A novel evidential reasoning based method for software trustworthiness evaluation under the uncertain and unreliable environment

      2012, Expert Systems with Applications
      Citation Excerpt :

      In the existing researches, STE has been usually considered a multiple attribute decision analysis (MADA) problem and implemented by different methods or models. Representative methods or models include unified model of dependability (UMD) (Basili et al., 2004), markov usage model (Water, 2000), fuzzy theory model (Shi, Ma, & Zou, 2008), trustworthy degree computation model (Ma & Zhang, 2008), adaptive dependability model (Mei & Xu, 2003), trustworthy metrics model (Zhang, Fang, & Xu, 2008) and open source components trustworthiness model (Anne & Marko, 2007). In practice, many STE problems include not only quantitative analysis but also qualitative judgment.

    • An Interval Approach for the Availability Optimization of Multi-State Systems in the Presence of Aleatory and Epistemic Uncertainties

      2022, ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part B: Mechanical Engineering
    • Analysis of cost and benefit on safety critical systems

      2021, Proceedings of the International Conference on Industrial Engineering and Operations Management
    View all citing articles on Scopus

    Walter J. Gutjahr received his M.Sc. and Ph.D. degrees in mathematics from the University of Vienna, Austria, in 1980 and 1985, respectively. From 1980 to 1988 he was with Siemens Corporate, working in technical and in management positions on diverse software development projects. Since 1988, he is at the University of Vienna, currently as an Associate Professor of computer science and applied mathematics. His research interests include analysis of algorithms, optimization, software engineering, and project management.

    View full text