Software dependability evaluation based on Markov usage models
Introduction
In the last years, there is a growing number of publications on Markov-type software usage models (see, e.g., [1], [18], [22], [24], [25], [26], [27]). The reason seems to be that models of this type have turned out as flexible, easily understandable descriptions of the operational profiles of given programs or software systems, providing both a wide range of applications and a potential for deep formal analysis.
A software usage model characterizes the (estimated) distribution of possible uses of a given program or software system in its intended environment. Such a model is required whenever unbiased estimates of quantities like reliability or risk are to be derived from the results of tests. The testing method allowing such estimates is random testing with suitably chosen input distributions, and the most direct way to achieve the goal of unbiased estimation consists in choosing the operational profile, as described by the usage model, as the input distribution (cf. [14], [22] or [2]).
In its simplest form, a (partial) usage model is already specified by a table assigning to each program function the corresponding probability of a call. (To complete the model, it has to be supplied with a component that describes the selection of inputs for each program function, a problem which is usually solved in a pragmatic way.) However, the usage structure of many programs includes time dependencies of function calls. Then, from a probabilistic point of view, the usage model will rather be based on a stochastic process than on a single discrete random variable. Markov chains are, as stated above, very convenient model processes for a large spectrum of applications.
In this paper, we consider three different quantities to be estimated from test results, namely (i) risk, (ii) safety, and (iii) reliability. These quantities are partial aspects of the more general notion of dependability (see [11]), which additionally includes other features like availability and security. We think that the presented approach can also be extended to the last-mentioned properties, but exclude these topics for the sake of brevity.
We have mentioned that most usage-model-based attempts to predict dependability properties of a given program rely on testing with the operational profile, which is (hopefully) specified adequately by the usage model. However, as it will be argued, this approach has severe drawbacks. It will be shown that a software usage model can be applied for generating non-operational test cases as well, and which is of vital interest, the result of executing these tests can nevertheless be used to derive unbiased estimates of the interesting quantities in a statistically sound way.
Let us start by recapitulating the limitations of testing with the operational profile. In the context of the Markov usage model literature, these limitations have been formulated very clearly by Walton et al. [22, pp. 105–106]. They state that it is often important to ensure the exercise of specific operations of the software irrespective of their usage probabilities. For example, there may be operations of high criticality (i.e., high loss in case of failure); nevertheless, if these operations are infrequently executed in real applications, they are possibly not tested at all during operational-profile testing. Another example are newly implemented operations which have a high prior failure probability and should therefore be given a good chance of being tested, no matter whether they are frequently or infrequently used. Walton et al. suggest modifications and extensions to operational-profile testing in order to overcome the mentioned drawbacks, but, as they admit, the suggested measures are not able to allow unbiased estimates of, say, the reliability of the software under test.
The aim of the present paper is to develop a general method of sampling test cases from a Markov software usage model, such that
- 1.
test results can still be used to compute unbiased estimates of a dependability measure under consideration (risk, safety or reliability), although
- 2.
the distribution of test cases is shifted from the operational distribution to a test profile designed for exploiting prior information on possibly error-prone or critical operations.
For some special cases, the suggested approach has already been indicated in other recent publications: the simple, non-Markovian case of risk estimation is treated in [6]; a generalization to Markov models under specific model assumptions (again for risk) is shortly outlined in [7], and some aspects of reliability estimation are discussed in [8]. The present paper develops the approach within a unified framework in full generality (2 Formal pre-requisites, 3 Changing the test profile), specifies it, for two different dependability models, to the three particular dependability measures, including safety (Section 4), and elaborates an efficient procedure for the iterative computation of optimal transition probabilities (Section 5). In Section 6, we develop an algorithm for computing a test distribution guaranteeing for each operation a certain minimal probability of being executed; this algorithm supports the procedure of Section 5, but is also of interest for structural statistical testing [19], [20] outside the context of our approach. Finally, Section 7 contains a short discussion of the results.
Section snippets
Formal description of a Markov model
First of all, a definition of a Markov usage model will be given (cf. also [22], [26]). By a Markov usage model, we understand a Markov chain with a unique initial state, symbolizing program invocation, a unique final state, symbolizing program termination, and other states symbolizing intermediate usage or processing states of the program under consideration. As customary, the Markov chain will be represented by a directed graph G=(V,A) and a function p:V×V→[0,1] with the following properties:
- •
V
Changing the test profile
The simplest way of obtaining unbiased estimates for risk(π), safety(π) and reliability(π) is operational profile testing. Let c(x,π) be the cost function under consideration. Then, an unbiased estimate of EP(c(X,π)) is obtained by selecting X according to the operational profile P, performing a test of π with path X, and determining c(X,π) from the test result. In practical applications, this process is always iterated (cf., e.g., [22]): Draw N test paths X(1),…,X(N) according to P, perform
Optimal probability shifts for specific measures
In this section, the problem of computing optimal shifts of the transition probabilities of the Markov chain is reduced to the solution of specific stochastic optimization problems by deriving explicit representations of the function H(x) defined by (10). We shall do this separately for the three considered measures of interest (risk, safety and reliability). The distribution Q will be taken into account by two alternative models, the parameters of which have to be estimated, e.g., by educated
A numerical solution algorithm
In this section, we turn to the question how to solve the stochastic optimization problem from Lemma 3.1,numerically, where denotes the set of all transition matrices with the propertiesIn order to avoid technical complications, we further restrict ourselves to the set instead of , which obviously does not change the optimizers of (17).
The crucial
Constructing a distribution with arc traversal probability guarantees
In this section, we show how a distribution P0, defined by transition probabilities p0(i,j), can be found that guarantees each fixed arc a probability of at least 1/(n+#(arcs)) of being traversed. Let us emphasize that an algorithm for computing such a distribution is not only applicable within the context of our approach, but also useful in a broader range of application: Thévenod-Fosse has introduced the concept of structural statistical testing (see, e.g., [19], [20]), whereby the
Example
As an illustration for the application of our technique, we re-investigate a (relatively small) Markov usage model described in [7] (see Fig. 4).
The model reflects part of a train schedule program, and the nodes 1–12 correspond to the following program states respectively screens: (1) start, (2) initial screen, (3) main menu, (4) information request, (5) departure information, (6) arrival information, (7) board service information, (8) booking places, (9) preferences, (10) place reservation,
Conclusion
We have developed a general technique for computing optimal test transition probabilities in a Markov software usage model, based on pre-information on failure probabilities and losses in case of failure of single operations. The optimization criterion is maximum precision of dependability estimates obtained from the test results. Two different basic model (a static and a dynamic model) for the dependability aspects of the given program have been used. The optimization itself is done by means
Acknowledgements
The author would like to thank the anonymous referees for their helpful comments to a previous version of the paper.
Walter J. Gutjahr received his M.Sc. and Ph.D. degrees in mathematics from the University of Vienna, Austria, in 1980 and 1985, respectively. From 1980 to 1988 he was with Siemens Corporate, working in technical and in management positions on diverse software development projects. Since 1988, he is at the University of Vienna, currently as an Associate Professor of computer science and applied mathematics. His research interests include analysis of algorithms, optimization, software
References (27)
- et al.
Composite performance and dependability analysis
Performance Evaluation
(1992) - et al.
The automatic generation of load test suites and the assessment of the resulting software
IEEE Trans. Software Engrg.
(1995) - A. Bertolino, L. Strigini, Acceptance criteria for critical software based on testability estimates and test results,...
- et al.
A theoretical basis for the analysis of multiversion software subject to coincident errors
IEEE Trans. Software Engrg.
(1985) - W.D. Ehrenberger, Combining probabilistic and deterministic verification efforts, in: H. Frey (Ed.), Proceedings of the...
- et al.
Importance sampling for stochastic simulations
Management Sci.
(1989) Optimal test distributions for software failure cost estimation
IEEE Trans. Software Engrg.
(1995)- W.J. Gutjahr, Failure risk estimation via Markov software usage models, in: E. Schoitsch (Ed.), SAFECOMP’96,...
Importance sampling of test cases in Markovian software usage models
Prob. Eng. Informat. Sci.
(1997)- R. Hamlet, R. Taylor, Partition testing does not inspire confidence, Proceedings of the Second Workshop on Software...
Fast simulation of rare events in queuing and reliability models
ACM Trans. Modeling Comput. Simul.
Cited by (44)
A Systematic Literature Review on prioritizing software test cases using Markov chains
2022, Information and Software TechnologyQuality assurance through rigorous software specification and testing: A case study
2015, Procedia Computer ScienceUsage-pattern based statistical web testing and reliability measurement
2013, Procedia Computer ScienceA novel evidential reasoning based method for software trustworthiness evaluation under the uncertain and unreliable environment
2012, Expert Systems with ApplicationsCitation Excerpt :In the existing researches, STE has been usually considered a multiple attribute decision analysis (MADA) problem and implemented by different methods or models. Representative methods or models include unified model of dependability (UMD) (Basili et al., 2004), markov usage model (Water, 2000), fuzzy theory model (Shi, Ma, & Zou, 2008), trustworthy degree computation model (Ma & Zhang, 2008), adaptive dependability model (Mei & Xu, 2003), trustworthy metrics model (Zhang, Fang, & Xu, 2008) and open source components trustworthiness model (Anne & Marko, 2007). In practice, many STE problems include not only quantitative analysis but also qualitative judgment.
An Interval Approach for the Availability Optimization of Multi-State Systems in the Presence of Aleatory and Epistemic Uncertainties
2022, ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part B: Mechanical EngineeringAnalysis of cost and benefit on safety critical systems
2021, Proceedings of the International Conference on Industrial Engineering and Operations Management
Walter J. Gutjahr received his M.Sc. and Ph.D. degrees in mathematics from the University of Vienna, Austria, in 1980 and 1985, respectively. From 1980 to 1988 he was with Siemens Corporate, working in technical and in management positions on diverse software development projects. Since 1988, he is at the University of Vienna, currently as an Associate Professor of computer science and applied mathematics. His research interests include analysis of algorithms, optimization, software engineering, and project management.