State-space support for path-based reward variables

https://doi.org/10.1016/S0166-5316(99)00010-3Get rights and content

Abstract

Many sophisticated formalisms exist for specifying complex system behaviors, but methods for specifying performance and dependability variables have remained quite primitive. To cope with this problem, modelers often must augment system models with extra state information and event types to support particular variables. This often leads to models that are non-intuitive, and must be changed to support different variables. To address this problem, we extend the array of performance measures that may be derived from a given system model, by developing new performance measure specification and model construction techniques. Specifically, we introduce a class of path-based reward variables, and show how various performance measures may be specified using these variables. Path-based reward variables extend the previous work with reward structures to allow rewards to be accumulated based on sequences of states and transitions. To maintain the relevant history, we introduce the concept of a path automaton, whose state transitions are based on the system model state and transitions. Furthermore, we present a new procedure for constructing state spaces and the associated transition rate matrices that support path-based reward variables. Our new procedure takes advantage of the path automaton to allow a single system model to be used as the basis of multiple performance measures that would otherwise require separate models or a single more complicated model.

Introduction

Many sophisticated formalisms now exist for specifying complex system behaviors, and many tools exist that can convert a model specified in the formalism to an underlying stochastic process that can be solved. Specification methods for performance and dependability variables, on the other hand, have remained quite primitive by comparison. For example, most stochastic Petri net (SPN) tools require a user to specify performance and dependability variables in terms of a rate defined on the states of the model, and possibly, an impulse defined on each event (e.g., transition in an SPN). The trouble with this approach is that the model, as naturally defined, may not support the desired measure. To overcome this limitation, one often has to add extra components to a model (e.g. places and transitions if modeling using SPNs) to collect the desired information. These components are not part of the system being modeled, and must change whenever one desires new information from the model.

It is thus often the case that several different models of a system must be built in order to obtain the desired performance measures. Changing the model can be a time-consuming procedure, since it then must be validated to guarantee that it is still an accurate representation of the system under study. We address this problem by extending performance measure specification and state-space construction procedures to allow more flexible use of a given model. This is made possible by (1) extending current reward variable specification methods to include variables that have “state” and can capture behavior related to sequences of events and states, and (2) extending current state-space construction algorithms that build stochastic processes that are tailored to the variable(s) of interest.

The use of performance measures to direct state-space construction is not new, but has been limited to supporting lumping based on symmetries for analytic models. In particular, Sanders and Meyer [16] use standard rate and impulse-based reward variables to put limits on the lumping that can be achieved because of symmetries in a model, and to support impulses that depend on particular activity completions. Path-based reward variables for impulses have been considered, but only to the extent that their use did not change the state space that is generated. Specifically, Qureshi et al. [13] consider the use of such variables, but limit their use to impulses on sequences of instantaneous events in order not to change the set of (stable) states that is generated.

Our work extends previous work in two important ways. First, we provide support for a more general class of reward variables for a given system model. In particular, we support the definition of measures that depend on sequences of states and events that may occur. Examples of variables whose specification is facilitated by these methods include computations of probabilities of occurrence of particular recovery actions that have multiple steps, and computation of measures related to consecutive cell loss in ATM networks, among others. We do this by introducing the “path automaton”, a finite automaton that can be used to define rewards on sequences or sets of sequences of system model states and/or transitions (both timed and untimed). By building the required memory into the performance measure specification, we simultaneously accomplish two goals: we make more flexible the specification of complex performance and dependability variables, and we avoid the need to develop multiple system models. This approach offers the advantage of a single, smaller, system model that is easier to construct and validate. Multiple performance measures defined on multiple path automata may then be defined relative to the single system model.

Second, we provide procedures for automatic construction of a state space that supports the specified variables from the definition of the system model, path automata, and reward structures. Note that the choice of variables as well as system model determine the state space that is generated, and different variables result in different size state spaces for the same system model. In addition, these state generation procedures include automatic support for state-space truncation for the case of performance measures defined over intervals terminated upon satisfaction of a condition on the system model, such as entrance to a particular state or the occurrence of a sequence of states and/or transitions. This is made possible by the use of path automaton “final states”, which are interpreted by the state-space construction procedure as indicating that the model state reached upon entry to the final state should not be explored any further.

The rest of the paper is organized as follows. Section 2 reviews a simple model formalism that is used in subsequent sections. Section 3 introduces the new concept of a path-based reward variable, and Section 4 shows how various performance measures may be specified using path-based reward variables. Then, in Section 5, we present new procedures for automatically generating a state space that supports multiple path-based reward variables, and summarize the relevant numerical solution techniques. Section 6 gives an example model and some results on the variation of state space size for different performance measures.

Section snippets

Model specification

In this section we review a simple model description formalism, first defined in [10].

Definition 1

A model is a five-tuple (S,E,ε,λ,τ) where

  • S is a set of state variables {s1,s2,…,sn} that take values in N, the set of non-negative integers. The state of the model is defined as a mapping μ:S→N, where for all sS,μ(s) is the value of state variable s. Let M={μ∣μ:S→N} be the set of all such mappings.

  • E is the set of events that may occur.

  • ε:E×M→{0,1} is the event enabling function. For each eE and μM,ε(e,μ)=1

Path-based reward variables

We wish to evaluate a performance measure that is based on a sequence of model states and events. We call such a sequence a “path”, and such measures “path-based performance measures”. Formally, we have the following definition.

Definition 2

A path, (μ1,e1)(μ2,e2),…,(μn,en), is a sequence of ordered pairs where μi is a model state and ei is a model event.

We say a path is initialized when the model enters the first state in the path. A path completes when the last pair in the sequence is satisfied. A path is

Example performance measures

Given a model and a path, there are many different questions one might ask. First, we may want to know the probability of traversing the path. Given that the path is traversed, how long does it take? How many times was the path completed in some interval? What is the chance of finding the model in the middle of traversing the path, at some arbitrary time point? What is the total time spent traversing the path within some interval? In this section we show how all of these questions may be

State-space support

The first step in presenting the state-space construction method is to provide a precise definition of a state. We wish to allow multiple path-based reward variables to be associated with a given model. In general, this means that there will be multiple path automata and multiple reward structures to manage. Suppose that there are n different reward variables defined on a model. Each reward variable comprises a path automaton and a path-based reward structure. We index the path automaton and

Fault-tolerant computing example and results

The size of the state space that is needed to support a path-based reward variable clearly depends on the underlying model and the nature of the path automaton. In this section we introduce a larger model and investigate the variation in the size of the state space required for several path-based reward variables.

As mentioned in Section 2, the modeling formalism introduced there and used to develop the variable specification and state-space construction procedures is not intended to be used for

Conclusion

This paper presents a new performance/dependability measure specification technique and state space construction procedure that, together, solve the problem of supporting a broad array of performance measures from a given system model. Using this new approach, performance measures based on model states, model events, or sequences of (model state, model event) pairs can be supported by a single system model. Furthermore, the performance measures may be evaluated in steady state, at an instant of

Acknowledgements

We acknowledge Dan Deavours for giving us his SPN state space construction code, which served as a convenient starting point for our prototype implementation, and G.P. Kavanaugh for helpful discussions on the nature of path-based performance measures.

References (18)

  • G. Ciardo, A. Blakemore, P.F.J. Chimento, J.K. Muppala, K.S. Trivedi, Automated generation and analysis of Markov...
  • E. Çinlar, Introduction to Stochastic Processes, Prentice-Hall, Englewood Cliffs, NJ,...
  • E. de Souza e Silva, R. Gail, Calculating transient distributions of cumulative reward, in: Proceedings of...
  • L. Donatiello et al.

    On evaluating the cumulative performance distribution of fault-tolerant computer systems

    IEEE Trans. Comput.

    (1991)
  • R.A. Howard, Dynamic Probabilistic Systems, Vol. II: Semi-Markov and Decision Processes, Wiley, New York,...
  • J.F. Meyer, Probabilistic modeling, in: The Twenty-Fifth International Symposium on Fault-Tolerant Computing (FTCS-25),...
  • J.F. Meyer, A. Movaghar, W.H. Sanders, Stochastic activity networks: Structure, behavior, and application, in:...
  • H. Nabli et al.

    Performability analysis: A new algorithm

    IEEE Trans. Comput.

    (1996)
  • M.F. Neuts, Matrix-Geometric Solutions in Stochastic Models: An Algorithmic Approach, Johns Hopkins University Press,...
There are more references available in the full text version of this article.

Cited by (0)

This material is based upon work supported by DARPA/ITO under Contract no. DABT63-96-C-0069. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA/ITO.

1

This work was done while he was a Ph.D. student in the Electrical and Computer Engineering Department of the University of Arizona, and a Visiting Scholar in the Coordinated Science Laboratory of the University of Illinois.

View full text