Chapter 16 Simulation Algorithms for Regenerative Processes
Introduction
Let be a real-valued stochastic process in which represents the simulation output collected at time t. Roughly speaking, the process V is said to be (classically) regenerative if there exist random times at which the process “regenerates” (in the sense that V probabilistically starts afresh at each time , , and evolves independently of the process prior to time ). Such regenerative structure implies that V can be viewed conceptually as a sequence of independent “cycles” that are “pasted together” (where we adopt the convention that ). Thus, the infinite time behavior of V over is implicitly captured in the behavior of V over a cycle. Hence, in principle, virtually any expectation of V over can be alternatively described as an expectation involving cycle-related quantities. This observation is the key insight that underlies regenerative simulation.
The use of regenerative structure as an algorithmic tool in the simulation setting has primarily focused on its use in the context of steady-state simulation. The first suggestion that regenerative cycles could play a useful role in steady-state simulation output analysis came from Cox and Smith (1961), and the idea was further developed in Kabak (1968). However, the first comprehensive development of the regenerative method for steady-state simulation output analysis came in a series of papers of Crane and Iglehart, 1974a, Crane and Iglehart, 1974b, Crane and Iglehart, 1975, as well as concurrent work by Fishman, 1973, Fishman, 1974. The great majority of subsequent work on algorithmic exploitation of regeneration has followed the historic tradition of focusing on its application to steady-state simulation output analysis.
In this chapter we focus our discussion on the key theoretical and algorithmic issues underlying the use of regeneration in the steady-state simulation context. We start, in Section 2, by describing the key challenges that confront a simulationist in analyzing steady-state simulation output, while Section 3 discusses the basic regenerative approach to forming an estimator for the so-called “time-average variance constant”. Section 4 offers some discussion of how the particular choice of regenerative structure influences the efficiency of the method, and Section 5 describes the regenerative solution to the initial transient problem and the construction of low-bias steady-state estimators. In Sections 6 When is a simulation regenerative?, 7 When is a GSMP regenerative?, 8 Algorithmic identification of regenerative structure we discuss the theoretical issue of when a simulation is regenerative, with a particular focus on when a discrete-event simulation contains algorithmically identifiable regenerative structure. Section 9 then discusses steady-state regenerative analysis from the perspective of martingale theory.
The last two sections of the chapter are intended to give the reader a hint of the role that regeneration can play in the development of computationally efficient algorithms for other simulation problems. In particular, we show that in computing either steady-state gradients or infinite-horizon expected discounted reward that regeneration offers the simulationist the opportunity to not only construct asymptotically valid confidence statements but to also improve computational efficiency. While regeneration is primarily understood within the simulation community as offering a vehicle for analysis of simulation output, our two examples are intended to argue that regeneration has the potential to also play a significant role in the variance reduction context.
Section snippets
The steady-state simulation problem
Let be a real-valued stochastic process in which represents the value of the simulation output process at (simulated) time t. For example, could represent the total work-in-process at time t in a production context or the inventory position at time t in a supply chain setting. Throughout this chapter, we use a continuous time formulation to describe the relevant theory. (Note that any discrete-time sequence can be embedded into continuous time via the definition
The regenerative estimator for the TAVC
To obtain a TAVC estimator that converges to at rate , one needs to assume additional structure about the process V. To illustrate this idea, suppose that the simulation output process V is a (continuous-time) autoregressive process satisfying where and is a square integrable process with stationary independent increments for which and for . It is easily verified that and that V satisfies (2)
Choice of the optimal regeneration state
Given a simulation of V over the time interval , the natural point estimator for the steady-state mean α is, of course, the time-average . While it may be desirable to modify to deal with initial transient or initial bias effects, one would expect such modifications to be of small order asymptotically. Hence, any reasonable point estimator for α will either be exactly equal to or asymptotically equivalent to . Of course, the r.v. is not influenced in any way by
The regenerative approach to the initial transient and initial bias problems
As discussed in Section 2, one of the major challenges in steady-state simulation is the mitigation of effects due to the initial transient and initial bias. We deal first with the better understood issue of how to reduce biasing effects due to a nonstationary initialization.
It is usual, in the presence of (1), that there exists such that as , where represents a function that is bounded by a constant multiple of as . For example, (13) is known to
When is a simulation regenerative?
As has been seen in preceding sections, regenerative structure turns out to be algorithmically useful in developing solutions to various aspects of the steady-state simulation problem. Furthermore, regenerative structure can be easily identified in the setting of discrete state space Markov chains, in either discrete or continuous time.
Of course, most real-world discrete-event simulations do not involve simulating a discrete state space Markov processes. Much more complicated models are
When is a GSMP regenerative?
Section 6 makes clear that regeneration is the rule rather than the exception for well-behaved steady-state simulations. This, however, leaves open the question of when a specific simulation model has the structure necessary to guarantee that the associated steady-state simulation is well behaved.
We shall focus exclusively, in this section, on conditions under which discrete-event simulations possess the required structure. We take the point of view here that a discrete-event simulation is
Algorithmic identification of regenerative structure
Our discussion of Sections 6 When is a simulation regenerative?, 7 When is a GSMP regenerative? makes clear that regenerative structure exists within the typical discrete-event steady-sate simulation. On the other hand, the TAVC estimator of Section 3, as well as the low bias estimators of Section 5, all depend upon the ability of the simulationist to identify the associated regeneration times. Of course, this identification is trivial in the setting of discrete state space Markov chains, where
A martingale perspective on regeneration
To illustrate the connection between martingales and regeneration, we focus here on the case in which , where is an irreducible finite state continuous time Markov chain with rate matrix . Given the performance measure (where we choose to encode as a column vector), the linear system has a solution h. Here, e is the column vector in which all entries equal 1, and (19) is called Poisson's equation.
It is a standard fact in the
Efficiency improvement via regeneration: Computing steady state gradients
In many applications settings, it is of interest to compute the sensitivity of the system's performance to perturbations in an underlying parameter. For example, it may be that the arrival rate to a queue is only approximately known, so that computing the change in performance that corresponds to changing the arrival rate is relevant. In particular, computing the derivative (or, more generally, the gradient) of a steady-state performance measure with respect to the arrival rate is a
Efficiency improvement via regeneration: Computing infinite horizon discounted reward
We now offer a second illustration of the principle that the presence of regenerative structure can be usefully exploited to obtain efficiency improvements. Consider the infinite horizon expected discounted reward , where for some . From a computational standpoint, an algorithm based on simulating i.i.d. copies of the r.v. D cannot be operationalized, because it takes infinite time to generate the above r.v. As a consequence, one needs to consider computationally feasible
References (36)
- et al.
Limit theorems for cumulative processes
Stochastic Processes and Their Applications
(1993) - et al.
Necessary conditions in limit theorems for cumulative processes
Stochastic Processes and Their Applications
(2002) The Statistical Analysis of Time Series
(1971)- et al.
Accelerated regeneration for Markov chain simulations
Probability in the Engineering and Informational Sciences
(1995) Probability and Measure
(1995)Probability
(1968)Return state independent quantities in regenerative simulation
Operations Research
(1994)- Calvin, J.M., Glynn, P.W., Nakayama, M.K. (2006). The semi-regenerative method of simulation output analysis. ACM...
- et al.
Queues
(1961) Mathematical Methods of Statistics
(1946)
Simulating stable stochastic systems, I: General multi-server queues
Journal of the Association for Computing Machinery
Simulating stable stochastic systems, II: Markov chains
Journal of the Association for Computing Machinery
Simulating stable stochastic systems, III: Regenerative processes and discrete-event simulations
Operations Research
An Introduction to the Bootstrap
Markov Processes: Characterization and Convergence
Statistical analysis for queueing simulations
Management Science
Estimation in multiserver queueing simulations
Operations Research
Likelihood ratio gradient estimation for stochastic systems
Communications of the ACM
Cited by (16)
On the numerical solution of functional equations with application to response time distributions
2024, Applied Mathematics and ComputationCoupling from the past with randomized quasi-Monte Carlo
2010, Mathematics and Computers in SimulationEfficient Input Uncertainty Quantification for Regenerative Simulation
2023, Proceedings - Winter Simulation ConferenceResponse Time Distribution in a Tandem Pair of Queues with Batch Processing
2021, Journal of the ACM