Abstract
In this paper, an algorithm for the estimation of the gradient of the stationary performance of a Markov chain w.r.t. a real parameter is presented. The method works for discrete and continuous state spaces. A comparison with the efficient score method and extensions to semi-Markov processes and discrete event dynamical systems (DEDS) are made.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
P.W. Glynn, Stochastic approximation for Monte Carlo optimization,Proc. 1986 Winter Simulation Conf. (The Society for Computer Simulation (1986) pp. 356–364.
P.W. Glynn and D.L. Iglehart, Simulation methods for queues: an overview, Queueing Systems 3(1988)221–256.
W.B. Gong and Y.C. Ho, Smoothed (conditional) perturbation analysis of discrete event dynamical systems, IEEE Trans. Auto. Control (1988).
Y.M. Ermoliev and A.A. Gaivoronski, Stochastic programming techniques for optimization of discrete event systems, Ann. Oper. Res. (1991).
Y.C. Ho and X. Cao, Perturbation analysis and optimization of queueing networks, J. Optim. Theory Appl. 20(1983)559–589.
H.J. Kushner and D.S. Clark,Stochastic Approximation Methods for Constrained and Unconstrained Systems (Springer, 1978).
M. Metivier and P. Priouret, Théorie de convergence presque sûre pour une classe d'algoritms stochastiques à pas decroissants, Prob. Th. Rel. Fields 74(1987)403–428.
G. Ch. Pflug, On-line optimization of simulated Markovian processes, Math. Oper. Res. 15(1990)381–395.
G. Ch. Pflug,Derivatives of Probability Measures — Concepts and Applications to the Optimization of Stochastic Systems, Lecture Notes in Control and Information Sciences 103, ed. P. Varaiya and A. Kurzhanski (Springer, 1988) pp. 252–274.
G.Ch. Pflug and W. Schachermayer, Coefficients of ergodicity for order preserving Markov chains, Adv. Appl. Prob. (Dec. 1992).
S.T. Rachev, The Monge-Kantorovich mass transportation problem and its stochastic applications, Theory Prob. Appl. 29(1984)647–676.
R. Rubinstein,Simulation and the Monte Carlo Method (Wiley, 1987).
R. Rubinstein,Monte Carlo Optimization, Simulation and Sensitivity of Queueing Networks (Wiley, 1986).
R. Rubinstein, The score function approach for sensitivity analysis of computer simulation models, Math. Comput. Simul. 28(1986)351–379.
R. Rubinstein, Sensitivity analysis of computer simulation models via the efficient score, Oper. Res. 37(1989).
R. Rubinstein and A. Shapiro, Optimization of simulation models by the score function method, Math. Comput. Simul. 32(1990).
R. Suri, Infinitesimal perturbation analysis for general discrete event systems, J. ACM 34(1987)686–717.
R. Suri and Y.T. Leung, Single run optimization of discrete event simulations: An empirical study using theM/M/1 queue, Technical Report TR 87-3, University of Wisconsin, Madison (1987).
M.T. Wasan,Stochastic Approximation, Cambridge Tracts No. 58(Cambridge University Press, 1969).
M.A. Zazanis and R. Suri, Estimating first and second derivatives of response time forG/G/1 queues from a single sample path, submitted.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Pflug, G.C. Gradient estimates for the performance of markov chains and discrete event processes. Ann Oper Res 39, 173–194 (1992). https://doi.org/10.1007/BF02060941
Issue Date:
DOI: https://doi.org/10.1007/BF02060941