Abstract
Continuous-time Markov decision process are an important variant of labelled transition systems having nondeterminism through labels and stochasticity through exponential fire-time distributions. Nondeterministic choices are resolved using the notion of a scheduler. In this paper we characterize the class of measurable schedulers, which is the most general one, and show how a measurable scheduler induces a unique probability measure on the sigma-algebra of infinite paths. We then give evidence that for particular reachability properties it is sufficient to consider a subset of measurable schedulers. Having analyzed schedulers and their induced probability measures we finally show that each probability measure on the sigma-algebra of infinite paths is indeed induced by a measurable scheduler which proves that this class is complete.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. II. Athena Scientific (1995)
Feinberg, E.A.: Continuous Time Discounted Jump Markov Decision Processes: A Discrete-Event Approach. Mathematics of Operations Research 29, 492–524 (2004)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Chichester (1994)
Sennot, L.: Stochastic Dynamic Programming and the Control of Queueing Systems. John Wiley & Sons, Chichester (1999)
Abdeddaïm, Y., Asarin, E., Maler, O.: On Optimal Scheduling under Uncertainty. In: Garavel, H., Hatcliff, J. (eds.) ETAPS 2003 and TACAS 2003. LNCS, vol. 2619, pp. 240–253. Springer, Heidelberg (2003)
Bruno, J., Downey, P., Frederickson, G.N.: Sequencing Tasks with Exponential Service Times to Minimize the Expected Flow Time or Makespan. J. ACM 28, 100–113 (1981)
Qiu, Q., Pedram, M.: Dynamic power managment based on continuous-time Markov decision processes. In: Proceedings DAC, pp. 555–561 (1999)
Baier, C., Haverkort, B., Hermanns, H., Katoen, J.P.: Efficient Computation of Time-Bounded Reachability Probabilities in Uniform Continuous-Time Markov Decision Processes. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 61–76. Springer, Heidelberg (2004)
Cattani, S., Segala, R., Kwiatkowska, M.Z., Norman, G.: Stochastic Transition Systems for Continuous State Spaces and Non-determinism. In: Sassone, V. (ed.) FOSSACS 2005. LNCS, vol. 3441, pp. 125–139. Springer, Heidelberg (2005)
Kwiatkowska, M., Norman, G., Segala, R., Sproston, J.: Verifying quantitative properties of continuous probabilistic timed automata. In: Palamidessi, C. (ed.) CONCUR 2000. LNCS, vol. 1877, pp. 123–137. Springer, Heidelberg (2000)
Hernndez-Lerma, O., Lassere, J.B.: Discrete-time Markov control processes: Basic optimality criteria. Appl. Math., vol. 30. Springer, Heidelberg (1996)
Grabiszewski, K.: Type space with disintegrability. Draft (2005)
Valadier, M.: Désintégration d’une mesure sur un produit. C.R. Acad. Sc. Paris 276, 33–35 (1973) Serie A
Ash, R.B., Doléans-Dade, C.A.: Probability & Measure Theory, 2nd edn. Academic Press, London (2000)
Shiryaev, A.N.: Probability, 2nd edn. Springer, Heidelberg (1995)
Panangaden, P.: Measure and probability for concurrency theorists. TCS: Theoretical Computer Science 253 (2001)
Panangaden, P.: Stochastic Techniques in Concurrency. Lecture Notes from a course given at BRICS (unpublished, 1997)
Billingsley, P.: Probability and Measure, 2nd edn. Wiley, New York (1986)
Leao Jr., D., Fragoso, M., Ruffino, P.: Regular conditional probability, disintegration of probability and radon spaces. Proyecciones 23, 15–29 (2004)
Edalat, A.: Semi-pullbacks and bisimulation in categories of markov processes. Mathematical Structures in Computer Science 9, 523–543 (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wolovick, N., Johr, S. (2006). A Characterization of Meaningful Schedulers for Continuous-Time Markov Decision Processes. In: Asarin, E., Bouyer, P. (eds) Formal Modeling and Analysis of Timed Systems. FORMATS 2006. Lecture Notes in Computer Science, vol 4202. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11867340_25
Download citation
DOI: https://doi.org/10.1007/11867340_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-45026-9
Online ISBN: 978-3-540-45031-3
eBook Packages: Computer ScienceComputer Science (R0)