Abstract
Markov Decision Processes (MDPs) are a well known mathematical formalism that combines probabilities with decisions and allows one to compute optimal sequences of decisions, denoted as policies, for fairly large models in many situations. However, the practical application of MDPs is often faced with two problems: the specification of large models in an efficient and understandable way, which has to be combined with algorithms to generate the underlying MDP, and the inherent uncertainty on transition probabilities and rewards, of the resulting MDP. This paper introduces a new graphical formalism, called Markov Decision Petri Net with Uncertainty (MDPNU), that extends the Markov Decision Petri Net (MDPN) formalism, which has been introduced to define MDPs. MDPNUs allow one to specify MDPs where transition probabilities and rewards are defined by intervals rather than constant values. The resulting process is a Bounded Parameter MDP (BMDP). The paper shows how BMDPs are generated from MDPNUs, how analysis methods can be applied and which results can be derived from the models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
de Alfaro, L.: Stochastic transition systems. In: Sangiorgi, D., de Simone, R. (eds.) CONCUR 1998. LNCS, vol. 1466, pp. 423–438. Springer, Heidelberg (1998)
Alur, R., Henzinger, T.: Reactive modules. Formal Methods in System Design 15(1), 7–48 (1999)
Beccuti, M., Franceschinis, G., Haddad, S.: Markov decision Petri net and Markov decision well-formed net formalisms. In: Kleijn, J., Yakovlev, A. (eds.) ICATPN 2007. LNCS, vol. 4546, pp. 43–62. Springer, Heidelberg (2007)
Beccuti, M., Franceschinis, G., Codetta-Raiteri, D., Haddad, S.: Computing optimal repair strategies by means of NdRFT modeling and analysis. The Computer Journal 57(12), 1870–1892 (2014)
Bellman, R.: A Markovian Decision Process. Indiana Univ. Math. Journal 6 (1957)
Dean, T., Wellman, M.P.: Planning and Control. Morgan Kaufmann Pub, San Francisco (1991)
Dean, T., Givan, R., Leach, S.: Model reduction techniques for computing approximately optimal solutions for Markov decision processes. In: 13th Conference on Uncertainty in Artificial Intelligence, pp. 124–131. Morgan Kaufmann (1997)
Delgado, K.V., Sanner, S., de Barros, L.N.: Efficient solutions to factored MDPs with imprecise transition probabilities. Artif. Intell. 175(9–10), 1498–1527 (2011)
Eboli, M.G., Cozman, F.G.: Markov decision processes from colored Petri nets. In: da Rocha Costa, A.C., Vicari, R.M., Tonidandel, F. (eds.) SBIA 2010. LNCS, vol. 6404, pp. 72–81. Springer, Heidelberg (2010)
Givan, R., Leach, S.M., Dean, T.L.: Bounded-parameter Markov decision processes. Artif. Intell. 122(1–2), 71–109 (2000)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st edn. John Wiley & Sons Inc, New York (1994)
Scheftelowitsch, D., Buchholz, P.: Multi-criteria approaches to Markov decision processes with uncertain transition parameters. Tech. rep., TU Dortmund (2015). http://ls4-www.cs.uni-dortmund.de/download/buchholz/pareto.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Beccuti, M., Amparore, E.G., Donatelli, S., Scheftelowitsch, D., Buchholz, P., Franceschinis, G. (2015). Markov Decision Petri Nets with Uncertainty. In: Beltrán, M., Knottenbelt, W., Bradley, J. (eds) Computer Performance Engineering. EPEW 2015. Lecture Notes in Computer Science(), vol 9272. Springer, Cham. https://doi.org/10.1007/978-3-319-23267-6_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-23267-6_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23266-9
Online ISBN: 978-3-319-23267-6
eBook Packages: Computer ScienceComputer Science (R0)