Abstract
Verification of probabilistic systems is usually based on variants of Markov processes. For systems with continuous dynamics, Markov processes are generated using discrete approximation methods. These methods assume an exact model of the dynamic behavior. However, realistic systems operate in the presence of uncertainty and variability and they are described by uncertain models. In this paper, we address the problem of probabilistic verification of uncertain systems using Bounded-parameter Markov Decision Processes (BMDPs). Proposed by Givan, Leach and Dean [1], BMDPs are a generalization of MDPs that allow modeling uncertainty. In this paper, we first show how discrete approximation methods can be extended for modeling uncertain systems using BMDPs. Then, we focus on the problem of maximizing the probability of reaching a set of desirable states, we develop a iterative algorithm for probabilistic verification, and we present a detailed mathematical analysis of the convergence results. Finally, we use a robot path-finding application to demonstrate the approach.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Givan, R., Leach, S., Dean, T.: Bounded-parameter Markov decision process. Artificial Intelligence 122, 71–109 (2000)
Courcoubetis, C., Yannakakis, M.: Markov decision processes and regular events. IEEE Transaction on Automatic Control 43, 1399–1418 (1998)
de Alfaro, L.: Computing minimum and maximum reachability times in probabilistic systems. In: Baeten, J.C.M., Mauw, S. (eds.) CONCUR 1999. LNCS, vol. 1664, pp. 66–81. Springer, Heidelberg (1999)
Koutsoukos, X.D.: Optimal control of stochastic hybrid systems based on locally consistent Markov decision processes. International Journal of Hybrid Systems 4, 301–318 (2004)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York (1994)
Satia, J.K., Lave, R.E.: Markovian decision processes with uncertain transition probabilities. Operations Research 39, 1095–1100 (1953)
White, C.C., Eldeib, H.K.: Parameter imprecision in finite state, finite action dynamic programs. Operations Research 34, 120–129 (1986)
White, C.C., Eldeib, H.K.: Markov decision processes with imprecise transition probabilities. Operations Research 43, 739–749 (1994)
Tang, H., Liang, X., Gao, J., Liu, C.: Robust control policy for semi-Markov decision processes with dependent uncertain parameters. In: WCICA 2004: Proceedings of the Fifth World Congress on Intelligent Control and Automation (2004)
Kalyanasundaram, S., Chong, E.K.P., Shroff, N.B.: Markovian decision processes with uncertain transition rates: Sensitivity and robust control. In: CDC 2002: Proceedings of the 41th IEEE Conference on Decision and Control (2002)
Kushner, H.J., Dupuis, P.: Numerical Methods for Stochastic Control Problems in Continuous Time, 2nd edn. Springer, New York (2001)
Wu, D., Koutsoukos, X.D.: Probabilistic verification of bounded-parameter Markov decision processes. Technical Report ISIS-05-607, Institute for Software Integrated Systems, Vanderbilt University, Nashville, TN, USA (2005)
Dean, T., Givan, R., Leach, S.: Model reduction techniques for computing approximately optimal solutions for Markov decision processes. In: Proceedings of the 13th Annual Conference on Uncertainty in Artificial Intelligence (UAI 1997), pp. 124–131. Morgan Kaufmann Publishers, San Francisco (1997)
Rudin, W.: Real and Complex Analysis, 3rd edn. McGraw-Hill, New York (1994)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wu, D., Koutsoukos, X. (2006). Probabilistic Verification of Uncertain Systems Using Bounded-Parameter Markov Decision Processes. In: Torra, V., Narukawa, Y., Valls, A., Domingo-Ferrer, J. (eds) Modeling Decisions for Artificial Intelligence. MDAI 2006. Lecture Notes in Computer Science(), vol 3885. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11681960_28
Download citation
DOI: https://doi.org/10.1007/11681960_28
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-32780-6
Online ISBN: 978-3-540-32781-3
eBook Packages: Computer ScienceComputer Science (R0)