Skip to main content
SpringerLink
Log in

Towards Analysis of Semi-Markov Decision Processes

  • Conference paper
Book cover Artificial Intelligence and Computational Intelligence (AICI 2010)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 6319))

  • 1928 Accesses

Abstract

We investigate Semi-Markov Decision Processes (SMDPs). Two problems are studied, namely, the time-bounded reachability problem and the long-run average fraction of time problem. The former aims to compute the maximal (or minimum) probability to reach a certain set of states within a given time bound. We obtain a Bellman equation to characterize the maximal time-bounded reachability probability, and suggest two approaches to solve it based on discretization and randomized techniques respectively. The latter aims to compute the maximal (or minimum) average amount of time spent in a given set of states during the long run. We exploit a graph-theoretic decomposition of the given SMDP based on maximal end components and reduce it to linear programming problems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.-P.: Model-checking algorithms for continuous-time Markov chains. IEEE Trans. Software Eng. 29(6), 524–541 (2003)

    Article  MATH  Google Scholar 

  2. Baier, C., Hermanns, H., Katoen, J.-P., Haverkort, B.R.: Efficient computation of time-bounded reachability probabilities in uniform continuous-time markov decision processes. Theor. Comput. Sci. 345(1), 2–26 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bertsekas, D.P.: Dynamic Programming and Optimal Control. Athena Scientific, Belmont (2007)

    MATH  Google Scholar 

  4. Bertsekas, D.P., Tsitsiklis, J.N.: An analysis of stochastic shortest path problems. Mathematics of Operations Research 16(3), 580–595 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  5. de Alfaro, L.: How to specify and verify the long-run average behavior of probabilistic systems. In: LICS, pp. 454–465 (1998)

    Google Scholar 

  6. Guo, X., Hernández-Lerma, O.: Continuous-Time Markov Decision Processes. Springer, Heidelberg (2009)

    Book  MATH  Google Scholar 

  7. Jewell, W.S.: Markov-renewal programming I: Formulation, finite returen models; markov-renewal programming II: infinite return models, example. Operations Research 11, 938–971 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  8. Neuhaeusser, M.R.: Model Checking Nondeterministic and Randomly Timed Systems. PhD thesis (2010)

    Google Scholar 

  9. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994)

    Book  MATH  Google Scholar 

  10. Rust, J.: Using randomization to break the curse of dimensionality. Econometrica 65(3), 487–516 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  11. Wolovick, N., Johr, S.: A characterization of meaningful schedulers for continuous-time markov decision processes. In: Asarin, E., Bouyer, P. (eds.) FORMATS 2006. LNCS, vol. 4202, pp. 352–367. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  12. Zhang, L., Neuhaeusser, M.R.: Model checking interactive markov chains. In: Esparza, J., Majumdar, R. (eds.) TACAS 2010. LNCS, vol. 6015, pp. 53–68. Springer, Heidelberg (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. FMT, University of Twente, The Netherlands

    Taolue Chen

  2. State Key Laboratory of Novel Software Technology, Nanjing University, China

    Jian Lu

Authors
  1. Taolue Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Business Administration, Caritas Francis Hsu College, 18 Chui Ling Road, Tseung Kwan O, Hong Kong, China

    Fu Lee Wang

  2. School of Business Information Technology, RMIT University, City Campus, 124 La Trobe Street, 3000, Melbourne, Victoria, Australia

    Hepu Deng

  3. Department of Computer Science, Nanjing University, 210093, Nanjing, China

    Yang Gao

  4. School of Computer, Nanjing University of Posts and Telecommunications, 210003, Nanjing, China

    Jingsheng Lei

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Chen, T., Lu, J. (2010). Towards Analysis of Semi-Markov Decision Processes. In: Wang, F.L., Deng, H., Gao, Y., Lei, J. (eds) Artificial Intelligence and Computational Intelligence. AICI 2010. Lecture Notes in Computer Science(), vol 6319. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-16530-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-16530-6_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-16529-0

  • Online ISBN: 978-3-642-16530-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation