Skip to main content
Springer Nature Link
Log in

Optimal Continuous Time Markov Decisions

  • Conference paper
  • First Online:
Automated Technology for Verification and Analysis (ATVA 2015)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 9364))

Abstract

In the context of Markov decision processes running in continuous time, one of the most intriguing challenges is the efficient approximation of finite horizon reachability objectives. A multitude of sophisticated model checking algorithms have been proposed for this. However, no proper benchmarking has been performed thus far.

This paper presents a novel and yet simple solution: an algorithm, originally developed for a restricted subclass of models and a subclass of schedulers, can be twisted so as to become competitive with the more sophisticated algorithms in full generality. As the second main contribution, we perform a comparative evaluation of the core algorithmic concepts on an extensive set of benchmarks varying over all key parameters: model size, amount of non-determinism, time horizon, and precision.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Measurable with respect to the standard \(\sigma \)-algebra on the set of finite histories [25].

References

  1. Aziz, A., Sanwal, K., Singhal, V., Brayton, R.K.: Verifying continuous time Markov chains. In: Alur, R., Henzinger, T.A. (eds.) CAV 1996. LNCS, vol. 1102, pp. 269–276. Springer, Heidelberg (1996)

    Chapter  Google Scholar 

  2. Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.: Model-checking algorithms for continuous-time Markov chains. IEEE Trans. Softw. Eng. 29(6), 524–541 (2003)

    Article  MATH  Google Scholar 

  3. Baier, C., Hermanns, H., Katoen, J., Haverkort, B.R.: Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes. Theor. Comput. Sci. 345(1), 2–26 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  4. Brázdil, T., Forejt, V., Krcál, J., Kretínský, J., Kucera, A.: Continuous-time stochastic games with time-bounded reachability. Inf. Comput. 224, 46–70 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bruno, J.L., Downey, P.J., Frederickson, G.N.: Sequencing tasks with exponential service times to minimize the expected flow time or makespan. J. ACM 28(1), 100–113 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  6. Buchholz, P., Hahn, E.M., Hermanns, H., Zhang, L.: Model checking algorithms for CTMDPs. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 225–242. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  7. Buchholz, P., Schulz, I.: Numerical analysis of continuous time Markov decision processes over finite horizons. Comput. OR 38(3), 651–659 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  8. Butkova, Y., Hatefi, H., Hermanns, H., Krčál, J.: Optimal continuous time Markov decisions. CoRR abs/1507.02876 (2015). http://arxiv.org/abs/1507.02876

  9. Eisentraut, Christian, Hermanns, Holger, Katoen, Joost-Pieter, Zhang, Lijun: A semantics for every GSPN. In: Colom, José-Manuel, Desel, Jörg (eds.) PETRI NETS 2013. LNCS, vol. 7927, pp. 90–109. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  10. Fearnley, J., Rabe, M., Schewe, S., Zhang, L.: Efficient approximation of optimal control for continuous-time markov games. In: FSTTCS, pp. 399–410 (2011)

    Google Scholar 

  11. Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. In: SOSP, pp. 29–43. ACM (2003)

    Google Scholar 

  12. Guck, D.: Quantitative Analysis of Markov Automata. Master’s thesis, RWTH Aachen University, June 2012

    Google Scholar 

  13. Guck, D., Hatefi, H., Hermanns, H., Katoen, J.-P., Timmer, M.: Modelling, reduction and analysis of markov automata. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 55–71. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  14. Gurobi Optimization Inc: Gurobi optimizer reference manual, version 6.0 (2015)

    Google Scholar 

  15. Hatefi, H., Hermanns, H.: Model checking algorithms for Markov automata. In: ECEASST, vol. 53 (2012)

    Google Scholar 

  16. Hatefi, H., Hermanns, H.: Improving time bounded reachability computations in interactive markov chains. In: Arbab, F., Sirjani, M. (eds.) FSEN 2013. LNCS, vol. 8161, pp. 250–266. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  17. Haverkort, B.R., Hermanns, H., Katoen, J.: On the use of model checking techniques for dependability evaluation. In: SRDS 2000, pp. 228–237. IEEE CS (2000)

    Google Scholar 

  18. Hermanns, H., Katoen, J.-P.: The how and why of interactive markov chains. In: de Boer, F.S., Bonsangue, M.M., Hallerstede, S., Leuschel, M. (eds.) FMCO 2009. LNCS, vol. 6286, pp. 311–338. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  19. Jensen, A.: Markoff chains as an aid in the study of Markoff processes. Scand. Actuarial J. 1953, 87–91 (1953)

    Article  MathSciNet  MATH  Google Scholar 

  20. Katoen, J., Zapreev, I.S., Hahn, E.M., Hermanns, H., Jansen, D.N.: The ins and outs of the probabilistic model checker MRMC. Perform. Eval. 68(2), 90–104 (2011)

    Article  Google Scholar 

  21. Lefévre, C.: Optimal control of a birth and death epidemic process. Oper. Res. 29(5), 971–982 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  22. Marsan, M.A., Balbo, G., Conte, G., Donatelli, S., Franceschinis, G.: Modelling with Generalized Stochastic Petri Nets. Wiley, New York (1994)

    MATH  Google Scholar 

  23. Meyer, J.F., Movaghar, A., Sanders, W.H.: Stochastic activity networks: Structure, behavior, and application. In: PNPM, pp. 106–115 (1985)

    Google Scholar 

  24. Miller, B.L.: Finite state continuous time Markov decision processes with a finite planning horizon. SIAM J. Control 6(2), 266–280 (1968)

    Article  MathSciNet  MATH  Google Scholar 

  25. Neuhäußer, M.R.: Model checking nondeterministic and randomly timed systems. Ph.D. thesis, RWTH Aachen University (2010)

    Google Scholar 

  26. Neuhäußer, M.R., Stoelinga, M., Katoen, J.-P.: Delayed nondeterminism in continuous-time markov decision processes. In: de Alfaro, L. (ed.) FOSSACS 2009. LNCS, vol. 5504, pp. 364–379. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  27. Neuhäußer, M.R., Zhang, L.: Time-bounded reachability probabilities in continuous-time Markov decision processes. In: QEST, pp. 209–218 (2010)

    Google Scholar 

  28. Qiu, Q., Qu, Q., Pedram, M.: Stochastic modeling of a power-managed system-construction andoptimization. IEEE Trans. CAD Integr. Circ. Syst. 20(10), 1200–1217 (2001)

    Article  Google Scholar 

  29. Rabe, M.N., Schewe, S.: Finite optimal control for time-bounded reachability in CTMDPs and continuous-time Markov games. Acta Inf. 48(5–6), 291–315 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  30. Rabe, M.N., Schewe, S.: Optimal time-abstract schedulers for CTMDPs and continuous-time Markov games. Theor. Comput. Sci. 467, 53–67 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  31. Timmer, M., van de Pol, J., Stoelinga, M.I.A.: Confluence reduction for markov automata. In: Braberman, V., Fribourg, L. (eds.) FORMATS 2013. LNCS, vol. 8053, pp. 243–257. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  32. Zhang, L., Neuhäußer, M.R.: Model checking interactive markov chains. In: Esparza, J., Majumdar, R. (eds.) TACAS 2010. LNCS, vol. 6015, pp. 53–68. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

Download references

Acknowledgements

We are grateful to Moritz Hahn (ISCAS Beijing), Dennis Guck (Universiteit Twente), and Markus Rabe (UC Berkeley) for discussions and technical contributions. This work is supported by the EU 7th Framework Programme projects 295261 (MEALS) and 318490 (SENSATION), by the Czech Science Foundation project P202/12/G061, the DFG Transregional Collaborative Research Centre SFB/TR 14 AVACS, and by the CDZ project 1023 (CAP).

Author information

Authors and Affiliations

  1. Saarland University – Computer Science, Saarbrücken, Germany

    Yuliya Butkova, Hassan Hatefi, Holger Hermanns & Jan Krčál

Authors
  1. Yuliya Butkova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hassan Hatefi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Holger Hermanns
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jan Krčál
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuliya Butkova .

Editor information

Editors and Affiliations

  1. Universität des Saarlandes, Saarbrücken, Germany

    Bernd Finkbeiner

  2. East China Normal University, Shanghai, China

    Geguang Pu

  3. Chinese Academy of Sciences, Beijing, China

    Lijun Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Butkova, Y., Hatefi, H., Hermanns, H., Krčál, J. (2015). Optimal Continuous Time Markov Decisions. In: Finkbeiner, B., Pu, G., Zhang, L. (eds) Automated Technology for Verification and Analysis. ATVA 2015. Lecture Notes in Computer Science(), vol 9364. Springer, Cham. https://doi.org/10.1007/978-3-319-24953-7_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-24953-7_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24952-0

  • Online ISBN: 978-3-319-24953-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics