Skip to main content
Springer Nature Link
Log in

Graph-Based Reductions for Parametric and Weighted MDPs

  • Conference paper
  • First Online:
Automated Technology for Verification and Analysis (ATVA 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14215))

Abstract

We study the complexity of reductions for weighted reachability in parametric Markov decision processes. That is, we say a state p is never worse than q if for all valuations of the polynomial indeterminates it is the case that the maximal expected weight that can be reached from p is greater than the same value from q. In terms of computational complexity, we establish that determining whether p is never worse than q is \({\textbf {co}}{\textbf {ETR}}\)-complete. On the positive side, we give a polynomial-time algorithm to compute the equivalence classes of the order we study for Markov chains. Additionally, we describe and implement two inference rules to under-approximate the never-worse relation and empirically show that it can be used as an efficient preprocessing step for the analysis of large Markov decision processes.

This work was supported by the Belgian FWO “SAILor” (G030020N) and Flemish inter-university (iBOF) “DESCARTES” projects.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Technically, Le Roux and Pérez show that the problem is hard for target arenas. In appendix, we give reductions between target arenas and \(\tilde{\text {p}}\)MDPs.

  2. 2.

    This definition is inspired by [7] but it is not exactly the same as in that paper.

  3. 3.

    We were not able to properly compare our results to the ones in [2]. The sizes reported therein do not match those from the QComp models. The authors confirmed theirs are based on modified models of which the data and code have been misplaced.

References

  1. Baier, C., Katoen, J.: Principles of Model Checking. MIT Press, Cambridge (2008)

    Google Scholar 

  2. Bharadwaj, S., Roux, S.L., Pérez, G.A., Topcu, U.: Reduction techniques for model checking and learning in MDPs. In: Sierra, C. (ed.) Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, 19–25 August 2017, pp. 4273–4279. ijcai.org (2017). https://doi.org/10.24963/ijcai.2017/597

  3. Brázdil, T., et al.: Verification of Markov decision processes using learning algorithms. In: Cassez, F., Raskin, J.-F. (eds.) ATVA 2014. LNCS, vol. 8837, pp. 98–114. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11936-6_8

    Chapter  Google Scholar 

  4. Budde, C.E., et al.: On correctness, precision, and performance in quantitative verification. In: Margaria, T., Steffen, B. (eds.) ISoLA 2020. LNCS, vol. 12479, pp. 216–241. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-83723-5_15

    Chapter  Google Scholar 

  5. Ciesinski, F., Baier, C., Größer, M., Klein, J.: Reduction techniques for model checking Markov decision processes. In: Fifth International Conference on the Quantitative Evaluaiton of Systems (QEST 2008), 14–17 September 2008, Saint-Malo, France, pp. 45–54. IEEE Computer Society (2008). https://doi.org/10.1109/QEST.2008.45

  6. Clarke, E.M., Henzinger, T.A., Veith, H., Bloem, R. (eds.): Handbook of Model Checking. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-10575-8

  7. D’Argenio, P.R., Jeannet, B., Jensen, H.E., Larsen, K.G.: Reachability analysis of probabilistic systems by successive refinements. In: de Alfaro, L., Gilmore, S. (eds.) PAPM-PROBMIV 2001. LNCS, vol. 2165, pp. 39–56. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44804-7_3

    Chapter  MATH  Google Scholar 

  8. Engelen, K.: Code for graph-based reductions for parametric and weighted MDPs. https://doi.org/10.5281/zenodo.7915828

  9. Hartmanns, A., Hermanns, H.: The modest toolset: an integrated environment for quantitative modelling and verification. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014. LNCS, vol. 8413, pp. 593–598. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54862-8_51

    Chapter  Google Scholar 

  10. Hartmanns, A., Junges, S., Quatmann, T., Weininger, M.: A practitioner’s guide to MDP model checking algorithms. In: Sankaranarayanan, S., Sharygina, N. (eds.) Tools and Algorithms for the Construction and Analysis of Systems - 29th International Conference, TACAS 2023, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Paris, France, 22–27 April 2023, Proceedings, Part I. LNCS, vol. 13993, pp. 469–488. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30823-9_24

  11. Hartmanns, A., Klauck, M., Parker, D., Quatmann, T., Ruijters, E.: The quantitative verification benchmark set. In: Vojnar, T., Zhang, L. (eds.) TACAS 2019. LNCS, vol. 11427, pp. 344–350. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17462-0_20

    Chapter  Google Scholar 

  12. Hensel, C., Junges, S., Katoen, J., Quatmann, T., Volk, M.: The probabilistic model checker storm. Int. J. Softw. Tools Technol. Transf. 24(4), 589–610 (2022). https://doi.org/10.1007/s10009-021-00633-z

  13. Junges, S., Katoen, J., Pérez, G.A., Winkler, T.: The complexity of reachability in parametric Markov decision processes. J. Comput. Syst. Sci. 119, 183–210 (2021). https://doi.org/10.1016/j.jcss.2021.02.006

  14. Křetínský, J., Meggendorfer, T.: Efficient strategy iteration for mean payoff in Markov decision processes. In: D’Souza, D., Narayan Kumar, K. (eds.) ATVA 2017. LNCS, vol. 10482, pp. 380–399. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68167-2_25

    Chapter  MATH  Google Scholar 

  15. Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_47

    Chapter  Google Scholar 

  16. Le Roux, S., Pérez, G.A.: The complexity of graph-based reductions for reachability in Markov decision processes. In: Baier, C., Dal Lago, U. (eds.) FoSSaCS 2018. LNCS, vol. 10803, pp. 367–383. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89366-2_20

    Chapter  Google Scholar 

  17. Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach, 4th ed. Pearson, Hoboken (2020). http://aima.cs.berkeley.edu/

  18. Schaefer, M.: Complexity of some geometric and topological problems. In: Eppstein, D., Gansner, E.R. (eds.) GD 2009. LNCS, vol. 5849, pp. 334–344. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-11805-0_32

    Chapter  Google Scholar 

  19. Spel, J., Junges, S., Katoen, J.-P.: Are parametric Markov chains monotonic? In: Chen, Y.-F., Cheng, C.-H., Esparza, J. (eds.) ATVA 2019. LNCS, vol. 11781, pp. 479–496. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31784-3_28

    Chapter  Google Scholar 

  20. Sutton, R.S., Barto, A.G.: Reinforcement Learning - An Introduction. Adaptive Computation and Machine Learning, MIT Press, Cambridge (1998). https://www.worldcat.org/oclc/37293240

  21. Winkler, T., Lehmann, J., Katoen, J.-P.: Out of control: reducing probabilistic models by control-state elimination. In: Finkbeiner, B., Wies, T. (eds.) VMCAI 2022. LNCS, vol. 13182, pp. 450–472. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-94583-1_22

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Antwerp – Flanders Make, Antwerp, Belgium

    Kasper Engelen, Guillermo A. Pérez & Shrisha Rao

Authors
  1. Kasper Engelen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guillermo A. Pérez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shrisha Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shrisha Rao .

Editor information

Editors and Affiliations

  1. Université Sorbonne Paris Nord, Villetaneuse, France

    Étienne André

  2. Singapore Management University, Singapore, Singapore

    Jun Sun

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Engelen, K., Pérez, G.A., Rao, S. (2023). Graph-Based Reductions for Parametric and Weighted MDPs. In: André, É., Sun, J. (eds) Automated Technology for Verification and Analysis. ATVA 2023. Lecture Notes in Computer Science, vol 14215. Springer, Cham. https://doi.org/10.1007/978-3-031-45329-8_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45329-8_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45328-1

  • Online ISBN: 978-3-031-45329-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation