Skip to main content
Springer Nature Link
Log in

1–2–3–Go! Policy Synthesis for Parameterized Markov Decision Processes via Decision-Tree Learning and Generalization

  • Conference paper
  • First Online:
Verification, Model Checking, and Abstract Interpretation (VMCAI 2025)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15530))

  • 156 Accesses

Abstract

Despite the advances in probabilistic model checking, the scalability of the verification methods remains limited. In particular, the state space often becomes extremely large when instantiating parameterized Markov decision processes (MDPs) even with moderate values. Synthesizing policies for such huge MDPs is beyond the reach of available tools. We propose a learning-based approach to obtain a reasonable policy for such huge MDPs.

The idea is to generalize optimal policies obtained by model-checking small instances to larger ones using decision-tree learning. Consequently, our method bypasses the need for explicit state-space exploration of large models, providing a practical solution to the state-space explosion problem. We demonstrate the efficacy of our approach by performing extensive experimentation on the relevant models from the quantitative verification benchmark set. The experimental results indicate that our policies perform well, even when the size of the model is orders of magnitude beyond the reach of state-of-the-art analysis tools.

This research was funded in part by the DFG project 427755713 GOPro, the DFG GRK 2428 (ConVeY), the MUNI Award in Science and Humanities (MUNI/I/1757/2021) of the Grant Agency of Masaryk University, and the EU under MSCA grant agreement 101034413 (IST-BRIDGE).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The nature of our generalization-based policy synthesis is also portrayed by our quipping title “1–2–3–Go!”: Find out what works for cases 1, 2, and 3, then “Go!” and apply it for arbitrary large values of the parameters.

  2. 2.

    Axis-aligned predicates are of the form \(x > c\) where x is a state variable and \(c\in \mathbb {R}\). One can also consider DTs with richer predicates in the decision nodes.

References

  1. Ashok, P., Jackermeier, M., Jagtap, P., Kretínský, J., Weininger, M., Zamani, M.: dtControl: decision tree learning algorithms for controller representation. In: Ames, A.D., Seshia, S.A., Deshmukh, J. (eds.) HSCC ’20: 23rd ACM International Conference on Hybrid Systems: Computation and Control, Sydney, New South Wales, Australia, April 21–24, 2020, pp. 30:1–30:2. ACM (2020). https://doi.org/10.1145/3365365.3383468

  2. Ashok, P., Jackermeier, M., Kretínský, J., Weinhuber, C., Weininger, M., Yadav, M.: dtControl 2.0: explainable strategy representation via decision tree learning steered by experts. In: TACAS (2). Lecture Notes in Computer Science, vol. 12652, pp. 326–345. Springer (2021)

    Google Scholar 

  3. Ashok, P., Křetínský, J., Weininger, M.: PAC statistical model checking for Markov decision processes and stochastic games. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 497–519. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_29

    Chapter  Google Scholar 

  4. Azeem, M., Chakraborty, D., Kanav, S., Kretínský, J., Mohagheghi, M., Mohr, S., Weininger, M.: 1-2-3-Go! policy synthesis for parameterized Markov decision processes via decision-tree learning and generalization. CoRR abs/2410.18293 (2024). https://arxiv.org/abs/2410.18293

  5. Baier, C., Katoen, J.: Principles of Model Checking. MIT Press (2008)

    Google Scholar 

  6. Baier, C., Klein, J., Leuschner, L., Parker, D., Wunderlich, S.: Ensuring the reliability of your model checker: interval iteration for Markov decision processes. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 160–180. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_8

    Chapter  Google Scholar 

  7. Beyer, D., Löwe, S., Wendler, P.: Reliable benchmarking: requirements and solutions. Int. J. Softw. Tools Technol. Transf. 21(1), 1–29 (2019). https://doi.org/10.1007/s10009-017-0469-y

    Article  MATH  Google Scholar 

  8. Brázdil, T., Chatterjee, K., Chmelík, M., Fellner, A., Křetínský, J.: Counterexample explanation by learning small strategies in Markov decision processes. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 158–177. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_10

    Chapter  MATH  Google Scholar 

  9. Brázdil, T., et al.: Verification of Markov decision processes using learning algorithms. In: Cassez, F., Raskin, J.-F. (eds.) ATVA 2014. LNCS, vol. 8837, pp. 98–114. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11936-6_8

    Chapter  MATH  Google Scholar 

  10. Breiman, L.: Classification and Regression Trees. The Wadsworth statistics / probability series), Wadsworth International Group (1984)

    Google Scholar 

  11. Budde, C.E., D’Argenio, P.R., Hartmanns, A., Sedwards, S.: A statistical model checker for nondeterminism and rare events. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10806, pp. 340–358. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89963-3_20

    Chapter  MATH  Google Scholar 

  12. Budde, C.E., et al.: On correctness, precision, and performance in quantitative verification - QComp 2020 competition report. In: ISoLA (4). Lecture Notes in Computer Science, vol. 12479, pp. 216–241. Springer (2020)

    Google Scholar 

  13. Ciesinski, F., Baier, C., Größer, M., Klein, J.: Reduction techniques for model checking markov decision processes. In: QEST, pp. 45–54. IEEE Computer Society (2008)

    Google Scholar 

  14. Curtin, R.R., Edel, M., Lozhnikov, M., Mentekidis, Y., Ghaisas, S., Zhang, S.: mlpack 3: a fast, flexible machine learning library. J. Open Source Softw. 3(26), 726 (2018)

    Article  Google Scholar 

  15. D’Argenio, P.R., Legay, A., Sedwards, S., Traonouez, L.: Smart sampling for lightweight verification of Markov decision processes. Int. J. Softw. Tools Technol. Transf. 17(4), 469–484 (2015)

    Article  MATH  Google Scholar 

  16. Feng, L.: On learning assumptions for compositional verification of probabilistic systems. Ph.D. thesis, University of Oxford, UK (2014)

    Google Scholar 

  17. Feng, L., Kwiatkowska, M., Parker, D.: Automated learning of probabilistic assumptions for compositional reasoning. In: Giannakopoulou, D., Orejas, F. (eds.) Fundamental Approaches to Software Engineering, pp. 2–17. Springer, Heidelberg (2011)

    Chapter  MATH  Google Scholar 

  18. Groote, J.F., Verduzco, J.R., de Vink, E.P.: An efficient algorithm to determine probabilistic bisimulation. Algorithms 11(9), 131 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  19. Gros, T.P., Hermanns, H., Hoffmann, J., Klauck, M., Steinmetz, M.: Deep statistical model checking. In: Gotsman, A., Sokolova, A. (eds.) FORTE 2020. LNCS, vol. 12136, pp. 96–114. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50086-3_6

    Chapter  MATH  Google Scholar 

  20. Größer, M., Baier, C.: Partial order reduction for Markov decision processes: a survey. In: FMCO. Lecture Notes in Computer Science, vol. 4111, pp. 408–427. Springer (2005)

    Google Scholar 

  21. Haddad, S., Monmege, B.: Reachability in MDPs: refining convergence of value iteration. In: Ouaknine, J., Potapov, I., Worrell, J. (eds.) RP 2014. LNCS, vol. 8762, pp. 125–137. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11439-2_10

    Chapter  MATH  Google Scholar 

  22. Hahn, E.M., Perez, M., Schewe, S., Somenzi, F., Trivedi, A., Wojtczak, D.: Omega-regular objectives in model-free reinforcement learning. In: Vojnar, T., Zhang, L. (eds.) TACAS 2019. LNCS, vol. 11427, pp. 395–412. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17462-0_27

    Chapter  MATH  Google Scholar 

  23. Hartmanns, A.: MODEST - A unified language for quantitative models. In: FDL, pp. 44–51. IEEE (2012). https://ieeexplore.ieee.org/document/6336982/

  24. Hartmanns, A., Hermanns, H.: Explicit model checking of very large MDP using partitioning and secondary storage. In: Finkbeiner, B., Pu, G., Zhang, L. (eds.) Automated Technology for Verification and Analysis - 13th International Symposium, ATVA 2015, Shanghai, China, October 12–15, 2015, Proceedings. Lecture Notes in Computer Science, vol. 9364, pp. 131–147. Springer (2015). https://doi.org/10.1007/978-3-319-24953-7_10

  25. Hartmanns, A., Klauck, M., Parker, D., Quatmann, T., Ruijters, E.: The quantitative verification benchmark set. In: TACAS (1). Lecture Notes in Computer Science, vol. 11427, pp. 344–350. Springer (2019).https://doi.org/10.1007/978-3-030-17462-0_20

  26. Hartmanns, A., Timmer, M.: Sound statistical model checking for MDP using partial order and confluence reduction. Int. J. Softw. Tools Technol. Transf. 17(4), 429–456 (2015)

    Article  MATH  Google Scholar 

  27. Henriques, D., Martins, J.G., Zuliani, P., Platzer, A., Clarke, E.M.: Statistical model checking for Markov decision processes. In: QEST. pp. 84–93. IEEE Computer Society (2012)

    Google Scholar 

  28. Hensel, C., Junges, S., Katoen, J., Quatmann, T., Volk, M.: The probabilistic model checker storm. Int. J. Softw. Tools Technol. Transf. 24(4), 589–610 (2022). https://doi.org/10.1007/s10009-021-00633-z

    Article  MATH  Google Scholar 

  29. Hyafil, L., Rivest, R.L.: Constructing optimal binary decision trees is np-complete. Inf. Process. Lett. 5(1), 15–17 (1976). https://doi.org/10.1016/0020-0190(76)90095-8

    Article  MathSciNet  MATH  Google Scholar 

  30. Kamaleson, N.: Model reduction techniques for probabilistic verification of Markov chains. Ph.D. thesis, University of Birmingham, UK (2018)

    Google Scholar 

  31. Klein, J., et al.: Advances in probabilistic model checking with PRISM: variable reordering, quantiles and weak deterministic büchi automata. Int. J. Softw. Tools Technol. Transf. 20(2), 179–194 (2018)

    Article  MATH  Google Scholar 

  32. Křetínský, J.: Survey of statistical verification of linear unbounded properties: model checking and distances. In: Margaria, T., Steffen, B. (eds.) ISoLA 2016, Part I. LNCS, vol. 9952, pp. 27–45. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47166-2_3

    Chapter  MATH  Google Scholar 

  33. Kretínský, J., Meggendorfer, T.: Of cores: a partial-exploration framework for Markov decision processes. Log. Methods Comput. Sci. 16(4) (2020)

    Google Scholar 

  34. Kwiatkowska, M., Norman, G., Parker, D.: PRISM: probabilistic symbolic model checker. In: Field, T., Harrison, P.G., Bradley, J., Harder, U. (eds.) TOOLS 2002. LNCS, vol. 2324, pp. 200–204. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-46029-2_13

    Chapter  MATH  Google Scholar 

  35. Kwiatkowska, M.Z., Norman, G., Parker, D.: Game-based abstraction for Markov decision processes. In: QEST, pp. 157–166. IEEE Computer Society (2006)

    Google Scholar 

  36. Kwiatkowska, M., Norman, G., Parker, D.: Symmetry reduction for probabilistic model checking. In: Ball, T., Jones, R.B. (eds.) CAV 2006. LNCS, vol. 4144, pp. 234–248. Springer, Heidelberg (2006). https://doi.org/10.1007/11817963_23

    Chapter  MATH  Google Scholar 

  37. Kwiatkowska, M.Z., Norman, G., Parker, D.: The PRISM benchmark suite. In: QEST, pp. 203–204. IEEE Computer Society (2012). https://doi.org/10.1109/QEST.2012.14

  38. Kwiatkowska, M.Z., Parker, D., Qu, H.: Incremental quantitative verification for markov decision processes. In: DSN, pp. 359–370. IEEE Compute Society (2011)

    Google Scholar 

  39. Li, R., Liu, Y.: Compositional stochastic model checking probabilistic automata via symmetric assume-guarantee rule. In: 2019 IEEE 17th International Conference on Software Engineering Research, Management and Applications (SERA), pp. 110–115. IEEE (2019)

    Google Scholar 

  40. Lomuscio, A., Pirovano, E.: A counter abstraction technique for the verification of probabilistic swarm systems. In: AAMAS, pp. 161–169. International Foundation for Autonomous Agents and Multiagent Systems (2019)

    Google Scholar 

  41. Maisonneuve, V.: Automatic heuristic-based generation of MTBDD variable orderings for prism models. internship report (2009)

    Google Scholar 

  42. Mitchell, T.: Machine Learning, vol. 1. McGraw-Hill, New York (1997)

    Google Scholar 

  43. Mohagheghi, M., Salehi, K.: Machine learning and disk-based methods for qualitative verification of Markov decision processes. In: ICTERI Workshops. CEUR Workshop Proceedings, vol. 2732, pp. 74–88. CEUR-WS.org (2020)

    Google Scholar 

  44. Parker, D.A.: Implementation of symbolic model checking for probabilistic systems. Ph.D. thesis, University of Birmingham, UK (2003)

    Google Scholar 

  45. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics, Wiley (1994). https://doi.org/10.1002/9780470316887

  46. Pyeatt, L.D., Howe, A.E.: Decision tree function approximation in reinforcement learning (1999)

    Google Scholar 

  47. Rataj, A., Wozna-Szczesniak, B.: Extrapolation of an optimal policy using statistical probabilistic model checking. Fundam. Informaticae 157(4), 443–461 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  48. Shannon, C.E.: A mathematical theory of communication. The Bell system technical journal 27(3), 379–423 (1948)

    Article  MathSciNet  MATH  Google Scholar 

  49. Smolka, S., et al.: Scalable verification of probabilistic networks. In: PLDI, pp. 190–203. ACM (2019)

    Google Scholar 

  50. Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. Cambridge, MA, USA (1998)

    Google Scholar 

  51. Tappler, M., Aichernig, B.K., Bacci, G., Eichlseder, M., Larsen, K.G.: L\( ^{\text{*}}\)-based learning of Markov decision processes. In: FM. Lecture Notes in Computer Science, vol. 11800, pp. 651–669. Springer (2019)

    Google Scholar 

  52. Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 223–235. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45657-0_17

    Chapter  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Technical University of Munich, Munich, Germany

    Muqsit Azeem, Jan Křetínský, Stefanie Mohr & Maximilian Weininger

  2. Vali-e-Asr University of Rafsanjan, Rafsanjan, Iran

    Mohammadsadegh Mohagheghi

  3. Masaryk University, Brno, Czech Republic

    Debraj Chakraborty, Sudeep Kanav & Jan Křetínský

  4. Institute of Science and Technology Austria, Vienna, Austria

    Maximilian Weininger

Authors
  1. Muqsit Azeem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Debraj Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sudeep Kanav
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jan Křetínský
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mohammadsadegh Mohagheghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Stefanie Mohr
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Maximilian Weininger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jan Křetínský .

Editor information

Editors and Affiliations

  1. Indian Institute of Technology Bombay, Mumbai, Maharashtra, India

    Krishna Shankaranarayanan

  2. University of Colorado Boulder, Boulder, CO, USA

    Sriram Sankaranarayanan

  3. University of Colorado Boulder, Boulder, CO, USA

    Ashutosh Trivedi

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Azeem, M. et al. (2025). 1–2–3–Go! Policy Synthesis for Parameterized Markov Decision Processes via Decision-Tree Learning and Generalization. In: Shankaranarayanan, K., Sankaranarayanan, S., Trivedi, A. (eds) Verification, Model Checking, and Abstract Interpretation. VMCAI 2025. Lecture Notes in Computer Science, vol 15530. Springer, Cham. https://doi.org/10.1007/978-3-031-82703-7_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-82703-7_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-82702-0

  • Online ISBN: 978-3-031-82703-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics