Skip to main content
Springer Nature Link
Log in

A Scenario Approach for Parametric Markov Decision Processes

  • Chapter
  • First Online:
Principles of Verification: Cycling the Probabilistic Landscape

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15261))

  • 166 Accesses

Abstract

In this paper, we consider the parameter synthesis problem for parametric Markov decision processes (MDP). Computing the maximal expected value of satisfaction of a logical formula in parametric MDP is a challenging task. Thus, we adopt the scenario approach: instead of computing the precise rational function \(f_{\varphi }\) representing e.g. the maximal expected value, we aim at the approximation function \(\tilde{f}_{\varphi , \lambda }\) that is \(\lambda \)-probably approximately correct with respect to the desired statistical guarantees. The approximation function is based on a template chosen by the user, for instance a polynomial with fixed degree. By means of several theoretical results, we discuss the relation of \(\tilde{f}_{\varphi , \lambda }\) and \(f_{\varphi }\), and propose a framework for checking properties of the Markov model using \(\tilde{f}_{\varphi , \lambda }\). An extensive empirical evaluation show the effectiveness of our framework.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

eBook
USD 12.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/iscas-tis/PacPMA/.

  2. 2.

    https://github.com/iscas-tis/PacPMA/.

References

  1. Andova, S., Hermanns, H., Katoen, J.-P.: Discrete-time rewards model-checked. In: Larsen, K.G., Niebert, P. (eds.) FORMATS 2003. LNCS, vol. 2791, pp. 88–104. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-40903-8_8

    Chapter  Google Scholar 

  2. Badings, T.S., Cubuktepe, M., Jansen, N., Junges, S., Katoen, J.-P., Topcu, U.: Scenario-based verification of uncertain parametric MDPs. Int. J. Softw. Tools Technol. Transf. 24(5), 803–819 (2022). https://doi.org/10.1007/s10009-022-00673-z

    Article  Google Scholar 

  3. Baier, C., Hensel, C., Hutschenreiter, L., Junges, S., Katoen, J.-P., Klein, J.: Parametric Markov chains: PCTL complexity and fraction-free Gaussian elimination. Inf. Comput. 272, 104504 (2020). https://doi.org/10.1016/j.ic.2019.104504

    Article  MathSciNet  Google Scholar 

  4. Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press (2008)

    Google Scholar 

  5. Barbier, M., et al.: Validation of perception and decision-making systems for autonomous driving via statistical model checking. In: 2019 IEEE Intelligent Vehicles Symposium, pp. 252–259. IEEE (2019). https://doi.org/10.1109/IVS.2019.8813793

  6. Ben-Tal, A., Nemirovski, A.: Robust convex optimization. Math. Oper. Res. 23(4), 769–805 (1998). https://doi.org/10.1287/moor.23.4.769

    Article  MathSciNet  Google Scholar 

  7. Ben-Tal, A., Nemirovski, A.: Robust solutions of uncertain linear programs. Oper. Res. Lett. 25(1), 1–13 (1999). https://doi.org/10.1016/S0167-6377(99)00016-4

    Article  MathSciNet  Google Scholar 

  8. Beyer, D., Löwe, S., Wendler, P.: Reliable benchmarking: requirements and solutions. Int. J. Softw. Tools Technol. Transf. 21(1), 1–29 (2019). https://doi.org/10.1007/s10009-017-0469-y

    Article  Google Scholar 

  9. Bianco, A., de Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026, pp. 499–513. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-60692-0_70

    Chapter  Google Scholar 

  10. Billingsley, P.: Probability and Measure. Wiley (1995)

    Google Scholar 

  11. Calafiore, G.C., Campi, M.C.: Uncertain convex programs: randomized solutions and confidence levels. Math. Program. 102(1), 25–46 (2005). https://doi.org/10.1007/s10107-003-0499-y

    Article  MathSciNet  Google Scholar 

  12. Calafiore, G.C., Campi, M.C.: The scenario approach to robust control design. IEEE Trans. Autom. Control 51(5), 742–753 (2006). https://doi.org/10.1109/TAC.2006.875041

    Article  MathSciNet  Google Scholar 

  13. Campi, M.C., Garatti, S.: The exact feasibility of randomized solutions of uncertain convex programs. SIAM J. Optim. 19(3), 1211–1230 (2008). https://doi.org/10.1137/07069821X

    Article  MathSciNet  Google Scholar 

  14. Campi, M.C., Garatti, S., Prandini, M.: The scenario approach for systems and control design. Annu. Rev. Control. 33(2), 149–157 (2009). https://doi.org/10.1016/j.arcontrol.2009.07.001

    Article  Google Scholar 

  15. Daws, C.: Symbolic and parametric model checking of discrete-time Markov chains. In: Liu, Z., Araki, K. (eds.) ICTAC 2004. LNCS, vol. 3407, pp. 280–294. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-31862-0_21

    Chapter  Google Scholar 

  16. Dehnert, C., et al.: PROPhESY: A PRObabilistic ParamEter SYnthesis tool. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 214–231. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_13

    Chapter  Google Scholar 

  17. Fang, X., Calinescu, R., Gerasimou, S., Alhwikem, F.: Fast parametric model checking through model fragmentation. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: ICSE, pp. 835–846. IEEE (2021). https://doi.org/10.1109/ICSE43902.2021.00081

  18. Forejt, V., Kwiatkowska, M., Norman, G., Parker, D.: Automated verification techniques for probabilistic systems. In: Bernardo, M., Issarny, V. (eds.) SFM 2011. LNCS, vol. 6659, pp. 53–113. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21455-4_3

    Chapter  Google Scholar 

  19. Gainer, P., Hahn, E.M., Schewe, S.: Accelerated model checking of parametric Markov chains. In: Lahiri, S.K., Wang, C. (eds.) ATVA 2018. LNCS, vol. 11138, pp. 300–316. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01090-4_18

    Chapter  Google Scholar 

  20. Garavel, H., Lang, F., Mateescu, R., Serwe, W.: CADP 2011: a toolbox for the construction and analysis of distributed processes. Int. J. Softw. Tools Technol. Transf. 15(2), 89–107 (2013). https://doi.org/10.1007/s10009-012-0244-z

    Article  Google Scholar 

  21. Ghaoui, L.E., Oustry, F., Lebret, H.: Robust solutions to uncertain semidefinite programs. SIAM J. Optim. 9(1), 33–52 (1998). https://doi.org/10.1137/S1052623496305717

    Article  MathSciNet  Google Scholar 

  22. Givan, R., Leach, S.M., Dean, T.L.: Bounded-parameter Markov decision processes. Artif. Intell. 122(1–2), 71–109 (2000). https://doi.org/10.1016/S0004-3702(00)00047-3

    Article  MathSciNet  Google Scholar 

  23. Hahn, E.M., Han, T., Zhang, L.: Synthesis for PCTL in parametric Markov decision processes. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 146–161. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20398-5_12

    Chapter  Google Scholar 

  24. Hahn, E.M., Hermanns, H., Zhang, L.: Probabilistic reachability for parametric Markov models. Int. J. Softw. Tools Technol. Transf. 13(1), 3–19 (2011). https://doi.org/10.1007/s10009-010-0146-x

    Article  Google Scholar 

  25. Hahn, E.M., Li, Y., Schewe, S., Turrini, A., Zhang, L.: iscasMc: a web-based probabilistic model checker. In: Jones, C., Pihlajasaari, P., Sun, J. (eds.) FM 2014. LNCS, vol. 8442, pp. 312–317. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-06410-9_22

    Chapter  Google Scholar 

  26. Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects Comput. 6(5), 512–535 (1994). https://doi.org/10.1007/BF01211866

    Article  Google Scholar 

  27. Hensel, C., Junges, S., Katoen, J.-P., Quatmann, T., Volk, M.: The probabilistic model checker Storm. Int. J. Softw. Tools Technol. Transf. 24(4), 589–610 (2022). https://doi.org/10.1007/s10009-021-00633-z

    Article  Google Scholar 

  28. Hopcroft, J.E., Motwani, R., Ullman, J.D.: Introduction to automata theory, languages, and computation, 3rd edn., Pearson International Edition. Addison-Wesley (2007)

    Google Scholar 

  29. Jansen, N., et al.: Accelerating parametric probabilistic verification. In: Norman, G., Sanders, W. (eds.) QEST 2014. LNCS, vol. 8657, pp. 404–420. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10696-0_31

    Chapter  Google Scholar 

  30. Jansen, N., Junges, S., Katoen, J.-P.: Parameter synthesis in Markov models: a gentle survey. In: Raskin, J., Chatterjee, K., Doyen, L., Majumdar, R. (eds.) Principles of Systems Design. LNCS, vol. 13660, pp. 407–437. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-22337-2_20

  31. Junges, S., Katoen, J.-P., Pérez, G.A., Winkler, T.: The complexity of reachability in parametric Markov decision processes. J. Comput. Syst. Sci. 119, 183–210 (2021). https://doi.org/10.1016/j.jcss.2021.02.006

    Article  MathSciNet  Google Scholar 

  32. Katoen, J.-P., Zapreev, I.S., Hahn, E.M., Hermanns, H., Jansen, D.N.: The ins and outs of the probabilistic model checker MRMC. Perform. Eval. 68(2), 90–104 (2011). https://doi.org/10.1016/j.peva.2010.04.001

    Article  Google Scholar 

  33. Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_47

    Chapter  Google Scholar 

  34. Kwiatkowska, M.Z., Norman, G., Parker, D.: The PRISM benchmark suite. In: QEST 2012: Ninth International Conference on Quantitative Evaluation of Systems, pp. 203–204. IEEE Computer Society (2012). https://doi.org/10.1109/QEST.2012.14

  35. Legay, A., Lukina, A., Traonouez, L.M., Yang, J., Smolka, S.A., Grosu, R.: Statistical model checking. In: Steffen, B., Woeginger, G. (eds.) Computing and Software Science. LNCS, vol. 10000, pp. 478–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-91908-9_23

    Chapter  Google Scholar 

  36. Li, R., Yang, P., Huang, C., Sun, Y., Xue, B., Zhang, L.: Towards practical robustness analysis for DNNs based on PAC-model learning. In: 2022 ACM/IEEE 44th International Conference on Software Engineering: ICSE, pp. 2189–2201. ACM (2022). https://doi.org/10.1145/3510003.3510143

  37. Liu, Y., Turrini, A., Hahn, E.M., Xue, B., Zhang, L.: Scenario approach for parametric Markov models. In: André, É., Sun, J. (eds.) ATVA 2023. LNCS, vol. 14215, pp. 158–180. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-45329-8_8

  38. Paigwar, A., Baranov, E., Renzaglia, A., Laugier, C., Legay, A.: Probabilistic collision risk estimation for autonomous driving: validation via statistical model checking. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 737–743. IEEE (2020). https://doi.org/10.1109/IV47402.2020.9304821

  39. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics, Wiley (1994). https://doi.org/10.1002/9780470316887

  40. Quatmann, T., Dehnert, C., Jansen, N., Junges, S., Katoen, J.-P.: Parameter synthesis for Markov models: faster than ever. In: Artho, C., Legay, A., Peled, D. (eds.) ATVA 2016. LNCS, vol. 9938, pp. 50–67. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46520-3_4

    Chapter  Google Scholar 

  41. Rockafellar, R.T.: Convex Analysis, vol. 11. Princeton University Press (1997)

    Google Scholar 

  42. Spel, J., Junges, S., Katoen, J.-P.: Finding provably optimal Markov chains. In: TACAS 2021. LNCS, vol. 12651, pp. 173–190. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72016-2_10

    Chapter  Google Scholar 

  43. Xie, J., Tan, W., Fang, B., Huang, Z.: Towards a statistical model checking method for safety-critical cyber-physical system verification. Secur. Commun. Networks 2021, 5536722:1–5536722:12 (2021). https://doi.org/10.1155/2021/5536722

  44. Xue, B., Zhang, M., Easwaran, A., Li, Q.: PAC model checking of black-box continuous-time dynamical systems. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 39(11), 3944–3955 (2020). https://doi.org/10.1109/TCAD.2020.3012251

Download references

Acknowledgements

We thank Jianting Yang (CNRS@CREATE, 1 Create Way, #08-01 CREATE Tower, Singapore 138602) for improving a proof of the paper. Work supported in part by the CAS Project for Young Scientists in Basic Research under grant No. YSBR-040, NSFC under grant No. 61836005, the CAS Pioneer Hundred Talents Program, and the ISCAS New Cultivation Project ISCAS-PYFX-202201. This project is part of the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant no. 101008233.

Author information

Authors and Affiliations

  1. Key Laboratory of System Software (Chinese Academy of Sciences) and State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, China

    Zhiming Chi, Andrea Turrini, Lijun Zhang & David N. Jansen

  2. University of Chinese Academy of Sciences, Beijing, China

    Zhiming Chi, Lijun Zhang & David N. Jansen

  3. Institute of Optics and Electronics, Chinese Academy of Sciences, Chengdu, China

    Ying Liu

  4. Institute of Intelligent Software Guangzhou, Guangzhou, China

    Andrea Turrini, Lijun Zhang & David N. Jansen

Authors
  1. Zhiming Chi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Turrini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lijun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David N. Jansen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lijun Zhang .

Editor information

Editors and Affiliations

  1. Ruhr University Bochum and Radboud University Nijmegen, Bochum, Germany

    Nils Jansen

  2. Radboud University Nijmegen, Nijmegen, The Netherlands

    Sebastian Junges

  3. Saarland University and University College London, Saarbrücken, Germany

    Benjamin Lucien Kaminski

  4. University of Oldenburg, Oldenburg, Germany

    Christoph Matheja

  5. RWTH Aachen University, Aachen, Germany

    Thomas Noll

  6. RWTH Aachen University, Aachen, Germany

    Tim Quatmann

  7. University of Twente and Radboud University Nijmegen, Enschede, The Netherlands

    Mariëlle Stoelinga

  8. Eindhoven University of Technology, Eindhoven, The Netherlands

    Matthias Volk

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Chi, Z., Liu, Y., Turrini, A., Zhang, L., Jansen, D.N. (2025). A Scenario Approach for Parametric Markov Decision Processes. In: Jansen, N., et al. Principles of Verification: Cycling the Probabilistic Landscape . Lecture Notes in Computer Science, vol 15261. Springer, Cham. https://doi.org/10.1007/978-3-031-75775-4_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-75775-4_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-75774-7

  • Online ISBN: 978-3-031-75775-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics