Abstract
Formal techniques for verifying stochastic systems (e.g.., probabilistic model checking) do not generally scale well with respect to the system size. Therefore, simulation-based techniques such as statistical model checking are often used in practice. In this paper, we focus on stochastic hybrid systems and evaluate Monte Carlo and Quasi-Monte Carlo (QMC) methods for computing probabilistic reachability. We compare a number of interval estimation techniques based on the Central Limit Theorem (CLT), and we also introduce a new approach based on the CLT for computing confidence intervals for probabilities near the borders of the [0,1] interval. We empirically show that QMC techniques and our CLT approach are accurate and efficient in practice. Our results readily apply to any stochastic system and property that can be checked by simulation, and are hence relevant for statistical model checking.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Available at https://github.com/dreal/probreach/tree/master/model.
References
Agresti, A., Coull, B.A.: Approximate is better than “exact” for interval estimation of binomial proportions. Am. Stat. 52(2), 119–126 (1998)
Alur, R., Courcoubetis, C., Henzinger, T.A., Ho, P.-H.: Hybrid automata: an algorithmic approach to the specification and verification of hybrid systems. In: Grossman, R.L., Nerode, A., Ravn, A.P., Rischel, H. (eds.) HS 1991-1992. LNCS, vol. 736, pp. 209–229. Springer, Heidelberg (1993). https://doi.org/10.1007/3-540-57318-6_30
Anscombe, F.J.: The transformation of Poisson, binomial and negative-binomial data. Biometrika 35(3/4), 246–254 (1948)
Anscombe, F.J.: On estimating binomial response relations. Biometrika 43(3/4), 461–464 (1956)
Antonov, A.A., Ermakov, S.M.: Empirically estimating error of integration by Quasi-Monte Carlo method. Vestnik St. Petersburg Univ. Math. 47(1), 1–8 (2015)
Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.: Model-checking algorithms for continuous-time Markov chains. IEEE Trans. Softw. Eng. 29(6), 524–541 (2003)
Bianco, A., de Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026, pp. 499–513. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-60692-0_70
Clopper, C.J., Pearson, E.S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4), 404–413 (1934)
Brown, L.D., Cai, T.T., DasGupta, A.: Interval estimation for a binomial proportion. Stat. Sci. 16(2), 128–133 (2001)
Dean, N., Pagano, M.: Evaluating confidence interval methods for binomial proportions in clustered surveys. J. Surv. Stat. Methodol. 3(4), 484–503 (2015)
Edwin, W.B.: Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 22(158), 209–212 (1927)
Fränzle, M., Chen, M., Kröger, P.: In memory of Oded Maler: automatic reachability analysis of hybrid-state automata. ACM SIGLOG News 6(1), 19–39 (2019)
Gao, S., Avigad, J., Clarke, E.M.: Delta-decidability over the reals. In: LICS, pp. 305–314 (2012)
Gnewuch, M., Srivastav, A., Winzen, C.: Finding optimal volume subintervals with k points and calculating the star discrepancy are NP-hard problems. J. Complex. 25(2), 115–127 (2009)
Mahajan, K.K., Arora, S., Kaur, K.: Bayesian estimation for Gini index and a poverty measure in case of Pareto distribution using Jeffreys’ prior. MASA 10(1), 63–72 (2015)
Pradhan, V., Banerjee, T.: Confidence interval of the difference of two independent binomial proportions using weighted profile likelihood. Commun. Stati. - Simul. Comput. 37(4), 645–659 (2008)
Shmarov, F., Zuliani, P.: ProbReach: verified probabilistic \(\delta \)-reachability for stochastic hybrid systems. In: HSCC, pp. 134–139. ACM (2015)
Shmarov, F., Zuliani, P.: Probabilistic hybrid systems verification via SMT and Monte Carlo techniques. In: Bloem, R., Arbel, E. (eds.) HVC 2016. LNCS, vol. 10028, pp. 152–168. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49052-6_10
Sobol’, I.M.: On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput. Math. Math. Phys. 7(4), 86–112 (1967)
Traub, J.F., Wasilkowski, G.W., Woźniakowski, H.: Information-Based Complexity. Academic Press, Cambridge (1988)
Vardi, M.Y.: Automatic verification of probabilistic concurrent finite-state programs. In: FOCS, pp. 327–338 (1985)
Younes, H.L.S., Simmons, R.G.: Statistical probabilistic model checking with a focus on time-bounded properties. Inf. Comput. 204(9), 1368–1409 (2006)
Zuliani, P., Platzer, A., Clarke, E.M.: Bayesian statistical model checking with application to stateflow/simulink verification. Formal Methods Syst. Des. 43(2), 338–367 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Vasileva, M., Zuliani, P. (2020). An Evaluation of Estimation Techniques for Probabilistic Verification. In: Ben Hedia, B., Chen, YF., Liu, G., Yu, Z. (eds) Verification and Evaluation of Computer and Communication Systems. VECoS 2020. Lecture Notes in Computer Science(), vol 12519. Springer, Cham. https://doi.org/10.1007/978-3-030-65955-4_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-65955-4_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-65954-7
Online ISBN: 978-3-030-65955-4
eBook Packages: Computer ScienceComputer Science (R0)