Abstract
This paper presents a model-based testing framework for black-box probabilistic systems with stochastic continuous time. Markov automata are used as an underlying model. We show how to generate, execute and evaluate test cases automatically from a probabilistically timed requirements model. In doing so, we connect classical ioco-theory with statistical hypothesis testing; our ioco-style algorithms test for functional behaviour, while \(\chi ^2\) hypothesis tests and confidence interval estimations assess the statistical correctness of the system.
A crucial development are the classical soundness and completeness properties of our framework. Soundness states that test cases assign the correct verdict, while completeness states that our methods are powerful enough to discover each discrepancy in functional or statistical misbehaviour, up to arbitrary precision.
We illustrate our framework via the Bluetooth device discovery protocol.
This research has been partially funded by STW and ProRail under the project ArRangeer (12238), STW under the project SEQUOIA (15474), NWO under the project BEAT (612.001.303), NWO under the project SamSam (628.005.015), and the EU under the project SUCCESS (509-18240).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Beyer, M., Dulz, W.: Scenario-based statistical testing of quality of service requirements. In: Leue, S., Systä, T.J. (eds.) Scenarios: Models, Transformations and Tools. LNCS, vol. 3466, pp. 152–173. Springer, Heidelberg (2005). doi:10.1007/11495628_9
Bohnenkamp, H., Belinfante, A.: Timed testing with TorX. In: Fitzgerald, J., Hayes, I.J., Tarlecki, A. (eds.) FM 2005. LNCS, vol. 3582, pp. 173–188. Springer, Heidelberg (2005). doi:10.1007/11526841_13
Bohnenkamp, H., Stoelinga, M.: Quantitative testing. In: Proceedings of the 8th International Conference on Embedded Software, (EMSOFT), pp. 227–236. ACM (2008)
Böhr, F.: Model-based statistical testing of embedded systems. In: IEEE 4th International Conference on Software Testing, Verification and Validation, pp. 18–25 (2011)
Bozga, M., David, A. Hartmanns, H., Hermanns, H., Larsen, K.G., Legay, A., Tretmans, J.: State-of-the-art tools and techniques for quantitative modelling and analysis of embedded systems. In: DATE, pp. 370–375 (2012)
Briones, L.B., Brinksma, E.: A test generation framework for quiescent real-time systems. In: Grabowski, J., Nielsen, B. (eds.) FATES 2004. LNCS, vol. 3395, pp. 64–78. Springer, Heidelberg (2005). doi:10.1007/978-3-540-31848-4_5
Briones, L.B., Brinksma, E., Stoelinga, M.: A semantic framework for test coverage. In: Graf, S., Zhang, W. (eds.) ATVA 2006. LNCS, vol. 4218, pp. 399–414. Springer, Heidelberg (2006). doi:10.1007/11901914_30
Cheung, L., Stoelinga, M., Vaandrager, F.: A testing scenario for probabilistic processes. J. ACM 54(6), 29:1–29:45 (2007). Article 29
Choi, S.G., Dachman-Soled, D., Malkin, T., Wee, H.: Improved non-committing encryption with applications to adaptively secure protocols. In: Matsui, M. (ed.) ASIACRYPT 2009. LNCS, vol. 5912, pp. 287–302. Springer, Heidelberg (2009). doi:10.1007/978-3-642-10366-7_17
Cleaveland, R., Dayar, Z., Smolka, S.A., Yuen, S.: Testing preorders for probabilistic processes. Inf. Comput. 154(2), 93–148 (1999)
Cohn, D.L.: Measure Theory. Birkhäuser, Basel (1980)
D’Argenio, P.R., Katoen, J.-P.: A theory of stochastic systems part I: stochastic automata. Inf. Comput. 203(1), 1–38 (2005)
Deng, Y., Hennessy, M.: On the semantics of Markov automata. Inf. Comput. 222, 139–168 (2013)
Deng, Y., Hennessy, M., van Glabbeek, R.J., Morgan, C.: Characterising testing preorders for finite probabilistic processes. CoRR (2008)
Duflot, M., Kwiatkowska, M., Norman, G., Parker, D.: A formal analysis of bluetooth device discovery. Int. J. Softw. Tools Technol. Transf. 8(6), 621–632 (2006)
Eisentraut, C., Hermanns, H., Zhang, L.: On probabilistic automata in continuous time. In: IEEE 25th Annual Symposium on LICS, pp. 342–351 (2010)
Gerhold, M., Stoelinga, M.: Model-based testing of probabilistic systems. In: Stevens, P., Wąsowski, A. (eds.) FASE 2016. LNCS, vol. 9633, pp. 251–268. Springer, Heidelberg (2016). doi:10.1007/978-3-662-49665-7_15
Gerhold, M., Stoelinga, M.: Model-based testing of stochastic systems with IOCO theory. In: A-TEST 2016, Proceedings of the 7th International Workshop on Automating Test Case Design, Selection, and Evaluation, pp. 45–51. ACM (2016)
Guiotto, A., Acquaroli, B., Martelli, A.: MaTeLo: automated testing suite for software validation. In: DASIA, vol. 532 (2003)
Hartmanns, A., Hermanns, H.: The Modest Toolset: An Integrated Environment for Quantitative Modelling and Verification. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014. LNCS, vol. 8413, pp. 593–598. Springer, Heidelberg (2014). doi:10.1007/978-3-642-54862-8_51
Hermanns, H., Chains, I.M.: Interactive Markov Chains: and the Quest for Quantified Quality. Springer, Heidelberg (2002)
Hessel, A., Larsen, K.G., Mikucionis, M., Nielsen, B., Pettersson, P., Skou, A.: Testing real-time systems using UPPAAL. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. LNCS, vol. 4949, pp. 77–117. Springer, Heidelberg (2008). doi:10.1007/978-3-540-78917-8_3
Hierons, R.M., Merayo, M.G.: Mutation testing from probabilistic and stochastic finite state machines. J. Syst. Softw. 82(11), 1804–1818 (2009)
Hierons, R.M., Núñez, M.: Testing probabilistic distributed systems. In: Hatcliff, J., Zucca, E. (eds.) FMOODS/FORTE -2010. LNCS, vol. 6117, pp. 63–77. Springer, Heidelberg (2010). doi:10.1007/978-3-642-13464-7_6
Hierons, R.M., Núñez, M.: Implementation relations and probabilistic schedulers in the distributed test architecture. J. Syst. Softw. (2017)
Hwang, I., Cavalli, A.R.: Testing a probabilistic FSM using interval estimation. Comput. Netw. 54(7), 1108–1125 (2010)
Jegourel, C., Legay, A., Sedwards, S.: A Platform for High Performance Statistical Model Checking – PLASMA. In: Flanagan, C., König, B. (eds.) TACAS 2012. LNCS, vol. 7214, pp. 498–503. Springer, Heidelberg (2012). doi:10.1007/978-3-642-28756-5_37
Krichen, M., Tripakis, S.: Conformance testing for real-time systems. Form. Methods Syst. Des. 34(3), 238–304 (2009)
Kwiatkowska, M., Norman, G., Parker, D.: PRISM: probabilistic symbolic model checker. In: Field, T., Harrison, P.G., Bradley, J., Harder, U. (eds.) TOOLS 2002. LNCS, vol. 2324, pp. 200–204. Springer, Heidelberg (2002). doi:10.1007/3-540-46029-2_13
Larsen, K.G., Mikucionis, M., Nielsen, B.: Online testing of real-time systems using Uppaal. In: Grabowski, J., Nielsen, B. (eds.) FATES 2004. LNCS, vol. 3395, pp. 79–94. Springer, Heidelberg (2005). doi:10.1007/978-3-540-31848-4_6
Larsen, K.G., Skou, A.: Bisimulation through probabilistic testing, pp. 344–352. ACM Press (1989)
Lohrey, M., D’Argenio, P.R., Hermanns, H.: Axiomatising Divergence. In: Widmayer, P., Eidenbenz, S., Triguero, F., Morales, R., Conejo, R., Hennessy, M. (eds.) ICALP 2002. LNCS, vol. 2380. Springer, Heidelberg (2002)
Marsan, M.A., Balbo, G., Conte, G., Donatelli, S., Franceschinis, G.: Modelling with Generalized Stochastic Petri Nets. Wiley, Hoboken (1994)
Milner, R.: A Calculus of Communicating Systems. Springer, Heidelberg (1980)
Mostowski, W., Poll, E., Schmaltz, J., Tretmans, J., Wichers Schreur, R.: Model-based testing of electronic passports. In: Alpuente, M., Cook, B., Joubert, C. (eds.) FMICS 2009. LNCS, vol. 5825, pp. 207–209. Springer, Heidelberg (2009). doi:10.1007/978-3-642-04570-7_19
Nie, J., Demmel, J., Gu, M.: Global minimization of rational functions and the nearest GCDs. J. Glob. Optim. 40(4), 697–718 (2008)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (2014)
Kwiatkowska, M., Norman, G., Parker, D.: Stochastic model checking. In: Bernardo, M., Hillston, J. (eds.) SFM 2007. LNCS, vol. 4486, pp. 220–270. Springer, Heidelberg (2007). doi:10.1007/978-3-540-72522-0_6
Russell, N., Moore, R.: Explicit modelling of state occupancy in hidden markov models for automatic speech recognition. In: IEEE International Conference on Acoustics, Speech, and Signal Processing ICASSP, vol. 10, pp. 5–8 (1985)
Segala, R.: Modeling and verification of randomized distributed real-time systems. Ph.D. thesis, Cambridge, MA, USA (1995)
B. SIG. Bluetooth Specification, version 1.2 (2003). www.bluetooth.com
Song, L., Zhang, L., Godskesen, J.C., Hermanns, H., Eisentraut, C.: Late weak bisimulation for Markov automata. CoRR, abs/1202.4116 (2012)
Stoelinga, M.: Alea jacta est: verification of probabilistic, real-time and parametric systems. Ph.D. thesis, Radboud University of Nijmegen (2002)
Stoelinga, M., Vaandrager, F.: Root contention in IEEE 1394. In: Katoen, J.-P. (ed.) ARTS 1999. LNCS, vol. 1601, pp. 53–74. Springer, Heidelberg (1999). doi:10.1007/3-540-48778-6_4
Stokkink, W.G.J., Timmer, M., Stoelinga, M.I.A.: Divergent quiescent transition systems. In: Veanes, M., Viganò, L. (eds.) TAP 2013. LNCS, vol. 7942, pp. 214–231. Springer, Heidelberg (2013). doi:10.1007/978-3-642-38916-0_13
Thrun, S.: Probabilistic robotics. Commun. ACM 45(3), 52–57 (2002)
Timmer, M., Brinksma, H., Stoelinga, M., Testing, M.-B., Software, I., Safety, S.: Specification and verification, Volume 30 of NATO Science for Peace and Security, pp. 1–32. IOS Press (2011)
Timmer, M., Katoen, J.-P., Pol, J., Stoelinga, M.I.A.: Efficient modelling and generation of Markov automata. In: Koutny, M., Ulidowski, I. (eds.) CONCUR 2012. LNCS, vol. 7454, pp. 364–379. Springer, Heidelberg (2012). doi:10.1007/978-3-642-32940-1_26
Timmer, M., van de Pol, J., Stoelinga, M.I.A.: Confluence reduction for Markov automata. In: Braberman, V., Fribourg, L. (eds.) FORMATS 2013. LNCS, vol. 8053, pp. 243–257. Springer, Heidelberg (2013). doi:10.1007/978-3-642-40229-6_17
Tretmans, J.: Test generation with inputs, outputs and repetitive quiescence. Softw. - Concepts Tools 17(3), 103–120 (1996)
Veanes, M., Campbell, C., Grieskamp, W., Schulte, W., Tillmann, N., Nachmanson, L.: Model-based testing of object-oriented reactive systems with spec explorer. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. LNCS, vol. 4949, pp. 39–76. Springer, Heidelberg (2008). doi:10.1007/978-3-540-78917-8_2
Whittaker, J.A., Rekab, K., Thomason, M.G.: A Markov chain model for predicting the reliability of multi-build software. Inf. Softw. Technol. 42(12), 889–894 (2000)
Acknowledgements
We would like to thank David Huistra for his aid on the case study.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Gerhold, M., Stoelinga, M. (2017). Model-Based Testing of Probabilistic Systems with Stochastic Time. In: Gabmeyer, S., Johnsen, E. (eds) Tests and Proofs. TAP 2017. Lecture Notes in Computer Science(), vol 10375. Springer, Cham. https://doi.org/10.1007/978-3-319-61467-0_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-61467-0_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-61466-3
Online ISBN: 978-3-319-61467-0
eBook Packages: Computer ScienceComputer Science (R0)