Skip to main content
SpringerLink
Log in

TraceVis: Towards Visualization for Deep Statistical Model Checking

  • Conference paper
  • First Online:
Leveraging Applications of Formal Methods, Verification and Validation: Tools and Trends (ISoLA 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12479))

Included in the following conference series:

Abstract

With the proliferation of neural networks (NN), the need to analyze, and ideally verify, their behavior becomes more and more pressing. Significant progress has been made in the analysis of individual NN decision episodes, but the verification of NNs as part of larger systems remains a grand challenge. Deep statistical model checking (DSMC) is a recent approach addressing that challenge in the context of Markov decision processes (MDP) where a NN represents a policy taking action decisions. The NN determinizes the MDP, resulting in a Markov chain which is analyzed by statistical model checking. Initial results in a Racetrack case study (a simple abstract encoding of driving control) suggest that such a DSMC analysis can be useful for quality assurance in system approval or certification.

Here we explore the use of visualization to support DSMC users (human analysts, domain engineers). We implement an interactive visualization tool, TraceVis, for the Racetrack case study. The tool allows to explore crash probabilities into particular wall segments as a function of start position and velocity. It furthermore supports the in-depth examination of the policy traces generated by DSMC, in aggregate form as well as individually. This demonstrates how visualization can foster the effective analysis of DSMC results, and it forms a first step in combining model checking and visualization in the analysis of NN behavior.

Authors are listed alphabetically.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Barto, A.G., Bradtke, S.J., Singh, S.P.: Learning to act using real-time dynamic programming. Artif. Intell. 72(1–2), 81–138 (1995)

    Article  Google Scholar 

  2. Bogdoll, J., Ferrer Fioriti, L.M., Hartmanns, A., Hermanns, H.: Partial order methods for statistical model checking and simulation. In: Bruni, R., Dingel, J. (eds.) FMOODS/FORTE 2011. LNCS, vol. 6722, pp. 59–74. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21461-5_4

    Chapter  Google Scholar 

  3. Bonet, B., Geffner, H.: Labeled RTDP: improving the convergence of real-time dynamic programming. In: ICAPS, pp. 12–21 (2003)

    Google Scholar 

  4. Budde, C.E., D’Argenio, P.R., Hartmanns, A., Sedwards, S.: A statistical model checker for nondeterminism and rare events. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10806, pp. 340–358. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89963-3_20

    Chapter  Google Scholar 

  5. Budde, C.E., Dehnert, C., Hahn, E.M., Hartmanns, A., Junges, S., Turrini, A.: JANI: quantitative model and tool interaction. In: Legay, A., Margaria, T. (eds.) TACAS 2017. LNCS, vol. 10206, pp. 151–168. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-662-54580-5_9

    Chapter  Google Scholar 

  6. Croce, F., Andriushchenko, M., Hein, M.: Provable robustness of ReLU networks via maximization of linear regions. In: AISTATS, PMLR 89, pp. 2057–2066 (2019)

    Google Scholar 

  7. Dehnert, C., Junges, S., Katoen, J.-P., Volk, M.: A storm is coming: a modern probabilistic model checker. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10427, pp. 592–600. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63390-9_31

    Chapter  Google Scholar 

  8. Ehlers, R.: Formal verification of piece-wise linear feed-forward neural networks. In: D’Souza, D., Narayan Kumar, K. (eds.) ATVA 2017. LNCS, vol. 10482, pp. 269–286. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68167-2_19

    Chapter  Google Scholar 

  9. Etienne, L., Devogele, T., Buchin, M., McArdle, G.: Trajectory Box Plot: a new pattern to summarize movements. Int. J. Geograph. Inf. Sci. 30(5), 835–853 (2016). https://doi.org/10.1080/13658816.2015.1081205

  10. Gardner, M.: Mathematical games. Sci. Am. 229, 118–121 (1973)

    Article  Google Scholar 

  11. Gardner, M., Dorling, S.: Artificial neural networks (the multilayer perceptron)–a review of applications in the atmospheric sciences. Atmos. Environ. 32(14), 2627–2636 (1998)

    Article  Google Scholar 

  12. Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.T.: AI2: safety and robustness certification of neural networks with abstract interpretation. In: IEEE Symposium on Security and Privacy 2018, pp. 3–18 (2018)

    Google Scholar 

  13. Gros, T.P., Groß, D., Gumhold, S., Hoffmann, J., Klauck, M., Steinmetz, M.: TraceVis: Visualization for DSMC: tool, demonstration video, data (2020). https://doi.org/10.5281/zenodo.3961196

  14. Gros, T.P., Hermanns, H., Hoffmann, J., Klauck, M., Steinmetz, M.: Deep statistical model checking. In: Gotsman, A., Sokolova, A. (eds.) FORTE 2020. LNCS, vol. 12136, pp. 96–114. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50086-3_6

    Chapter  Google Scholar 

  15. Gros, T.P., Höller, D., Hoffmann, J., Wolf, V.: Tracking the race between deep reinforcement learning and imitation learning. In: Gribaudo M., Jansen, D.N., Remke, A. (eds.) Proceedings of the 17th International Conference on Quantitative Evaluation of SysTems (QEST). Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59854-9

  16. Gumhold, S.: The computer graphics and visualization framework. https://github.com/sgumhold/cgv. Accessed 18 May 2020

  17. Gumhold, S.: Splatting illuminated ellipsoids with depth correction. In: Ertl, T. (ed.) Proceedings of the Vision, Modeling, and Visualization Conference 2003 (VMV 2003), München, Germany, 19–21 November 2003, pp. 245–252. Aka GmbH (2003)

    Google Scholar 

  18. Hahn, E.M., Li, Y., Schewe, S., Turrini, A., Zhang, L.: iscasMc: a web-based probabilistic model checker. In: Jones, C., Pihlajasaari, P., Sun, J. (eds.) FM 2014. LNCS, vol. 8442, pp. 312–317. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-06410-9_22

    Chapter  Google Scholar 

  19. Hartmanns, A., Hermanns, H.: The modest toolset: an integrated environment for quantitative modelling and verification. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014. LNCS, vol. 8413, pp. 593–598. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54862-8_51

    Chapter  Google Scholar 

  20. Hérault, T., Lassaigne, R., Magniette, F., Peyronnet, S.: Approximate probabilistic model checking. In: Steffen, B., Levi, G. (eds.) VMCAI 2004. LNCS, vol. 2937, pp. 73–84. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24622-0_8

    Chapter  MATH  Google Scholar 

  21. Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)

    Article  Google Scholar 

  22. Hohman, F., Kahng, M., Pienta, R., Chau, D.H.: Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers. arXiv:1801.06889 [cs, stat], May 2018

  23. Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 3–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_1

    Chapter  Google Scholar 

  24. The JANI specification. http://www.jani-spec.org/. Accessed 28 Feb 2020

  25. Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_5

    Chapter  Google Scholar 

  26. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)

    Google Scholar 

  27. Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_47

    Chapter  Google Scholar 

  28. Kwiatkowska, M., Norman, G., Parker, D.: Stochastic model checking. In: Bernardo, M., Hillston, J. (eds.) SFM 2007. LNCS, vol. 4486, pp. 220–270. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72522-0_6

    Chapter  Google Scholar 

  29. Li, J., Liu, J., Yang, P., Chen, L., Huang, X., Zhang, L.: Analyzing deep neural networks with symbolic propagation: towards higher precision and faster verification. In: Chang, B.-Y.E. (ed.) SAS 2019. LNCS, vol. 11822, pp. 296–319. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32304-2_15

    Chapter  Google Scholar 

  30. McMahan, H.B., Gordon, G.J.: Fast exact planning in Markov decision processes. In: ICAPS, pp. 151–160 (2005)

    Google Scholar 

  31. Mirzargar, M., Whitaker, R.T., Kirby, R.M.: Curve Boxplot: generalization of boxplot for ensembles of curves. IEEE Trans. Vis. Comput. Graph. 20(12), 2654–2663 (2014). https://doi.org/10.1109/TVCG.2014.2346455. Conference Name: IEEE Transactions on Visualization and Computer Graphics

  32. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  33. Pineda, L.E., Lu, Y., Zilberstein, S., Goldman, C.V.: Fault-tolerant planning under uncertainty. In: IJCAI, pp. 2350–2356 (2013)

    Google Scholar 

  34. Pineda, L.E., Zilberstein, S.: Planning under uncertainty using reduced models: revisiting determinization. In: ICAPS, pp. 217–225 (2014)

    Google Scholar 

  35. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994)

    Google Scholar 

  36. Silver, D., et al.: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362(6419), 1140–1144 (2018)

    Article  MathSciNet  Google Scholar 

  37. Stoll, C., Gumhold, S., Seidel, H.P.: Incremental raycasting of piecewise quadratic surfaces on the GPU. In: 2006 IEEE Symposium on Interactive Ray Tracing, pp. 141–150. IEEE. https://doi.org/10.1109/RT.2006.280225. http://ieeexplore.ieee.org/document/4061556/

  38. Tominski, C., Schumann, H., Andrienko, G., Andrienko, N.: Stacking-based visualization of trajectory attribute data. IEEE Trans. Vis. Comput. Graph. 18(12), 2565–2574 (2012). https://doi.org/10.1109/TVCG.2012.265. Conference Name: IEEE Transactions on Visualization and Computer Graphics

  39. Tukey, J.W.: Mathematics and the picturing of data. In: Proceedings of the International Congress of Mathematicians, Vancouver, 1975, vol. 2, pp. 523–531 (1975)

    Google Scholar 

  40. Wang, J., Gou, L., Shen, H.W., Yang, H.: DQNViz: a visual analytics approach to understand deep Q-networks. IEEE Trans. Vis. Comput. Graph. 25(1), 288–298 (2019). https://doi.org/10.1109/TVCG.2018.2864504. https://ieeexplore.ieee.org/document/8454905/

  41. Wang, J., Hazarika, S., Li, C., Shen, H.W.: Visualization and visual analysis of ensemble data: a survey. IEEE Trans. Vis. Comput. Graph. 25(9), 2853–2872 (2019). https://doi.org/10.1109/TVCG.2018.2853721. Conference Name: IEEE Transactions on Visualization and Computer Graphics

  42. Wicker, M., Huang, X., Kwiatkowska, M.: Feature-guided black-box safety testing of deep neural networks. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10805, pp. 408–426. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89960-2_22

    Chapter  Google Scholar 

  43. Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 223–235. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45657-0_17

    Chapter  MATH  Google Scholar 

  44. Zahavy, T., Zrihem, N.B., Mannor, S.: Graying the black box: understanding DQNs. arXiv:1602.02658 [cs], April 2017

Download references

Acknowledgements

This work was partially supported by the ERC Advanced Investigators Grant 695614 (POWVER), by DFG grant 389792660 as part of TRR 248 (see https://perspicuous-computing.science) and by the two Clusters of Excellence CeTI (EXC 2050/1, grant 390696704) and PoL (EXC-2068, grant 390729961) of TU Dresden.

Author information

Authors and Affiliations

  1. Technical University Dresden, Dresden, Germany

    David Groß & Stefan Gumhold

  2. Saarland University, Saarland Informatics Campus, Saarbrücken, Germany

    Timo P. Gros, Jörg Hoffmann, Michaela Klauck & Marcel Steinmetz

Authors
  1. Timo P. Gros
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Groß
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Gumhold
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jörg Hoffmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michaela Klauck
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marcel Steinmetz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Timo P. Gros , David Groß , Stefan Gumhold , Jörg Hoffmann , Michaela Klauck or Marcel Steinmetz .

Editor information

Editors and Affiliations

  1. University of Limerick and Lero, Limerick, Ireland

    Tiziana Margaria

  2. TU Dortmund, Dortmund, Germany

    Bernhard Steffen

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gros, T.P., Groß, D., Gumhold, S., Hoffmann, J., Klauck, M., Steinmetz, M. (2021). TraceVis: Towards Visualization for Deep Statistical Model Checking. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation: Tools and Trends. ISoLA 2020. Lecture Notes in Computer Science(), vol 12479. Springer, Cham. https://doi.org/10.1007/978-3-030-83723-5_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-83723-5_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-83722-8

  • Online ISBN: 978-3-030-83723-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation