Skip to main content
SpringerLink
Log in

Robustness of Specifications and Its Applications to Falsification, Parameter Mining, and Runtime Monitoring with S-TaLiRo

  • Conference paper
  • First Online:
Runtime Verification (RV 2019)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11757))

Included in the following conference series:

Abstract

Logical specifications have enabled formal methods by carefully describing what is correct, desired or expected of a given system. They have been widely used in runtime monitoring and applied to domains ranging from medical devices to information security. In this tutorial, we will present the theory and application of robustness of logical specifications. Rather than evaluate logical formulas to Boolean valuations, robustness interpretations attempt to provide numerical valuations that provide degrees of satisfaction, in addition to true/false valuations to models. Such a valuation can help us distinguish between behaviors that “barely” satisfy a specification to those that satisfy it in a robust manner. We will present and compare various notions of robustness in this tutorial, centered primarily around applications to safety-critical Cyber-Physical Systems (CPS). We will also present key ways in which the robustness notions can be applied to problems such as runtime monitoring, falsification search for finding counterexamples, and mining design parameters for synthesis.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Simulink model discussed at: http://www.mathworks.com/help/simulink/examples/modeling-an-automatic-transmission-controller.html.

References

  1. Abbas, H., Fainekos, G.E., Sankaranarayanan, S., Ivancic, F., Gupta, A.: Probabilistic temporal logic falsification of cyber-physical systems. ACM Transactions on Embedded Computing Systems 12(s2) (2013)

    Article  Google Scholar 

  2. Abbas, H., Hoxha, B., Fainekos, G., Ueda, K.: Robustness-guided temporal logic testing and verification for stochastic cyber-physical systems. In: IEEE 4th Annual International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER) (2014)

    Google Scholar 

  3. Abbas, H., Mittelmann, H., Fainekos, G.: Formal property verification in a conformance testing framework. In: 12th ACM-IEEE International Conference on Formal Methods and Models for System Design (2014)

    Google Scholar 

  4. Abbas, H., O’Kelly, M., Rodionova, A., Mangharam, R.: Safe at any speed: a simulation-based test harness for autonomous vehicles. In: CyPhy 2017 (2017)

    Google Scholar 

  5. Akazaki, T., Liu, S., Yamagata, Y., Duan, Y., Hao, J.: Falsification of cyber-physical systems using deep reinforcement learning. In: Havelund, K., Peleska, J., Roscoe, B., de Vink, E. (eds.) FM 2018. LNCS, vol. 10951, pp. 456–465. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95582-7_27

    Chapter  Google Scholar 

  6. Alur, R., Courcoubetis, C., Dill, D.: Model-checking for real-time systems. In: Mitchell, J. (ed.) 5th Annual IEEE Symposium on Logic in Computer Science (LICS), pp. 414–425. IEEE Computer Society Press, June 1990

    Google Scholar 

  7. Annpureddy, Y., Liu, C., Fainekos, G., Sankaranarayanan, S.: S-TaLiRo: a tool for temporal logic falsification for hybrid systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 254–257. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19835-9_21

    Chapter  MATH  Google Scholar 

  8. Anonymous: Model-based testing and validation of control software with Reactis (2003). http://www.reactive-systems.com/papers/bcsf.pdf

  9. Asarin, E., Donzé, A., Maler, O., Nickovic, D.: Parametric identification of temporal properties. In: Khurshid, S., Sen, K. (eds.) RV 2011. LNCS, vol. 7186, pp. 147–160. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29860-8_12

    Chapter  Google Scholar 

  10. Bartocci, E., et al.: Specification-based monitoring of cyber-physical systems: a survey on theory, tools and applications. In: Bartocci, E., Falcone, Y. (eds.) Lectures on Runtime Verification. LNCS, vol. 10457, pp. 135–175. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75632-5_5

    Chapter  Google Scholar 

  11. Bojarski, M., Testa, D.D., Dworakowski, D., et al.: End to end learning for self-driving cars. CoRR abs/1604.07316 (2016)

    Google Scholar 

  12. Cameron, F., Fainekos, G., Maahs, D.M., Sankaranarayanan, S.: Towards a verified artificial pancreas: challenges and solutions for runtime verification. In: Bartocci, E., Majumdar, R. (eds.) RV 2015. LNCS, vol. 9333, pp. 3–17. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23820-3_1

    Chapter  Google Scholar 

  13. Claviere, A., Dutta, S., Sankaranarayanan, S.: Trajectory tracking control for robotic vehicles using counterexample guided training of neural networks. In: ICAPS, pp. 680–688. AAAI Press (2019)

    Google Scholar 

  14. Deshmukh, J.V., Sankaranarayanan, S.: Formal techniques for verification and testing of cyber-physical systems. In: Al Faruque, M.A., Canedo, A. (eds.) Design Automation of Cyber-Physical Systems, pp. 69–105. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-13050-3_4

    Chapter  Google Scholar 

  15. Diwakaran, R.D., Sankaranarayanan, S., Trivedi, A.: Analyzing neighborhoods of falsifying traces in cyber-physical systems. In: International Conference on Cyber-Physical Systems (ICCPS), pp. 109–119. ACM Press (2017)

    Google Scholar 

  16. Dokhanchi, A., Amor, H.B., Deshmukh, J.V., Fainekos, G.: Evaluating perception systems for autonomous vehicles using quality temporal logic. In: Colombo, C., Leucker, M. (eds.) RV 2018. LNCS, vol. 11237, pp. 409–416. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03769-7_23

    Chapter  Google Scholar 

  17. Dokhanchi, A., Hoxha, B., Fainekos, G.: On-line monitoring for temporal logic robustness. In: Bonakdarpour, B., Smolka, S.A. (eds.) RV 2014. LNCS, vol. 8734, pp. 231–246. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11164-3_19

    Chapter  Google Scholar 

  18. Dokhanchi, A., Hoxha, B., Fainekos, G.: Formal requirement debugging for testing and verification of cyber-physical systems. ACM Trans. Embed. Comput. Syst. (TECS) 17(2), 34 (2018)

    Google Scholar 

  19. Dokhanchi, A., et al.: ARCH-COMP18 category report: results on the falsification benchmarks. In: ARCH@ ADHS, pp. 104–109 (2018)

    Google Scholar 

  20. Dokhanchi, A., Zutshi, A., Sriniva, R.T., Sankaranarayanan, S., Fainekos, G.: Requirements driven falsification with coverage metrics. In: Proceedings of the 12th International Conference on Embedded Software, pp. 31–40. IEEE Press (2015)

    Google Scholar 

  21. Donzé, A.: Breach, a toolbox for verification and parameter synthesis of hybrid systems. In: Touili, T., Cook, B., Jackson, P. (eds.) CAV 2010. LNCS, vol. 6174, pp. 167–170. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14295-6_17

    Chapter  Google Scholar 

  22. Donzé, A., Maler, O.: Robust satisfaction of temporal logic over real-valued signals. In: Chatterjee, K., Henzinger, T.A. (eds.) FORMATS 2010. LNCS, vol. 6246, pp. 92–106. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15297-9_9

    Chapter  MATH  Google Scholar 

  23. Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A., Seshia, S.A.: Systematic testing of convolutional neural networks for autonomous driving (2017). Reliable Machine Learning in the Wild (RMLW) workshop

    Google Scholar 

  24. Ernst, G., Arcaini, P., Donze, A., Fainekos, G., Mathesen, L., Pedrielli, G., Yaghoubi, S., Yamagata, Y., Zhang, Z.: ARCH-COMP 2019 category report: falsification. EPiC Ser. Comput. 61, 129–140 (2019)

    Article  Google Scholar 

  25. Fainekos, G., Sankaranarayanan, S., Ueda, K., Yazarel, H.: Verification of automotive control applications using s-TaLiRo. In: Proceedings of the American Control Conference (2012)

    Google Scholar 

  26. Fainekos, G.E., Girard, A., Kress-Gazit, H., Pappas, G.J.: Temporal logic motion planning for dynamic robots. Automatica 45(2), 343–352 (2009)

    Article  MathSciNet  Google Scholar 

  27. Fainekos, G.E., Pappas, G.J.: Robustness of temporal logic specifications. In: Havelund, K., Núñez, M., Roşu, G., Wolff, B. (eds.) FATES/RV -2006. LNCS, vol. 4262, pp. 178–192. Springer, Heidelberg (2006). https://doi.org/10.1007/11940197_12

    Chapter  Google Scholar 

  28. Fainekos, G.E., Pappas, G.J.: Robustness of temporal logic specifications for continuous-time signals. Theoret. Comput. Sci. 410(42), 4262–4291 (2009)

    Article  MathSciNet  Google Scholar 

  29. Ferrère, T., Nickovic, D., Donzé, A., Ito, H., Kapinski, J.: Interface-aware signal temporal logic. In: 22nd ACM International Conference on Hybrid Systems: Computation and Control, pp. 57–66 (2019)

    Google Scholar 

  30. Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: PLDI, pp. 63–78 (2019)

    Google Scholar 

  31. Gregg, A., MacMillan, D.: Airlines cancel thousands of flights as Boeing works to fix 737 max software problems. The Washington Post July 14 (2019)

    Google Scholar 

  32. Hoxha, B., Abbas, H., Fainekos, G.: Benchmarks for temporal logic requirements for automotive systems. In: Workshop on Applied Verification for Continuous and Hybrid Systems (2014)

    Google Scholar 

  33. Hoxha, B., Dokhanchi, A., Fainekos, G.: Mining parametric temporal logic properties in model based design for cyber-physical systems. Int. J. Softw. Tools Technol. Transfer 20, 79–93 (2018)

    Article  Google Scholar 

  34. Hoxha, B., Mavridis, N., Fainekos, G.: VISPEC: a graphical tool for elicitation of MTL requirements. In: IEEE/RSJ IROS (2015)

    Google Scholar 

  35. Johnson, T.T., Gannamaraju, R., Fischmeister, S.: A survey of electrical and electronic (E/E) notifications for motor vehicles. In: ESV 2015 (2015)

    Google Scholar 

  36. Kapinski, J., Deshmukh, J.V., Jin, X., Ito, H., Butts, K.: Simulation-based approaches for verification of embedded control systems: an overview of traditional and advanced modeling, testing, and verification techniques. IEEE Control Syst. 36(6), 45–64 (2016)

    Article  MathSciNet  Google Scholar 

  37. Koymans, R.: Specifying real-time properties with metric temporal logic. Real Time Syst. 2(4), 255–299 (1990)

    Article  Google Scholar 

  38. LeCun, Y., Kavukcuoglu, K., Farabet, C.: Convolutional networks and applications in vision. In: Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pp. 253–256, May 2010

    Google Scholar 

  39. Lee, T.B.: Report: software bug led to death in Uber’s self-driving crash. Ars Technica May 07 (2018)

    Google Scholar 

  40. Leitner, F., Leue, S.: Simulink design verifier vs. SPIN - a comparative case study (short paper). In: Formal Methods for Industrial Critical Systems (2008)

    Google Scholar 

  41. Maler, O., Nickovic, D.: Monitoring temporal properties of continuous signals. In: Lakhnech, Y., Yovine, S. (eds.) FORMATS/FTRTFT -2004. LNCS, vol. 3253, pp. 152–166. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30206-3_12

    Chapter  MATH  Google Scholar 

  42. Mathesen, L., Yaghoubi, S., Pedrielli, G., Fainekos, G.: Falsification of cyber-physical systems with robustness uncertainty quantification through stochastic optimization with adaptive restart. In: IEEE CASE (2019)

    Google Scholar 

  43. Nghiem, T., Sankaranarayanan, S., Fainekos, G.E., Ivancic, F., Gupta, A., Pappas, G.J.: Monte-Carlo techniques for falsification of temporal properties of non-linear hybrid systems. In: Proceedings of the 13th ACM International Conference on Hybrid Systems: Computation and Control, pp. 211–220. ACM Press (2010)

    Google Scholar 

  44. S-TaLiRo Tools. https://sites.google.com/a/asu.edu/s-taliro/

  45. Sandler, K., et al.: Killed by code: software transparency in implantable medical devices. Technical report, Software Freedom Law Center (2010)

    Google Scholar 

  46. Tuncali, C.E., Fainekos, G., Ito, H., Kapinski, J.: Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: IEEE Intelligent Vehicles Symposium (IV) (2018)

    Google Scholar 

  47. Tuncali, C.E., Hoxha, B., Ding, G., Fainekos, G., Sankaranarayanan, S.: Experience report: application of falsification methods on the UxAS system. In: Dutle, A., Muñoz, C., Narkawicz, A. (eds.) NFM 2018. LNCS, vol. 10811, pp. 452–459. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77935-5_30

    Chapter  Google Scholar 

  48. Yaghoubi, S., Fainekos, G.: Gray-box adversarial testing for control systems with machine learning components. In: ACM International Conference on Hybrid Systems: Computation and Control (HSCC) (2019)

    Google Scholar 

  49. Yaghoubi, S., Fainekos, G.: Worst-case satisfaction of STL specifications using feedforward neural network controllers: a Lagrange multipliers approach. In: International Conference on Embedded Software (EMSOFT) (2019)

    Google Scholar 

  50. Zhang, Z., Ernst, G., Sedwards, S., Arcaini, P.: Two-layered falsification of hybrid systems guided by Monte Carlo tree search. IEEE Trans. CADIntegr. Circ.Syst. 37(11), 2894–2905 (2018)

    Article  Google Scholar 

Download references

Acknowledgments

GF acknowledges support from NSF award 1350420. SS acknowledges support from NSF award numbers 1646556, 1815983 and the Air Force Research Laboratory (AFRL). All opinions expressed are those of the authors and not necessarily of the US NSF or AFRL.

Author information

Authors and Affiliations

  1. Arizona State University, Tempe, AZ, USA

    Georgios Fainekos

  2. Southern Illinois University, Carbondale, IL, USA

    Bardh Hoxha

  3. University of Colorado, Boulder, CO, USA

    Sriram Sankaranarayanan

Authors
  1. Georgios Fainekos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bardh Hoxha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sriram Sankaranarayanan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sriram Sankaranarayanan .

Editor information

Editors and Affiliations

  1. Universität des Saarlandes, Saarbrücken, Germany

    Bernd Finkbeiner

  2. University of Milano Bicocca, Milan, Italy

    Leonardo Mariani

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fainekos, G., Hoxha, B., Sankaranarayanan, S. (2019). Robustness of Specifications and Its Applications to Falsification, Parameter Mining, and Runtime Monitoring with S-TaLiRo. In: Finkbeiner, B., Mariani, L. (eds) Runtime Verification. RV 2019. Lecture Notes in Computer Science(), vol 11757. Springer, Cham. https://doi.org/10.1007/978-3-030-32079-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32079-9_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32078-2

  • Online ISBN: 978-3-030-32079-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation