Skip to main content
SpringerLink
Log in

RTAMT – Runtime Robustness Monitors with Application to CPS and Robotics

  • General
  • Regular
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

In this paper, we present the Real-Time Analog Monitoring Tool (RTAMT), a tool for quantitative monitoring of Signal Temporal Logic (STL) specifications. The library implements a flexible architecture that supports: (1) various environments connected by an Application Programming Interface (API) in Python, (2) various flavors of temporal logic specification and robustness notion such as STL, including an interface-aware variant that distinguishes between input and output variables, and (3) discrete-time and dense-time interpretation of STL with generation of online and offline monitors. We specifically focus on robotics and Cyber-Physical System (CPS) applications, showing how to integrate RTAMT into (1) the Robot Operating System (ROS) and (2) MATLAB/Simulink® environments. We evaluate the tool by demonstrating several use scenarios involving service robotic and avionic applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Lee, E.A., Seshia, S.A.: Introduction to Embedded Systems: A Cyber-Physical Systems Approach (2016)

    Google Scholar 

  2. Mitra, S.: Verifying cyber-physical systems: a path to safe autonomy (2021)

  3. Alur, R.: Principles of Cyber-Physical Systems (2015)

    Google Scholar 

  4. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3, p. 5. Kobe, Japan (2009)

    Google Scholar 

  5. Sha, L.: Using simplicity to control complexity. IEEE Softw. 4, 20–28 (2001)

    Google Scholar 

  6. Maler, O., Nickovic, D.: Monitoring temporal properties of continuous signals. In: Formal Techniques, Modelling and Analysis of Timed and Fault-Tolerant Systems, Joint International Conferences on Formal Modelling and Analysis of Timed Systems, FORMATS 2004 and Formal Techniques in Real-Time and Fault-Tolerant Systems, FTRTFT 2004, Proceedings, Grenoble, France, September 22-24, 2004, pp. 152–166 (2004)

  7. Ferrère, T., Nickovic, D., Donzé, A., Ito, H., Kapinski, J.: Interface-aware signal temporal logic. In: Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, HSCC 2019, Montreal, QC, Canada, April 16-18, 2019, pp. 57–66 (2019)

    Google Scholar 

  8. Nickovic, D., Yamaguchi, T.: RTAMT: online robustness monitors from STL. In: Automated Technology for Verification and Analysis – 18th International Symposium, ATVA 2020, Proceedings, Hanoi, Vietnam, October 19-23, 2020, pp. 564–571 (2020). https://doi.org/10.1007/978-3-030-59152-6_34

    Chapter  Google Scholar 

  9. Maler, O., Nickovic, D., Pnueli, A.: On synthesizing controllers from bounded-response properties. In: Computer Aided Verification, 19th International Conference, CAV 2007, Proceedings, Berlin, Germany, July 3-7, 2007, pp. 95–107 (2007)

    Google Scholar 

  10. Jaksic, S., Bartocci, E., Grosu, R., Kloibhofer, R., Nguyen, T., Nickovic, D.: From signal temporal logic to FPGA monitors. In: 13. ACM/IEEE International Conference on Formal Methods and Models for Codesign, MEMOCODE 2015, Austin, TX, USA, September 21-23, 2015, pp. 218–227 (2015)

    Chapter  Google Scholar 

  11. Parr, T.: The definitive antlr 4 reference (2013)

  12. Henzinger, T.A., Manna, Z., Pnueli, A.: What good are digital clocks? In: International Colloquium on Automata, Languages, and Programming, pp. 545–558. Springer, Berlin (1992)

    Chapter  Google Scholar 

  13. Donzé, A., Ferrère, T., Maler, O.: Efficient robust monitoring for STL. In: Computer Aided Verification (CAV), pp. 264–279 (2013)

    Chapter  Google Scholar 

  14. Nickovic, D., Maler, O.: AMT: a property-based monitoring tool for analog systems. In: Formal Modeling and Analysis of Timed Systems, 5th International Conference, FORMATS 2007, Proceedings, Salzburg, Austria, October 3-5, 2007, pp. 304–319 (2007)

    Google Scholar 

  15. Yamaguchi, T., Hoxha, B., Prokhorov, D., Deshmukh, J.V.: Specification-guided software fault localization for autonomous mobile systems. In: 2020 18th ACM-IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE), pp. 1–12. IEEE, Los Alamitos (2020)

    Google Scholar 

  16. Yamamoto, T., Terada, K., Ochiai, A., Saito, F., Asahara, Y., Murase, K.: Development of human support robot as the research platform of a domestic mobile manipulator. ROBOMECH J. 6(1), 4 (2019)

    Article  Google Scholar 

  17. Koenig, N., Howard, A.: Design and use paradigms for gazebo, an open-source multi-robot simulator. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Sendai, Japan, pp. 2149–2154 (2004)

    Google Scholar 

  18. Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M., Dolan, J., Duggins, D., Galatali, T., Geyer, C., et al.: Autonomous driving in urban environments: boss and the urban challenge. J. Field Robot. 25(8), 425–466 (2008)

    Article  Google Scholar 

  19. Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics (2005)

    Google Scholar 

  20. Marder-Eppstein, E., Berger, E., Foote, T., Gerkey, B., Konolige, K.: The office marathon: robust navigation in an indoor office environment. In: International Conference on Robotics and Automation (2010)

    Google Scholar 

  21. Kuffner, J.J., LaValle, S.M.: Rrt-connect: an efficient approach to single-query path planning. In: Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), vol. 2, pp. 995–1001. IEEE, Los Alamitos (2000)

    Chapter  Google Scholar 

  22. Pnueli, A.: The temporal logic of programs. In: 18th Annual Symposium on Foundations of Computer Science (SFCS 1977), pp. 46–57. IEEE, Los Alamitos (1977)

    Chapter  Google Scholar 

  23. Fainekos, G.E., Pappas, G.J.: Robustness of temporal logic specifications. In: Formal Approaches to Software Testing and Runtime Verification, First Combined International Workshops, FATES 2006 and RV 2006, Revised Selected Papers, Seattle, WA, USA, August 15-16, 2006, pp. 178–192 (2006)

    Google Scholar 

  24. Fainekos, G.E., Pappas, G.J.: Robustness of temporal logic specifications for continuous-time signals. Theor. Comput. Sci. 410(42), 4262–4291 (2009). https://doi.org/10.1016/j.tcs.2009.06.021

    Article  MathSciNet  Google Scholar 

  25. Donzé, A., Maler, O.: Robust satisfaction of temporal logic over real-valued signals. In: Formal Modeling and Analysis of Timed Systems (FORMATS), pp. 92–106 (2010)

    Chapter  Google Scholar 

  26. Abbas, H., Mittelmann, H., Fainekos, G.: Formal property verification in a conformance testing framework. In: 2014 Twelfth ACM/IEEE Conference on Formal Methods and Models for Codesign (MEMOCODE), pp. 155–164. IEEE, Los Alamitos (2014)

    Google Scholar 

  27. Akazaki, T., Tasuo, I.: Time robustness in MTL and expressivity in hybrid system falsification. In: Computer Aided Verification, 27th International Conference, CAV 2015, Proceedings, San Francisco, CA, USA, July 18-24, 2015, pp. 356–374, (2015).

    Google Scholar 

  28. Annpureddy, Y., Liu, C., Fainekos, G.E., Sankaranarayanan, S.: S-taliro: a tool for temporal logic falsification for hybrid systems. In: Tools and Algorithms for the Construction and Analysis of Systems – 17th International Conference, TACAS 2011, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2011, Proceedings, Saarbrücken, Germany, March 26-April 3, pp. 254–257 (2011)

    Google Scholar 

  29. Donzé, A.: Breach, a toolbox for verification and parameter synthesis of hybrid systems. In: Computer Aided Verification, 22nd International Conference, CAV 2010, Proceedings, Edinburgh, UK, July 15-19, 2010, pp. 167–170 (2010)

    Google Scholar 

  30. Havelund, K., Roşu, G.: Synthesizing monitors for safety properties. In: International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pp. 342–356. Springer, Berlin (2002)

    Google Scholar 

  31. Reinbacher, T., Függer, M., Brauer, J.: Real-time runtime verification on chip. In: Proc. of RV 2012. LNCS, vol. 7687, pp. 110–125 (2013). https://doi.org/10.1007/978-3-642-35632-2-13

    Chapter  Google Scholar 

  32. Reinbacher, T., Függer, M., Brauer, J.: Runtime verification of embedded real-time systems. Form. Methods Syst. Des. 44(3), 230–239 (2014)

    Article  Google Scholar 

  33. Reinbacher, T., Rozier, K.Y., Schumann, J.: Temporal-logic based runtime observer pairs for system health management of real-time systems. In: International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pp. 357–372. Springer, Berlin (2014)

    Google Scholar 

  34. Schumann, J., Moosbrugger, P., Rozier, K.Y.: Runtime analysis with R2U2: a tool exhibition report. In: Runtime Verification – 16th International Conference, RV 2016, Proceedings, Madrid, Spain, September 23-30, 2016, pp. 504–509 (2016)

    Google Scholar 

  35. Rozier, K.Y., Schumann, J.: R2U2: tool overview. In: RV-CuBES 2017. An International Workshop on Competitions, Usability, Benchmarks, Evaluation, and Standardisation for Runtime Verification Tools, September 15, 2017, Seattle, WA, USA, pp. 138–156 (2017)

    Google Scholar 

  36. Hariharan, G., Kempa, B., Wongpiromsarn, T., Jones, P.H., Rozier, K.Y.: MLTL multi-type (MLTLM): a logic for reasoning about signals of different types. In: Software Verification and Formal Methods for ML-Enabled Autonomous Systems – 5th International Workshop, FoMLAS 2022, and 15th International Workshop, NSV 2022, Proceedings, Haifa, Israel, July 31–August 1, and August 11, 2022, pp. 187–204 (2022)

    Google Scholar 

  37. Finkbeiner, B., Sankaranarayanan, S., Sipma, H.: Collecting statistics over runtime executions. In: Runtime Verification 2002, RV 2002, FLoC Satellite Event, Kopenhagen, Denmark, July 26, 2002, pp. 36–54 (2002). https://doi.org/10.1016/S1571-0661(04)80576-0

    Chapter  Google Scholar 

  38. D’Angelo, B., Sankaranarayanan, S., Sánchez, C., Robinson, W., Finkbeiner, B., Sipma, H.B., Mehrotra, S., Manna, Z.: LOLA: runtime monitoring of synchronous systems. In: 12th International Symposium on Temporal Representation and Reasoning (TIME 2005), Burlington, Vermont, USA, 23-25 June 2005, pp. 166–174 (2005)

    Chapter  Google Scholar 

  39. Faymonville, P., Finkbeiner, B., Schirmer, S., Torfah, H.: A stream-based specification language for network monitoring. In: Runtime Verification: 16th International Conference, RV 2016, Proceedings, Madrid, Spain, September 23–30, 2016, pp. 152–168. Springer, Berlin (2016)

    Chapter  Google Scholar 

  40. Faymonville, P., Finkbeiner, B., Schwenger, M., Torfah, H.: Real-time stream-based monitoring (2017). arXiv:1711.03829. ArXiv preprint

  41. Convent, L., Hungerecker, S., Leucker, M., Scheffel, T., Schmitz, M., Thoma, D.: Tessla: temporal stream-based specification language. In: Formal Methods: Foundations and Applications: 21st Brazilian Symposium, SBMF 2018, Proceedings, Salvador, Brazil, November 26–30, 2018, pp. 144–162. Springer, Berlin (2018)

    Google Scholar 

  42. Gorostiaga, F., Sánchez, C.: Striver: stream runtime verification for real-time event-streams. In: Runtime Verification: 18th International Conference, RV 2018, Proceedings 18, Limassol, Cyprus, November 10–13, 2018, pp. 282–298. Springer, Berlin (2018)

    Google Scholar 

  43. Dokhanchi, A., Hoxha, B., Fainekos, G.: On-line monitoring for temporal logic robustness. In: International Conference on Runtime Verification, pp. 231–246. Springer, Berlin (2014)

    Chapter  Google Scholar 

  44. Deshmukh, J.V., Donzé, A., Ghosh, S., Jin, X., Juniwal, G., Seshia, S.A.: Robust online monitoring of signal temporal logic. Form. Methods Syst. Des. 51(1), 5–30 (2017)

    Article  Google Scholar 

  45. Mamouras, K., Wang, Z.: Online signal monitoring with bounded lag. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 39(11), 3868–3880 (2020). https://doi.org/10.1109/TCAD.2020.3013053

    Article  Google Scholar 

  46. Jaksic, S., Bartocci, E., Grosu, R., Nickovic, D.: An algebraic framework for runtime verification. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 37(11), 2233–2243 (2018). https://doi.org/10.1109/TCAD.2018.2858460

    Article  Google Scholar 

  47. Mamouras, K., Chattopadhyay, A., Wang, Z.: Algebraic quantitative semantics for efficient online temporal monitoring. In: Tools and Algorithms for the Construction and Analysis of Systems – 27th International Conference, TACAS 2021, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2021, Proceedings, Part I, Luxembourg City, Luxembourg, March 27–April 1, 2021, pp. 330–348 (2021)

    Google Scholar 

  48. Asarin, E., Caspi, P., Maler, O.: Timed regular expressions. J. ACM 49(2), 172–206 (2002)

    Article  MathSciNet  Google Scholar 

  49. Asarin, E., Caspi, P., Maler, O.: A Kleene theorem for timed automata. In: Logic in Computer Science (LICS), pp. 160–171 (1997)

    Google Scholar 

  50. Ulus, D., Ferrère, T., Asarin, E., Maler, O.: Timed pattern matching. In: Formal Modeling and Analysis of Timed Systems (FORMATS), pp. 222–236 (2014)

    Chapter  Google Scholar 

  51. Ulus, D., Ferrère, T., Asarin, E., Maler, O.: Online timed pattern matching using derivatives. In: Tools and Algorithms for the Construction and Analysis of Systems – 22nd International Conference, TACAS 2016, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2016, Proceedings, Eindhoven, The Netherlands, April 2-8, 2016, pp. 736–751 (2016)

    Google Scholar 

  52. Ulus, D.: Montre: a tool for monitoring timed regular expressions. In: Computer Aided Verification – 29th International Conference, CAV 2017, Proceedings, Part I, Heidelberg, Germany, July 24-28, 2017, pp. 329–335 (2017)

    Google Scholar 

  53. Waga, M., Hasuo, I.: Moore-machine filtering for timed and untimed pattern matching. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 37(11), 2649–2660 (2018)

    Article  Google Scholar 

  54. Waga, M., Hasuo, I., Suenaga, K.: Efficient online timed pattern matching by automata-based skipping. In: Formal Modeling and Analysis of Timed Systems – 15th International Conference, FORMATS 2017, Proceedings, Berlin, Germany, September 5-7, 2017, pp. 224–243 (2017)

    Google Scholar 

  55. Waga, M., Hasuo, I., Suenaga, K.: MONAA: a tool for timed pattern matching with automata-based acceleration. In: 3rd Workshop on Monitoring and Testing of Cyber-Physical Systems, MT@CPSWeek 2018, Porto, Portugal, April 10, 2018, pp. 14–15 (2018)

    Chapter  Google Scholar 

  56. Kapinski, J., Jin, X., Deshmukh, J., Donze, A., Yamaguchi, T., Ito, H., Kaga, T., Kobuna, S., Seshia, S.: St-lib: a library for specifying and classifying model behaviors. Technical report, SAE Technical Paper, (2016)

  57. Najm, W.G., Smith, J.D., Yanagisawa, M., et al.: Pre-crash scenario typology for crash avoidance research. Technical report, United States. National Highway Traffic Safety Administration (2007)

  58. Koopman, P., Osyk, B., Weast, J.: Autonomous vehicles meet the physical world: Rss, variability, uncertainty, and proving safety. In: International Conference on Computer Safety, Reliability, and Security, pp. 245–253. Springer, Berlin (2019)

    Chapter  Google Scholar 

  59. Hekmatnejad, M., Yaghoubi, S., Dokhanchi, A., Amor, H.B., Shrivastava, A., Karam, L., Fainekos, G.: Encoding and monitoring responsibility sensitive safety rules for automated vehicles in signal temporal logic. In: Proceedings of the 17th ACM-IEEE International Conference on Formal Methods and Models for System Design, pp. 1–11 (2019)

    Google Scholar 

  60. Dreossi, T., Fremont, D.J., Ghosh, S., Kim, E., Ravanbakhsh, H., Vazquez-Chanlatte, M., Seshia, S.A.: Verifai: a toolkit for the design and analysis of artificial intelligence-based systems (2019). arXiv:1902.04245. ArXiv preprint

  61. Rong, G., Shin, B.H., Tabatabaee, H., Lu, Q., Lemke, S., Možeiko, M., Boise, E., Uhm, G., Gerow, M., Mehta, S., et al.: Lgsvl simulator: a high fidelity simulator for autonomous driving. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–6. IEEE, Los Alamitos (2020)

    Google Scholar 

  62. Vitelli, M., Chang, Y., Ye, Y., Wołczyk, M., Osiński, B., Niendorf, M., Grimmett, H., Huang, Q., Jain, A., Ondruska, P.: Safetynet: Safe planning for real-world self-driving vehicles using machine-learned policies (2021). arXiv:2109.13602. ArXiv preprint

  63. Liu, C., Arnon, T., Lazarus, C., Barrett, C., Kochenderfer, M.J.: Algorithms for verifying deep neural networks (2019). arXiv:1903.06758. ArXiv preprint

  64. Tuncali, C.E., Fainekos, G., Ito, H., Kapinski, J.: Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1555–1562. IEEE, Los Alamitos (2018)

    Chapter  Google Scholar 

  65. Date, Y., Baba, T., Hoxha, B., Yamaguchi, T., Prokhorov, D.: Application of simulation-based methods on autonomous vehicle control with deep neural network: work-in-progress. In: 2020 International Conference on Embedded Software (EMSOFT), pp. 1–3. IEEE, Los Alamitos (2020)

    Google Scholar 

  66. Ghosh, S., Pant, Y.V., Ravanbakhsh, H., Seshia, S.A.: Counterexample-guided synthesis of perception models and control. In: 2021 American Control Conference (ACC), pp. 3447–3454. IEEE, Los Alamitos (2021)

    Chapter  Google Scholar 

  67. Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A., Seshia, S.A.: A formalization of robustness for deep neural networks (2019). arXiv:1903.10033. ArXiv preprint

Download references

Author information

Authors and Affiliations

  1. TRINA, Toyota Motor NA R&D, 1555 Woodridge Ave, Ann Arbor, 48105, Michigan, USA

    Tomoya Yamaguchi & Bardh Hoxha

  2. AIT Austrian Institute of Technology, Giefinggasse 4, 1210, Vienna, Austria

    Dejan Ničković

Authors
  1. Tomoya Yamaguchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bardh Hoxha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dejan Ničković
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dejan Ničković.

Ethics declarations

Competing Interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yamaguchi, T., Hoxha, B. & Ničković, D. RTAMT – Runtime Robustness Monitors with Application to CPS and Robotics. Int J Softw Tools Technol Transfer 26, 79–99 (2024). https://doi.org/10.1007/s10009-023-00720-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-023-00720-3

Keywords

Search

Navigation