Skip to main content

Sampling-Based and Gradient-Based Efficient Scenario Generation

  • Conference paper
  • First Online:
Runtime Verification (RV 2024)

Abstract

Safety-critical autonomous systems often operate in highly uncertain environments. These environments consist of many agents, some of which are being designed, and some represent the uncertain aspects of the environment. Testing autonomous systems requires generating diverse scenarios. However, the space of scenarios is very large, and many scenarios do not represent edge cases of the system. We want to develop a framework for automatically generating interesting scenarios. We propose to describe scenarios using a formal language. We show how we can extract interesting scenarios using scenario specifications from sampling-based approaches for scenario generation. We also introduce another technique for edge-case scenario generation using the gradient computation over STL. We demonstrate the capability of our framework in scenario generation in two case studies of autonomous systems involving the autonomous driving domain and the safety of human-robot systems in an industrial manufacturing context.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The function \(\textsf{swish}(x;\gamma _1)\) = \(\frac{x}{1+e^{-\gamma _1 x}}\), and the function \(\textsf{softplus}(x;\gamma _2)\) = \(\frac{1}{\gamma _2}\log (1+e^{\gamma _2 x})\). Here, \(\gamma _1,\gamma _2 > 0\) are user-defined parameters trading off smoothness and the accuracy of approximation.

  2. 2.

    Let Y be a metric space with the metric d. Then, \(X \subseteq Y\) is called \(\epsilon \)-separated if for all \(x_1,x_2 \in X\), \(d(x_1,x_2) > \epsilon \).

  3. 3.

    A scenario can be viewed as a vector of length T, and any vector norm can be thus be used to define distance between scenarios.

  4. 4.

    For two vectors \(V_1, V_2 \in \mathbb {R}^n\), we have: \(\ \text {sup-norm}(V_1,V_2) = \max _i( \mid V_1(i)-V_2(i) \mid )\).

References

  1. Asam openscenario. https://www.asam.net/standards/detail/openscenario/. Accessed 19 May 2024

  2. AG, S.: Siemens offers real-time locating system for a safe production environment and optimized production processes (2020). https://press.siemens.com/global/en/pressrelease/siemens-offers-real-time-locating-system-safe-production-environment-and-optimized

  3. Akazaki, T., Hasuo, I.: Time robustness in MTL and expressivity in hybrid system falsification. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9207, pp. 356–374. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21668-3_21

    Chapter  Google Scholar 

  4. Baier, C., Katoen, J.P.: Principles of Model Checking. MIT Press, Cambridge (2008)

    Google Scholar 

  5. Bak, S., Betz, J., Chawla, A., Zheng, H., Mangharam, R.: Stress testing autonomous racing overtake maneuvers with RRT. In: 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 806–812. IEEE (2022)

    Google Scholar 

  6. Branicky, M.S., Curtiss, M.M., Levine, J., Morgan, S.: Sampling-based planning, control and verification of hybrid systems. IEE Proc.-Control Theory Appl. 153(5), 575–590 (2006)

    Article  Google Scholar 

  7. Dang, T., Donzé, A., Maler, O., Shalev, N.: Sensitive state-space exploration. In: 2008 47th IEEE Conference on Decision and Control, pp. 4049–4054. IEEE (2008)

    Google Scholar 

  8. Donzé, A., Maler, O.: Robust satisfaction of temporal logic over real-valued signals. In: Chatterjee, K., Henzinger, T.A. (eds.) FORMATS 2010. LNCS, vol. 6246, pp. 92–106. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15297-9_9

    Chapter  Google Scholar 

  9. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: an open urban driving simulator. In: Conference on Robot Learning, pp. 1–16. PMLR (2017)

    Google Scholar 

  10. Dreossi, T., Dang, T., Donzé, A., Kapinski, J., Jin, X., Deshmukh, J.V.: Efficient guiding strategies for testing of temporal properties of hybrid systems. In: Havelund, K., Holzmann, G., Joshi, R. (eds.) NFM 2015. LNCS, vol. 9058, pp. 127–142. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-17524-9_10

    Chapter  Google Scholar 

  11. Emerson, E.A., Lei, C.L.: Modalities for model checking (extended abstract) branching time strikes back. In: Proceedings of the 12th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, pp. 84–96 (1985)

    Google Scholar 

  12. Fainekos, G.E., Pappas, G.J.: Robustness of temporal logic specifications. In: Havelund, K., Núñez, M., Roşu, G., Wolff, B. (eds.) FATES/RV -2006. LNCS, vol. 4262, pp. 178–192. Springer, Heidelberg (2006). https://doi.org/10.1007/11940197_12

    Chapter  Google Scholar 

  13. Freeman, C.D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., Bachem, O.: Brax–a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281 (2021)

  14. Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 63–78 (2019)

    Google Scholar 

  15. Fremont, D.J., et al.: Scenic: a language for scenario specification and data generation. Mach. Learn. 112(10), 3805–3849 (2023)

    Article  MathSciNet  Google Scholar 

  16. Fronda, N., Abbas, H.: Differentiable inference of temporal logic formulas. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 41(11), 4193–4204 (2022)

    Article  Google Scholar 

  17. Frostig, R., Johnson, M.J., Leary, C.: Compiling machine learning programs via high-level tracing. Syst. Mach. Learn. 4(9) (2018)

    Google Scholar 

  18. Gulino, C., et al.: Waymax: an accelerated, data-driven simulator for large-scale autonomous driving research. Adv. Neural Inf. Process. Syst. 36 (2024)

    Google Scholar 

  19. Hansen, N., Müller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11(1), 1–18 (2003)

    Article  Google Scholar 

  20. Hashemi, N., Hoxha, B., Prokhorov, D., Fainekos, G., Deshmukh, J.: Scaling learning based policy optimization for temporal tasks via dropout. arXiv preprint arXiv:2403.15826 (2024)

  21. Heiden, E., Millard, D., Coumans, E., Sheng, Y., Sukhatme, G.S.: NeuralSim: augmenting differentiable simulators with neural networks. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) (2021). https://github.com/google-research/tiny-differentiable-simulator

  22. Jaillet, L., Cortés, J., Siméon, T.: Transition-based RRT for path planning in continuous cost spaces. In: 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2145–2150. IEEE (2008)

    Google Scholar 

  23. Karaman, S., Frazzoli, E.: Sampling-based algorithms for optimal motion planning. Int. J. Robot. Res. 30(7), 846–894 (2011)

    Article  Google Scholar 

  24. Kavraki, L.E., Kolountzakis, M.N., Latombe, J.C.: Analysis of probabilistic roadmaps for path planning. IEEE Trans. Robot. Autom. 14(1), 166–171 (1998)

    Article  Google Scholar 

  25. Kim, J., Esposito, J.M., Kumar, V.: An RRT-based algorithm for testing and validating multi-robot controllers. In: Robotics: Science and Systems, pp. 249–256. Boston, MA (2005)

    Google Scholar 

  26. Kim, J., Esposito, J.M., Kumar, V.: Sampling-based algorithm for testing and validating robot controllers. Int. J. Robot. Res. 25(12), 1257–1272 (2006)

    Article  Google Scholar 

  27. Koschi, M., Pek, C., Maierhofer, S., Althoff, M.: Computationally efficient safety falsification of adaptive cruise control systems. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 2879–2886. IEEE (2019)

    Google Scholar 

  28. Lagarias, J.C., Reeds, J.A., Wright, M.H., Wright, P.E.: Convergence properties of the nelder-mead simplex method in low dimensions. SIAM J. Optim. 9(1), 112–147 (1998)

    Article  MathSciNet  Google Scholar 

  29. LaValle, S.: Rapidly-exploring random trees: a new tool for path planning. Research Report, no. 9811 (1998)

    Google Scholar 

  30. Leung, K., Aréchiga, N., Pavone, M.: Backpropagation through signal temporal logic specifications: infusing logical structure into gradient-based methods. Int. J. Robot. Res. 42(6), 356–370 (2023)

    Article  Google Scholar 

  31. Majumdar, R., Mathur, A., Pirron, M., Stegner, L., Zufferey, D.: Paracosm: a test framework for autonomous driving simulations. In: FASE 2021. LNCS, vol. 12649, pp. 172–195. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-71500-7_9

    Chapter  Google Scholar 

  32. Marini, F., Walczak, B.: Particle swarm optimization (PSO). A tutorial. Chemom. Intell. Lab. Syst. 149, 153–165 (2015)

    Google Scholar 

  33. Queiroz, R., Berger, T., Czarnecki, K.: Geoscenario: an open dsl for autonomous driving scenario representation. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 287–294. IEEE (2019)

    Google Scholar 

  34. Riedmaier, S., Ponn, T., Ludwig, D., Schick, B., Diermeyer, F.: Survey on scenario-based safety assessment of automated vehicles. IEEE Access 8, 87456–87477 (2020)

    Article  Google Scholar 

  35. Rodionova, A., Lindemann, L., Morari, M., Pappas, G.J.: Combined left and right temporal robustness for control under STL specifications. IEEE Control Syst. Lett. (2022)

    Google Scholar 

  36. Rong, G., et al.: LGSVL simulator: a high fidelity simulator for autonomous driving. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–6. IEEE (2020)

    Google Scholar 

  37. Schütt, B., Braun, T., Otten, S., Sax, E.: SceML: a graphical modeling framework for scenario-based testing of autonomous vehicles. In: Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp. 114–120 (2020)

    Google Scholar 

  38. Tuncali, C.E., Fainekos, G.: Rapidly-exploring random trees for testing automated vehicles. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 661–666. IEEE (2019)

    Google Scholar 

  39. Vin, E., et al.: 3D environment modeling for falsification and beyond with scenic 3.0. In: Enea, C., Lal, A. (eds.) Computer Aided Verification. CAV 2023. LNCS, vol. 13964, pp. 253–265. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-37706-8_13

  40. Zhang, L., Peng, Z., Li, Q., Zhou, B.: Cat: closed-loop adversarial training for safe end-to-end driving. In: Conference on Robot Learning, pp. 2357–2372. PMLR (2023)

    Google Scholar 

  41. Zhong, Z., Tang, Y., Zhou, Y., Neves, V.D.O., Liu, Y., Ray, B.: A survey on scenario-based testing for automated driving systems in high-fidelity simulation. arXiv preprint arXiv:2112.00964 (2021)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Vidisha Kudalkar .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kudalkar, V. et al. (2025). Sampling-Based and Gradient-Based Efficient Scenario Generation. In: Ábrahám, E., Abbas, H. (eds) Runtime Verification. RV 2024. Lecture Notes in Computer Science, vol 15191. Springer, Cham. https://doi.org/10.1007/978-3-031-74234-7_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-74234-7_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-74233-0

  • Online ISBN: 978-3-031-74234-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics