Abstract
Safety-critical autonomous systems often operate in highly uncertain environments. These environments consist of many agents, some of which are being designed, and some represent the uncertain aspects of the environment. Testing autonomous systems requires generating diverse scenarios. However, the space of scenarios is very large, and many scenarios do not represent edge cases of the system. We want to develop a framework for automatically generating interesting scenarios. We propose to describe scenarios using a formal language. We show how we can extract interesting scenarios using scenario specifications from sampling-based approaches for scenario generation. We also introduce another technique for edge-case scenario generation using the gradient computation over STL. We demonstrate the capability of our framework in scenario generation in two case studies of autonomous systems involving the autonomous driving domain and the safety of human-robot systems in an industrial manufacturing context.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The function \(\textsf{swish}(x;\gamma _1)\) = \(\frac{x}{1+e^{-\gamma _1 x}}\), and the function \(\textsf{softplus}(x;\gamma _2)\) = \(\frac{1}{\gamma _2}\log (1+e^{\gamma _2 x})\). Here, \(\gamma _1,\gamma _2 > 0\) are user-defined parameters trading off smoothness and the accuracy of approximation.
- 2.
Let Y be a metric space with the metric d. Then, \(X \subseteq Y\) is called \(\epsilon \)-separated if for all \(x_1,x_2 \in X\), \(d(x_1,x_2) > \epsilon \).
- 3.
A scenario can be viewed as a vector of length T, and any vector norm can be thus be used to define distance between scenarios.
- 4.
For two vectors \(V_1, V_2 \in \mathbb {R}^n\), we have: \(\ \text {sup-norm}(V_1,V_2) = \max _i( \mid V_1(i)-V_2(i) \mid )\).
References
Asam openscenario. https://www.asam.net/standards/detail/openscenario/. Accessed 19 May 2024
AG, S.: Siemens offers real-time locating system for a safe production environment and optimized production processes (2020). https://press.siemens.com/global/en/pressrelease/siemens-offers-real-time-locating-system-safe-production-environment-and-optimized
Akazaki, T., Hasuo, I.: Time robustness in MTL and expressivity in hybrid system falsification. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9207, pp. 356–374. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21668-3_21
Baier, C., Katoen, J.P.: Principles of Model Checking. MIT Press, Cambridge (2008)
Bak, S., Betz, J., Chawla, A., Zheng, H., Mangharam, R.: Stress testing autonomous racing overtake maneuvers with RRT. In: 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 806–812. IEEE (2022)
Branicky, M.S., Curtiss, M.M., Levine, J., Morgan, S.: Sampling-based planning, control and verification of hybrid systems. IEE Proc.-Control Theory Appl. 153(5), 575–590 (2006)
Dang, T., Donzé, A., Maler, O., Shalev, N.: Sensitive state-space exploration. In: 2008 47th IEEE Conference on Decision and Control, pp. 4049–4054. IEEE (2008)
Donzé, A., Maler, O.: Robust satisfaction of temporal logic over real-valued signals. In: Chatterjee, K., Henzinger, T.A. (eds.) FORMATS 2010. LNCS, vol. 6246, pp. 92–106. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15297-9_9
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: an open urban driving simulator. In: Conference on Robot Learning, pp. 1–16. PMLR (2017)
Dreossi, T., Dang, T., Donzé, A., Kapinski, J., Jin, X., Deshmukh, J.V.: Efficient guiding strategies for testing of temporal properties of hybrid systems. In: Havelund, K., Holzmann, G., Joshi, R. (eds.) NFM 2015. LNCS, vol. 9058, pp. 127–142. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-17524-9_10
Emerson, E.A., Lei, C.L.: Modalities for model checking (extended abstract) branching time strikes back. In: Proceedings of the 12th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, pp. 84–96 (1985)
Fainekos, G.E., Pappas, G.J.: Robustness of temporal logic specifications. In: Havelund, K., Núñez, M., Roşu, G., Wolff, B. (eds.) FATES/RV -2006. LNCS, vol. 4262, pp. 178–192. Springer, Heidelberg (2006). https://doi.org/10.1007/11940197_12
Freeman, C.D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., Bachem, O.: Brax–a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281 (2021)
Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 63–78 (2019)
Fremont, D.J., et al.: Scenic: a language for scenario specification and data generation. Mach. Learn. 112(10), 3805–3849 (2023)
Fronda, N., Abbas, H.: Differentiable inference of temporal logic formulas. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 41(11), 4193–4204 (2022)
Frostig, R., Johnson, M.J., Leary, C.: Compiling machine learning programs via high-level tracing. Syst. Mach. Learn. 4(9) (2018)
Gulino, C., et al.: Waymax: an accelerated, data-driven simulator for large-scale autonomous driving research. Adv. Neural Inf. Process. Syst. 36 (2024)
Hansen, N., Müller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11(1), 1–18 (2003)
Hashemi, N., Hoxha, B., Prokhorov, D., Fainekos, G., Deshmukh, J.: Scaling learning based policy optimization for temporal tasks via dropout. arXiv preprint arXiv:2403.15826 (2024)
Heiden, E., Millard, D., Coumans, E., Sheng, Y., Sukhatme, G.S.: NeuralSim: augmenting differentiable simulators with neural networks. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) (2021). https://github.com/google-research/tiny-differentiable-simulator
Jaillet, L., Cortés, J., Siméon, T.: Transition-based RRT for path planning in continuous cost spaces. In: 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2145–2150. IEEE (2008)
Karaman, S., Frazzoli, E.: Sampling-based algorithms for optimal motion planning. Int. J. Robot. Res. 30(7), 846–894 (2011)
Kavraki, L.E., Kolountzakis, M.N., Latombe, J.C.: Analysis of probabilistic roadmaps for path planning. IEEE Trans. Robot. Autom. 14(1), 166–171 (1998)
Kim, J., Esposito, J.M., Kumar, V.: An RRT-based algorithm for testing and validating multi-robot controllers. In: Robotics: Science and Systems, pp. 249–256. Boston, MA (2005)
Kim, J., Esposito, J.M., Kumar, V.: Sampling-based algorithm for testing and validating robot controllers. Int. J. Robot. Res. 25(12), 1257–1272 (2006)
Koschi, M., Pek, C., Maierhofer, S., Althoff, M.: Computationally efficient safety falsification of adaptive cruise control systems. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 2879–2886. IEEE (2019)
Lagarias, J.C., Reeds, J.A., Wright, M.H., Wright, P.E.: Convergence properties of the nelder-mead simplex method in low dimensions. SIAM J. Optim. 9(1), 112–147 (1998)
LaValle, S.: Rapidly-exploring random trees: a new tool for path planning. Research Report, no. 9811 (1998)
Leung, K., Aréchiga, N., Pavone, M.: Backpropagation through signal temporal logic specifications: infusing logical structure into gradient-based methods. Int. J. Robot. Res. 42(6), 356–370 (2023)
Majumdar, R., Mathur, A., Pirron, M., Stegner, L., Zufferey, D.: Paracosm: a test framework for autonomous driving simulations. In: FASE 2021. LNCS, vol. 12649, pp. 172–195. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-71500-7_9
Marini, F., Walczak, B.: Particle swarm optimization (PSO). A tutorial. Chemom. Intell. Lab. Syst. 149, 153–165 (2015)
Queiroz, R., Berger, T., Czarnecki, K.: Geoscenario: an open dsl for autonomous driving scenario representation. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 287–294. IEEE (2019)
Riedmaier, S., Ponn, T., Ludwig, D., Schick, B., Diermeyer, F.: Survey on scenario-based safety assessment of automated vehicles. IEEE Access 8, 87456–87477 (2020)
Rodionova, A., Lindemann, L., Morari, M., Pappas, G.J.: Combined left and right temporal robustness for control under STL specifications. IEEE Control Syst. Lett. (2022)
Rong, G., et al.: LGSVL simulator: a high fidelity simulator for autonomous driving. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–6. IEEE (2020)
Schütt, B., Braun, T., Otten, S., Sax, E.: SceML: a graphical modeling framework for scenario-based testing of autonomous vehicles. In: Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp. 114–120 (2020)
Tuncali, C.E., Fainekos, G.: Rapidly-exploring random trees for testing automated vehicles. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 661–666. IEEE (2019)
Vin, E., et al.: 3D environment modeling for falsification and beyond with scenic 3.0. In: Enea, C., Lal, A. (eds.) Computer Aided Verification. CAV 2023. LNCS, vol. 13964, pp. 253–265. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-37706-8_13
Zhang, L., Peng, Z., Li, Q., Zhou, B.: Cat: closed-loop adversarial training for safe end-to-end driving. In: Conference on Robot Learning, pp. 2357–2372. PMLR (2023)
Zhong, Z., Tang, Y., Zhou, Y., Neves, V.D.O., Liu, Y., Ray, B.: A survey on scenario-based testing for automated driving systems in high-fidelity simulation. arXiv preprint arXiv:2112.00964 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kudalkar, V. et al. (2025). Sampling-Based and Gradient-Based Efficient Scenario Generation. In: Ábrahám, E., Abbas, H. (eds) Runtime Verification. RV 2024. Lecture Notes in Computer Science, vol 15191. Springer, Cham. https://doi.org/10.1007/978-3-031-74234-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-74234-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-74233-0
Online ISBN: 978-3-031-74234-7
eBook Packages: Computer ScienceComputer Science (R0)