Abstract
Novel methods for safety validation of autonomous vehicles are needed in order to enable a successful release of self-driving cars to the public. Decomposition of safety validation is one promising strategy for replacing blunt test mileage conducted by real world drives and can be applied in multiple dimensions: shifting to a scenario-based testing process, assuring safety of individual subsystems as well as combining different validation methods. In this paper, we facilitate such a decomposed safety validation strategy by simulation-based elicitation of accuracy requirements for the environmental perception for a given planning function in a defined urban scenario. Our contribution is threefold: a methodology based on exploring perceptual inaccuracy spaces and identifying safety envelopes, perceptual error models to construct such inaccuracy spaces, and an exemplary application that utilizes the proposed methodology in a simulation-based test process. In a case study, we elicit quantitative perception requirements for a prototypical planning function, which has been deployed for real test drives in the city of Hamburg, Germany. We consider requirements regarding tracking and the position of an oncoming vehicle in a concrete scenario. Finally, we conclude our methodology to be useful for a first elicitation of quantifiable and measurable requirements.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Failure Mode Effect Analysis.
- 2.
Fault Tree Analysis.
- 3.
In this work we follow the definition of accuracy given by ISO 5725-1 [12].
- 4.
- 5.
Metric refers to a variable defined on either an interval or ratio scale.
- 6.
Categorical refers to a variable defined on either a nominal or ordinal scale.
References
SJ3016: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (2018)
Bohrer, B., Tan, Y.K., Mitsch, S., Sogokon, A., Platzer, A.: A formal safety net for waypoint-following in ground robots. IEEE RA-L 4(3), 2910–2917 (2019)
Brahmi, M., Siedersberger, K.H., Siegel, A., Maurer, M.: Reference systems for environmental perception: requirements, validation and metric-based evaluation. In: 6. Tagung Fahrerassistenzsysteme (2013)
Brat, G., Bushnell, D., Davies, M., Giannakopoulou, D., Howar, F., Kahsai, T.: Verifying the safety of a flight-critical system. In: FM: Formal Methods (2015)
Bussler, A., Hartjen, L., Philipp, R., Schuldt, F.: Application of evolutionary algorithms and criticality metrics for the verification and validation of automated driving systems at urban intersections. In: IEEE IV Symposium (2020)
Chechik, M., Salay, R., Viger, T., Kokaly, S., Rahimi, M.: Software assurance in an uncertain world. In: Fundamental Approaches to Software Engineering (2019)
Dreossi, T., Donzé, A., Seshia, S.A.: Compositional falsification of cyber-physical systems with machine learning components. J. Autom. Reason. 63, 1031–1053 (2019)
Eiras, F., Lahijanian, M., Kwiatkowska, M.: Correct-by-construction advanced driver assistance systems based on a cognitive architecture. In: IEEE CAVS Symposium (2019)
Fremont, D.J., Chiu, J., Margineantu, D.D., Osipychev, D., Seshia, S.A.: Formal analysis and redesign of a neural network-based aircraft taxiing system with VerifAI. In: Computer Aided Verification (2020)
Guissouma, H., Leiner, S., Sax, E.: Towards design and verification of evolving cyber physical systems using contract-based methodology. In: IEEE ISSE (2019)
Hoxha, B., Dokhanchi, A., Fainekos, G.: Mining parametric temporal logic properties in model-based design for cyber-physical systems. In: STTT, vol. 20 (2017)
ISO 5725–1:1994: Accuracy (trueness and precision) of measurement methods and results - part 1: General principles and definitions. Standard (1994)
ISO/DIS 21448: Road vehicles - Safety of the intended functionality. Standard, International Organization for Standardization (ISO), Geneva, Switzerland (2021)
Jeannin, J.B., et al.: A formally verified hybrid system for the next-generation airborne collision avoidance system. In: TACAS (2015)
Jin, X., Donze, A., Deshmukh, J.V., Seshia, S.A.: Mining requirements from closed-loop control models. In: IEEE TCAD, vol. 34 (2015)
Julian, K.D., Kochenderfer, M.J.: Guaranteeing safety for neural network-based aircraft collision avoidance systems. In: IEEE DASC (2019)
Klamann, B., Lippert, M., Amersbach, C., Winner, H.: Defining pass-/fail-criteria for particular tests of automated driving functions. In: IEEE ITSC. IEEE, October 2019. https://doi.org/10.1109/ITSC.2019.8917483
Luckcuck, M., Farrell, M., Dennis, L.A., Dixon, C., Fisher, M.: Formal specification and verification of autonomous robotic systems. ACM CSUR 52(5), 1–41 (2019)
Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. (eds.): Autonomes Fahren (2015)
Moradi, M., Oakes, B.J., Saraoglu, M., Morozov, A., Janschek, K., Denil, J.: Exploring fault parameter space using reinforcement learning-based fault injection. In: IEEE/IFIP DSN-W (2020)
Nilsson, P., et al.: Correct-by-construction adaptive cruise control: two approaches. IEEE Trans. Control Syst. Technol. 24(4), 1294–1307 (2016)
Philion, J., Kar, A., Fidler, S.: Learning to evaluate perception models using planner-centric metrics. In: Proceedings of the IEEE/CVF CVPR (2020)
Philipp, R., Schuldt, F., Howar, F.: Functional decomposition of automated driving systems for the classification and evaluation of perceptual threats. 13. Uni-DAS e.V. Workshop Fahrerassistenz und automatisiertes Fahren (2020)
Piazzoni, A., Cherian, J., Slavik, M., Dauwels, J.: Modeling perception errors towards robust decision making in autonomous vehicles. In: IJCAI (2020)
Piazzoni, A., Cherian, J., Slavik, M., Dauwels, J.: Modeling sensing and perception errors towards robust decision making in autonomous vehicles (2020). https://arxiv.org/abs/2001.11695
Sangiovanni-Vincentelli, A., Damm, W., Passerone, R.: Taming dr. frankenstein: contract-based design for cyber-physical systems. Eur. J. Control 18(3), 217–238 (2012)
Schönemann, V., et al.: Fault tree-based derivation of safety requirements for automated driving on the example of cooperative valet parking. In: 26th International Conference on the Enhanced Safety of Vehicles (ESV) (2019)
Shalev-Shwartz, S., Shammah, S., Shashua, A.: On a formal model of safe and scalable self-driving cars (2017). https://arxiv.org/abs/1708.06374
Stahl, T., Diermeyer, F.: Online verification enabling approval of driving functions–implementation for a planner of an autonomous race vehicle. IEEE Open J. Intell. Transp. Syst. 2, 97–110 (2021)
Stahl, T., Eicher, M., Betz, J., Diermeyer, F.: Online verification concept for autonomous vehicles – illustrative study for a trajectory planning module (2020)
Stellet, J.E., Woehrle, M., Brade, T., Poddey, A., Branz, W.: Validation of automated driving - a structured analysis and survey of approaches. In: 13. Uni-DAS e.V. Workshop Fahrerassistenz und automatisiertes Fahren (2020)
Tomlin, C., Mitchell, I., Bayen, A., Oishi, M.: Computational techniques for the verification of hybrid systems. Proc. IEEE 91(7), 986–1001 (2003)
Tuncali, C.E., Fainekos, G., Ito, H., Kapinski, J.: Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: IEEE IV Symposium (2018)
Tuncali, C.E., Fainekos, G., Prokhorov, D., Ito, H., Kapinski, J.: Requirements-driven test generation for autonomous vehicles with machine learning components. IEEE Trans. Intell. Veh. 5, 265–280 (2020)
UNECE: Proposal for a new UN Regulation on uniform provisions concerning the approval of vehicles with regards to Automated Lane Keeping System. Tech. rep. (2020). https://undocs.org/ECE/TRANS/WP.29/2020/81
Volk, G., Gamerdinger, J., von Bernuth, A., Bringmann, O.: A comprehensive safety metric to evaluate perception in autonomous systems. In: IEEE ITSC (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Philipp, R., Qian, H., Hartjen, L., Schuldt, F., Howar, F. (2021). Simulation-Based Elicitation of Accuracy Requirements for the Environmental Perception of Autonomous Vehicles. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation. ISoLA 2021. Lecture Notes in Computer Science(), vol 13036. Springer, Cham. https://doi.org/10.1007/978-3-030-89159-6_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-89159-6_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89158-9
Online ISBN: 978-3-030-89159-6
eBook Packages: Computer ScienceComputer Science (R0)