Abstract
Novel methods for safety validation of autonomous vehicles are needed in order to enable a successful release of self-driving cars to the public. Decomposition of safety validation is one promising strategy for replacing blunt test mileage conducted by real world drives and can be applied in multiple dimensions: shifting to a scenario-based testing process, assuring safety of individual subsystems as well as combining different validation methods. In this paper, we facilitate such a decomposed safety validation strategy by simulation-based elicitation of accuracy requirements for the environmental perception for a given planning function in a defined urban scenario. Our contribution is threefold: a methodology based on exploring perceptual inaccuracy spaces and identifying safety envelopes, perceptual error models to construct such inaccuracy spaces, and an exemplary application that utilizes the proposed methodology in a simulation-based test process. In a case study, we elicit quantitative perception requirements for a prototypical planning function, which has been deployed for real test drives in the city of Hamburg, Germany. We consider requirements regarding tracking and the position of an oncoming vehicle in a concrete scenario. Finally, we conclude our methodology to be useful for a first elicitation of quantifiable and measurable requirements.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Failure Mode Effect Analysis.
- 2.
Fault Tree Analysis.
- 3.
In this work we follow the definition of accuracy given by ISO 5725-1 [12].
- 4.
- 5.
Metric refers to a variable defined on either an interval or ratio scale.
- 6.
Categorical refers to a variable defined on either a nominal or ordinal scale.
References
SJ3016: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (2018)
Bohrer, B., Tan, Y.K., Mitsch, S., Sogokon, A., Platzer, A.: A formal safety net for waypoint-following in ground robots. IEEE RA-L 4(3), 2910–2917 (2019)
Brahmi, M., Siedersberger, K.H., Siegel, A., Maurer, M.: Reference systems for environmental perception: requirements, validation and metric-based evaluation. In: 6. Tagung Fahrerassistenzsysteme (2013)
Brat, G., Bushnell, D., Davies, M., Giannakopoulou, D., Howar, F., Kahsai, T.: Verifying the safety of a flight-critical system. In: FM: Formal Methods (2015)
Bussler, A., Hartjen, L., Philipp, R., Schuldt, F.: Application of evolutionary algorithms and criticality metrics for the verification and validation of automated driving systems at urban intersections. In: IEEE IV Symposium (2020)
Chechik, M., Salay, R., Viger, T., Kokaly, S., Rahimi, M.: Software assurance in an uncertain world. In: Fundamental Approaches to Software Engineering (2019)
Dreossi, T., Donzé, A., Seshia, S.A.: Compositional falsification of cyber-physical systems with machine learning components. J. Autom. Reason. 63, 1031–1053 (2019)
Eiras, F., Lahijanian, M., Kwiatkowska, M.: Correct-by-construction advanced driver assistance systems based on a cognitive architecture. In: IEEE CAVS Symposium (2019)
Fremont, D.J., Chiu, J., Margineantu, D.D., Osipychev, D., Seshia, S.A.: Formal analysis and redesign of a neural network-based aircraft taxiing system with VerifAI. In: Computer Aided Verification (2020)
Guissouma, H., Leiner, S., Sax, E.: Towards design and verification of evolving cyber physical systems using contract-based methodology. In: IEEE ISSE (2019)
Hoxha, B., Dokhanchi, A., Fainekos, G.: Mining parametric temporal logic properties in model-based design for cyber-physical systems. In: STTT, vol. 20 (2017)
ISO 5725–1:1994: Accuracy (trueness and precision) of measurement methods and results - part 1: General principles and definitions. Standard (1994)
ISO/DIS 21448: Road vehicles - Safety of the intended functionality. Standard, International Organization for Standardization (ISO), Geneva, Switzerland (2021)
Jeannin, J.B., et al.: A formally verified hybrid system for the next-generation airborne collision avoidance system. In: TACAS (2015)
Jin, X., Donze, A., Deshmukh, J.V., Seshia, S.A.: Mining requirements from closed-loop control models. In: IEEE TCAD, vol. 34 (2015)
Julian, K.D., Kochenderfer, M.J.: Guaranteeing safety for neural network-based aircraft collision avoidance systems. In: IEEE DASC (2019)
Klamann, B., Lippert, M., Amersbach, C., Winner, H.: Defining pass-/fail-criteria for particular tests of automated driving functions. In: IEEE ITSC. IEEE, October 2019. https://doi.org/10.1109/ITSC.2019.8917483
Luckcuck, M., Farrell, M., Dennis, L.A., Dixon, C., Fisher, M.: Formal specification and verification of autonomous robotic systems. ACM CSUR 52(5), 1–41 (2019)
Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. (eds.): Autonomes Fahren (2015)
Moradi, M., Oakes, B.J., Saraoglu, M., Morozov, A., Janschek, K., Denil, J.: Exploring fault parameter space using reinforcement learning-based fault injection. In: IEEE/IFIP DSN-W (2020)
Nilsson, P., et al.: Correct-by-construction adaptive cruise control: two approaches. IEEE Trans. Control Syst. Technol. 24(4), 1294–1307 (2016)
Philion, J., Kar, A., Fidler, S.: Learning to evaluate perception models using planner-centric metrics. In: Proceedings of the IEEE/CVF CVPR (2020)
Philipp, R., Schuldt, F., Howar, F.: Functional decomposition of automated driving systems for the classification and evaluation of perceptual threats. 13. Uni-DAS e.V. Workshop Fahrerassistenz und automatisiertes Fahren (2020)
Piazzoni, A., Cherian, J., Slavik, M., Dauwels, J.: Modeling perception errors towards robust decision making in autonomous vehicles. In: IJCAI (2020)
Piazzoni, A., Cherian, J., Slavik, M., Dauwels, J.: Modeling sensing and perception errors towards robust decision making in autonomous vehicles (2020). https://arxiv.org/abs/2001.11695
Sangiovanni-Vincentelli, A., Damm, W., Passerone, R.: Taming dr. frankenstein: contract-based design for cyber-physical systems. Eur. J. Control 18(3), 217–238 (2012)
Schönemann, V., et al.: Fault tree-based derivation of safety requirements for automated driving on the example of cooperative valet parking. In: 26th International Conference on the Enhanced Safety of Vehicles (ESV) (2019)
Shalev-Shwartz, S., Shammah, S., Shashua, A.: On a formal model of safe and scalable self-driving cars (2017). https://arxiv.org/abs/1708.06374
Stahl, T., Diermeyer, F.: Online verification enabling approval of driving functions–implementation for a planner of an autonomous race vehicle. IEEE Open J. Intell. Transp. Syst. 2, 97–110 (2021)
Stahl, T., Eicher, M., Betz, J., Diermeyer, F.: Online verification concept for autonomous vehicles – illustrative study for a trajectory planning module (2020)
Stellet, J.E., Woehrle, M., Brade, T., Poddey, A., Branz, W.: Validation of automated driving - a structured analysis and survey of approaches. In: 13. Uni-DAS e.V. Workshop Fahrerassistenz und automatisiertes Fahren (2020)
Tomlin, C., Mitchell, I., Bayen, A., Oishi, M.: Computational techniques for the verification of hybrid systems. Proc. IEEE 91(7), 986–1001 (2003)
Tuncali, C.E., Fainekos, G., Ito, H., Kapinski, J.: Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: IEEE IV Symposium (2018)
Tuncali, C.E., Fainekos, G., Prokhorov, D., Ito, H., Kapinski, J.: Requirements-driven test generation for autonomous vehicles with machine learning components. IEEE Trans. Intell. Veh. 5, 265–280 (2020)
UNECE: Proposal for a new UN Regulation on uniform provisions concerning the approval of vehicles with regards to Automated Lane Keeping System. Tech. rep. (2020). https://undocs.org/ECE/TRANS/WP.29/2020/81
Volk, G., Gamerdinger, J., von Bernuth, A., Bringmann, O.: A comprehensive safety metric to evaluate perception in autonomous systems. In: IEEE ITSC (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Philipp, R., Qian, H., Hartjen, L., Schuldt, F., Howar, F. (2021). Simulation-Based Elicitation of Accuracy Requirements for the Environmental Perception of Autonomous Vehicles. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation. ISoLA 2021. Lecture Notes in Computer Science(), vol 13036. Springer, Cham. https://doi.org/10.1007/978-3-030-89159-6_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-89159-6_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89158-9
Online ISBN: 978-3-030-89159-6
eBook Packages: Computer ScienceComputer Science (R0)