Abstract
Various quantitative methods and associated metrics for evaluating safety-related properties of ML functions have been proposed. However, it is often not clear how these metrics relate to safety requirements, how suitable target values can be selected to demonstrate that the safety requirements are met, and with which confidence can the results be used to reason about safety. This paper presents an uncertainty-aware method for using quantitative evidence to evaluate safety requirements of an ML-based function. To achieve this, we make use of Subjective Logic to describe opinions related to properties of the ML function and its associated evidence. We then show how combining these opinions can allow us to reason about our confidence in the statements we make based on this evidence. The approach is illustrated with a practical example and leads to some general observations related to the confidence that can be achieved in safety arguments for ML-based systems based on such evidence.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
https://www.iso.org/standard/83303.html(in development at time of writing).
- 3.
The non-informative prior weight ensures that when evidence begins to accumulate, uncertainty decreases accordingly. W is typically set to the same value as the cardinality of the domain (2 in our binary case), thus artificially adding one “success” and one “failure”. Higher values require more evidence for uncertainty to decrease.
- 4.
- 5.
We assume \(a=0.5\) and omit it from this and subsequent opinions for clarity.
References
Ashmore, R., Calinescu, R., Paterson, C.: Assuring the machine learning lifecycle: desiderata, methods, and challenges. ACM Comput. Surv. (CSUR) 54(5), 1–39 (2021)
Bloomfield, R., Netkachova, K., Rushby, J.: Defeaters and eliminative argumentation in assurance 2.0. arXiv preprint arXiv:2405.15800 (2024)
Brier, G.W.: Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 78(1), 1–3 (1950)
Burton, S., Hellert, C., Hüger, F., Mock, M., Rohatschek, A.: Safety assurance of machine learning for perception functions. In: Fingscheidt, T., Gottschalk, H., Houben, S. (eds.) Deep Neural Networks and Data for Automated Driving: Robustness, Uncertainty Quantification, and Insights Towards Safety, pp. 335–358. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-031-01233-4_12
Burton, S., Herd, B.: Addressing uncertainty in the safety assurance of machine-learning. Frontiers Comput. Sci. 5, 1132580 (2023). https://doi.org/10.3389/fcomp.2023.1132580
Dow, S.C., Dow, S.C.: Uncertainty about uncertainty. Springer (2012)
Gruber, C., Schenk, P.O., Schierholz, M., Kreuter, F., Kauermann, G.: Sources of uncertainty in machine learning–a statisticians’ view. arXiv preprint arXiv:2305.16703 (2023)
Hawkins, R., Kelly, T., Knight, J., Graydon, P.: A new approach to creating clear safety arguments. In: Dale, C., Anderson, T. (eds.) Advances in Systems Safety: Proceedings of the Nineteenth Safety-Critical Systems Symposium, Southampton, UK, 8-10th February 2011, pp. 3–23. Springer London, London (2011). https://doi.org/10.1007/978-0-85729-133-2_1
Hawkins, R., Paterson, C., Picardi, C., Jia, Y., Calinescu, R., Habli, I.: Guidance on the assurance of machine learning in autonomous systems (AMLAS). arXiv preprint arXiv:2102.01564 (2021)
Henne, M., Schwaiger, A., Roscher, K., Weiss, G.: Benchmarking uncertainty estimation methods for deep learning with safety-related metrics. In: SafeAI@ AAAI, pp. 83–90 (2020)
Herd, B., Burton, S.: Can you trust your ML metrics? Using subjective logic to determine the true contribution of ML metrics for safety. In: Proceedings of the the 39th ACM/SIGAPP Symposium On Applied Computing (SAC24), pp. 1579–1586 (2024)
Houben, S. et al.: Inspect, understand, overcome: a survey of practical methods for AI safety. In: Fingscheidt, T., Gottschalk, H., Houben, S. (eds.) Deep Neural Networks and Data for Automated Driving. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-01233-4_1
Huang, X., et al.: A survey of safety and trustworthiness of deep neural networks: verification, testing, adversarial attack and defence, and interpretability. Comput. Sci. Rev. 37, 100270 (2020)
Jøsang, Audun: Subjective Logic. Springer International Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-42337-1
Kläs, M., Vollmer, A.M.: Uncertainty in machine learning applications: a practice-driven classification of uncertainty. In: Gallina, B., Skavhaug, A., Schoitsch, E., Bitsch, F. (eds.) Computer Safety, Reliability, and Security. SAFECOMP 2018. Lecture Notes in Computer Science(), vol. 11094. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99229-7_36
Wang, R., Guiochet, J., Motet, G., Schön, W.: Safety case confidence propagation based on Dempster-Shafer theory. Int. J. Approximate Reasoning 107, 46–64 (2019)
Wang, R., Guiochet, J., Motet, G., Schön, W.: Safety case confidence propagation based on Dempster–Shafer theory. Int. J. Approximate Reasoning 107, 46–64 (2019). https://doi.org/10.1016/j.ijar.2019.02.002, https://www.sciencedirect.com/science/article/pii/S0888613X18303505
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Burton, S., Herd, B., Zacchi, JV. (2024). Uncertainty-Aware Evaluation of Quantitative ML Safety Requirements. In: Ceccarelli, A., Trapp, M., Bondavalli, A., Schoitsch, E., Gallina, B., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2024 Workshops. SAFECOMP 2024. Lecture Notes in Computer Science, vol 14989. Springer, Cham. https://doi.org/10.1007/978-3-031-68738-9_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-68738-9_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-68737-2
Online ISBN: 978-3-031-68738-9
eBook Packages: Computer ScienceComputer Science (R0)