Abstract
We investigate the role of ensemble methods in learning runtime monitors for operational design domains of autonomous systems. An operational design domain (ODD) of a system captures the conditions under which we can trust the components of the system to maintain its safety. A runtime monitor of an ODD predicts, based on a sequence of monitorable observations, whether the system is about to exit the ODD. For black-box systems, a key challenge in learning an ODD monitor is obtaining a monitor with a high degree of accuracy. While statistical theories such as that of probably approximate learning (PAC) allow us to provide guarantees on the accuracy of a learned ODD monitor up to a certain confidence probability (by bounding the number of needed training examples), practically, there will always remain a chance, that using such a one-shot approach will result in monitors with a high misclassification rate. To address this challenge we consider well-known ensemble learning algorithms and utilize them for learning ODD ensembles. We derive theoretical bounds on the estimated misclassification risk of ensembles, showing that it reduces exponentially with the number of monitors and linearly with the risk of individual monitors. An empirical evaluation of the impact of different ensemble learning methods on a case study from autonomous driving demonstrates the advantage of this approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For the interested reader, the CNN had three convolutional layers with 24, 48, 96 filters, respectively, with a \(5\times 5\) kernel size, composed with an inner dense layer with 512 units using ReLU activation. The CNN was trained on 99k images collected at restricted weather and time of the day conditions, and labeled with the correct CTE.
References
Avellaneda, F.: Efficient inference of optimal decision trees. In: AAAI 2020, pp. 3195–3202. AAAI Press (2020)
Blumenthal, M.S., Fraade-Blanar, L., Best, R., Irwin, J.L.: Safe Enough: Approaches to Assessing Acceptable Safety for Automated Vehicles. RAND Corporation, Santa Monica (2020). https://doi.org/10.7249/RRA569-1
Colwell, I., Phan, B., Saleem, S., Salay, R., Czarnecki, K.: An automated vehicle safety concept based on runtime restriction of the operational design domain. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1910–1917 (2018). https://doi.org/10.1109/IVS.2018.8500530
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning, pp. 1–16 (2017)
Dreossi, T., et al.: VerifAI: a toolkit for the formal design and analysis of artificial intelligence-based systems. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 432–442. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_25
Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: PLDI, pp. 63–78. ACM (2019)
Fremont, D.J., et al.: Scenic: a language for scenario specification and data generation (2020). https://arxiv.org/abs/1809.09310
Germain, P., Lacasse, A., Laviolette, F., Marchand, M., Roy, J.: Risk bounds for the majority vote: from a PAC-Bayesian analysis to a learning algorithm. CoRR abs/1503.08329 (2015). http://arxiv.org/abs/1503.08329
Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963). https://doi.org/10.1080/01621459.1963.10500830. https://www.tandfonline.com/doi/abs/10.1080/01621459.1963.10500830
The British Standards Institution: Operational design domain (ODD) taxonomy for an automated driving system (ADS) - specification. BSI PAS 1883 (2020)
Irvine, P., Zhang, X., Khastgir, S., Schwalb, E., Jennings, P.: A two-level abstraction ODDdefinition language: Part i*. In: 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2614–2621. IEEE Press (2021). https://doi.org/10.1109/SMC52423.2021.9658751
Jha, S., Sahai, T., Raman, V., Pinto, A., Francis, M.: Explaining AI decisions using efficient methods for learning sparse boolean formulae. J. Autom. Reason. 63(4), 1055–1075 (2019)
Khastgir, S., Birrell, S.A., Dhadyalla, G., Jennings, P.A.: Calibrating trust through knowledge: introducing the concept of informed safety for automation in vehicles. Transp. Res. Part C Emerg. Technol. 96, 290–303 (2018)
Khastgir, S., Brewerton, S., Thomas, J., Jennings, P.: Systems approach to creating test scenarios for automated driving systems. Reliab. Eng. Syst. Saf. 215, 107610 (2021). https://doi.org/10.1016/j.ress.2021.107610. https://www.sciencedirect.com/science/article/pii/S0951832021001551
McAllester, D.: Simplified PAC-Bayesian margin bounds. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT-Kernel 2003. LNCS (LNAI), vol. 2777, pp. 203–215. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-45167-9_16
Narodytska, N., Ignatiev, A., Pereira, F., Marques-Silva, J.: Learning optimal decision trees with SAT. In: Lang, J. (ed.) International Joint Conference on Artificial Intelligence, IJCAI 2018. ijcai.org (2018)
Pedreschi, D., Giannotti, F., Guidotti, R., Monreale, A., Ruggieri, S., Turini, F.: Meaningful explanations of black box AI decision systems. In: AAAI (2019)
Ribeiro, M.T., Singh, S., Guestrin, C.: “why should i trust you?”: explaining the predictions of any classifier. In: Knowledge Discovery and Data Mining, KDD 2016. Association for Computing Machinery (2016)
Russo, D.J., Van Roy, B., Kazerouni, A., Osband, I., Wen, Z.: A tutorial on thompson sampling. Found. Trends Mach. Learn. 11(1), 1–96 (2018). https://doi.org/10.1561/2200000070
SAE on-Road Automated Driving Committee and others: SAE J3016. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Technical report
Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, Cambridge (2014)
Thorn, E., Kimmel, S.C., Chaka, M.: A framework for automated driving system testable cases and scenarios (2018)
Torfah, H., Junges, S., Fremont, D.J., Seshia, S.A.: Formal analysis of AI-based autonomy: from modeling to runtime assurance. In: Feng, L., Fisman, D. (eds.) RV 2021. LNCS, vol. 12974, pp. 311–330. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88494-9_19
Torfah, H., Seshia, S.A.: Runtime monitors for operational design domains of black-box ML-models. In: NeurIPS ML Safety Workshop (2022). https://openreview.net/forum?id=6_AtjSBhqx
Torfah, H., Shah, S., Chakraborty, S., Akshay, S., Seshia, S.A.: Synthesizing pareto-optimal interpretations for black-box models. In: Proceedings of the IEEE International Conference on Formal Methods in Computer-Aided Design (FMCAD), pp. 153–162. IEEE (2021)
Torfah, H., Xie, C., Junges, S., Vazquez-Chanlatte, M., Seshia, S.A.: Learning monitorable operational design domains for assured autonomy. In: Bouajjani, A., Holík, L., Wu, Z. (eds.) ATVA 2022. LNCS, vol. 13505, pp. 3–22. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19992-9_1
Zhang, X., Khastgir, S., Jennings, P.: Scenario description language for automated driving systems: a two level abstraction approach. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 973–980 (2020). https://doi.org/10.1109/SMC42975.2020.9283417
Acknowledgments
This work is partially supported by NSF grants 1545126 (VeHICaL, including an NSF-TiH grant) and 1837132, by the DARPA contracts FA8750-18-C-0101 (AA) and FA8750-20-C-0156 (SDCPS), by Berkeley Deep Drive, by C3DTI, and by Toyota under the iCyPhy center. Financial support from TiH-IoT, IIT Bombay vide grant TIH-IOT/06-2022/IC/NSF/SL/NIUC-2022-05/001 under TiH-IoT US-India Collaborative Research Program 2022 is gratefully acknowledged. Funds from the latter grant were used to partially support Shetal Shah, Aniruddha Joshi, S. Akshay and Supratik Chakraborty for work reported in the current paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Torfah, H., Joshi, A., Shah, S., Akshay, S., Chakraborty, S., Seshia, S.A. (2023). Learning Monitor Ensembles for Operational Design Domains. In: Katsaros, P., Nenzi, L. (eds) Runtime Verification. RV 2023. Lecture Notes in Computer Science, vol 14245. Springer, Cham. https://doi.org/10.1007/978-3-031-44267-4_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-44267-4_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44266-7
Online ISBN: 978-3-031-44267-4
eBook Packages: Computer ScienceComputer Science (R0)