Abstract
For reliable situation awareness in autonomous vehicle applications, we need to develop robust and reliable image processing and machine learning algorithms. Currently, there is no general framework for reasoning about the performance of perception systems. This paper introduces Timed Quality Temporal Logic (TQTL) as a formal language for monitoring and testing the performance of object detection and situation awareness algorithms for autonomous vehicle applications. We demonstrate that it is possible to describe interesting properties as TQTL formulas and detect cases where the properties are violated.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Autonomous vehicle disengagement reports 2016. https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2016
IEEE Connected Vehicles: Google reports self-driving car disengagements. http://sites.ieee.org/connected-vehicles/2015/12/15/google-reports-self-driving-car-disengagements
S-TaLiRo Toolbox. https://sites.google.com/a/asu.edu/s-taliro/s-taliro
Alur, R., Henzinger, T.A.: A really temporal logic. J. ACM 41(1), 181–204 (1994)
Annpureddy, Y., Liu, C., Fainekos, G., Sankaranarayanan, S.: S-TaLiRo: a tool for temporal logic falsification for hybrid systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 254–257. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19835-9_21
Bartocci, E., Deshmukh, J., Donzé, A., Fainekos, G., Maler, O., Ničković, D., Sankaranarayanan, S.: Specification-based monitoring of cyber-physical systems: a survey on theory, tools and applications. In: Bartocci, E., Falcone, Y. (eds.) Lectures on Runtime Verification - Introductory and Advanced Topics. LNCS, vol. 10457, pp. 135–175. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75632-5_5
Deshmukh, J.V., Majumdar, R., Prabhu, V.S.: Quantifying conformance using the skorokhod metric. In: 27th International Conference on Computer Aided Verification (CAV), pp. 234–250 (2015)
Dokhanchi, A., Hoxha, B., Tuncali, C.E., Fainekos, G.: An efficient algorithm for monitoring practical TPTL specifications. In: The ACM/IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE), pp. 184–193 (2016)
Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Systematic testing of convolutional neural networks for autonomous driving. In: ICML Workshop on Reliable Machine Learning in the Wild (RMLW) (2017)
Fainekos, G., Pappas, G.J.: Robustness of temporal logic specifications for continuous-time signals. Theor. Comput. Sci. 410(42), 4262–4291 (2009)
Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Robot. Res. (IJRR) 32, 1229–1235 (2013)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org
Maler, O., Nickovic, D.: Monitoring temporal properties of continuous signals. In: Lakhnech, Y., Yovine, S. (eds.) FORMATS/FTRTFT -2004. LNCS, vol. 3253, pp. 152–166. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30206-3_12
Nguyen, L.V., Kapinski, J., Jin, X., Deshmukh, J.V., Johnson, T.T.: Hyperproperties of real-valued signals. In: The ACM-IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE), pp. 104–113 (2017)
Tuncali, C.E., Fainekos, G., Ito, H., Kapinski, J.: Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: IEEE Intelligent Vehicles Symposium (IV) (2018)
Wu, B., Iandola, F., Jin, P.H., Keutzer, K.: Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving (2016)
Xu, H., Gao, Y., Yu, F., Darrell, T.: End-to-end learning of driving models from large-scale video datasets. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017). https://doi.org/10.1109/CVPR.2017.376
Acknowledgements
This work was partially supported by the NSF I/UCRC Center for Embedded Systems and by NSF grants 1350420, 1361926 and 1446730.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Dokhanchi, A., Amor, H.B., Deshmukh, J.V., Fainekos, G. (2018). Evaluating Perception Systems for Autonomous Vehicles Using Quality Temporal Logic. In: Colombo, C., Leucker, M. (eds) Runtime Verification. RV 2018. Lecture Notes in Computer Science(), vol 11237. Springer, Cham. https://doi.org/10.1007/978-3-030-03769-7_23
Download citation
DOI: https://doi.org/10.1007/978-3-030-03769-7_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03768-0
Online ISBN: 978-3-030-03769-7
eBook Packages: Computer ScienceComputer Science (R0)