skip to main content
research-article
Public Access

A Systematic Framework to Identify Violations of Scenario-dependent Driving Rules in Autonomous Vehicle Software

Published: 04 June 2021 Publication History

Abstract

Safety compliance is paramount to the safe deployment of autonomous vehicle (AV) technologies in real-world transportation systems. As AVs will share road infrastructures with human drivers and pedestrians, it is an important requirement for AVs to obey standard driving rules. Existing AV software testing methods, including simulation and road testing, only check fundamental safety rules such as collision avoidance and safety distance. Scenario-dependent driving rules, including crosswalk and intersection rules, are more complicated because the expected driving behavior heavily depends on the surrounding circumstances. However, a testing framework is missing for checking scenario-dependent driving rules on various AV software.
In this paper, we design and implement a systematic framework AVChecker for identifying violations of scenario-dependent driving rules in AV software using formal methods. AVChecker represents both the code logic of AV software and driving rules in proposed formal specifications and leverages satisfiability modulo theory (SMT) solvers to identify driving rule violations. To improve the automation of systematic rule-based checking, AVChecker provides a powerful user interface for writing driving rule specifications and applies static code analysis to extract rule-related code logic from the AV software codebase. Evaluations on two open-source AV software platforms, Baidu Apollo and Autoware, uncover 19 true violations out of 28 real-world driving rules covering crosswalks, traffic lights, stop signs, and intersections. Seven of the violations can lead to severe risks of a collision with pedestrians or blocking traffic.

References

[1]
2017. Automated Driving Systems 2.0: A Vision for Safety. https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/13069a-ads2.0_090617_v9a_tag.pdf.
[2]
2019. A Matter of Trust Ford's Approach to Developing Self-driving Vehicles. https://media.ford.com/content/dam/fordmedia/pdf/Ford_AV_LLC_FINAL_HR_2.pdf.
[3]
2019. Apollo Auto: An open autonomous driving platform. https://github.com/ApolloAuto/apollo.
[4]
2019. General Motors 2018 Self-Driving Safety Report. https://www.gm.com/content/dam/company/docs/us/en/gmcom/gmsafetyreport.pdf .
[5]
2020. 2010 Georgia Code Title 40 - Motor Vehicles and Traffic. https://law.justia.com/codes/georgia/2010/title-40/chapter-6/article-5/40--6--91.
[6]
2020. Autoware: Open-source software for self-driving vehicles. https://gitlab.com/autowarefoundation/autoware.ai.
[7]
2020. Baidu Apollo's fuzzing support. https://github.com/ApolloAuto/apollo/commit/7aca63966211ceada44981d96b35a1252f9f1729.
[8]
2020. Louisiana DMV Handbook. https://driving-tests.org/wp-content/uploads/2018/03/LA_Guide-2017.pdf .
[9]
2020. New York State DMV: Driver's Manual and Practice Tests. https://dmv.ny.gov/driver-license/drivers-manual-practice-tests.
[10]
2020. ROS.org | Powering the world's robots. https://www.ros.org/.
[11]
2020. State of California DMV: California Driver Handbook. https://www.dmv.ca.gov/web/eng_pdf/dl600.pdf.
[12]
2020. State of Michigan DMV: What Every Driver Must Know. https://www.michigan.gov/documents/wedmk_16312_7.pdf.
[13]
2020. Waymo Safety Report. https://waymo.com/safety.
[14]
Matthias Althoff and John M Dolan. 2011. Set-based computation of vehicle behaviors for the online verification of autonomous vehicles. In 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 1162--1167.
[15]
Matthias Althoff and Sebastian Lutz. 2018. Automatic generation of safety-critical test scenarios for collision avoidance of road vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1326--1333.
[16]
Matthias Althoff, Olaf Stursberg, and Martin Buss. 2009. Model-based probabilistic collision detection in autonomous driving. IEEE Transactions on Intelligent Transportation Systems10, 2 (2009), 299--310.
[17]
Alexander Amini, Igor Gilitschenski, Jacob Phillips, Julia Moseyko, Rohan Banerjee, Sertac Karaman, and Daniela Rus. 2020. Learning Robust Control Policies for End-to-End Autonomous Driving From Data-Driven Simulation. IEEE Robotics and Automation Letters 5, 2 (2020), 1143--1150.
[18]
Mathieu Barbier, Alessandro Renzaglia, Jean Quilbeuf, Lukas Rummelhard, Anshul Paigwar, Christian Laugier, Axel Legay, Javier Ibañez-Guzmán, and Olivier Simonin. 2019. Validation of perception and decision-making systems for autonomous driving via statistical model checking. In 2019 IEEE Intelligent Vehicles Symposium (IV). IEEE, 252--259.
[19]
Dirk Beyer and Matthias Dangl. 2016. SMT-based software model checking: an experimental comparison of four algorithms. In Working Conference on Verified Software: Theories, Tools, and Experiments. Springer, 181--198.
[20]
Dirk Beyer, Matthias Dangl, and Philipp Wendler. 2018. A unifying view on SMT-based software verification. Journal of automated reasoning60, 3 (2018), 299--335.
[21]
Nikolaj Bjørner. 2010. Linear quantifier elimination as an abstract decision procedure. In International Joint Conference on Automated Reasoning. Springer, 316--330.
[22]
NTS Board. 2018. Preliminary report, highway, hwy18mh010.National Transpotation Safety Board, https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf, accessed(2018), 11--15.
[23]
Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In 8th USENIX Symposium on Operating Systems Design and Implementation(OSDI 08). 209--224.
[24]
Silviu S Craciunas and Ramon Serna Oliver. 2014. SMT-based task-and network-level static schedule generation for time-triggered networked systems. In Proceedings of the 22nd international conference on real-time networks and systems.45--54.
[25]
Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Proceedings of the 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems.
[26]
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. arXiv preprint arXiv:1711.03938(2017).
[27]
Hassan Eldib, Chao Wang, and Patrick Schaumont. 2014. SMT-based verification of software countermeasures against side-channel attacks. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 62--77.
[28]
Haoyang Fan, Fan Zhu, Changchun Liu, Liangliang Zhang, Li Zhuang, Dong Li, Weicheng Zhu, Jiangtao Hu, Hongye Li, and Qi Kong. 2018. Baidu apollo em motion planner.arXiv preprint arXiv:1807.08048(2018).
[29]
Daniel S Fowler, Jeremy Bryans, Siraj Ahmed Shaikh, and Paul Wooderson. 2018. Fuzz testing for automotive cyber-security. In 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops(DSN-W). IEEE, 239--246.
[30]
David Ke Hong, Qi Alfred Chen, and Z. Morley Mao. 2017. An Initial Investigation of Protocol Customization. In Proceedings of the 2017 Workshop on Forming an Ecosystem Around Software Transformation.
[31]
David Ke Hong, John Kloosterman, Yuqi Jin, Yulong Cao, Qi Alfred Chen, Scott Mahlke, and Z Morley Mao. 2020. AVGuardian: Detecting and Mitigating Publish-Subscribe Overprivilege for Autonomous Vehicle Systems. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P). 445--459.
[32]
Ke Hong. 2019. Performance, Security, and Safety Requirements Testing for Smart Systems Through Systematic Software Analysis. Ph.D. Dissertation. University of Michigan, Ann Arbor, MI.
[33]
Eunkyoung Jee, Donghwan Shin, Sungdeok Cha, Jang-Soo Lee, and Doo-Hwan Bae. 2014. Automated test case generation for FBD programs implementing reactor protection system software.Software Testing, Verification andReliability24, 8 (2014), 608--628.
[34]
Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solverfor verifying deep neural networks. In International Conference on Computer Aided Verification. Springer, 97--117.
[35]
BaekGyu Kim, Akshay Jarandikar, Jonathan Shum, Shinichi Shiraishi, and Masahiro Yamaura. 2016. The SMT-based automatic road network generation in vehicle simulation environment. In2016 International Conference on Embedded Software (EMSOFT). IEEE, 1--10.
[36]
Philip Koopman and Michael Wagner. 2016. Challenges in autonomous vehicle testing and validation. SAE International Journal of Transportation Safety(2016).
[37]
Philip Koopman and Michael Wagner. 2017. Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intelligent Transportation Systems Magazine(2017).
[38]
Chris Lattner and Vikram Adve. 2004. LLVM: A compilation framework for lifelong program analysis & transformation. In International Symposium on Code Generation and Optimization, 2004. CGO 2004. IEEE, 75--86.
[39]
John Leonard, Jonathan How, Seth Teller, Mitch Berger, Stefan Campbell, Gaston Fiore, Luke Fletcher, Emilio Frazzoli, Albert Huang, Sertac Karaman, Olivier Koch, Yoshiaki Kuwata, David Moore, Edwin Olson, Steve Peters, Justin Teo,Robert Truax, Matthew Walter, David Barrett, Alexander Epstein, Keoni Maheloni, Katy Moyer, Troy Jones, Ryan Buckley, Matthew Antone, Robert Galejs, Siddhartha Krishnamurthy, and Jonathan Williams. 2008. A Perception-Driven Autonomous Urban Vehicle. Journal of Field Robotic(2008).
[40]
Guodong Li and Ganesh Gopalakrishnan. 2010. Scalable SMT-based verification of GPU kernel functions. In Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering. 187--196.
[41]
Guanpeng Li, Yiran Li, Saurabh Jha, Timothy Tsai, Siva Hari, Michael Sullivan, Zbigniew T. Kalbarczyk, and Ravis-hankar K. Iyer. 2020. AV-FUZZER: Finding safety violations in autonomous driving systems. ISSRE'20, Proceedings of the IEEE International Conference on Software Reliability Engineering(Nov 2020).
[42]
Nan Li, Dave W Oyler, Mengxuan Zhang, Yildiray Yildiz, Ilya Kolmanovsky, and Anouck R Girard. 2017. Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems. IEEE Transactions on control systems technology 26, 5 (2017), 1782--1797.
[43]
Satoshi Masuda, Hiroaki Nakamura, and Kohichi Kajitani. 2018. Rule-based searching for collision test cases of autonomous vehicles simulation. IET Intelligent Transport Systems 12, 9 (2018), 1088--1095.
[44]
M. Montemerlo, J. Becker, S. Bhat, H. Dahlkamp, D. Dolgov, S. Ettinger, D. Haehnel, T. Hilden, G. Hoffmann, B.Huhnke, D. Johnston, S. Klumpp, D. Langer, A. Levandowski, J. Levinson, J. Marcil, D. Orenstein, J. Paefgen, I. Penny,A. Petrovskaya, M. Pflueger, G. Stanek, D. Stavens, A. Vogt, and S. Thrun. 2008. Junior: The Stanford Entry in the Urban Challenge. Journal of Field Robotics, Special Issue on the 2007 DARPA Urban Challenge, Part II(2008).
[45]
Luke Nelson, Helgi Sigurbjarnarson, Kaiyuan Zhang, Dylan Johnson, James Bornholt, Emina Torlak, and Xi Wang.2017. Hyperkernel: Push-button verification of an OS kernel. In Proceedings of the 26th Symposium on Operating Systems Principles. 252--269.
[46]
Andres Nötzli, Jehandad Khan, Andy Fingerhut, Clark Barrett, and Peter Athanas. 2018. P4pktgen: Automated testcase generation for P4 programs. In Proceedings of the Symposium on SDN Research. 1--7.
[47]
Matthew O'Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, and John C Duchi. 2018. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In Advances in Neural Information Processing Systems. 9827--9838.
[48]
Anshul Paigwar, Eduard Baranov, Alessandro Renzaglia, Christian Laugier, and Axel Legay. 2020. Probabilistic Collision Risk Estimation for Autonomous Driving: Validation via Statistical Model Checking. In 31st IEEE Intelligent Vehicles Symposium.
[49]
Christian Pek, Peter Zahn, and Matthias Althoff. 2017. Verifying the safety of lane change maneuvers of self-driving vehicles based on formalized traffic rules. In 2017 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1477--1483.
[50]
Michael Pradel, Ciera Jaspan, Jonathan Aldrich, and Thomas R Gross. 2012. Statically checking API protocol conformance with mined multi-object specifications. In 2012 34th International Conference on Software Engineering (ICSE). IEEE, 925--935.
[51]
Albert Rizaldi, Fabian Immler, and Matthias Althoff. 2016. A formally verified checker of the safe distance traffic rulesfor autonomous vehicles. In NASA Formal Methods Symposium. Springer, 175--190.
[52]
Guodong Rong, Byung Hyun Shin, Hadi Tabatabaee, Qiang Lu, Steve Lemke, Martins Mozeiko, Eric Boise, Geehoon Uhm, Mark Gerow, Shalin Mehta, et al.[n.d.]. LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving.([n. d.]).
[53]
Tao B Schardl, Tyler Denniston, Damon Doucet, Bradley C Kuszmaul, I-Ting Angelina Lee, and Charles E Leiserson. 2017. The CSI framework for compiler-inserted program instrumentation. Proceedings of the ACM on Measurement and Analysis of Computing Systems 1, 2 (2017), 1--25.
[54]
Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. 2017. On a Formal Model of Safe and Scalable Self-driving Cars. CoRR(2017).
[55]
Helgi Sigurbjarnarson, James Bornholt, Emina Torlak, and Xi Wang. 2016. Push-Button Verification of File Systems via Crash Refinement. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16).
[56]
Yulei Sui and Jingling Xue. 2016. SVF: interprocedural static value-flow analysis in LLVM. In Proceedings of the 25th international conference on compiler construction. 265--266.
[57]
Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th international conference on software engineering. 303--314.
[58]
Chris Urmson, Joshua Anhalt, Drew Bagnell, Christopher Baker, Robert Bittner, M. N. Clark, John Dolan, Dave Duggins, Tugrul Galatali, Chris Geyer, Michele Gittleman, Sam Harbaugh, Martial Hebert, Thomas M. Howard, Sascha Kolski, Alonzo Kelly, Maxim Likhachev, Matt McNaughton, Nick Miller, Kevin Peterson, Brian Pilnick, Raj Rajkumar, Paul Rybski, Bryan Salesky, Young-Woo Seo, Sanjiv Singh, Jarrod Snider, Anthony Stentz, William "Red" Whittaker, Ziv Wolkowicki, Jason Ziglar, Hong Bae, Thomas Brown, Daniel Demitrish, Bakhtiar Litkouhi, Jim Nickolaou, Varsha Sadekar, Wende Zhang, Joshua Struble, Michael Taylor, Michael Darms, and Dave Ferguson. 2008. Autonomous Driving in Urban Environments: Boss and the Urban Challenge. Journal of Field Robotics(2008).
[59]
Niki Vazou, Anish Tondwalkar, Vikraman Choudhury, Ryan G Scott, Ryan R Newton, Philip Wadler, and Ranjit Jhala.2017. Refinement reflection: complete verification with SMT. Proceedings of the ACM on Programming Languages 2, POPL (2017), 1--31.
[60]
J. Wei, J. M. Snider, T. Gu, J. M. Dolan, and B. Litkouhi. 2014. A behavioral planning framework for autonomous driving. In 2014 IEEE Intelligent Vehicles Symposium Proceedings.
[61]
Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 132--142.
[62]
Marc René Zofka, Sebastian Klemm, Florian Kuhnt, Thomas Schamm, and J Marius Zöllner. 2016. Testing and validating high level components for automated driving: simulation framework for traffic scenarios. In 2016 IEEE Intelligent Vehicles Symposium (IV). IEEE, 144--150.

Cited By

View all
  • (2025)Machine-readable specification and intelligent cloud-based execution of logical test cases for automated driving functionsAutomated Software Engineering10.1007/s10515-024-00481-632:1Online publication date: 1-May-2025
  • (2024)Network Fairness Ambivalence: When does social network capital mitigate or amplify unfairness?Proceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36560178:2(1-28)Online publication date: 29-May-2024
  • (2024)Identifying Cross-User Privacy Leakage in Mobile Mini-Apps at a Large ScaleIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.335619719(3135-3147)Online publication date: 1-Jan-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Measurement and Analysis of Computing Systems
Proceedings of the ACM on Measurement and Analysis of Computing Systems  Volume 5, Issue 2
POMACS
June 2021
424 pages
EISSN:2476-1249
DOI:10.1145/3469656
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 June 2021
Published in POMACS Volume 5, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. autonomous vehicle
  2. formal methods
  3. software system

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)463
  • Downloads (Last 6 weeks)51
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Machine-readable specification and intelligent cloud-based execution of logical test cases for automated driving functionsAutomated Software Engineering10.1007/s10515-024-00481-632:1Online publication date: 1-May-2025
  • (2024)Network Fairness Ambivalence: When does social network capital mitigate or amplify unfairness?Proceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36560178:2(1-28)Online publication date: 29-May-2024
  • (2024)Identifying Cross-User Privacy Leakage in Mobile Mini-Apps at a Large ScaleIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.335619719(3135-3147)Online publication date: 1-Jan-2024
  • (2024)CommonRoad-CARLA Interface: Bridging the Gap between Motion Planning and 3D Simulation2024 IEEE Intelligent Vehicles Symposium (IV)10.1109/IV55156.2024.10588815(2702-2708)Online publication date: 2-Jun-2024
  • (2024)TR2MTL: LLM based framework for Metric Temporal Logic Formalization of Traffic Rules2024 IEEE Intelligent Vehicles Symposium (IV)10.1109/IV55156.2024.10588650(1206-1213)Online publication date: 2-Jun-2024
  • (2024)Responsible automatically processable regulationAI & SOCIETY10.1007/s00146-024-01901-7Online publication date: 28-Mar-2024
  • (2023)TrustedDomain Compromise Attack in App-in-app EcosystemsProceedings of the 2023 ACM Workshop on Secure and Trustworthy Superapps10.1145/3605762.3624430(51-57)Online publication date: 26-Nov-2023
  • (2023)MUID: Detecting Sensitive User Inputs in Miniapp EcosystemsProceedings of the 2023 ACM Workshop on Secure and Trustworthy Superapps10.1145/3605762.3624429(17-21)Online publication date: 26-Nov-2023
  • (2023)Towards a Better Super-App Architecture from a Browser Security PerspectiveProceedings of the 2023 ACM Workshop on Secure and Trustworthy Superapps10.1145/3605762.3624427(23-28)Online publication date: 26-Nov-2023
  • (2023)A Survey on Automated Driving System Testing: Landscapes and TrendsACM Transactions on Software Engineering and Methodology10.1145/357964232:5(1-62)Online publication date: 24-Jul-2023
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media