skip to main content
research-article

Test Specification and Generation for Connected and Autonomous Vehicle in Virtual Environments

Published: 02 November 2019 Publication History

Abstract

The trend of connected/autonomous features adds significant complexity to the traditional automotive systems to improve driving safety and comfort. Engineers are facing significant challenges in designing test environments that are more complex than ever. We propose a test framework that allows one to automatically generate various virtual road environments from the path and behavior specifications. The path specification intends to characterize geometric paths that an environmental object (e.g., a roadway or a pedestrian) needs to be visualized or move over. We characterize this aspect in the form of constraints of 3-Dimensional (3D) coordinates. Then, we introduce a test coverage, called an area coverage, to quantify the quality of the generated paths in terms of how diverse of an area the generated paths can cover. We propose an algorithm that automatically generates such paths using an SMT (Satisfiability Modulo Theories) solver. However, the behavioral specification intends to characterize how an environmental object changes its mode over time by interacting with other objects (e.g., a pedestrian waits for a signal or starts crossing). We characterize this aspect in the form of timed automata. Then, we introduce a test coverage, called an edge/location coverage, to quantify the quality of the generated mode changes in terms of how many modes or transitions are visited. We propose a method that automatically generates many different mode changes using a model-checking. To demonstrate the test framework, we developed the right-turn pedestrian warning system in intersection scenarios and generated many different types of pedestrian paths and behaviors to analyze the effectiveness of the system.

References

[1]
Rajeev Alur and David L. Dill. 1994. A theory of timed automata. Theor. Comput. Sci. 126, 2 (1994), 183--235.
[2]
ristian Cadar, Patrice Godefroid, Sarfraz Khurshid, Corina S. Păsăreanu, Koushik Sen, Nikolai Tillmann, and Willem Visser. 2011. Symbolic execution for software testing in practice: Preliminary assessment. In Proceedings of the 33rd International Conference on Software Engineering (ICSE’11).
[3]
A. En-Nouaary, R. Dssouli, and F. Khendek. 2002. Timed Wp-method: Testing real-time systems. IEEE Trans. Softw. Eng. 28, 11 (Nov. 2002), 1023--1038.
[4]
Eric Galin, Adrien Peytavie, Nicolas Maréchal, and Eric Guérin. 2010. Procedural generation of roads. Comput. Graph. For. 29, 2 (2010), 429--438.
[5]
Patrice Godefroid, Michael Y. Levin, and David Molnar. 2012. SAGE: Whitebox fuzzing for security testing. Queue 10, 1, Article 20 (Jan. 2012), 8 pages.
[6]
Anders Hessel, Kim G. Larsen, Marius Mikucionis, Brian Nielsen, Paul Pettersson, and Arne Skou. 2008. Testing real-time systems using UPPAAL. Formal Methods and Testing. Springer-Verlag, Berlin, 77--117. Retrieved from: http://dl.acm.org/citation.cfm?id=1806209.1806212.
[7]
Texas Transportation Institute and Southwest Region University Transportation Center (U.S.). 2011. Automated Generation of Virtual Scenarios in Driving Simulator from Highway Design Data. https://rosap.ntl.bts.gov/view/dot/23113.
[8]
ISO15622. 2018. Intelligent transport systems—Adaptive Cruise Control systems—Performance requirements and test procedures. https://www.iso.org/standard/71515.html.
[9]
ISO17387. 2008. Intelligent transport systems—Lane change decision aid systems (LCDAS)—Performance requirements and test procedures. https://www.iso.org/standard/43654.html.
[10]
ISO19237. 2017. Intelligent transport systems—Pedestrian detection and collision mitigation systems (PDCMS)—Performance requirements and test procedures. https://www.iso.org/standard/64111.html.
[11]
Song Jian and Jin Ruichen. 1999. Generation of virtual road surfaces and simulation of nonlinear vibration of vehicles. In Proceedings of the Vehicle Electronics Conference (IVEC’99).
[12]
Felipe Jiménez, Francisco Aparicio, and Gonzalo Estrada. 2009. Measurement uncertainty determination and curve-fitting algorithms for development of accurate digital maps for advanced driver assistance systems. Transport. Res. Part C: Emerg. Technol. 17, 3 (2009), 225--239.
[13]
R. J. Kavanagh and R. Ramanathan. 1982. Computer simulation of road surface profiles for a four-wheeled vehicle. In Proceedings of the 14th Conference on Winter Simulation—Volume 1.
[14]
B. Kim, A. Jarandikar, J. Shum, S. Shiraishi, and M. Yamaura. 2016. The SMT-based automatic road network generation in vehicle simulation environment. In Proceedings of the International Conference on Embedded Software (EMSOFT’16). 1--10.
[15]
B. Kim, Y. Kashiba, S. Dai, and S. Shiraishi. 2017. Testing autonomous vehicle software in the virtual prototyping environment. IEEE Embed. Syst. Lett. 9, 1 (Mar. 2017), 5--8.
[16]
Moez Krichen and Stavros Tripakis. 2004. Black-box conformance testing for real-time systems. In Proceedings of the 11th International SPIN Workshop: Model Checking Software. Springer Berlin, 109--126.
[17]
Kim G. Larsen, Marius Mikucionis, and Brian Nielsen. 2005. Online testing of real-time systems using Uppaal. In Proceedings of the 4th International Workshop on Formal Approaches to Software Testing (FATES’04). Springer Berlin, 79--94.
[18]
Kim G. Larsen, Paul Pettersson, and Wang Yi. 1997. UPPAAL in a nutshell. Int. J. Softw. Tools Technol. Trans. 1, 1 (1997), 134--152.
[19]
Kim G. Larsen and Wang Yi. 1994. Time Abstracted Bisimulation: Implicit Specifications and Decidability. Springer Berlin, 160--176.
[20]
Guodong Li, Peng Li, Geof Sawaya, Ganesh Gopalakrishnan, Indradeep Ghosh, and Sreeranga P. Rajan. 2012. GKLEE: Concolic verification and test generation for GPUs. SIGPLAN Not. 47, 8 (Feb. 2012), 215--224.
[21]
A. Madrigal. 2017. Inside Waymo’s secret world for training self-driving cars. The Atlantic, Aug. 23 (2017). Retrieved from: https://www.theatlantic.com/technology/archive/2017/08/inside-waymos-secret-testing-and-simulation-facilities/537648/.
[22]
Euro NCAP. 2017. European New Car Assessment Programme (Euro NCAP)—Test Protocol—AEB VRU systems. Technical Report.
[23]
Federal Highway Administration Office. 2009. Pedestrian Safety at Intersections. Technical Report.
[24]
Corina S. Păsăreanu and Willem Visser. 2009. A survey of new trends in symbolic execution for software testing and analysis. Int. J. Softw. Tools Technol. Trans. 11, 4 (2009), 339--353.
[25]
Jan Peleska, Elena Vorobev, and Florian Lapschies. 2011. Automated test case generation with SMT-solving and abstract interpretation. In Proceedings of the NASA Formal Methods Symposium (NFM’11).
[26]
Stefan Ratschan. 2006. Efficient solving of quantified inequality constraints over the real numbers. ACM Trans. Comput. Log. 7, 4 (2006), 723--748.
[27]
Mikio Yanagisawa, Elizabeth D. Swanson, Philip Azeredo, and Wassim Najm. 2017. Estimation of Potential Safety Benefits for Pedestrian Crash Avoidance/Mitigation Systems (Report No. DOT HS 812 400). Technical Report. John A. Volpe National Transportation Systems Center U.S. Department of Transportation. https://rosap.ntl.bts.gov/view/dot/12475/dot_12475_DS1.pdf.
[28]
S. Zhao and J. A. Farrell. 2011. Optimization-based road curve fitting. In Proceedings of the 50th IEEE Conference on Decision and Control and European Control Conference.

Cited By

View all
  • (2023)Toward Large-Scale Test for Certifying Autonomous Driving Software in Collaborative Virtual EnvironmentIEEE Access10.1109/ACCESS.2023.329550011(72641-72654)Online publication date: 2023
  • (2022)Generating Critical Test Scenarios for Autonomous Driving Systems via Influential Behavior PatternsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3560430(1-12)Online publication date: 10-Oct-2022
  • (2022)MOSAT: finding safety violations of autonomous driving systems using multi-objective genetic algorithmProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549100(94-106)Online publication date: 7-Nov-2022
  • Show More Cited By

Index Terms

  1. Test Specification and Generation for Connected and Autonomous Vehicle in Virtual Environments

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Cyber-Physical Systems
    ACM Transactions on Cyber-Physical Systems  Volume 4, Issue 1
    Special Issue on Transportation CPS
    January 2020
    311 pages
    ISSN:2378-962X
    EISSN:2378-9638
    DOI:10.1145/3371149
    • Editor:
    • Tei-Wei Kuo
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Journal Family

    Publication History

    Published: 02 November 2019
    Accepted: 01 February 2019
    Revised: 01 August 2018
    Received: 01 September 2017
    Published in TCPS Volume 4, Issue 1

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. SMT solver
    2. Test specification
    3. autonomous vehicle
    4. connected vehicle
    5. system safety
    6. test coverage
    7. test generation
    8. timed automata
    9. virtual prototyping

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)35
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Toward Large-Scale Test for Certifying Autonomous Driving Software in Collaborative Virtual EnvironmentIEEE Access10.1109/ACCESS.2023.329550011(72641-72654)Online publication date: 2023
    • (2022)Generating Critical Test Scenarios for Autonomous Driving Systems via Influential Behavior PatternsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3560430(1-12)Online publication date: 10-Oct-2022
    • (2022)MOSAT: finding safety violations of autonomous driving systems using multi-objective genetic algorithmProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549100(94-106)Online publication date: 7-Nov-2022
    • (2020)Survey on Scenario-Based Safety Assessment of Automated VehiclesIEEE Access10.1109/ACCESS.2020.29937308(87456-87477)Online publication date: 2020

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media