Towards Automated Test Scenario Generation for Assuring COLREGs Compliance of Autonomous Surface Vehicles
Authors: 
Ulf Kargén, Dániel VarróAuthors Info & Claims
Ulf Kargén, Dániel VarróAuthors Info & ClaimsPages 249 - 256
Abstract
International maritime traffic is controlled by collision-avoidance regulations (COLREGs) with 41 standardized rules describing how a vessel should navigate in the proximity of other vessels. Since some rules can be overridden by human judgement when resolving critical encounters of vessels, justifying COLREGs compliance has become a significant challenge in the increasing presence of autonomous surface vehicles (ASVs) operated without (or with only remote) human control. This paper provides a high-level framework and long-term research agenda towards the automated synthesis of test scenarios to assure COLREGs compliance for ASVs by exploiting various model-driven engineering techniques. By adapting ideas from testing self-driving cars, we envisage a multi-layered test scenario generation approach involving functional, logical and concrete scenarios. In the current paper, we demonstrate how functional scenarios of COLREGs situations between given vessels can be precisely formalized by using metamodels, domain-specific graph models and first-order logic graph constraints. By using automated model generation techniques, we derive a complete set of functional-level test scenarios, which includes all possible COLREGs situations that may arise between given vessels. As initial result, we provide several dangerous situations involving only three vessels where a potential collision may occur even when all vessels follow the COLREGs, which showcases that some COLREGs rules need further clarification for the safe regulation of ASVs.
References
[1]
2018. Xcore. https://wiki.eclipse.org/Xcore
[2]
Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2016. Testing advanced driver assistance systems using multi-objective search and neural networks. ASE 2016 - Proceedings of the 31st IEEE/ACM Int'l Conf. on Automated Software Engineering (2016), 63--74.
[3]
Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing vision-based control systems using learnable evolutionary algorithms. In Proceedings - Int'l Conf. on Software Engineering, Vol. 11. IEEE Computer Society, 1016--1026.
[4]
Raja Ben Abdessalem, Annibale Panichella, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing autonomous cars for feature interaction failures using many-objective search. In ASE 2018 - Proceedings of the 33rd ACM/IEEE Int'l Conf. on Automated Software Engineering, Vol. 18. ACM, Inc, New York, New York, USA, 143--154.
[5]
Aren A. Babikian, Oszkár Semeráth, Anqi Li, Kristóf Marussy, and Dániel Varró. 2022. Automated generation of consistent models using qualitative abstractions and exploration strategies. Softw. Syst. Model. 21, 5 (2022), 1763--1787.
[6]
Aren A. Babikian, Oszkár Semeráth, and Dániel Varró. 2024. Concretization of Abstract Traffic Scene Specifications Using Metaheuristic Search. IEEE Trans. Software Eng. 50, 1 (2024), 48--68.
[7]
Johannes Bach, Stefan Otten, and Eric Sax. 2016. Model based scenario specification for development and test of automated driving functions. In IEEE Intelligent Vehicles Symposium. IEEE Inc., 1149--1155.
[8]
Gerrit Bagschik, Till Menzel, and Markus Maurer. 2018. Ontology based Scene Creation for the Development of Automated Vehicles. In IEEE Intelligent Vehicles Symposium. IEEE, 1813--1820.
[9]
Ivan R. Bertaska, Brual Shah, Karl von Ellenrieder, Petr Švec, Wilhelm Klinger, Armando J. Sinisterra, Manhar Dhanak, and Satyandra K. Gupta. 2015. Experimental evaluation of automatically-generated behaviors for USV operations. Ocean Engineering 106 (2015), 496--514.
[10]
Victor Bolbot, Christos Gkerekos, Gerasimos Theotokatos, and Evangelos Boulougouris. 2022. Automatic traffic scenarios generation for autonomous ships collision avoidance system testing. Ocean Engineering 254 (2022).
[11]
Alessandro Calò, Paolo Arcaini, Shaukat Ali, Florian Hauer, and Fuyuki Ishikawa. 2020. Generating Avoidable Collision Scenarios for Testing Autonomous Driving Systems. In 2020 IEEE 13th Int. Conf. on Software Testing, Validation and Verification (ICST). 375--386.
[12]
Hao-Tien Lewis Chiang and Lydia Tapia. 2018. COLREG-RRT: An RRT-Based COLREGS-Compliant Motion Planner for Surface Vehicle Navigation. IEEE Robotics and Automation Letters 3, 3 (2018), 2024--2031.
[13]
A.N. Cockcroft and J.N.F. Lameijer. 2012. Part B- Steering and sailing rules. In A Guide to the Collision Avoidance Rules (Seventh Edition) (seventh edition ed.), A.N. Cockcroft and J.N.F. Lameijer (Eds.). Butterworth-Heinemann, Oxford, 11--104.
[14]
Daniel J. Fremont, Xiangyu Yue, Tommaso Dreossi, Alberto L. Sangiovanni-Vincentelli, Shromona Ghosh, and Sanjit A. Seshia. 2019. Scenic: A language for scenario specification and scene generation. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). ACM, 63--78. arXiv:2010.06580
[15]
Yahei Fujii and Kenichi Tanaka. 1971. Traffic Capacity. Journal of Navigation 24, 4 (1971), 543--552.
[16]
Fitash Ul Haq, Donghwan Shin, and Lionel Briand. 2022. Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization. Proceedings - International Conference on Software Engineering 2022-May (2022), 811--822.
[17]
Yixiong He, Yi Jin, Liwen Huang, Yong Xiong, Pengfei Chen, and Junmin Mou. 2017. Quantitative analysis of COLREG rules and seamanship for autonomous collision avoidance at open sea. Ocean Engineering 140 (2017), 281--291.
[18]
Thomas Helmer, Lei Wang, Klaus Kompass, and Ronald Kates. 2015. Safety Performance Assessment of Assisted and Automated Driving by Virtual Experiments: Stochastic Microscopic Traffic Simulation as Knowledge Synthesis. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems. 2019--2023.
[19]
Thomas Hempen, Sanjana Biank, Werner Huber, and Christian Diedrich. 2017. Model Based Generation of Driving Scenarios. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST 222 (2017), 153--163.
[20]
Helmut Hilgert and Michael Baldauf. 1997. A common risk model for the assessment of encounter situations on board ships. Ocean Dynamics 49, 4 (1997), 531--542.
[21]
Liang Hu, Wasif Naeem, Eshan Rajabally, Graham Watson, Terry Mills, Zakirul Bhuiyan, Craig Raeburn, Ivor Salter, and Claire Pekcan. 2020. A Multiobjective Optimization Approach for COLREGs-Compliant Path Planning of Autonomous Surface Vehicles Verified on Networked Bridge Simulators. IEEE Transactions on Intelligent Transportation Systems 21, 3 (2020), 1167--1179.
[22]
Tor Arne Johansen, Tristan Perez, and Andrea Cristofaro. 2016. Ship Collision Avoidance and COLREGS Compliance Using Simulation-Based Control Behavior Selection With Predictive Hazard Assessment. IEEE Transactions on Intelligent Transportation Systems 17, 12 (2016), 3407--3422.
[23]
Chijung Jung, Ali Ahad, Yuseok Jeon, and Yonghwi Kwon. 2022. SWARM-FLAWFINDER: Discovering and Exploiting Logic Flaws of Swarm Algorithms. In 2022 IEEE Symposium on Security and Privacy (SP). 1808--1825.
[24]
Hanna Krasowski and Matthias Althoff. 2021. Temporal Logic Formalization of Marine Traffic Rules. In 2021 IEEE Intelligent Vehicles Symposium (IV). 186--192.
[25]
Arne Kreutzmann, Diedrich Wolter, Frank Dylla, and Jae Hee Lee. 2013. Towards safe navigation by formalizing navigation rules. TransNav: International Journal on Marine Navigation and Safety of Sea Transportation 7, 2 (2013), 161--168.
[26]
D. K. M. Kufoalor, T. A. Johansen, E. F. Brekke, A. Hepsø, and K. Trnka. 2020. Autonomous maritime collision avoidance: Field verification of autonomous surface vehicle behavior in challenging scenarios. Journal of Field Robotics 37, 3 (2020), 387--403.
[27]
Yoshiaki Kuwata, Michael T. Wolf, Dimitri Zarzhitsky, and Terrance L. Huntsberger. 2014. Safe Maritime Autonomous Navigation With COLREGS, Using Velocity Obstacles. IEEE Journal of Oceanic Engineering 39, 1 (2014), 110--119.
[28]
Rupak Majumdar, Aman Mathur, Marcus Pirron, Laura Stegner, and Damien Zufferey. 2021. Paracosm: A Test Framework for Autonomous Driving Simulations. Springer, Cham, 172--195.
[29]
Kristóf Marussy, Attila Ficsor, Oszkár Semeráth, and Dániel Varró. 2024. Refinery: Graph solver as a service. In 46th Int. Conf. on Software Engineering, ICSE 2024: Tool Demonstration Track.
[30]
Kristóf Marussy, Oszkár Semeráth, Aren A. Babikian, and Dániel Varró. 2020. A Specification Language for Consistent Model Generation based on Partial Models. Journal of Object Technology 19, 3 (2020), 3:1--22. https://doi.org/10.5381/jot.2020.19.3.a12
[31]
Kristóf Marussy, Oszkár Semeráth, and Dániel Varró. 2022. Automated Generation of Consistent Graph Models with Multiplicity Reasoning. IEEE Transactions on Software Engineering 48 (2022), 1610--1629. Issue 5. https://doi.org/10.1109/TSE.2020.3025732
[32]
Till Menzel, Gerrit Bagschik, and Markus Maurer. 2018. Scenarios for Development, Test and Validation of Automated Vehicles. In IEEE Intelligent Vehicles Symposium, Proceedings, Vol. 2018-June. IEEE Inc., 1821--1827. arXiv:1801.08598
[33]
Galen E. Mullins, Paul G. Stankiewicz, R. Chad Hawthorne, and Satyandra K. Gupta. 2018. Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles. Journal of Systems and Software 137 (2018), 97--215.
[34]
Wasif Naeem, George W. Irwin, and Aolei Yang. 2012. COLREGs-based collision avoidance strategies for unmanned surface vehicles. Mechatronics 22, 6 (2012), 669--678.
[35]
Matthew O'Kelly, John Duchi, Aman Sinha, Hongseok Namkoong, and Russ Tedrake. 2018. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In Advances in Neural Information Processing Systems, Vol. 2018-Decem. Neural information processing systems foundation, 9827--9838. arXiv:1811.00145
[36]
International Maritime Organization. 2003. Convention on the International Regulations for Preventing Collisions at Sea, 1972. IMO.
[37]
Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2015. Reformulating branch coverage as a many-objective optimization problem. 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings (2015).
[38]
Rodrigo Queiroz, Thorsten Berger, and Krzysztof Czarnecki. 2019. GeoScenario: An open DSL for autonomous driving scenario representation. In IEEE Intelligent Vehicles Symposium. IEEE, 287--294.
[39]
Vincenzo Riccio and Paolo Tonella. 2020. Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing. ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering (2020), 876--888.
[40]
Elias Rocklage, Heiko Kraft, Abdullah Karatas, and Jorg Seewig. 2018. Automated scenario generation for regression testing of autonomous vehicles. In IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, Vol. 2018-March. IEEE Inc., 476--483.
[41]
Maike Scholtes, Lukas Westhofen, Lara Ruth Turner, Katrin Lotto, Michael Schuldes, Hendrik Weber, Nicolas Wagener, Christian Neurohr, Martin Herbert Bollmann, Franziska Kortke, Johannes Hiller, Michael Hoss, Julian Bock, and Lutz Eckstein. 2021. 6-Layer Model for a Structured Description and Categorization of Urban Traffic and Environment. IEEE Access 9 (2021), 59131--59147. arXiv:2012.06319
[42]
Fabian Schuldt, Andreas Reschka, and Markus Maurer. 2018. A method for an efficient, systematic test case generation for advanced driver assistance systems in virtual environments. In Automotive Systems Engineering II. Taylor and Francis, 147--175.
[43]
Barbara Schütt, Thilo Braun, Stefan Otten, and Eric Sax. 2020. SceML: A Graphical Modeling Framework for Scenario-Based Testing of Autonomous Vehicles. In 23rd ACM/IEEE Int'l Conf. on Model Driven Engineering Languages and Systems. ACM, 114--120.
[44]
Oszkár Semeráth, András Szabolcs Nagy, and Dániel Varró. 2018. A graph solver for the automated generation of consistent domain-specific models. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018. ACM, 969--980. https://doi.org/10.1145/3180155.3180186 CORE Rank: A*; Acceptance rate: 21%.
[45]
Paul G Stankiewicz and Galen E Mullins. 2019. Improving evaluation methodology for autonomous surface vessel COLREGSs compliance. In OCEANS 2019-Marseille. IEEE.
[46]
Markus Steimle, Till Menzel, and Markus Maurer. 2021. Toward a Consistent Taxonomy for Scenario-Based Development and Test Approaches for Automated Vehicles: A Proposal for a Structuring Framework, a Basic Vocabulary, and its Application. IEEE Access 9 (2021), 147828--147854. arXiv:2104.09097v3
[47]
Rafal Szlapczynski and Joanna Szlapczynska. 2017. Review of ship safety domains: Models and applications. Ocean Engineering 145 (2017), 277--289.
[48]
Simon Ulbrich, Till Menzel, Andreas Reschka, Fabian Schuldt, and Markus Maurer. 2015. Defining and Substantiating the Terms Scene, Situation, and Scenario for Automated Driving. In IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, Vol. 2015-Octob. IEEE Inc., 982--988.
[49]
Itziar Urbieta, Marcos Nieto, Mikel García, Oihana Otaegui, Miguel Clavijo, Felipe Jiménez, and Jose Eugenio Naranjo. 2021. Design and Implementation of an Ontology for Semantic Labeling and Testing: Automotive Global Ontology (AGO). Applied Sciences 11, 17 (2021), 7782.
[50]
Anete Vagale, Rachid Oucheikh, Robin T Bye, Ottar L Osen, and Thor I Fossen. 2021. Path planning and collision avoidance for autonomous surface vehicles I: a review. Journal of Marine Science and Technology 26, 4 (2021), 1292--1306.
[51]
Dániel Varró, Oszkár Semeráth, Gábor Szárnyas, and Ákos Horváth. 2018. Towards the Automated Generation of Consistent, Diverse, Scalable and Realistic Graph Models. In Graph Transformation, Specifications, and Nets - In Memory of Hartmut Ehrig. LNCS, Vol. 10800. Springer, 285--312. https://doi.org/10.1007/978-3-319-75396-6_16
[52]
Kyle Woerner, Michael R Benjamin, Michael Novitzky, and John J Leonard. 2019. Quantifying protocol evaluation for autonomous collision avoidance: Toward establishing COLREGS compliance metrics. Autonomous Robots 43 (2019), 967--991.
[53]
Kyle L. Woerner, Michael R. Benjamin, Michael Novitzky, and John J. Leonard. 2016. Collision avoidance road test for COLREGS-constrained autonomous vehicles. In OCEANS 2016 MTS/IEEE Monterey.
[54]
Siyu Wu, Hong Wang, Wenhao Yu, Kai Yang, Dongpu Cao, and Feiyue Wang. 2021. A new SOTIF scenario hierarchy and its critical test case generation based on potential risk assessment. IEEE 1st Int'l Conf. on Digital Twins and Parallel Intelligence, DTPI 2021 (2021), 399--409.
[55]
Songan Zhang, Huei Peng, Ding Zhao, and H. Eric Tseng. 2018. Accelerated Evaluation of Autonomous Vehicles in the Lane Change Scenario Based on Subset Simulation Technique. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). 3935--3940.
[56]
Luman Zhao and Myung-Il Roh. 2019. COLREGs-compliant multiship collision avoidance based on deep reinforcement learning. Ocean Engineering 191 (2019).
[57]
Feixiang Zhu, Zhengyu Zhou, and Hongrui Lu. 2022. Randomly Testing an Autonomous Collision Avoidance System with Real-World Ship Encounter Scenario from AIS Data. Journal of Marine Science and Engineering 10, 11 (2022).
Recommendations
Critical scenario identification for realistic testing of autonomous driving systems
AbstractAutonomous driving has become an important research area for road traffic, whereas testing of autonomous driving systems to ensure a safe and reliable operation remains an open challenge. Substantial real-world testing or massive driving data ...
DFS-KeyLevel: A Two-Layer Test Scenario Generation Approach for UML Activity Diagram
AbstractFor automatic generation of test scenarios from UML (Unified Modeling Language) activity diagrams (ADs) are very important for improving test efficiency. However, state-of-the-art approaches mainly focus on simple approaches, without specifically ...
Constrained permutation-based test scenario generation from concurrent activity diagrams
AbstractConcurrency in application systems can be designed and visualized using concurrent activity diagrams. Such diagrams are useful to design concurrency test scenarios for testing. However, the number of test scenarios inside a fork-join construct ...
Comments
Information & Contributors
Information
Published In
September 2024
311 pages
ISBN:9798400705045
DOI:10.1145/3640310
- General Chairs:
- Alexander Egyed,
- Manuel Wimmer,
- Program Chairs:
- Marsha Chechik,
- Benoit Combemale
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
- Johannes Kepler University, Linz, Austria
- SIGSOFT: ACM Special Interest Group on Software Engineering
- IEEE CS
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 September 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
MODELS '24
Sponsor:
MODELS '24: ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems
September 22 - 27, 2024
Linz, Austria
Acceptance Rates
MODELS '24 Paper Acceptance Rate 26 of 124 submissions, 21%;
Overall Acceptance Rate 144 of 506 submissions, 28%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 159Total Downloads
- Downloads (Last 12 months)159
- Downloads (Last 6 weeks)44
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in