skip to main content
research-article

Trustworthy Autonomous System Development

Published: 18 April 2023 Publication History

Abstract

Autonomous systems emerge from the need to progressively replace human operators by autonomous agents in a wide variety of application areas. We offer an analysis of the state of the art in developing autonomous systems, focusing on design and validation and showing that the multi-faceted challenges involved go well beyond the limits of weak AI. We argue that traditional model-based techniques are defeated by the complexity of the problem, while solutions based on end-to-end machine learning fail to provide the necessary trustworthiness. We advocate a hybrid design approach, which combines the two, adopting the best of each, and seeks tradeoffs between trustworthiness and performance. We claim that traditional risk analysis and mitigation techniques fail to scale and discuss the trend of moving away from correctness at design time and toward reliance on runtime assurance techniques. We argue that simulation and testing remain the only realistic approach for global validation and show how current methods can be adapted to autonomous systems. We conclude by discussing the factors that will play a decisive role in the acceptance of autonomous systems and by highlighting the urgent need for new theoretical foundations.

References

[1]
David Harel, Assaf Marron, and Joseph Sifakis. 2022. Creating a foundation for next-generation autonomous systems. IEEE Des. Test 39, 1 (2022), 49–56.
[2]
D. Harel, Assaf Marron, and J. Sifakis. 2020. Autonomics: In search of a foundation for next generation autonomous systems. Proc. Natl. Acad. Sci. U.S.A. 117, 30 (2020), 17491–17498.
[3]
Autonomic Computing. 2006. An architectural blueprint for autonomic computing. IBM White Paper 31, (2006), 1–6. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=0e99837d9b1e70bb35d516e32ecfc345cd30e795.
[4]
Michael I. Jordan. Artificial Intelligence—The Revolution Hasn't Happened Yet. Retrieved from https://hdsr.mitpress.mit.edu/pub/wot7mkc1/release/9.
[5]
David Harel and Amir Pnueli. 1985. On the Eevelopment of Reactive Systems, Logics and Models of Concurrent Systems (K. R. Apt, Ed.). NATO ASI Series, F-13, Springer-Verlag, New York, 477–498.
[6]
Joseph Sifakis. 2018. Autonomous systems an architectural characterization. arXiv:1811.10277. Retrieved from https://arxiv.org/abs/1811.10277.
[7]
S. Efroni, D. Harel, and I. R. Cohen. 2005. Reactive animation: Realistic modeling of complex dynamic systems. Computer 38, 1, (2005), 38–47. DOI:
[8]
Simon Bliudze, Sébastien Furic, Joseph Sifakis, and Antoine Viel. 2019. Antoine viel: Rigorous design of cyber-physical systems—Linking physicality and computation. Softw. Syst. Model 18, 3 (2019), 1613–1636.
[9]
R. Bloem, S. Jacobs, A. Khalimov, I. Konnov, S. Rubin, H. Veith, and J. Widder. 2015. Decidability of Parameterized Verification, Synthesis, Lectures on Distributed Computing Theory. Morgan & Claypool.
[10]
J. Sifakis. 2012. Rigorous system design. Foundations and Trends in Electronic Design Automation 6, 4 (2012), 293–362.
[11]
ISO Online Browsing Platform. Road vehicles— Functional safety— Part9: Automotive Safety Integrity Level (ASIL)-oriented and safety-oriented analyses. Retrieved from https://www.iso.org/obp/ui/#iso:std:iso:26262:-9:ed-1:v1:en.
[12]
Wikipedia. V-model. Retrieved from https://en.wikipedia.org/wiki/V-model.
[13]
Agile Alliance. Agile 101. Retrieved from https://www.agilealliance.org/agile101/.
[14]
Rajmohan Madhavan, Elena R. Messina, and James S. Albus. 2006. Intelligent Vehicle Systems: A 4D/RCS Approach. Nova Science.
[15]
James S. Albus and Anthony J. Barbera. 2005. RCS: A cognitive architecture for intelligent multi-agent systems. Annu. Rev. Contr. 29, 1 (2005), 87–99.
[16]
Simon Ulbrich, Andreas Reschka, Jens Rieken, Susanne Ernst, Gerrit Bagschik, Frank Dierkes, Marcus Nolte, and Markus Maurer. 2017. Towards a functional system architecture for automated vehicles, arXiv:1703.08557 [cs.SY]. Retrieved from https://arxiv.org/abs/1703.08557.
[17]
Sara Dersten, Jakob Axelsson, and Joakim Fröberg. 2015. An analysis of a layered system architecture for autonomous construction vehicles. In Proceedings of the Annual IEEE Systems Conference (SysCon’15). 582–588.
[18]
Thomas Braud, Jordan Ivanchev, Corvin Deboeser, Alois C. Knoll, David Eckhoff, and Alberto L. Sangiovanni-Vincentelli. 2021. AVDM: A hierarchical command-and-control system architecture for cooperative autonomous vehicles in highways scenario using microscopic simulations. Auton. Agents Multi Agent Syst. 35, 1 (2021), 16.
[19]
Jonathan Aldrich, David Garlan, Christian Kästner, Claire Le Goues, Anahita Mohseni-Kabir, Ivan Ruchkin, Selva Samuel, Bradley R. Schmerl, Christopher Steven Timperley, Manuela Veloso, Ian Voysey, Joydeep Biswas, Arjun Guha, Jarrett Holtz, Javier Cámara, and Pooyan Jamshidi. 2019. Model-based adaptation for robotics software. IEEE Softw. 36, 2 (2019), 83–90.
[20]
VIRES Simulationstechnologie GmbH. 2006. OpenDRIVE Format Specification. Tech. Rep. V 1.4.
[21]
ASAM e.V. 2020. ASAM OpenDRIVE—Open Dynamic Road Information for Vehicle Environment. Tech. Rep. V 1.6.0.
[22]
J. Beetz and A. Borrmann. 2018. Benefits and limitations of linked data approaches for road modeling and data exchange. In Proceedings of the 25th EG-ICE International Workshop Advanced Computing Strategies for Engineering, Lecture Notes in Computer Science, Vol. 10864I. F. C. Smith and B.Domer (Eds.). Springer, 245–261.
[23]
G. Bagschik, T. Menzel, and M. Maurer. 2018. Ontology based scene creation for the development of automated vehicles. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV’18) IEEE, 1813–1820.
[24]
F. Poggenhans, J. Pauls, J. Janosovits, S. Orf, M. Naumann, F. Kuhnt, and M. Mayr. 2018. Lanelet2: A high-definition map framework for the future of automated driving. In Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC’18), W. Zhang, A. M. Bayen, J. J. S. Medina, and M. J. Barth (Eds.), 1672–1679.
[25]
Marius Bozga and Joseph Sifakis. 2021. Specification and validation of autonomous driving systems: A multilevel semantic framework, arXiv:2109.06478 [cs.MA]. Retrieved from https://arxiv.org/abs/2109.06478.
[26]
Jean-Claude Laprie. 1992. Dependability: Basic concepts and terminology. In Dependable Computing and Fault-Tolerant Systems. Springer, Berlin, (1992).
[27]
George Apostolakis. 2004. How useful is quantitative risk assessment? Risk Anal. 24, 3 (2004).
[28]
W. S. Lee, D. L. Grosh, F. A. Tillman, and C. H. Lie. 1985. Fault tree analysis, methods, and applications a review. IEEE Trans. Reliabil. R-34, 3 (1985).
[29]
Asim Abdulkhaleq, Stefan Wagner, and Nancy Leveson. 2015. A comprehensive safety engineering approach for software-intensive systems based on STPA. arXiv:1612.03109 [cs.SE].
[30]
Malcolm Wallace. 2005. Modular architectural representation and analysis of fault propagation and transformation. Electr. Not. Theor. Comput. Sci. 141 (2005), 53–71.
[31]
NHTSA. 2007. Pre-crash scenario typology for crash avoidance research, DOT HS 810 767.
[32]
A Zolghadri. 2012. Advanced model-based FDIR techniques for aerospace systems: Today challenges and opportunities. In Progress in Aerospace Sciences. Vol. 53, Elsevier, 18–29.
[33]
Asim Abdulkhaleq, Daniel Lammering, Stefan Wagner, Jürgen Rôder, Norbert Balbierer, Ludwig Ramsauer, Thomas Rastec, and Hagen Boehmert. 2017. A systematic approach based on STPA for developing a dependable architecture for fully automated driving vehicles. Proceedings of the 4th European STAMP Workshop 179 (2017), 41–51.
[34]
John D. Schierman, Michael D. DeVore, Nathan D. Richards, Neha Gandhi, Jared K. Cooper, and Kenneth R. Horneman. 2015. Runtime assurance framework development for highly adaptive flight control systems, Barron associates. AFRL-RQ-WP-TR-2016-0001Final Report. Stony Brook University.
[35]
L. Sha. 2001. Using simplicity to control complexity. IEEE Softw. 18, 4 (2001), 20–28.
[36]
J. R. Mayo, R. C. Armstrong, G. C. Hulette, M. Salloum, and A. M. Smith. 2018. Robust digital computation in the physical world. In Cyber-Physical Systems Security. Springer, 1–21. DOI:
[37]
M. Althoff, S. Maierhofer, and C. Pek. 2021. Provably-correct and comfortable adaptive cruise control. IEEE Trans. Intell. Vehic. 6, 1 (2021).
[38]
Andreas Bauer, Martin Leucker, and Christian Schallhart. 2011. Runtime verification for LTL and TLTL. ACM Trans. Softw. Eng. Methodol. 20, 4 (2011), 1–64.
[39]
Antonio Bucchiarone and Juan P. Galeotti. 2008. Dynamic software architectures verification using dynalloy. Electron. Commun. Eur. Assoc. Softw. Sci. Technol. 10 (2008). DOI:
[40]
WAYMO. Waymo reaches 5 million self-driven miles. February 27, 2018. Retrieved from https://blog.waymo.com/2019/08/waymo-reaches-5-million-self-driven.html.
[41]
Y. Setty, I. R. Cohen, Y. Dor, and D. Harel. 2008. Four-dimensional realistic modeling of pancreatic organogenesis. Proc. Natl. Acad. Sci. U.S.A. 105, 51 (2008), 20374–20379.
[42]
N. Bloch, G. Weiss, S. Szekely, and D. Harel. 2015. An interactive tool for animating biology, and its use in spatial and temporal modeling of a cancerous tumor and its microenvironment. PLoS ONE 10, 7 (2015), e0133484. DOI:
[43]
ASAM Open. 2020. Scenario—Dynamic content in driving simulation, UML Modeling Rules. Tech. Rep. V 1.0.0, ASAM e.V.
[44]
W. Damm and D. Harel. 2001. LSCs: Breathing life into message sequence charts. Form. Methods Syst. Des. 19, 1 (2001), 45–80.
[45]
W. Damm, S. Kemper, E. Môhlmann, T. Peikenkamp, and A. Rakow. 2018. Using traffic sequence charts for the development of HAVs. In Proceedings of the European Congress on Embedded Real Time Systems (ERTS’18).
[46]
D. Harel and R. Marelly. 2003. Come, let's play: Scenario-based programming using lscs and the play-engine. Springer-Verlag, Berlin.
[47]
D. J. Fremont, E. Kim, Y. V. Pant, S. A. Seshia, A. Acharya, X. Bruso, P. Wells, S. Lemke, Q. Lu, and S. Mehta. 2020. Formal scenario-based testing of autonomous vehicles: From simulation ta the real world. In Proceedings of the 23rd IEEE International Conference on Intelligent Transportation Systems (ITSC’20). IEEE, 1–8.
[48]
D. J. Fremont, E. Kim, T. Dreossi, S. Ghosh, X. Yue, A. L. Sangiovanni-Vincentelli, and S. A. Seshia. 2020. Scenic: A language for scenario specification and data generation. arXiv:2010.06580. Retrieved from https://arxiv.org/abs/2010.06580.
[49]
Antoine El-Hokayem, Marius Bozga, and Joseph Sifakis. 2021. A temporal configuration logic for dynamic reconfigurable systems. SAC'21. ACM, 1419–1428.
[50]
Klemens Esterle, Vincent Aravantinos, and Alois Knoll. 2019. From specifications to behavior: Maneuver verification in a semantic state space. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV’19). IEEE, 2140–2147.
[51]
A. Rizaldi and M. Althoff. 2015. Formalising traffic rules for accountability of autonomous vehicles. In Proceedings of the IEEE 18th International Conference on Intelligent Transportation Systems (ITSC’15). IEEE, 1658–1665.
[52]
A. Rizaldi, J. Keinholz, M. Huber, J. Feldle, F. Immler, M. Althoff, E. Hilgendorf, and T. Nipkow. 2017. Formalising and monitoring traffic rules for autonomous vehicles in Isabelle/HOL. In Proceedings of the 13th International Conference on Integrated Formal Methods (IFM’17), Lecture Notes in Computer Science, Vol. 10510, N. Polikarpova and S. A. Schneider (Eds.). Springer, 50–66.
[53]
A. Karimi and P. S. Duggirala. 2020. Formalizing traffic rules for uncontrolled intersections. In Proceedings of the 11th ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS’20). IEEE, 41–50.
[54]
Z. Q. Zhou and L. Sun. 2019. Metamorphic testing of driverless cars. Commun. ACM 62, 3 (2019), 61–67.
[55]
Antoine El-Hokayem, Saddek Bensalem, and Marius Bozga. 2020. Joseph Sifakis: A layered implementation of DR-BIP supporting run-time monitoring and analysis. In Proceedings of the International Conference on Software Engineering and Formal Methods (SEFM’20). 284–302.
[56]
SAE International Releases Updated Visual Chart for Its “Levels of Driving Automation” Standard for Self-Driving Vehicles. Retrieved from https://www.sae.org/news/press-room/2018/12/sae-international-releases-updated-visual-chart-for-its-%E2%80%9Clevels-of-driving-automation%E2%80%9D-standard-for-self-driving-vehicles.
[57]
M. Jerrold and A. Grochow. 2020. Taxonomy of automated assistants. Commun. ACM 63 (2020), 39–41.
[58]
Fernando Galdon, Ashley Hall, and Stephen Jia Wang. 2020. Designing trust in highly automated virtual assistants: A taxonomy of levels of autonomy. In Artificial Intelligence in Industry 4.0: A Collection of Innovative Research Case-studies.
[59]
Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM 58, 9 (2015), 92–103.
[60]
NDTV Watch: Tesla Autopilot Feature Mistakes Moon For Yellow Traffic Light. Retrieved from July 27 2021 https://www.ndtv.com/offbeat/watch-tesla-autopilot-feature-mistakes-moon-for-yellow-trafficlight-2495804.
[61]
P. G. Neumann. 2017. Trustworthiness and truthfulness are essential. Commun. ACM 60, 6 (2017), 26–28.
[62]
Wikipedia. Precautionary principle. Retrieved from https://en.wikipedia.org/wiki/Precautionary_principle.
[63]
Stuart Mason Dambrot, Derrick de Kerchove, Francesco Flammini, Witold Kinsner, Linda MacDonald Glenn, and Roberto Saracco. 2018. IEEE Symbiotic Autonomous Systems White Paper ii.
[64]
Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. arXiv:1702.01135v2 [cs.AI]. Retrieved from https://arxiv.org/abs/1702.01135v2.
[65]
Nicola Franco, Tom Wollschläger, Nicholas Gao, Jeanette Miriam Lorenz, and Stephan Günnemann. 2022. Quantum robustness verification: A hybrid quantum-classical neural network certification algorithm. arXiv:2205.00900v1 [quant-ph]. Retrieved from https:arxiv.org/abs/2205.00900v1.

Cited By

View all
  • (2025)Enhancing Autonomous System Security With AI and Secure Computation TechnologiesAI Developments for Industrial Robotics and Intelligent Drones10.4018/979-8-3693-2707-4.ch008(159-186)Online publication date: 3-Jan-2025
  • (2024)Taxonomy Describing Levels of Autonomous Drilling Systems: Incorporating Complexity, Uncertainty, Sparse Data, With Human InteractionDay 3 Thu, March 07, 202410.2118/217754-MSOnline publication date: 27-Feb-2024
  • (2024)Decision-making based on Markov decision process in integrated artificial reasoning framework—Part I: TheoryNuclear Science and Technology Open Research10.12688/nuclscitechnolopenres.17491.12(64)Online publication date: 21-Aug-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Embedded Computing Systems
ACM Transactions on Embedded Computing Systems  Volume 22, Issue 3
May 2023
519 pages
ISSN:1539-9087
EISSN:1558-3465
DOI:10.1145/3592782
  • Editor:
  • Tulika Mitra
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

Publication History

Published: 18 April 2023
Online AM: 29 June 2022
Accepted: 20 June 2022
Revised: 20 June 2022
Received: 07 April 2022
Published in TECS Volume 22, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Autonomous system
  2. trustworthy system development
  3. dependability
  4. machine learning
  5. critical systems engineering
  6. validation
  7. simulation and testing
  8. requirement and scenario specification

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)730
  • Downloads (Last 6 weeks)47
Reflects downloads up to 18 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Enhancing Autonomous System Security With AI and Secure Computation TechnologiesAI Developments for Industrial Robotics and Intelligent Drones10.4018/979-8-3693-2707-4.ch008(159-186)Online publication date: 3-Jan-2025
  • (2024)Taxonomy Describing Levels of Autonomous Drilling Systems: Incorporating Complexity, Uncertainty, Sparse Data, With Human InteractionDay 3 Thu, March 07, 202410.2118/217754-MSOnline publication date: 27-Feb-2024
  • (2024)Decision-making based on Markov decision process in integrated artificial reasoning framework—Part I: TheoryNuclear Science and Technology Open Research10.12688/nuclscitechnolopenres.17491.12(64)Online publication date: 21-Aug-2024
  • (2024)Surrogate Adaptive Controller Tuning Based on DE in a 3R Serial Robot: A Comparative Analysis2024 IEEE 19th Conference on Industrial Electronics and Applications (ICIEA)10.1109/ICIEA61579.2024.10664799(1-7)Online publication date: 5-Aug-2024
  • (2024)Navigating Energy: Unveiling Power Consumption Patterns in Autonomous Vehicles2024 1st International Conference on Emerging Technologies for Dependable Internet of Things (ICETI)10.1109/ICETI63946.2024.10777158(1-6)Online publication date: 25-Nov-2024
  • (2024)Comparative Analysis of Indirect Adaptive Controller Tuning Strategies Using Surrogate and Model-Based Techniques Applied to the Omnidirectional Mobile Robot2024 10th International Conference on Control, Decision and Information Technologies (CoDIT)10.1109/CoDIT62066.2024.10708577(91-96)Online publication date: 1-Jul-2024
  • (2024)Philosophical Investigations into AI Alignment: A Wittgensteinian FrameworkPhilosophy & Technology10.1007/s13347-024-00761-937:3Online publication date: 1-Jul-2024
  • (2024)Evaluating the Effectiveness of Digital Twins Through Statistical Model Checking with Feedback and PerturbationsFormal Methods for Industrial Critical Systems10.1007/978-3-031-68150-9_2(21-39)Online publication date: 9-Sep-2024
  • (2024)Fault‐tolerance approaches for distributed and cloud computing environments: A systematic review, taxonomy and future directionsConcurrency and Computation: Practice and Experience10.1002/cpe.808136:13Online publication date: 18-Mar-2024
  • (2023)Failure Detection for Motion Prediction of Autonomous Driving: An Uncertainty Perspective2023 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA48891.2023.10160596(12721-12728)Online publication date: 29-May-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media