research-article

Trustworthy Autonomous System Development

Authors:
Joseph Sifakis

Verimag Laboratory

Verimag Laboratory

0000-0003-2447-7981
View Profile

,
David Harel

Weizmann Institute

Weizmann Institute

0000-0001-7240-3931
View Profile

Authors Info & Claims

ACM Transactions on Embedded Computing Systems Volume 22 Issue 3Article No.: 40pp 1–24https://doi.org/10.1145/3545178

Published:18 April 2023Publication History

ACM Transactions on Embedded Computing Systems

Abstract

Autonomous systems emerge from the need to progressively replace human operators by autonomous agents in a wide variety of application areas. We offer an analysis of the state of the art in developing autonomous systems, focusing on design and validation and showing that the multi-faceted challenges involved go well beyond the limits of weak AI. We argue that traditional model-based techniques are defeated by the complexity of the problem, while solutions based on end-to-end machine learning fail to provide the necessary trustworthiness. We advocate a hybrid design approach, which combines the two, adopting the best of each, and seeks tradeoffs between trustworthiness and performance. We claim that traditional risk analysis and mitigation techniques fail to scale and discuss the trend of moving away from correctness at design time and toward reliance on runtime assurance techniques. We argue that simulation and testing remain the only realistic approach for global validation and show how current methods can be adapted to autonomous systems. We conclude by discussing the factors that will play a decisive role in the acceptance of autonomous systems and by highlighting the urgent need for new theoretical foundations.

REFERENCES

[1] Harel David, Marron Assaf, and Sifakis Joseph. 2022. Creating a foundation for next-generation autonomous systems. IEEE Des. Test 39, 1 (2022), 49–56.Google ScholarCross Ref
[2] Harel D., Marron Assaf, and Sifakis J.. 2020. Autonomics: In search of a foundation for next generation autonomous systems. Proc. Natl. Acad. Sci. U.S.A. 117, 30 (2020), 17491–17498.Google ScholarCross Ref
[3] Autonomic Computing. 2006. An architectural blueprint for autonomic computing. IBM White Paper 31, (2006), 1–6. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=0e99837d9b1e70bb35d516e32ecfc345cd30e795.Google Scholar
[4] Jordan Michael I.. Artificial Intelligence—The Revolution Hasn't Happened Yet. Retrieved from https://hdsr.mitpress.mit.edu/pub/wot7mkc1/release/9.Google Scholar
[5] Harel David and Pnueli Amir. 1985. On the Eevelopment of Reactive Systems, Logics and Models of Concurrent Systems (K. R. Apt, Ed.). NATO ASI Series, F-13, Springer-Verlag, New York, 477–498.Google ScholarCross Ref
[6] Sifakis Joseph. 2018. Autonomous systems an architectural characterization. arXiv:1811.10277. Retrieved from https://arxiv.org/abs/1811.10277.Google Scholar
[7] Efroni S., Harel D., and Cohen I. R.. 2005. Reactive animation: Realistic modeling of complex dynamic systems. Computer 38, 1, (2005), 38–47. DOI:Google ScholarDigital Library
[8] Simon Bliudze, Sébastien Furic, Joseph Sifakis, and Antoine Viel. 2019. Antoine viel: Rigorous design of cyber-physical systems—Linking physicality and computation. Softw. Syst. Model 18, 3 (2019), 1613–1636.Google ScholarDigital Library
[9] Bloem R., Jacobs S., Khalimov A., Konnov I., Rubin S., Veith H., and Widder J.. 2015. Decidability of Parameterized Verification, Synthesis, Lectures on Distributed Computing Theory. Morgan & Claypool.Google ScholarCross Ref
[10] Sifakis J.. 2012. Rigorous system design. Foundations and Trends in Electronic Design Automation 6, 4 (2012), 293–362.Google ScholarDigital Library
[11] ISO Online Browsing Platform. Road vehicles— Functional safety— Part9: Automotive Safety Integrity Level (ASIL)-oriented and safety-oriented analyses. Retrieved from https://www.iso.org/obp/ui/#iso:std:iso:26262:-9:ed-1:v1:en.Google Scholar
[12] Wikipedia. V-model. Retrieved from https://en.wikipedia.org/wiki/V-model.Google Scholar
[13] Agile Alliance. Agile 101. Retrieved from https://www.agilealliance.org/agile101/.Google Scholar
[14] Madhavan Rajmohan, Messina Elena R., and Albus James S.. 2006. Intelligent Vehicle Systems: A 4D/RCS Approach. Nova Science.Google Scholar
[15] Albus James S. and Barbera Anthony J.. 2005. RCS: A cognitive architecture for intelligent multi-agent systems. Annu. Rev. Contr. 29, 1 (2005), 87–99.Google ScholarCross Ref
[16] Ulbrich Simon, Reschka Andreas, Rieken Jens, Ernst Susanne, Bagschik Gerrit, Dierkes Frank, Nolte Marcus, and Maurer Markus. 2017. Towards a functional system architecture for automated vehicles, arXiv:1703.08557 [cs.SY]. Retrieved from https://arxiv.org/abs/1703.08557.Google Scholar
[17] Dersten Sara, Axelsson Jakob, and Fröberg Joakim. 2015. An analysis of a layered system architecture for autonomous construction vehicles. In Proceedings of the Annual IEEE Systems Conference (SysCon’15). 582–588.Google ScholarCross Ref
[18] Braud Thomas, Ivanchev Jordan, Deboeser Corvin, Knoll Alois C., Eckhoff David, and Sangiovanni-Vincentelli Alberto L.. 2021. AVDM: A hierarchical command-and-control system architecture for cooperative autonomous vehicles in highways scenario using microscopic simulations. Auton. Agents Multi Agent Syst. 35, 1 (2021), 16.Google ScholarDigital Library
[19] Aldrich Jonathan, Garlan David, Kästner Christian, Goues Claire Le, Mohseni-Kabir Anahita, Ruchkin Ivan, Samuel Selva, Schmerl Bradley R., Timperley Christopher Steven, Veloso Manuela, Voysey Ian, Biswas Joydeep, Guha Arjun, Holtz Jarrett, Cámara Javier, and Jamshidi Pooyan. 2019. Model-based adaptation for robotics software. IEEE Softw. 36, 2 (2019), 83–90.Google ScholarDigital Library
[20] VIRES Simulationstechnologie GmbH. 2006. OpenDRIVE Format Specification. Tech. Rep. V 1.4.Google Scholar
[21] ASAM e.V. 2020. ASAM OpenDRIVE—Open Dynamic Road Information for Vehicle Environment. Tech. Rep. V 1.6.0. Google Scholar
[22] Beetz J. and Borrmann A.. 2018. Benefits and limitations of linked data approaches for road modeling and data exchange. In Proceedings of the 25th EG-ICE International Workshop Advanced Computing Strategies for Engineering, Lecture Notes in Computer Science, Vol. 10864I. F. C. Smith and B.Domer (Eds.). Springer, 245–261.Google Scholar
[23] Bagschik G., Menzel T., and Maurer M.. 2018. Ontology based scene creation for the development of automated vehicles. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV’18) IEEE, 1813–1820.Google ScholarDigital Library
[24] Poggenhans F., Pauls J., Janosovits J., Orf S., Naumann M., Kuhnt F., and Mayr M.. 2018. Lanelet2: A high-definition map framework for the future of automated driving. In Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC’18), W. Zhang, A. M. Bayen, J. J. S. Medina, and M. J. Barth (Eds.), 1672–1679.Google ScholarDigital Library
[25] Bozga Marius and Sifakis Joseph. 2021. Specification and validation of autonomous driving systems: A multilevel semantic framework, arXiv:2109.06478 [cs.MA]. Retrieved from https://arxiv.org/abs/2109.06478.Google Scholar
[26] Laprie Jean-Claude. 1992. Dependability: Basic concepts and terminology. In Dependable Computing and Fault-Tolerant Systems. Springer, Berlin, (1992).Google Scholar
[27] George Apostolakis. 2004. How useful is quantitative risk assessment? Risk Anal. 24, 3 (2004).Google Scholar
[28] Lee W. S., Grosh D. L., Tillman F. A., and Lie C. H.. 1985. Fault tree analysis, methods, and applications a review. IEEE Trans. Reliabil. R-34, 3 (1985).Google ScholarCross Ref
[29] Asim Abdulkhaleq, Stefan Wagner, and Nancy Leveson. 2015. A comprehensive safety engineering approach for software-intensive systems based on STPA. arXiv:1612.03109 [cs.SE]. .Google ScholarCross Ref
[30] Wallace Malcolm. 2005. Modular architectural representation and analysis of fault propagation and transformation. Electr. Not. Theor. Comput. Sci. 141 (2005), 53–71.Google ScholarDigital Library
[31] NHTSA. 2007. Pre-crash scenario typology for crash avoidance research, DOT HS 810 767.Google Scholar
[32] Zolghadri A. 2012. Advanced model-based FDIR techniques for aerospace systems: Today challenges and opportunities. In Progress in Aerospace Sciences. Vol. 53, Elsevier, 18–29.Google Scholar
[33] Abdulkhaleq Asim, Lammering Daniel, Wagner Stefan, Rôder Jürgen, Balbierer Norbert, Ramsauer Ludwig, Rastec Thomas, and Boehmert Hagen. 2017. A systematic approach based on STPA for developing a dependable architecture for fully automated driving vehicles. Proceedings of the 4th European STAMP Workshop 179 (2017), 41–51.Google Scholar
[34] Schierman John D., DeVore Michael D., Richards Nathan D., Gandhi Neha, Cooper Jared K., and Horneman Kenneth R.. 2015. Runtime assurance framework development for highly adaptive flight control systems, Barron associates. AFRL-RQ-WP-TR-2016-0001Final Report. Stony Brook University.Google Scholar
[35] Sha L.. 2001. Using simplicity to control complexity. IEEE Softw. 18, 4 (2001), 20–28.Google ScholarDigital Library
[36] Mayo J. R., Armstrong R. C., Hulette G. C., Salloum M., and Smith A. M.. 2018. Robust digital computation in the physical world. In Cyber-Physical Systems Security. Springer, 1–21. DOI:Google ScholarCross Ref
[37] Althoff M., Maierhofer S., and Pek C.. 2021. Provably-correct and comfortable adaptive cruise control. IEEE Trans. Intell. Vehic. 6, 1 (2021).Google Scholar
[38] Bauer Andreas, Leucker Martin, and Schallhart Christian. 2011. Runtime verification for LTL and TLTL. ACM Trans. Softw. Eng. Methodol. 20, 4 (2011), 1–64. Google ScholarDigital Library
[39] Bucchiarone Antonio and Galeotti Juan P.. 2008. Dynamic software architectures verification using dynalloy. Electron. Commun. Eur. Assoc. Softw. Sci. Technol. 10 (2008). DOI:Google ScholarCross Ref
[40] WAYMO. Waymo reaches 5 million self-driven miles. February 27, 2018. Retrieved from https://blog.waymo.com/2019/08/waymo-reaches-5-million-self-driven.html.Google Scholar
[41] Setty Y., Cohen I. R., Dor Y., and Harel D.. 2008. Four-dimensional realistic modeling of pancreatic organogenesis. Proc. Natl. Acad. Sci. U.S.A. 105, 51 (2008), 20374–20379.Google ScholarCross Ref
[42] Bloch N., Weiss G., Szekely S., and Harel D.. 2015. An interactive tool for animating biology, and its use in spatial and temporal modeling of a cancerous tumor and its microenvironment. PLoS ONE 10, 7 (2015), e0133484. DOI:Google ScholarCross Ref
[43] ASAM Open. 2020. Scenario—Dynamic content in driving simulation, UML Modeling Rules. Tech. Rep. V 1.0.0, ASAM e.V.Google Scholar
[44] Damm W. and Harel D.. 2001. LSCs: Breathing life into message sequence charts. Form. Methods Syst. Des. 19, 1 (2001), 45–80.Google ScholarDigital Library
[45] Damm W., Kemper S., Môhlmann E., Peikenkamp T., and Rakow A.. 2018. Using traffic sequence charts for the development of HAVs. In Proceedings of the European Congress on Embedded Real Time Systems (ERTS’18).Google Scholar
[46] Harel D. and Marelly R.. 2003. Come, let's play: Scenario-based programming using lscs and the play-engine. Springer-Verlag, Berlin.Google ScholarCross Ref
[47] Fremont D. J., Kim E., Pant Y. V., Seshia S. A., Acharya A., Bruso X., Wells P., Lemke S., Lu Q., and Mehta S.. 2020. Formal scenario-based testing of autonomous vehicles: From simulation ta the real world. In Proceedings of the 23rd IEEE International Conference on Intelligent Transportation Systems (ITSC’20). IEEE, 1–8.Google ScholarDigital Library
[48] Fremont D. J., Kim E., Dreossi T., Ghosh S., Yue X., Sangiovanni-Vincentelli A. L., and Seshia S. A.. 2020. Scenic: A language for scenario specification and data generation. arXiv:2010.06580. Retrieved from https://arxiv.org/abs/2010.06580.Google Scholar
[49] El-Hokayem Antoine, Bozga Marius, and Sifakis Joseph. 2021. A temporal configuration logic for dynamic reconfigurable systems. SAC'21. ACM, 1419–1428.Google ScholarDigital Library
[50] Klemens Esterle, Vincent Aravantinos, and Alois Knoll. 2019. From specifications to behavior: Maneuver verification in a semantic state space. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV’19). IEEE, 2140–2147.Google Scholar
[51] Rizaldi A. and Althoff M.. 2015. Formalising traffic rules for accountability of autonomous vehicles. In Proceedings of the IEEE 18th International Conference on Intelligent Transportation Systems (ITSC’15). IEEE, 1658–1665.Google ScholarDigital Library
[52] Rizaldi A., Keinholz J., Huber M., Feldle J., Immler F., Althoff M., Hilgendorf E., and Nipkow T.. 2017. Formalising and monitoring traffic rules for autonomous vehicles in Isabelle/HOL. In Proceedings of the 13th International Conference on Integrated Formal Methods (IFM’17), Lecture Notes in Computer Science, Vol. 10510, N. Polikarpova and S. A. Schneider (Eds.). Springer, 50–66.Google ScholarCross Ref
[53] Karimi A. and Duggirala P. S.. 2020. Formalizing traffic rules for uncontrolled intersections. In Proceedings of the 11th ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS’20). IEEE, 41–50.Google ScholarCross Ref
[54] Zhou Z. Q. and Sun L.. 2019. Metamorphic testing of driverless cars. Commun. ACM 62, 3 (2019), 61–67.Google ScholarDigital Library
[55] El-Hokayem Antoine, Bensalem Saddek, and Bozga Marius. 2020. Joseph Sifakis: A layered implementation of DR-BIP supporting run-time monitoring and analysis. In Proceedings of the International Conference on Software Engineering and Formal Methods (SEFM’20). 284–302.Google ScholarDigital Library
[56] SAE International Releases Updated Visual Chart for Its “Levels of Driving Automation” Standard for Self-Driving Vehicles. Retrieved from https://www.sae.org/news/press-room/2018/12/sae-international-releases-updated-visual-chart-for-its-%E2%80%9Clevels-of-driving-automation%E2%80%9D-standard-for-self-driving-vehicles.Google Scholar
[57] Jerrold M. and Grochow A.. 2020. Taxonomy of automated assistants. Commun. ACM 63 (2020), 39–41.Google ScholarDigital Library
[58] Galdon Fernando, Hall Ashley, and Wang Stephen Jia. 2020. Designing trust in highly automated virtual assistants: A taxonomy of levels of autonomy. In Artificial Intelligence in Industry 4.0: A Collection of Innovative Research Case-studies.Google Scholar
[59] Davis Ernest and Marcus Gary. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM 58, 9 (2015), 92–103.Google ScholarDigital Library
[60] NDTV Watch: Tesla Autopilot Feature Mistakes Moon For Yellow Traffic Light. Retrieved from July 27 2021 https://www.ndtv.com/offbeat/watch-tesla-autopilot-feature-mistakes-moon-for-yellow-trafficlight-2495804.Google Scholar
[61] Neumann P. G.. 2017. Trustworthiness and truthfulness are essential. Commun. ACM 60, 6 (2017), 26–28.Google ScholarDigital Library
[62] Wikipedia. Precautionary principle. Retrieved from https://en.wikipedia.org/wiki/Precautionary_principle.Google Scholar
[63] Dambrot Stuart Mason, Kerchove Derrick de, Flammini Francesco, Kinsner Witold, Glenn Linda MacDonald, and Saracco Roberto. 2018. IEEE Symbiotic Autonomous Systems White Paper ii.Google Scholar
[64] Katz Guy, Barrett Clark, Dill David, Julian Kyle, and Kochenderfer Mykel. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. arXiv:1702.01135v2 [cs.AI]. Retrieved from https://arxiv.org/abs/1702.01135v2.Google Scholar
[65] Franco Nicola, Wollschläger Tom, Gao Nicholas, Lorenz Jeanette Miriam, and Günnemann Stephan. 2022. Quantum robustness verification: A hybrid quantum-classical neural network certification algorithm. arXiv:2205.00900v1 [quant-ph]. Retrieved from https:arxiv.org/abs/2205.00900v1.Google Scholar

Index Terms

Trustworthy Autonomous System Development

Recommendations

The Value of Trustworthy AI
AIES '19: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society

Trust is one of the most critical relations in our human lives, whether trust in one another, trust in the artifacts that we use everyday, or trust of an AI system. Even a cursory examination of the literatures in human-computer interaction, human-robot ...
Read More
The relationship between trust in AI and trustworthy machine learning technologies
FAT* '20: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

To design and develop AI-based systems that users and the larger public can justifiably trust, one needs to understand how machine learning technologies impact trust. To guide the design and implementation of trusted AI-based systems, this paper ...
Read More
Trustworthy Software Development
CMS 2013: 14th IFIP TC 6/TC 11 International Conference on Communications and Multimedia Security - Volume 8099

This paper presents an overview on how existing development methodologies and practices support the creation of trustworthy software. Trustworthy software is key for a successful and trusted usage of software, specifically in the Cloud. To better ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Transactions on Embedded Computing Systems Volume 22, Issue 3
May 2023
546 pages
ISSN:1539-9087
EISSN:1558-3465
DOI:10.1145/3592782
Editor:
Tulika Mitra
National University of Singapore, Singapore
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States

Journal Family
ACM Journals for the Design of Smart and Connected Systems
Publication History
- Published: 18 April 2023
- Online AM: 29 June 2022
- Revised: 20 June 2022
- Accepted: 20 June 2022
- Received: 7 April 2022
Published in tecs Volume 22, Issue 3

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Autonomous system
trustworthy system development
dependability
machine learning
critical systems engineering
validation
simulation and testing
requirement and scenario specification
Qualifiers
- research-article
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 0
  Total Citations
  View Citations
- 1,787
  Total Downloads
- Downloads (Last 12 months)1,488
- Downloads (Last 6 weeks)78
Other Metrics
View Author Metrics
Cited By
This publication has not been cited yet

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

View Full Text

HTML Format

View this article in HTML Format .

View HTML Format

Trustworthy Autonomous System Development

ACM Transactions on Embedded Computing Systems

Abstract

REFERENCES

Cited By

Index Terms

Recommendations

The Value of Trustworthy AI

The relationship between trust in AI and trustworthy machine learning technologies

Trustworthy Software Development