research-article

Towards classes of architectural dependability assurance for machine-learning-based systems

Authors:

Jonas Klamroth,

Bernhard BeckertAuthors Info & Claims

SEAMS '20: Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems

Pages 31 - 37

https://doi.org/10.1145/3387939.3388613

Published: 18 September 2020 Publication History

SEAMS '20: Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems

Towards classes of architectural dependability assurance for machine-learning-based systems

Pages 31 - 37

Abstract
References

Abstract

Advances in Machine Learning (ML) have brought previously hard to handle problems within arm's reach. However, this power comes at the cost of unassured reliability and lacking transparency. Overcoming this drawback is very hard due to the probabilistic nature of ML. Current approaches mainly tackle this problem by developing more robust learning procedures. Such algorithmic approaches, however, are limited to certain types of uncertainties and cannot deal with all of them, e.g., hardware failure. This paper discusses how this problem can be addressed at architectural rather than algorithmic level to assess systems dependability properties in early development stages. Moreover, we argue that Self-Adaptive Systems (SAS) are more suited to safeguard ML w.r.t. various uncertainties. As a step towards this we propose classes of dependability in which ML-based systems may be categorized and discuss which and how assurances can be made for each class.

References

[1]

Vahdat Abdelzad, Krzysztof Czarnecki, Rick Salay, Taylor Denounden, Sachin Vernekar, and Buu Phan. 2019. Detecting Out-of-Distribution Inputs in Deep Neural Networks Using an Early-Layer Output. arXiv preprint arXiv:1910.10307 (2019).

[2]

Mehdi Amoui, Mazeiar Salehie, Siavash Mirarab, and Ladan Tahvildari. 2008. Adaptive action selection in autonomic software using reinforcement learning. In Fourth International Conference on Autonomic and Autonomous Systems (ICAS'08). IEEE, 175--181.

Digital Library

[3]

Matthias Becker, Markus Luckey, and Steffen Becker. 2013. Performance analysis of self-adaptive systems for requirements validation at design-time. In Proceedings of the 9th international ACM Sigsoft conference on Quality of software architectures. ACM, 43--52.

Digital Library

[4]

David M Blei, Alp Kucukelbir, and Jon D McAuliffe. 2017. Variational inference: A review for statisticians. Journal of the American statistical Association 112, 518 (2017), 859--877.

[5]

Javier Cámara and Rogério de Lemos. 2012. Evaluation of resilience in self-adaptive systems using probabilistic model-checking. In Proceedings of the 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems. IEEE Press, 53--62.

Digital Library

[6]

Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. 2015. Deepdriving: Learning affordance for direct perception in autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision. 2722--2730.

Digital Library

[7]

Chih-Hong Cheng, Georg Nührenberg, and Hirotoshi Yasuoka. 2019. Runtime monitoring neuron activation patterns. In 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 300--303.

[8]

Rogerio de Lemos and Marek Grzes. 2019. Self-adaptive Artificial Intelligence. (2019).

[9]

Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. (2017).

[10]

Tommaso Dreossi, Alexandre Donzé, and Sanjit A Seshia. 2019. Compositional falsification of cyber-physical systems with machine learning components. Journal of Automated Reasoning 63, 4 (2019), 1031--1053.

Digital Library

[11]

Naeem Esfahani and Sam Malek. 2013. Uncertainty in self-adaptive software systems. In Software Engineering for Self-Adaptive Systems II. Springer, 214--238.

[12]

Simos Gerasimou, Radu Calinescu, Stepan Shevtsov, and Danny Weyns. 2017. Undersea: an exemplar for engineering self-adaptive unmanned underwater vehicles. In 2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). IEEE, 83--89.

Digital Library

[13]

Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. 2017. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA). IEEE, 3389--3396.

Digital Library

[14]

Xiaozhe Gu and Arvind Easwaran. 2019. Towards safe machine learning for CPS: infer uncertainty from training data. In Proceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems. ACM, 249--258.

Digital Library

[15]

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2018), 93.

[16]

Muhammad Abdullah Hanif, Faiq Khalid, Rachmad Vidya Wicaksana Putra, Semeen Rehman, and Muhammad Shafique. 2018. Robust machine learning systems: Reliability and security for deep neural networks. In 2018 IEEE 24th International Symposium on On-Line Testing And Robust System Design (IOLTS). IEEE, 257--260.

[17]

Le-Ha Hoang, Muhammad Abdullah Hanif, and Muhammad Shafique. 2019. FT-ClipAct: Resilience Analysis of Deep Neural Networks and Improving their Fault Tolerance using Clipped Activation. arXiv preprint arXiv:1912.00941 (2019).

[18]

M Usman Iftikhar, Gowri Sankar Ramachandran, Pablo Bollansée, Danny Weyns, and Danny Hughes. 2017. DeltaIoT: A self-adaptive Internet of Things exemplar. In Proceedings of the 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems. IEEE Press, 76--82.

Digital Library

[19]

Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification. Springer, 97--117.

[20]

Guy Katz, Derek A Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljić, et al. 2019. The marabou framework for verification and analysis of deep neural networks. In International Conference on Computer Aided Verification. Springer, 443--452.

[21]

Daphne Koller and Nir Friedman. 2009. Probabilistic graphical models: principles and techniques. MIT press.

Digital Library

[22]

Anne Martens, Heiko Koziolek, Steffen Becker, and Ralf Reussner. 2010. Automatically improve software architecture models for performance, reliability, and cost using evolutionary algorithms. In Proceedings of the first joint WOSP/SIPEW international conference on Performance engineering. ACM, 105--116.

Digital Library

[23]

Gabriel A Moreno, Javier Cámara, David Garlan, and Bradley Schmerl. 2015. Proactive self-adaptation under uncertainty: a probabilistic model checking approach. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 1--12.

Digital Library

[24]

Christian Murphy and Gail E Kaiser. 2008. Improving the dependability of machine learning applications. (2008).

[25]

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2017. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security. ACM, 506--519.

Digital Library

[26]

Georg Püschel, Christian Piechnick, Sebastian Götz, Christoph Seidl, Sebastian Richly, Thomas Schlegel, and Uwe Aßmann. 2014. A combined simulation and test case generation strategy for self-adaptive systems. Journal On Advances in Software 7, 3&4 (2014), 686--696.

[27]

Markus Schumacher, Eduardo Fernandez-Buglioni, Duane Hybertson, FrankBuschmann, and Peter Sommerlad. 2006. Security Patterns - Integrating Security and Systems Engineering.

[28]

Alexandru Constantin Serban. 2019. Designing Safety Critical Software Systems to Manage Inherent Uncertainty. In 2019 IEEE International Conference on Software Architecture Companion (ICSA-C). IEEE, 246--249.

[29]

Sanjit A Seshia, Dorsa Sadigh, and S Shankar Sastry. 2016. Towards verified artificial intelligence. arXiv preprint arXiv:1606.08514 (2016).

[30]

Sina Shafaei, Stefan Kugele, Mohd Hafeez Osman, and Alois Knoll. 2018. Uncertainty in machine learning: A safety perspective on autonomous driving. In International Conference on Computer Safety, Reliability, and Security. Springer, 458--464.

[31]

Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.

Digital Library

[32]

Marc Toussaint, Amos Storkey, and Stefan Harmeling. 2010. Expectation-Maximization methods for solving (PO) MDPs and optimal control problems. Inference and Learning in Dynamic Models (2010).

[33]

Kush R Varshney and Homa Alemzadeh. 2017. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big data 5, 3 (2017), 246--255.

[34]

David Verstraete, Andrés Ferrada, Enrique López Droguett, Viviana Meruane, and Mohammad Modarres. 2017. Deep learning enabled fault diagnosis using time-frequency image analysis of rolling element bearings. Shock and Vibration 2017 (2017).

[35]

Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Efficient formal safety analysis of neural networks. In Advances in Neural Information Processing Systems. 6367--6377.

[36]

Danny Weyns. 2012. Towards an integrated approach for validating qualities of self-adaptive systems. In Proceedings of the Ninth International Workshop on Dynamic Analysis. ACM, 24--29.

Digital Library

[37]

Pamela Zave and Michael Jackson. 1997. Four dark corners of requirements engineering. ACM transactions on Software Engineering and Methodology (TOSEM) 6, 1 (1997), 1--30.

[38]

Jie M Zhang, Earl T Barr, Benjamin Guedj, Mark Harman, and John Shawe-Taylor. 2019. Perturbed Model Validation: A New Framework to Validate Model Relevance. arXiv preprint arXiv:1905.10201 (2019).

Cited By

Nazir RBucaioni APelliccione P(2024)Architecting ML-enabled systems: Challenges, best practices, and design decisionsJournal of Systems and Software10.1016/j.jss.2023.111860207(111860)Online publication date: Jan-2024
https://doi.org/10.1016/j.jss.2023.111860
Rapp MScheerer MReussner RFeng DBecker SHerbst NLeitner P(2022)Design-time Performability Optimization of Runtime Adaptation StrategiesCompanion of the 2022 ACM/SPEC International Conference on Performance Engineering10.1145/3491204.3527471(113-120)Online publication date: 14-Jul-2022
https://dl.acm.org/doi/10.1145/3491204.3527471
Serban AVisser J(2022)Adapting Software Architectures to Machine Learning Challenges2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00029(152-163)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00029
Show More Cited By

Index Terms

Towards classes of architectural dependability assurance for machine-learning-based systems
1. Computing methodologies
  1. Machine learning
2. Software and its engineering
  1. Software organization and properties
    1. Software functional properties
      1. Formal methods
    2. Software system structures
      1. Software architectures

Recommendations

Construction of a quality model for machine learning systems
Abstract
Nowadays, systems containing components based on machine learning (ML) methods are becoming more widespread. In order to ensure the intended behavior of a software system, there are standards that define necessary qualities of the system and its ...
Synthetic Deliberation: Can Emulated Imagination Enhance Machine Ethics?
Abstract
Artificial intelligence is becoming increasingly entwined with our daily lives: AIs work as assistants through our phones, control our vehicles, and navigate our vacuums. As these objects become more complex and work within our societies in ways ...
A Unified Model of Dependability: Capturing Dependability in Context

In contemporary societies, individuals and organizations increasingly depend on services delivered by sophisticated software-intensive systems.Dependability has become a key systems property, which needs to be engineered and guaranteed regardless of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SEAMS '20: Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems

June 2020

211 pages

ISBN:9781450379625

DOI:10.1145/3387939

General Chair:
Shinichi Honiden
Waseda University/National Institute of Informatics, Japan
,
Program Chairs:
Elisabetta Di Nitto
Politecnico di Milano, Italy
,
Radu Calinescu
University of York, UK

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SEAMS '20

Sponsor:

SIGSOFT

SEAMS '20: IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems

June 29 - July 3, 2020

Seoul, Republic of Korea

Acceptance Rates

Overall Acceptance Rate 17 of 31 submissions, 55%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
233
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nazir RBucaioni APelliccione P(2024)Architecting ML-enabled systems: Challenges, best practices, and design decisionsJournal of Systems and Software10.1016/j.jss.2023.111860207(111860)Online publication date: Jan-2024
https://doi.org/10.1016/j.jss.2023.111860
Rapp MScheerer MReussner RFeng DBecker SHerbst NLeitner P(2022)Design-time Performability Optimization of Runtime Adaptation StrategiesCompanion of the 2022 ACM/SPEC International Conference on Performance Engineering10.1145/3491204.3527471(113-120)Online publication date: 14-Jul-2022
https://dl.acm.org/doi/10.1145/3491204.3527471
Serban AVisser J(2022)Adapting Software Architectures to Machine Learning Challenges2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00029(152-163)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00029
Elkhatib YElhabbash A(2021)If a System is Learning to Self-adapt, Who's Teaching?2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)10.1109/SEAMS51251.2021.00043(256-257)Online publication date: May-2021
https://doi.org/10.1109/SEAMS51251.2021.00043
Scheerer MReussner R(2021)Reliability Prediction of Self-Adaptive Systems Managing Uncertain AI Black-Box Components2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)10.1109/SEAMS51251.2021.00024(111-117)Online publication date: May-2021
https://doi.org/10.1109/SEAMS51251.2021.00024
Giray G(2021)A software engineering perspective on engineering machine learning systems: State of the art and challengesJournal of Systems and Software10.1016/j.jss.2021.111031(111031)Online publication date: Jun-2021
https://doi.org/10.1016/j.jss.2021.111031

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten