research-article

Explanations for human-on-the-loop: a probabilistic model checking approach

Authors:

David GarlanAuthors Info & Claims

SEAMS '20: Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems

Pages 181 - 187

https://doi.org/10.1145/3387939.3391592

Published: 18 September 2020 Publication History

Abstract

Many self-adaptive systems benefit from human involvement and oversight, where a human operator can provide expertise not available to the system and can detect problems that the system is unaware of. One way of achieving this is by placing the human operator on the loop - i.e., providing supervisory oversight and intervening in the case of questionable adaptation decisions. To make such interaction effective, explanation is sometimes helpful to allow the human to understand why the system is making certain decisions and calibrate confidence from the human perspective. However, explanations come with costs in terms of delayed actions and the possibility that a human may make a bad judgement. Hence, it is not always obvious whether explanations will improve overall utility and, if so, what kinds of explanation to provide to the operator. In this work, we define a formal framework for reasoning about explanations of adaptive system behaviors and the conditions under which they are warranted. Specifically, we characterize explanations in terms of explanation content, effect, and cost. We then present a dynamic adaptation approach that leverages a probabilistic reasoning technique to determine when the explanation should be used in order to improve overall system utility.

References

[1]

B. H. C. Cheng and et al., "Software engineering for self-adaptive systems: A research roadmap" in Software Engineering for Self-Adaptive Systems. Lecture Notes in Computer Science, vol 5525. Springer, Berlin, Heidelberg.

[2]

R. de Lemos and et al., "Software engineering for self-adaptive systems: A second research roadmap," in Software Engineering for Self-Adaptive Systems II - International Seminar, Dagstuhl Castle, Germany, October 24--29, 2010 Revised Selected and Invited Papers, 2010, pp. 1--32.

[3]

R. Sukkerd, D. Garlan, and R. G. Simmons, "Task planning of cyber-human systems," in Software Engineering and Formal Methods - 13th International Conference, SEFM 2015, York, UK, September 7--11, 2015. Proceedings, 2015, pp. 293--309.

[4]

J. Cámara, G. A. Moreno, and D. Garlan, "Reasoning about human participation in self-adaptive systems," in 10th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS, Florence, Italy, May 18--19, 2015, 2015, pp. 146--156.

[5]

E. Lloyd, S. Huang, and E. Tognoli, "Improving human-in-the-loop adaptive systems using brain-computer interaction," in 12th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS@ICSE 2017, Buenos Aires, Argentina, May 22--23, 2017, 2017, pp. 163--174.

[6]

J. E. Fischer, C. Greenhalgh, W. Jiang, S. D. Ramchurn, F. Wu, and T. Rodden, "In-the-loop or on-the-loop? interactional arrangements to support team coordination with a planning agent," Concurrency and Computation: Practice and Experience, pp. 1--16, 2017.

[7]

O. Biran and C. Cotton, "Explanation and justification in machine learning: A survey" in IJCAI-17 workshop on explainable AI (XAI), vol. 8, 2017, p. 1.

[8]

T. Nomura and K. Kawakami, "Relationships between robot's self-disclosures and human's anxiety toward robots," in Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology-Volume 03. IEEE Computer Society, 2011, pp. 66--69.

[9]

M. Kwiatkowska, G. Norman, and D. Parker, Probabilistic Model Checking: Advances and Applications. Cham: Springer International Publishing, 2018, pp. 73--121.

[10]

"Tesla's trouble with semi trucks & another shakeup of the autopilot team --- is there a connection?" https://cleantechnica.com/2019/05/21/teslas-trouble-with-trucks-and-another-shakeup-of-the-autopilot-team-is-there-a-connection/, accessed: 2019-05-21.

[11]

S. Combéfis, D. Giannakopoulou, C. Pecheur, and M. Feary, "Learning system abstractions for human operators," in MALETS Proceedings of the International Workshop on Machine Learning Technologies in Software Engineering, 2011, pp. 3--10.

Digital Library

[12]

E. Palmer, "Oops, it didn't arm. - a case study of two automation surprises." in Proceedings of the 8th International Symposium on Aviation Psychology, 1996, pp. 227--232.

[13]

G. A. Moreno, J. Cámara, D. Garlan, and B. R. Schmerl, "Proactive self-adaptation under uncertainty: a probabilistic model checking approach," in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015, 2015, pp. 1--12.

Digital Library

[14]

A. Bianco and L. de Alfaro, "Model checking of probabilistic and nondeterministic systems," in Foundations of Software Technology and Theoretical Computer Science, P. S. Thiagarajan, Ed. Springer Berlin Heidelberg, 1995.

[15]

M. Z. Kwiatkowska, G. Norman, and D. Parker, "PRISM 4.0: Verification of probabilistic real-time systems," in Computer Aided Verification - 23rd International Conference, CAV, July 14--20, 2011. Proceedings, 2011, pp. 585--591.

[16]

M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, ser. Wiley Series in Probability and Statistics. Wiley, 1994.

[17]

G. Norman, D. Parker, and J. Sproston, "Model checking for probabilistic timed automata" Formal Methods in System Design, vol. 43, no. 2, pp. 164--190, 2013.

Digital Library

[18]

K. Soetaert, "plot3d : Tools for plotting 3-d and 2-d data." https://cran.r-project.org/web/packages/plot3D/vignettes/plot3D.pdf, 2018.

[19]

N. Li, A. Sridhar, E. Kang, and D. Garlan, "Analysis and synthesis of explanations for a secure industrial control system using probabilistic model checking," Institute for Software Research, Carnegie Mellon University, Tech. Rep. CMUISR-20--102, June 2020.

[20]

R. Sukkerd, "Improving transparency and understandability of multi- objective probabilistic planning," Thesis Proposal - School of Computer Science Institute for Software Research Software Engineering, Carnegie Mellon University, pp. 1--41, 2018.

[21]

B. Y. Lim, A. K. Dey, and D. Avrahami, "Why and why not explanations improve the intelligibility of context-aware intelligent systems," in Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, Boston, MA, USA, April 4--9, 2009, 2009, pp. 2119--2128.

[22]

S. Mahdavi-Hezavehi, V. H. S. Durelli, D. Weyns, and P. Avgeriou, "A systematic literature review on methods that handle multiple quality attributes in architecture-based self-adaptive systems," Information & Software Technology, vol. 90, pp. 1--26, 2017. [Online]. Available

[23]

T. Miller, "Explanation in artificial intelligence: Insights from the social sciences," Artif. Intell., vol. 267, pp. 1--38, 2019.

[24]

B. Chandrasekaran, M. C. Tanner, and J. R. Josephson, "Explaining control strategies in problem solving," IEEE Expert, vol. 4, no. 1, pp. 9--24, 1989.

Digital Library

[25]

B. Hayes and J. A. Shah, "Improving robot controller transparency through autonomous policy explanation," in 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI. IEEE, 2017, pp. 303--312.

[26]

R. W. Wohleber, K. Stowers, J. Y. Chen, and M. Barnes, "Effects of agent transparency and communication framing on human-agent teaming," in 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2017, pp. 3427--3432.

[27]

M. Harbers, K. Van Den Bosch, and J.-J. Meyer, "A methodology for developing self-explaining agents for virtual training," in International Workshop on Languages, Methodologies and Development Tools for Multi-Agent Systems. Springer, 2009, pp. 168--182.

[28]

F. Kaptein, J. Broekens, K. Hindriks, and M. Neerincx, "The role of emotion in self-explanations by cognitive agents," in 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 2017, pp. 88--93.

[29]

T. Kulesza, S. Stumpf, M. Burnett, S. Yang, I. Kwan, and W.-K. Wong, "Too much, too little, or just right? ways explanations impact end users' mental models," in 2013 IEEE Symposium on Visual Languages and Human Centric Computing. IEEE, 2013, pp. 3--10.

[30]

P. Carey., Data protection: a practical guide to UK and EU law. Oxford University Press, Inc., 2018.

[31]

T. Hellström and S. Bensch, "Understandable robots-what, why, and how" Paladyn, Journal of Behavioral Robotics, vol. 9, no. 1, pp. 110--123, 2018.

[32]

C. L. Bethel, "Robots without faces: non-verbal social human-robot interaction," 2009.

[33]

J. Broekens, M. Harbers, K. Hindriks, K. Van Den Bosch, C. Jonker, and J.-J. Meyer, "Do you get it? user-evaluated explainable bdi agents" in German Conference on Multiagent System Technologies. Springer 2010 pp. 28--39.

[34]

T. Chakraborti S. Sreedharan Y. Zhang and S. Kambhampati "Plan explanations as model reconciliation: Moving beyond explanation as soliloquy" in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19--25, 2017 2017 pp. 156--163.

[35]

F. Elizalde L. E. Sucar M. Luque J. Diez and A. Reyes "Policy explanation in factored markov decision processes." in In Proc European Workshop on Probabilistic Graphical Models (PGM) 2008 pp. 97--104.

[36]

O. Z. Khan, P. Poupart, and J. P. Black, "Minimal sufficient explanations for factored markov decision processes," in Proceedings of the 19th International Conference on Automated Planning and Scheduling, ICAPS 2009, Thessaloniki, Greece, September 19--23, 2009, 2009.

[37]

R. Sukkerd, R. G. Simmons, and D. Garlan, "Towards explainable multi-objective probabilistic planning," in Proceedings of the 4th International Workshop on Software Engineering for Smart Cyber-Physical Systems, ICSE 2018, Gothenburg, Sweden, May 27, 2018, 2018, pp. 19--25.

Cited By

Jähde OWeber TBuchkremer R(2025)Unraveling media perspectives: a comprehensive methodology combining large language models, topic modeling, sentiment analysis, and ontology learning to analyse media biasJournal of Computational Social Science10.1007/s42001-025-00372-08:2Online publication date: 25-Feb-2025
https://doi.org/10.1007/s42001-025-00372-0
Li JZhang MLi NWeyns DJin ZTei K(2024)Generative AI for Self-Adaptive Systems: State of the Art and Research RoadmapACM Transactions on Autonomous and Adaptive Systems10.1145/368680319:3(1-60)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3686803
Gavidia-Calderon CKordoni ABennaceur ALevine MNuseibeh B(2024)The IDEA of Us: An Identity-Aware Architecture for Autonomous SystemsACM Transactions on Software Engineering and Methodology10.1145/365443933:6(1-38)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3654439
Show More Cited By

Index Terms

Explanations for human-on-the-loop: a probabilistic model checking approach
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
      1. Probabilistic reasoning
2. Mathematics of computing
  1. Probability and statistics

Index terms have been assigned to the content through auto-classification.

Recommendations

Machine Explanations and Human Understanding
FAccT '23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency

Explanations are hypothesized to improve human understanding of machine learning models and achieve a variety of desirable outcomes, ranging from model debugging to enhancing human decision making. However, empirical studies have found mixed and even ...
Do Explanations Improve the Quality of AI-assisted Human Decisions? An Algorithm-in-the-Loop Analysis of Factual & Counterfactual Explanations
AAMAS '23: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems

The increased use of AI algorithmic aids in high-stakes decision making has prompted interest in explainable AI (xAI), and the role of counterfactual explanations to increase trust in human-algorithm collaborations and to mitigate unfair outcomes. ...
Counterfactual explanations for misclassified images: How human and machine explanations differ
Abstract
Counterfactual explanations have emerged as a popular solution for the eXplainable AI (XAI) problem of elucidating the predictions of black-box deep-learning systems because people easily understand them, they apply across different problem ...
Highlights
- A novel user-centric methodology where users create counterfactual explanations.
- Human-generated counterfactuals differ significantly from machine-generated ones.
- A novel interpretation of this difference using the notion of “...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SEAMS '20: Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems

June 2020

211 pages

ISBN:9781450379625

DOI:10.1145/3387939

General Chair:
Shinichi Honiden
Waseda University/National Institute of Informatics, Japan
,
Program Chairs:
Elisabetta Di Nitto
Politecnico di Milano, Italy
,
Radu Calinescu
University of York, UK

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SEAMS '20

Sponsor:

SIGSOFT

SEAMS '20: IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems

June 29 - July 3, 2020

Seoul, Republic of Korea

Acceptance Rates

Overall Acceptance Rate 17 of 31 submissions, 55%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
434
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)9

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jähde OWeber TBuchkremer R(2025)Unraveling media perspectives: a comprehensive methodology combining large language models, topic modeling, sentiment analysis, and ontology learning to analyse media biasJournal of Computational Social Science10.1007/s42001-025-00372-08:2Online publication date: 25-Feb-2025
https://doi.org/10.1007/s42001-025-00372-0
Li JZhang MLi NWeyns DJin ZTei K(2024)Generative AI for Self-Adaptive Systems: State of the Art and Research RoadmapACM Transactions on Autonomous and Adaptive Systems10.1145/368680319:3(1-60)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3686803
Gavidia-Calderon CKordoni ABennaceur ALevine MNuseibeh B(2024)The IDEA of Us: An Identity-Aware Architecture for Autonomous SystemsACM Transactions on Software Engineering and Methodology10.1145/365443933:6(1-38)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3654439
Li NZhang MLi JAdepu SKang EJin Z(2024)A Game-Theoretical Self-Adaptation Framework for Securing Software-Intensive SystemsACM Transactions on Autonomous and Adaptive Systems10.1145/365294919:2(1-49)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3652949
Negri FNicolosi NCamilli MMirandola RBaresi LMa XPasquale L(2024)Explanation-driven Self-adaptation using Model-agnostic Interpretable Machine LearningProceedings of the 19th International Symposium on Software Engineering for Adaptive and Self-Managing Systems10.1145/3643915.3644085(189-199)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643915.3644085
Castelnovo ACrupi RMombelli NNanino GRegoli D(2024)Evaluative Item-Contrastive Explanations in RankingsCognitive Computation10.1007/s12559-024-10311-216:6(3035-3050)Online publication date: 10-Jul-2024
https://doi.org/10.1007/s12559-024-10311-2
Yaqot MMenezes BMohammed AMoloney K(2024)A State-of-the-Art Review and Framework for Human-Centric Automation in Industry 5.0Advances in Production Management Systems. Production Management Systems for Volatile, Uncertain, Complex, and Ambiguous Environments10.1007/978-3-031-65894-5_27(385-400)Online publication date: 6-Sep-2024
https://doi.org/10.1007/978-3-031-65894-5_27
Dubslaff CKlös VPäßler J(2024)Template Decision Diagrams for Meta Control and ExplainabilityExplainable Artificial Intelligence10.1007/978-3-031-63797-1_12(219-242)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-63797-1_12
Cleland-Huang JChambers TZudaire SChowdhury MAgrawal AVierhauser M(2023)Human-Machine Teaming with small Unmanned Aerial Systems in a MAPE-K EnvironmentACM Transactions on Autonomous and Adaptive Systems10.1145/3618001Online publication date: 4-Sep-2023
https://dl.acm.org/doi/10.1145/3618001
Li NZhang MLi JKang ETei K(2023)Preference Adaptation: user satisfaction is all you need!2023 IEEE/ACM 18th Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)10.1109/SEAMS59076.2023.00027(133-144)Online publication date: May-2023
https://doi.org/10.1109/SEAMS59076.2023.00027
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten