Skip to main content

Advertisement

SpringerLink
Log in

An explanation space to align user studies with the technical development of Explainable AI

  • Open Forum
  • Published:
AI & SOCIETY Aims and scope Submit manuscript

Abstract

Providing meaningful and actionable explanations for end-users is a situated problem requiring the intersection of multiple disciplines to address social, operational, and technical challenges. However, the explainable artificial intelligence community has not commonly adopted or created tangible design tools that allow interdisciplinary work to develop reliable AI-powered solutions. This paper proposes a formative architecture that defines the explanation space from a user-inspired perspective. The architecture comprises five intertwined components to outline explanation requirements for a task: (1) the end-users’ mental models, (2) the end-users’ cognitive process, (3) the user interface, (4) the Human-Explainer Agent, and (5) the agent process. We first define each component of the architecture. Then, we present the Abstracted Explanation Space, a modeling tool that aggregates the architecture’s components to support designers in systematically aligning explanations with end-users’ work practices, needs, and goals. It guides the specifications of what needs to be explained (content: end-users’ mental model), why this explanation is necessary (context: end-users’ cognitive process), to delimit how to explain it (format: Human-Explainer Agent and user interface), and when the explanations should be given. We then exemplify the tool’s use in an ongoing case study in the aircraft maintenance domain. Finally, we discuss possible contributions of the tool, known limitations or areas for improvement, and future work to be done.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://aiir.nl/

  2. https://www.canada.ca/en/government/system/digital-government/digital-government-innovations/responsible-use-ai.html

  3. In this article, the word end-user refers to the actors who interact directly with the technical device.

  4. A user’s understanding and representation of a process, a phenomenon, or a system (Hoffman et al., 2018).

  5. See Hoffman et al. (2018), Wilson & Sharples (2015), and Milton (2007) for a complete description of existing data collection methods and techniques.

  6. https://www.mitacs.ca/en/projects/automated-visual-inspection-sentencing-dressing.

  7. This simplified process does not list the upstream tasks related to the preparation of the workstation, or the downstream operations after the response.

References

  • Akatsuka J, Yamamoto Y, Sekine T, Numata Y, Morikawa H, Tsutsumi K, Yanagi M, Endo Y, Takeda H, Hayashi T (2019) Illuminating clues of cancer buried in prostate mr image: deep learning and expert approaches. Biomolecules 9(11):673

    Article  Google Scholar 

  • Bisantz A, Roth EM, Watts-Englert J (2015) Study and analysis of complex cognitive work. In: Evaluation of Human Work, edited by John R. Wilson and Sarah Sharples, 61–82. CRC Press.

  • Cabitza F, Zeitoun J-D (2019) The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence. Ann Trans Med 7 (8). https://doi.org/10.21037/atm.2019.04.07.

  • Cabour G, Ledoux É, Bassetto S (2021a) Extending system performance past the boundaries of technical maturity: human-agent teamwork perspective for industrial inspection. In: Proceedings of the 21st Congress of the International Ergonomics Association (IEA 2021a), edited by Nancy L. Black, W. Patrick Neumann, and Ian Noy, 75–83. Cham: Springer International Publishing.

  • Cabour G, Ledoux É, Bassetto S (2021b) A work-centered approach for cyber-physical-social system design: applications in aerospace industrial inspection. ArXiv:2101.05385 [Cs], January.

  • Cabour G, Ledoux É, Bassetto S (2022) Aligning work analysis and modeling with the engineering goals of a cyber-physical-social system for industrial inspection. Appl Ergonomics.

  • Chen JYC, Barnes MJ (2014) Human–agent teaming for multirobot control: a review of human factors issues. IEEE Trans Hum-Mach Syst 44(1):13–29. https://doi.org/10.1109/THMS.2013.2293535

    Article  Google Scholar 

  • Chen JY, Procci K, Boyce M, Wright J, Garcia A, Barnes M (2014) Situation Awareness-Based Agent Transparency. Army research lab aberdeen proving ground md human research and engineering

  • Clancey WJ (1983) The epistemology of a rule-based expert system—a framework for explanation. Artif Intell 20(3):215–251. https://doi.org/10.1016/0004-3702(83)90008-5

    Article  Google Scholar 

  • Crabtree A, Rouncefield M, Tolmie P (2012) Doing design ethnography. Springer

    Book  Google Scholar 

  • Cutillo CM, Sharma KR, Foschini L, Kundu S, Mackintosh M, Mandl KD (2020) Machine intelligence in healthcare—perspectives on trustworthiness, explainability, usability, and transparency. Npjdigital Medicine 3(1):1–5. https://doi.org/10.1038/s41746-020-0254-2

    Article  Google Scholar 

  • Darius A, Damaševičius R (2014) Gamification of a Project Management System. In: Proc. of Int. conference on advances in computer-human interactions ACHI2014, 200–207. Citeseer.

  • Demir M, McNeese NJ, Cooke NJ (2020) Understanding human-robot teams in light of all-human teams: aspects of team interaction and shared cognition. Int J Hum Comput Stud 140:102436

    Article  Google Scholar 

  • Dhanorkar S, Wolf CT, Qian K, Xu A, Popa L, Li Y (2021) Who needs to know what, when?: Broadening the explainable AI (XAI) design space by looking at explanations across the ai lifecycle. In: Designing Interactive Systems Conference 2021, 1591–1602. Virtual Event USA: ACM. https://doi.org/10.1145/3461778.3462131.

  • Dong H, Kechen Song Yu, He JX, Yan Y, Meng Q (2019) PGA-Net: pyramid feature fusion and global context attention network for automated surface defect detection. IEEE Trans Industr Inf 16(12):7448–7458

    Article  Google Scholar 

  • Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpretable machine learning. ArXiv:1702.08608 [Cs, Stat], February. http://arxiv.org/abs/1702.08608.

  • Elsawah S, Guillaume JHA, Filatova T, Rook J, Jakeman AJ (2015) A methodology for eliciting, representing, and analysing stakeholder knowledge for decision making on complex socio-ecological systems: from cognitive maps to agent-based models. J Environ Manage 151(March):500–516. https://doi.org/10.1016/j.jenvman.2014.11.028

    Article  Google Scholar 

  • Endsley MR, Hoffman R, Kaber D, Roth E (2007) Cognitive engineering and decision making: an overview and future course. J Cognit Eng Decision Making 1(1):1–21

    Article  Google Scholar 

  • Fidel G, Bitton R, Shabtai A (2020) When Explainability Meets Adversarial Learning: Detecting Adversarial Examples Using SHAP Signatures. In: 2020 International Joint Conference on Neural Networks (IJCNN), 1–8. IEEE.

  • Friedman S, Forbus K, Sherin B (2018) Representing, running, and revising mental models: a computational model. Cogn Sci 42(4):1110–1145

    Article  Google Scholar 

  • Goh YM, Micheler S, Sanchez-Salas A, Case K, Bumblauskas D, Monfared R (2020) A variability taxonomy to support automation decision-making for manufacturing processes. Prod Planning Control 31(5):383–399

    Article  Google Scholar 

  • Government of Canada (2021). Responsible use of artificial intelligence (ai). Retrieved from https://www.canada.ca/en/government/system/digital-government/digital-government-innovations/responsible-use-ai.html

  • Haberfellner R, de Weck O, Fricke E, Vössner S (2019) Process models: systems engineering and others. In: Systems Engineering, 27–98. Springer.

  • Habli I, Lawton T, Porter Z (2020) Artificial intelligence in health care: accountability and safety. Bull World Health Organ 98(4):251–256. https://doi.org/10.2471/BLT.19.237487

    Article  Google Scholar 

  • He Yu, Song K, Meng Q, Yan Y (2019) An end-to-end steel surface defect detection approach via fusing multiple hierarchical features. IEEE Trans Instrum Meas 69(4):1493–1504

    Article  Google Scholar 

  • Henelius A, Puolamäki K, Ukkonen A (2017) Interpreting classifiers through attribute interactions in datasets. ArXiv Preprint ArXiv:1707.07576.

  • Hoffman RR, Mueller ST, Klein G, Litman J (2018) Metrics for explainable AI: challenges and prospects. ArXiv Preprint ArXiv:1812.04608.

  • Imran A, Jeon G, Piccialli F (2022) From artificial intelligence to explainable artificial intelligence in industry 4.0: a survey on what, how, and where. IEEE Trans Ind Inform 1–1. https://doi.org/10.1109/TII.2022.3146552

  • Johnson M, Bradshaw JM, Feltovich PJ (2017) Tomorrow’s human-machine design tools: from levels of automation to Interdependencies. J Cognit Eng Decision Making, October. https://doi.org/10.1177/1555343417736462

    Article  Google Scholar 

  • Johnson M, Bradshaw JM (2021) The role of interdependence in trust. In: Trust in Human-Robot Interaction, 379–403. Elsevier.

  • Klein G, Wright C (2016) Macrocognition: from theory to toolbox. Front Psychol 7. https://doi.org/10.3389/fpsyg.2016.00054

  • Kobrin JL, Sinharay S, Haberman SJ, Chajewski M (2011) An investigation of the fit of linear regression models to data from an SAT® validity study. ETS Research Report Series 2011(1):i–21

    Article  Google Scholar 

  • Konig R, Johansson U, Niklasson L (2008) G-REX: A versatile framework for evolutionary data mining. In: 2008 IEEE International Conference on Data Mining Workshops, 971–74. IEEE.

  • Lakkaraju H, Bach SH, Leskovec J (2016) Interpretable decision sets: a joint framework for description and prediction. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1675–84.

  • Langer M, Landers RN (2021) The future of artificial intelligence at work: a review on effects of decision automation and augmentation on workers targeted by algorithms and third-party observers. Comput Hum Behav 123(October):106878. https://doi.org/10.1016/j.chb.2021.106878

    Article  Google Scholar 

  • Lapuschkin S, Wäldchen S, Binder A, Montavon G, Samek W, Müller K-R (2019) Unmasking clever hans predictors and assessing what machines really learn. Nat Commun 10(1):1–8

    Article  Google Scholar 

  • Lewis M, Li H, Sycara K (2021) Deep learning, transparency, and trust in human robot teamwork. In: Trust in Human-Robot Interaction, 321–52. Elsevier.

  • Lockton D, Brawley L, Aguirre Ulloa M, Prindible M, Forlano L, Rygh K, Fass J, Herzog K, Nissen B (2019) Tangible thinking: materializing how we imagine and understand systems, experiences, and relationships.

  • Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. Adv Neural Inform Process Syst 30.

  • Marcus G, Davis E (2019) Rebooting AI: Building Artificial Intelligence We Can Trust. Vintage.

  • Matthews G, Panganiban AR, Lin J, Long M, Schwing M (2021) Super-Machines or Sub-Humans: Mental Models and Trust in Intelligent Autonomous Systems. In: Trust in Human-Robot Interaction, 59–82. Elsevier.

  • McMeekin N, Olivia Wu, Germeni E, Briggs A (2020) How methodological frameworks are being developed: evidence from a scoping review. BMC Med Res Methodol 20(1):1–9

    Article  Google Scholar 

  • Miller T (2019) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1–38

    Article  MathSciNet  MATH  Google Scholar 

  • Milton NR (2007) Knowledge Acquisition in Practice: A Step-by-Step Guide. Springer Science & Business Media.

  • Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B, Spitzer E, Deborah Raji I, Gebru T (2019) Model Cards for Model Reporting. In: Proceedings of the conference on fairness, accountability, and transparency, 220–29. FAT* ‘19. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3287560.3287596.

  • Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  • Mohseni S, Zarei N, Ragan ED (2021) A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Trans Interactive Intell Syst (TiiS) 11(3–4):1–45

    Google Scholar 

  • Moore J (1988) Explanation in expert systems: a survey. In.

  • Morales-Forero A, Bassetto S (2019) Case study: a semi-supervised methodology for anomaly detection and diagnosis. In: 2019 IEEE international conference on industrial engineering and engineering management (IEEM), 1031–37. IEEE.

  • Morales-Forero A, Bassetto S, Coatanea E (in press). Toward safe AI. AI & Society.

  • Mor-Yosef S, Samueloff A, Modan B, Navot D, Schenker JG (1990) Ranking the risk factors for cesarean: logistic regression analysis of a nationwide study. Obstet Gynecol 75(6):944–947

    Google Scholar 

  • Mueller ST, Hoffman RR, Clancey W, Emrey A, Klein G (2019) Explanation in human-ai systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. ArXiv:1902.01876 [Cs], February. http://arxiv.org/abs/1902.01876.

  • Mueller ST, Veinott ES, Hoffman RR, Klein G, Alam L, Mamun T, Clancey WJ (2021) Principles of explanation in human-AI systems ArXiv:2102.04972 [Cs], February. http://arxiv.org/abs/2102.04972.

  • Muller M, Wolf CT, Andres J, Desmond M, Joshi NN, Ashktorab Z, Sharma A et al. (2021) Designing ground truth and the social life of labels. In: Proceedings of the 2021 CHI conference on human factors in computing systems, 1–16. Yokohama Japan: ACM. https://doi.org/10.1145/3411764.3445402.

  • Naiseh M, Jiang N, Ma J, Ali R (2020) Personalising explainable recommendations: literature and conceptualisation. In: World conference on information systems and technologies, 518–33. Springer.

  • National academies of sciences, engineering, and medicine. 2021. Human-AI Teaming: State of the Art and Research Needs. Washington, DC: The National Academies Press. https://doi.org/10.17226/26355.

  • Nickerson RC, Varshney U, Muntermann J (2013) A method for taxonomy development and its application in information systems. Eur J Inf Syst 22(3):336–359

    Article  Google Scholar 

  • Nunes I, Jannach D (2017) A systematic review and taxonomy of explanations in decision support and recommender systems. User Model User-Adap Inter 27(3):393–444

    Article  Google Scholar 

  • Pekka AP, Bauer W, Bergmann U, Bieliková M, Bonefeld-Dahl C, Bonnet Y, Bouarfa L (2018) The European commission’s high-level expert group on artificial intelligence: ethics guidelines for trustworthy Ai. Working Document for Stakeholders’ Consultation. Brussels, 1–37.

  • Poursabzi-Sangdeh F, Goldstein DG, Hofman JM, Wortman Vaughan JW, Wallach H (2021) Manipulating and measuring model interpretability. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–52. CHI ‘21. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3411764.3445315.

  • Preece A, Harborne D, Braines D, Tomsett R, Chakraborty S (2018) Stakeholders in explainable AI. ArXiv Preprint ArXiv:1810.00184.

  • Rajabiyazdi F, Jamieson GA (2020) A review of transparency (seeing-into) models. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 302–8. https://doi.org/10.1109/SMC42975.2020.9282970.

  • Rathi S (2019) Generating counterfactual and contrastive explanations using SHAP.” ArXiv Preprint ArXiv:1906.09293.

  • Ribeiro MT, Singh S, Guestrin C (2016) Why Should I Trust You?’: Explaining the Predictions of Any Classifier. ArXiv:1602.04938 [Cs, Stat], August. http://arxiv.org/abs/1602.04938.

  • Ribera M, Lapedriza A (2019) Can we do better explanations? A proposal of user-centered explainable AI. Los Angeles, 7.

  • Rolls-Royce (2021) The aletheia framework. Retrieved 2021–07–01, from https://www.rolls-royce.com/sustainability/ethics-and-compliance/the-aletheia-framework.aspx

  • Roth EM, Bennett KB, Woods DD (1987) Human interaction with an ‘Intelligent’ machine. Int J Man Mach Stud 27(5):479–525. https://doi.org/10.1016/S0020-7373(87)80012-3

    Article  Google Scholar 

  • Roth EM, Bisantz AM, Wang X, Kim T, Hettinger AZ (2021) A work-centered approach to system user-evaluation. J Cognit Eng Decision Making 15(4):155–174

    Article  Google Scholar 

  • Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1(5):206–215

    Article  Google Scholar 

  • Salembier P, Wagner I (2021) Studies of work ‘in the wild.’ Computer Supported Cooperative Work (CSCW) 30(2):169–188

    Article  Google Scholar 

  • Sanneman L, Shah JA (2020) A situation awareness-based framework for design and evaluation of explainable AI. In: Calvaresi D, Najjar A, Winikoff M, Främling K (eds) Explainable, transparent autonomous agents and multi-agent systems. Springer International Publishing, Cham, pp 94–110

    Chapter  Google Scholar 

  • Seeber I, Bittner E, Briggs RO, de Vreede T, de Vreede G-J, Elkins A, Maier R et al (2020) Machines as teammates: a research agenda on AI in team collaboration. Information & Management 57(2):103174. https://doi.org/10.1016/j.im.2019.103174

    Article  Google Scholar 

  • Selbst AD, Boyd D, Friedler SA, Venkatasubramanian S, Vertesi J (2019) Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, 59–68. FAT* ‘19. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3287560.3287598.

  • Shadbolt N, Smart PR (2015) Knowledge elicitation: methods, tools and techniques. In: Evaluation of human work, edited by John R. Wilson and Sarah Sharples, 163–200. CRC Press. https://eprints.soton.ac.uk/359638/.

  • Shahri A, Hosseini M, Phalp K, Taylor J, Ali R (2014) Towards a code of ethics for gamification at enterprise. In: IFIP working conference on the practice of enterprise modeling, 235–45. Springer.

  • Shepherd A (2015) Task analysis. In: Evaluation of Human Work, 4th ed. CRC Press.

  • Shmelova T, Sterenharz A, Dolgikh S (2020) Artificial intelligence in aviation industries: methodologies, education, applications, and opportunities. In: Handbook of research on artificial intelligence applications in the aviation and aerospace industries, 1–35. IGI Global.

  • Shneiderman B (2020) Human-centered artificial intelligence: reliable, safe & trustworthy. Int J Hum-Comput Interact 36(6):495–504. https://doi.org/10.1080/10447318.2020.1741118

    Article  Google Scholar 

  • Song K, Yan Y (2013) A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects. Appl Surf Sci 285:858–864

    Article  Google Scholar 

  • St-Vincent M, Vézina N, Bellemare M, Denis D, Ledoux É, Imbeau D (2014) Ergonomic intervention. Institut de recherche Robert-Sauvé en santé et en sécurité du travail.

  • Tomsett R, Widdicombe A, Xing T, Chakraborty S, Julier S, Gurram P, Rao R, Srivastava M (2018) Why the failure? How adversarial examples can provide insights for interpretable machine learning. In: 2018 21st international conference on information fusion (FUSION), 838–45. IEEE.

  • Tramer F, Boneh D (2019) Adversarial training and robustness for multiple perturbations. Advances in Neural Information Processing Systems 32.

  • Vasey B, Clifton DA, Collins GS, Denniston AK, Faes L, Geerts BF, Liu X et al (2021) DECIDE-AI: New reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence. Nat Med 27(2):186–187. https://doi.org/10.1038/s41591-021-01229-5

    Article  Google Scholar 

  • Vicente KJ (1999) Cognitive work analysis: toward safe, productive, and healthy computer-based work. CRC Press

    Book  Google Scholar 

  • Wilson JR, Sharples S (2015) Evaluation of human work. CRC Press

    Book  Google Scholar 

  • Xiao C, Li B, Zhu J-Y, He W, Liu M, Song D (2018) Generating adversarial examples with adversarial networks. ArXiv Preprint ArXiv:1801.02610.

  • Yeung K (2020) Recommendation of the council on artificial intelligence (OECD). Int Leg Mater 59(1):27–34

    Article  Google Scholar 

  • Zouinar M (2020) Évolutions de l’Intelligence Artificielle : quels enjeux pour l’activité humaine et la relation Humain-Machine au travail? Activités, no. 17–1 (April). https://doi.org/10.4000/activites.4941.

  • Zsambok CE, Klein G (2014) Naturalistic decision making. Psychology Press

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Polytechnique Montreal, Montreal, Canada

    Garrick Cabour, Andrés Morales-Forero & Samuel Bassetto

  2. Université du Québec À Montréal, Montreal, Canada

    Élise Ledoux

Authors
  1. Garrick Cabour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrés Morales-Forero
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Élise Ledoux
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Samuel Bassetto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Garrick Cabour.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cabour, G., Morales-Forero, A., Ledoux, É. et al. An explanation space to align user studies with the technical development of Explainable AI. AI & Soc 38, 869–887 (2023). https://doi.org/10.1007/s00146-022-01536-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00146-022-01536-6

Keywords

Search

Navigation