Skip to main content
SpringerLink
Log in

What Does It Mean to Explain? A User-Centered Study on AI Explainability

  • Conference paper
  • First Online:
Artificial Intelligence in HCI (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12797))

Included in the following conference series:

Abstract

One frequent concern associated with the development of AI models is their perceived lack of transparency. Consequently, the AI academic community has been active in exploring mathematical approaches that can increase the explainability of models. However, ensuring explainability thoroughly in the real world remains an open question. Indeed, besides data scientists, a variety of users is involved in the model lifecycle with varying motivations and backgrounds. In this paper, we sought to better characterize these explanations needs. Specifically, we conducted a user research study within a large institution that routinely develops and deploys AI model. Our analysis led to the identification of five explanation focuses and three standard user profiles that together enable to better describe what explainability means in real life. We also propose a mapping between explanation focuses and a set of existing explainability approaches as a way to link the user view and AI-born techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://eli5.readthedocs.io/en/latest/.

  2. 2.

    https://aix360.mybluemix.net/.

  3. 3.

    https://ethicalml.github.io/xai/index.html.

  4. 4.

    https://xai-aniti.github.io/ethik/.

  5. 5.

    https://www.ibm.com/cloud/watson-openscale.

References

  1. Ribeiro, M.T., Singh, S., Guestrin, C.: Why Should I Trust You?: Explaining the Predictions of Any Classifier (2016)

    Google Scholar 

  2. Lundberg, S.M., Lee, S.: A Unified Approach to Interpreting Model Predictions

    Google Scholar 

  3. Kim, B., Austin, U.T.: Examples are not enough, learn to criticize! criticism for interpretability. In: NIPS (2016)

    Google Scholar 

  4. Wang, D., Yang, Q., Abdul, A., Lim, B.Y.: Designing theory-driven user-centric explainable AI (2019). https://doi.org/10.1145/3290605.3300831

  5. Liao, Q.V., Gruen, D., Miller, S.: Questioning the AI: informing design practices for explainable AI user experiences, pp. 1–15 (2020). https://doi.org/10.1145/3313831.3376590

  6. Gunning, D.: Explainable artificial intelligence (XAI). In: Defense Advanced Research Projects Agency (DARPA) (2017)

    Google Scholar 

  7. Lim, B.Y., Dey, A.K.: Assessing demand for intelligibility in context-aware applications. In: ACM's International Conference Proceedings Series, pp. 195–204 (2009). https://doi.org/10.1145/1620545.1620576

  8. F. Doshi-Velez and B. Kim, “A Roadmap for a Rigorous Science of Interpretability,” no. Ml, pp. 1–13, 2017.

    Google Scholar 

  9. Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2018). https://doi.org/10.1016/j.artint.2018.07.007

    Article  MathSciNet  MATH  Google Scholar 

  10. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Pedreschi, D., Giannotti, F.:A survey of methods for explaining black box models, arXiv, pp. 1–45 (2018)

    Google Scholar 

  11. Chari, S., Seneviratne, O., Gruen, D.M., Foreman, M.A., Das, A.K., McGuinness, D.L.: Explanation ontology: a model of explanations for user-centered AI. In: Pan, J.Z., et al. (eds.) ISWC. LNCS, vol. 12507, pp. 228–243. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62466-8_15

    Chapter  Google Scholar 

  12. Barredo Arrieta, A., et al.: Explainable explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020). https://doi.org/10.1016/j.inffus.2019.12.012

  13. Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., Kagal, L.: Explaining explanations: An overview of interpretability of machine learning. In: Proceedings of 2018 IEEE 5th International Conference on Data Science and Advanced Analytics, DSAA 2018, pp. 80–89 (2019). https://doi.org/10.1109/DSAA.2018.00018

  14. Arya, V., et al.: One explanation does not fit all: a toolkit and taxonomy of AI explainability techniques (2019). http://arxiv.org/abs/1909.03012.

  15. Parekh, J., Mozharovskyi, P., d’Alche-Buc, F.: A framework to learn with interpretation (2020)

    Google Scholar 

  16. Sokol, K., Flach, P.: Explainability fact sheets: a framework for systematic assessment of explainable approaches. In: FAT*2020 - Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 56–67 (2020). https://doi.org/10.1145/3351095.3372870

  17. Belle, V., Papantonis, I.: Principles and practice of explainable machine learning. arXiv (2020)

    Google Scholar 

  18. Molnar, C., Casalicchio, G., Bischl, B.: Interpretable machine learning – a brief history, state-of-the-art and challenges. In: Koprinska, I., et al. (eds.) ECML. CCIS, vol. 1323, pp. 417–431. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65965-3_28

    Chapter  Google Scholar 

  19. Vilone, G., Longo, L.: Explainable artificial intelligence: a systematic review. arXiv, no. Dl (2020)

    Google Scholar 

  20. Freitas, A.A.: Comprehensible classification models. ACM SIGKDD Explor. Newsl. 15(1), 1 (2014). https://doi.org/10.1145/2594473.2594475

    Article  Google Scholar 

  21. Fisher, A., Rudin, C., Dominici, F.: All models are wrong, but many are useful: learning a variable’s importance by studying an entire class of prediction models simultaneously. arXiv (2018)

    Google Scholar 

  22. Apley, D.W., Zhu, J.: Visualizing the effects of predictor variables in black box supervised learning models. J. R. Stat. Soc. Ser. B Stat. Methodol. 82(4), 1059–1086 (2020). https://doi.org/10.1111/rssb.12377

  23. Zhao, Q., Hastie, T.: Causal interpretations of black-box models. Department of Statistics, Stanford University (2016)

    Google Scholar 

  24. Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: high-precision model-agnostic explanations. In: 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 1527–1535 (2018)

    Google Scholar 

  25. Goldstein, A., Kapelner, A., Bleich, J., Pitkin, E.: Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. J. Comput. Graph. Stat. 24(1), 44–65 (2015). https://doi.org/10.1080/10618600.2014.907095

    Article  MathSciNet  Google Scholar 

  26. Gurumoorthy, K.S., Dhurandhar, A., Cecchi, G., Aggarwal, C.: Efficient data representation by selecting prototypes with importance weights. In: Proceedings of IEEE International Conference on Data Mining, ICDM, vol. 2019, pp. 260–269 (2019). https://doi.org/10.1109/ICDM.2019.00036

  27. Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explanations without opening the black box: automated decisions and the GDPR. SSRN Electron. J. 31, 1–52 (2017). https://doi.org/10.2139/ssrn.3063289

  28. Dandl, S., Molnar, C., Binder, M., Bischl, B.: Multi-objective counterfactual explanations. In: Bäck, T., et al. (eds.) PPSN. LNCS, vol. 12269, pp. 448–469. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58112-1_31

    Chapter  Google Scholar 

  29. Samek, W., Wiegand, T., Müller, K.R.: Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. arXiv (2017)

    Google Scholar 

  30. Nunes, I., Jannach, D.: A systematic review and taxonomy of explanations in decision support and recommender systems

    Google Scholar 

  31. Hilton, D.J., Slugoski, B.R.: Knowledge-based causal attribution. The abnormal conditions focus model. Psychol. Rev. 93(1), 75–88 (1986). https://doi.org/10.1037/0033-295X.93.1.75

    Article  Google Scholar 

  32. Lim, B.Y., Dey, A.K.: Investigating intelligibility for uncertain context-aware applications. In: UbiComp 2011, Proceedings of the 2011 ACM Conference on Ubiquitous Computing, pp. 415–424 (2011). https://doi.org/10.1145/2030112.2030168

  33. Lim, B.Y., Dey, A.K.: Evaluating intelligibility usage and usefulness in a context-aware application. In: Kurosu, M. (ed.) HCI. LNCS, vol. 8008, pp. 92–101. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39342-6_11

    Chapter  Google Scholar 

  34. Krause, J., Perer, A., Ng, K.: Interacting with predictions: visual inspection of black-box machine learning models. In: Conference on Human Factors Computing Systems - Proceedings, pp. 5686–5697 (2016). https://doi.org/10.1145/2858036.2858529

  35. Coppers, S., et al.: Intellingo: an intelligible translation environment. In: Conference on Human Factors Computing System - Proc., vol. 2018-April, (2018). https://doi.org/10.1145/3173574.3174098

  36. Lim, B.Y., Dey, A.K.: Toolkit to support intelligibility in context-aware applications, p. 13 (2010). https://doi.org/10.1145/1864349.1864353

  37. Eiband, M., Schneider, H., Bilandzic, M., Fazekas-Con, J., Haug, M., Hussmann, H.: Bringing transparency design into practice, pp. 211–223 (2018). https://doi.org/10.1145/3172944.3172961

  38. Kulesza, C., Principles, S.: Principles of explanatory debugging to personalize interactive machine learning (2015). https://doi.org/10.1145/2678025.2701399

  39. Maria, R.: How to analyze qualitative data from UX research : thematic analysis. Nielsen Norman Group Publication (2019)

    Google Scholar 

  40. Ribera, M., Lapedriza, A.: Can we do better explanations? A proposal of user-centered explainable AI. In: CEUR Workshop Proceedings, vol. 2327 (2019)

    Google Scholar 

  41. Lim, B.Y., Dey, A.K.: Evaluating intelligibility usage and usefulness in a context-aware application

    Google Scholar 

  42. Lim, B.Y., Dey, A.K.: Investigating intelligibility for uncertain context-aware applications (2011)

    Google Scholar 

  43. Dhurandhar, A., et al.: Explanations based on the missing: towards contrastive explanations with pertinent negatives. arXiv, NeurIPS (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. BNP Paribas, Paris, France

    Lingxue Yang, Hongrun Wang & Léa A. Deleris

Authors
  1. Lingxue Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongrun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Léa A. Deleris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lingxue Yang .

Editor information

Editors and Affiliations

  1. Siemens Corporation, Princeton, NJ, USA

    Helmut Degen

  2. Foundation for Research and Technology – Hellas (FORTH), Heraklion, Greece

    Stavroula Ntoa

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, L., Wang, H., Deleris, L.A. (2021). What Does It Mean to Explain? A User-Centered Study on AI Explainability. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2021. Lecture Notes in Computer Science(), vol 12797. Springer, Cham. https://doi.org/10.1007/978-3-030-77772-2_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77772-2_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77771-5

  • Online ISBN: 978-3-030-77772-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation