Skip to main content
SpringerLink
Log in

Color for Characters - Effects of Visual Explanations of AI on Trust and Observability

  • Conference paper
  • First Online:
Artificial Intelligence in HCI (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12217))

Included in the following conference series:

Abstract

The present study investigates the effects of prototypical visualization approaches aimed at increasing the explainability of machine learning systems in regard to perceived trustworthiness and observability. As the amount of processes automated by artificial intelligence (AI) increases, so does the need to investigate users’ perception. Previous research on explainable AI (XAI) tends to focus on technological optimization. The limited amount of empirical user research leaves key questions unanswered, such as which XAI designs actually improve perceived trustworthiness and observability. We assessed three different visual explanation approaches, consisting of either only a table with classification scores used for classification, or, additionally, one of two different backtraced visual explanations. In a within-subjects design with N = 83 we examined the effects on trust and observability in an online experiment. While observability benefitted from visual explanations, information-rich explanations also led to decreased trust. Explanations can support human-AI interaction, but differentiated effects on trust and observability have to be expected. The suitability of different explanatory approaches for individual AI applications should be further examined to ensure a high level of trust and observability in e.g. automated image processing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access. 6, 52138–52160 (2018). https://doi.org/10.1109/ACCESS.2018.2870052

    Article  Google Scholar 

  2. Weld, D.S., Bansal, G.: The Challenge of Crafting Intelligible Intelligence. ArXiv180304263 Cs. (2018)

    Google Scholar 

  3. Ancona, M., Ceolini, E., Öztireli, C., Gross, M.: Towards better understanding of gradient-based attribution methods for Deep Neural Networks. ArXiv171106104 Cs Stat. (2017)

    Google Scholar 

  4. Lee, J.D., See, K.A.: Trust in automation: designing for appropriate reliance. Hum. Factors 46, 50–80 (2004)

    Article  Google Scholar 

  5. Lee, J., Moray, N.: Trust, control strategies and allocation of function in human-machine systems. Ergonomics 35, 1243–1270 (1992). https://doi.org/10.1080/00140139208967392

    Article  Google Scholar 

  6. Muir, B.M., Moray, N.: Trust in automation. Part II experimental studies of trust and human intervention in a process control simulation. Ergonomics. 39, 429–460 (1996). https://doi.org/10.1080/00140139608964474

  7. Nushi, B., Kamar, E., Horvitz, E.: Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure. ArXiv180907424 Cs Stat. (2018)

    Google Scholar 

  8. Lim, B.Y., Dey, A.K.: Assessing demand for intelligibility in context-aware applications. In: Proceedings of the 11th international conference on Ubiquitous computing (Ubicomp 2009). p. 195. ACM Press, Orlando (2009). https://doi.org/10.1145/1620545.1620576

  9. Montavon, G., Samek, W., Müller, K.-R.: Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 73, 1–15 (2018). https://doi.org/10.1016/j.dsp.2017.10.011

    Article  MathSciNet  Google Scholar 

  10. Ribeiro, M.T., Singh, S., Guestrin, C.: “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. ArXiv160204938 Cs Stat. (2016)

    Google Scholar 

  11. Kruger, J., Wirtz, D., Van Boven, L., Altermatt, T.W.: The effort heuristic. J. Exp. Soc. Psychol. 40, 91–98 (2004). https://doi.org/10.1016/S0022-1031(03)00065-9

    Article  Google Scholar 

  12. Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: an HCI research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI 2018), pp. 1–18. ACM Press, Montreal QC (2018). https://doi.org/10.1145/3173574.3174156

  13. Amershi, S., et al.: Guidelines for human-AI interaction. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI 2019), pp. 1–13. ACM Press, Glasgow (2019). https://doi.org/10.1145/3290605.3300233

  14. Miller, T.: Explanation in Artificial Intelligence: Insights from the Social Sciences. ArXiv170607269 Cs. (2017)

    Google Scholar 

  15. Lapuschkin, S., Binder, A., Montavon, G., Müller, K.-R., Samek, W.: The LRP toolbox for artificial neural networks. J. Mach. Learn. Res. 17(1), 3938–3942 (2016)

    Google Scholar 

  16. Hoffman, R.R., Mueller, S.T., Klein, G., Litman, J.: Metrics for explainable AI: Challenges and prospects. ArXiv Prepr. ArXiv181204608. (2018)

    Google Scholar 

  17. Ras, G., van Gerven, M., Haselager, P.: Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges. ArXiv180307517 Cs Stat. (2018)

    Google Scholar 

  18. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626. IEEE, Venice (2017). https://doi.org/10.1109/ICCV.2017.74

  19. Samek, W., Wiegand, T., Müller, K.-R.: Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models. ArXiv170808296 Cs Stat. (2017)

    Google Scholar 

  20. Binder, A., Bach, S., Montavon, G., Müller, K.-R., Samek, W.: Layer-Wise Relevance Propagation for Deep Neural Network Architectures. Information Science and Applications (ICISA) 2016. LNEE, vol. 376, pp. 913–922. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-0557-2_87

    Chapter  Google Scholar 

  21. Timmermans, D.: The impact of task complexity on information use in multi-attribute decision making. J. Behav. Decis. Mak. 6, 95–111 (1993). https://doi.org/10.1002/bdm.3960060203

  22. Furner, C.P., Zinko, R.A.: The influence of information overload on the development of trust and purchase intention based on online product reviews in a mobile vs. web environment: an empirical investigation. Electron. Mark. 27, 211–224 (2017). https://doi.org/10.1007/s12525-016-0233-2

  23. Roese, N.J., Morrison, M.: The psychology of counterfactual thinking. Hist. Soc. Res. Sozialforschung 16–26 (2009)

    Google Scholar 

  24. Sokol, K., Flach, P.: Glass-Box: explaining AI decisions with counterfactual statements through conversation with a voice-enabled virtual assistant. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence Organization, Stockholm, Sweden, pp. 5868–5870 (2018). https://doi.org/10.24963/ijcai.2018/865

  25. Kulesza, T., Stumpf, S., Burnett, M., Kwan, I.: Tell me more? The effects of mental model soundness on personalizing an intelligent agent. In: Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems (CHI 2012), p. 1. ACM Press, Austin (2012). https://doi.org/10.1145/2207676.2207678

  26. Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., Lee, S.: Counterfactual Visual Explanations. ArXiv190407451 Cs Stat. (2019)

    Google Scholar 

  27. Bigras, E., et al.: In AI we trust: characteristics influencing assortment planners’ perceptions of AI based recommendation agents. In: Nah, F.F.-H., Xiao, B.S. (eds.) HCIBGO 2018. LNCS, vol. 10923, pp. 3–16. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91716-0_1

    Chapter  Google Scholar 

  28. Breuer, C., Hüffmeier, J., Hibben, F., Hertel, G.: Trust in teams: a taxonomy of perceived trustworthiness factors and risk-taking behaviors in face-to-face and virtual teams. Hum. Relat. (2019). https://doi.org/10.1177/0018726718818721

  29. Zanker, M.: The influence of knowledgeable explanations on users’ perception of a recommender system. In: Proceedings of the sixth ACM conference on Recommender systems (RecSys 2012), p. 269. ACM Press, Dublin (2012). https://doi.org/10.1145/2365952.2366011

  30. Springer, A., Whittaker, S.: “I had a solid theory before but it’s falling apart”: polarizing effects of algorithmic transparency. arXiv preprint arXiv:1811.02163 (2018)

    Google Scholar 

  31. Hengstler, M., Enkel, E., Duelli, S.: Applied artificial intelligence and trust—The case of autonomous vehicles and medical assistance devices. Technol. Forecast. Soc. Change. 105, 105–120 (2016). https://doi.org/10.1016/j.techfore.2015.12.014

    Article  Google Scholar 

  32. Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N.: Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2015), pp. 1721–1730. ACM Press, Sydney (2015). https://doi.org/10.1145/2783258.2788613

  33. Krause, J., Perer, A., Bertini, E.: A user study on the effect of aggregating explanations for interpreting machine learning models. In: ACM KDD Workshop on Interactive Data Exploration and Analytics (2018)

    Google Scholar 

  34. Kalman, R.E.: A new approach to linear filtering and prediction problems. J. Basic Eng. 82, 35–45 (1960)

    Article  MathSciNet  Google Scholar 

  35. Billings, C.E.: Human-centered aviation automation: principles and guidelines (1996)

    Google Scholar 

  36. Johnson, M., Bradshaw, J.M., Feltovich, P.J.: Tomorrow’s human–machine design tools: from levels of automation to interdependencies. J. Cogn. Eng. Decis. Mak. 12, 77–82 (2018). https://doi.org/10.1177/1555343417736462

    Article  Google Scholar 

  37. Rovira, E., McGarry, K., Parasuraman, R.: Effects of imperfect automation on decision making in a simulated command and control task. Hum. Factors J. Hum. Factors Ergon. Soc. 49, 76–87 (2007). https://doi.org/10.1518/001872007779598082

  38. Franke, T., Attig, C., Wessel, D.: A personal resource for technology interaction development and validation of the affinity for technology interaction (ATI) Scale. Int. J. Hum.-Comput. Inter. 35, 456–467 (2019). https://doi.org/10.1080/10447318.2018.1456150

    Article  Google Scholar 

  39. Deng, L.: The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. 29, 141–142 (2012). https://doi.org/10.1109/MSP.2012.2211477

    Article  Google Scholar 

  40. Franke, T., Trantow, M., Günther, M., Krems, J.F., Zott, V., Keinath, A.: Advancing electric vehicle range displays for enhanced user experience: the relevance of trust and adaptability. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI 2015). pp. 249–256. ACM Press, Nottingham (2015). https://doi.org/10.1145/2799250.2799283

  41. Jian, J.-Y., Bisantz, A.M., Drury, C.G.: Foundations for an empirically determined scale of trust in automated systems. Int. J. Cogn. Ergon. 4, 53–71 (2000). https://doi.org/10.1207/S15327566IJCE0401_04

    Article  Google Scholar 

  42. Mauchly, J.W.: Significance test for sphericity of a normal n-Variate distribution. Ann. Math. Stat. 11, 204–209 (1940). https://doi.org/10.1214/aoms/1177731915

    Article  MathSciNet  MATH  Google Scholar 

  43. Greenhouse, S.W., Geisser, S.: On methods in the analysis of profile data. Psychometrika 24, 95–112 (1959). https://doi.org/10.1007/BF02289823

    Article  MathSciNet  MATH  Google Scholar 

  44. Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979)

    MathSciNet  MATH  Google Scholar 

  45. Cohen, J.: A power primer. Psychol. Bull. 112, 155–159 (1992). https://doi.org/10.1037/0033-2909.112.1.155

    Article  Google Scholar 

  46. Dunlap, W.P., Cortina, J.M., Vaslow, J.B., Burke, M.J.: Meta-analysis of experiments with matched groups or repeated measures designs. Psychol. Methods 1(2), 170 (1996)

    Article  Google Scholar 

  47. Kizilcec, R.F.: How much information? Effects of transparency on trust in an algorithmic interface. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016). pp. 2390–2395. ACM Press, Santa Clara (2016). https://doi.org/10.1145/2858036.2858402

  48. Biros, D.P., Fields, G., Gunsch, G.: The effect of external safeguards on human-information system trust in an information warfare environment. In: Proceedings of the 36th Annual Hawaii International Conference on System Sciences 2003, p. 10. IEEE, Big Island (2003). https://doi.org/10.1109/HICSS.2003.1173894

  49. Christoffersen, K., Woods, D.: How to make automated systems team players. Adv. Hum. Perform. Cogn. Eng. Res. pp. 1–12 (2002). https://doi.org/10.1016/S1479-3601(02)02003-9

  50. Mueller, S.T., Hoffman, R.R., Clancey, W., Emrey, A., Klein, G.: Explanation in human-AI systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. arXiv preprint arXiv:1902.01876 (2019)

    Google Scholar 

  51. Hoffman, R.R., Klein, G., Mueller, S.T.: Explaining explanation for “Explainable Ai”. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 62, 197–201 (2018). https://doi.org/10.1177/1541931218621047

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Universität zu Lübeck, Ratzeburger Allee 160, 23562, Lübeck, Germany

    Tim Schrills & Thomas Franke

Authors
  1. Tim Schrills
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Franke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tim Schrills .

Editor information

Editors and Affiliations

  1. Siemens, Princeton, USA

    Helmut Degen

  2. University of Central Florida, Orlando, FL, USA

    Lauren Reinerman-Jones

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Schrills, T., Franke, T. (2020). Color for Characters - Effects of Visual Explanations of AI on Trust and Observability. In: Degen, H., Reinerman-Jones, L. (eds) Artificial Intelligence in HCI. HCII 2020. Lecture Notes in Computer Science(), vol 12217. Springer, Cham. https://doi.org/10.1007/978-3-030-50334-5_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-50334-5_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-50333-8

  • Online ISBN: 978-3-030-50334-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation