Abstract
An algorithm effects a causal representation of relations between features and labels in the human’s perception. Such a representation might conflict with the human’s prior belief. Explanations can direct the human’s attention to the conflicting feature and away from other relevant features. This leads to causal overattribution and may adversely affect the human’s information processing. In a field experiment we implemented an XGBoost-trained model as a decision-making aid for counselors at a public employment service to predict candidates’ risk of long-term unemployment. The treatment group of counselors was also provided with SHAP. The results show that the quality of the human’s decision-making is worse when a feature on which the human holds a conflicting prior belief is displayed as part of the explanation.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Setting X to \(x_0\), Y would be \(y_0\).
- 2.
Had X been \(x_0\), Y would have been \(y_0\).
References
Bundorf, K., Polyakova, M., Tai-Seale, M.: How do humans interact with algorithms? experimental evidence from health insurance. Technical report, National Bureau of Economic Research (2019)
Colin, J., Fel, T., Cadène, R., Serre, T.: What i cannot predict, i do not understand: a human-centered evaluation framework for explainability methods. Adv. Neural. Inf. Process. Syst. 35, 2832–2845 (2022)
Cyert, R.M., March, J.G., et al.: A Behavioral Theory of the Firm, vol. 2. Englewood Cliffs, NJ (1963)
DeLanda, M.: Materialist Phenomenology: A Philosophy of Perception. Bloomsbury Publishing (2021)
Dietvorst, B.J., Simmons, J.P., Massey, C.: Algorithm aversion: people erroneously avoid algorithms after seeing them err. J. Exp. Psychol. Gen. 144(1), 114 (2015)
Dietvorst, B.J., Simmons, J.P., Massey, C.: Overcoming algorithm aversion: people will use imperfect algorithms if they can (even slightly) modify them. Manage. Sci. 64(3), 1155–1170 (2018)
Fiske, S.T., Kenny, D.A., Taylor, S.E.: Structural models for the mediation of salience effects on attribution. J. Exp. Soc. Psychol. 18(2), 105–127 (1982)
Fügener, A., Grahl, J., Gupta, A., Ketter, W.: Will humans-in-the-loop become borgs? merits and pitfalls of working with AI. Manage. Inf. Syst. Q. (MISQ)-Vol 45 (2021)
Fügener, A., Grahl, J., Gupta, A., Ketter, W.: Cognitive challenges in human-artificial intelligence collaboration: investigating the path toward productive delegation. Inf. Syst. Res. 33(2), 678–696 (2022)
Galhotra, S., Pradhan, R., Salimi, B.: Explaining black-box algorithms using probabilistic contrastive counterfactuals. In: Proceedings of the 2021 International Conference on Management of Data, pp. 577–590 (2021)
Gao, R., Saar-Tsechansky, M., De-Arteaga, M., Han, L., Lee, M.K., Lease, M.: Human-AI collaboration with bandit feedback. arXiv preprint arXiv:2105.10614 (2021)
Gigerenzer, G., Goldstein, D.G.: Reasoning the fast and frugal way: models of bounded rationality. Psychol. Rev. 103(4), 650 (1996)
Gigerenzer, G., Todd, P., Group, A.: Simple heuristics that make us smart (1999)
Glikson, E., Woolley, A.W.: Human trust in artificial intelligence: review of empirical research. Acad. Manag. Ann. 14(2), 627–660 (2020)
Guidotti, R.: Evaluating local explanation methods on ground truth. Artif. Intell. 291, 103428 (2021)
Han, T., Srinivas, S., Lakkaraju, H.: Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. arXiv preprint arXiv:2206.01254 (2022)
Jacovi, A., Marasović, A., Miller, T., Goldberg, Y.: Formalizing trust in artificial intelligence: prerequisites, causes and goals of human trust in AI. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 624–635 (2021)
Kawaguchi, K.: When will workers follow an algorithm? a field experiment with a retail business. Manage. Sci. 67(3), 1670–1695 (2021)
Lagnado, D.A.: Explaining the Evidence: How the Mind Investigates the World. Cambridge University Press, Cambridge (2021). https://doi.org/10.1017/9780511794520
Lebovitz, S., Lifshitz-Assaf, H., Levina, N.: To engage or not to engage with AI for critical judgments: how professionals deal with opacity when using AI for medical diagnosis. Organ. Sci. 33(1), 126–148 (2022)
Lombrozo, T.: Simplicity and probability in causal explanation. Cogn. Psychol. 55(3), 232–257 (2007)
Lu, H., Yuille, A.L., Liljeholm, M., Cheng, P.W., Holyoak, K.J.: Bayesian generic priors for causal learning. Psychol. Rev. 115(4), 955 (2008)
Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, pp. 4765–4774 (2017)
Nauta, M., et al.: From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. arXiv preprint arXiv:2201.08164 (2022)
Pearl, J., Mackenzie, D.: The Book of Why: The New Science of Cause and Effect. Basic books (2018)
Ribeiro, M.T., Singh, S., Guestrin, C.: " why should i trust you?" explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1135–1144 (2016)
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)
Simon, H.A.: Administrative Behavior. Simon and Schuster, New York (2013)
Slack, D.Z., Hilgard, S., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: modeling uncertainty in explainability. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021). https://openreview.net/forum?id=rqfq0CYIekd
Sun, J., Zhang, D.J., Hu, H., Van Mieghem, J.A.: Predicting human discretion to adjust algorithmic prescription: a large-scale field experiment in warehouse operations. Manage. Sci. 68(2), 846–865 (2022)
Taylor, S.E., Crocker, J., Fiske, S.T., Sprinzen, M., Winkler, J.D.: The generalizability of salience effects. J. Pers. Soc. Psychol. 37(3), 357 (1979)
Taylor, S.E., Fiske, S.T.: Point of view and perceptions of causality. J. Pers. Soc. Psychol. 32(3), 439 (1975)
Taylor, S.E., Fiske, S.T.: Salience, attention, and attribution: top of the head phenomena. In: Advances in Experimental Social Psychology, vol. 11, pp. 249–288. Elsevier (1978)
Ullman, D., Malle, B.F.: What does it mean to trust a robot? steps toward a multidimensional measure of trust. In: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 263–264 (2018)
Wan, C., Belo, R., Zejnilovic, L.: Explainability’s gain is optimality’s loss? how explanations bias decision-making. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 778–787 (2022)
Woodward, J.: Causation with A Human Face: Normative Theory and Descriptive Psychology. Oxford University Press, Oxford (2021)
Xiang, Y., Vélez, N., Gershman, S.J.: Collaborative decision making is grounded in representations of other people’s competence and effort (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wan, C., Belo, R., Zejnilović, L., Lavado, S. (2023). The Duet of Representations and How Explanations Exacerbate It. In: Longo, L. (eds) Explainable Artificial Intelligence. xAI 2023. Communications in Computer and Information Science, vol 1902. Springer, Cham. https://doi.org/10.1007/978-3-031-44067-0_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-44067-0_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44066-3
Online ISBN: 978-3-031-44067-0
eBook Packages: Computer ScienceComputer Science (R0)