Skip to main content
SpringerLink
Log in

Let Me Think! Investigating the Effect of Explanations Feeding Doubts About the AI Advice

  • Conference paper
  • First Online:
Machine Learning and Knowledge Extraction (CD-MAKE 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14065))

Abstract

Augmented Intelligence (AuI) refers to the use of artificial intelligence (AI) to amplify certain cognitive tasks performed by human decision-makers. However, there are concerns that AI’s increasing capability and alignment with human values may undermine user agency, autonomy, and responsible decision-making. To address these concerns, we conducted a user study in the field of orthopedic radiology diagnosis, introducing a reflective XAI (explainable AI) support that aimed to stimulate human reflection, and we evaluated its impact of in terms of decision performance, decision confidence and perceived utility. Specifically, the reflective XAI support system prompted users to reflect on the dependability of AI-generated advice by presenting evidence both in favor of and against its recommendation. This evidence was presented via two cases that closely resembled a given base case, along with pixel attribution maps. These cases were associated with the same AI advice for the base case, but one case was accurate while the other was erroneous with respect to the ground truth. While the introduction of this support system did not significantly enhance diagnostic accuracy, it was highly valued by more experienced users. Based on the findings of this study, we advocate for further research to validate the potential of reflective XAI in fostering more informed and responsible decision-making, ultimately preserving human agency.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    This performance was set since in the previous study described above, this was found to be the average accuracy of the expert x-rays readers therein involved. Human average accuracy was then 78% (N = 16), SD 7%, median 80%, max 89%, min 67%.

References

  1. Arnott, D., Gao, S.: Behavioral economics for decision support systems researchers. Decis. Support Syst. 122, 113063 (2019)

    Article  Google Scholar 

  2. Baselli, G., Codari, M., Sardanelli, F.: Opening the black box of machine learning in radiology: can the proximity of annotated cases be a way? European Radiology Experimental 4, 1–7 (2020). https://doi.org/10.1186/s41747-020-00159-0

    Article  Google Scholar 

  3. Bertrand, A., Belloum, R., Eagan, J.R., Maxwell, W.: How cognitive biases affect XAI-assisted decision-making: a systematic review. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 78–91 (2022)

    Google Scholar 

  4. Bhandari, M., Reddiboina, M.: Augmented intelligence: a synergy between man and the machine. Indian J. Urol. IJU: J. Urol. Soc. India 35(2), 89 (2019)

    Google Scholar 

  5. Bhattacharya, C., et al.: An application of the Mann-Whitney “U” test. Technical report. Indian Institute of Management Ahmedabad, Research and Publication Department (1981)

    Google Scholar 

  6. Buçinca, Z., Malaya, M.B., Gajos, K.Z.: To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proc. ACM Hum.-Comput. Interact. 5(CSCW1), 1–21 (2021)

    Article  Google Scholar 

  7. Byrne, R.M.: Counterfactuals in explainable artificial intelligence (XAI): evidence from human reasoning. In: IJCAI, pp. 6276–6282 (2019)

    Google Scholar 

  8. Cabitza, F., Campagner, A., Angius, R., Natali, C., Reverberi, C.: AI shall have no dominion: on how to measure technology dominance in AI-supported human decision-making. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI 2023). Association for Computing Machinery, New York, USA, Article 354, pp. 1–20 (2023). https://doi.org/10.1145/3544548.3581095

  9. Cabitza, F., Campagner, A., Natali, C., Famiglini, L., Caccavella, V., Gallazzi, E.: Never tell me the odds. Investigating the concept of similarity and its use in pro-hoc explanations in radiological AI settings (2023, Submitted)

    Google Scholar 

  10. Cabitza, F.: Cobra AI: exploring some unintended consequences of our most powerful technology. In: Machines We Trust: Perspectives on Dependable AI, MIT Press (2021). ISBN 978-0262542098

    Google Scholar 

  11. Cabitza, F., et al.: Quod erat demonstrandum?-Towards a typology of the concept of explanation for the design of explainable AI. Expert Syst. Appl. 213, 118888 (2023)

    Article  Google Scholar 

  12. Cabitza, F., et al.: Rams, hounds and white boxes: investigating human-AI collaboration protocols in medical diagnosis. Artif. Intell. Med. 138, 102506 (2023)

    Article  Google Scholar 

  13. Cabitza, F., Campagner, A., Simone, C.: The need to move away from agential-AI: empirical investigations, useful concepts and open issues. Int. J. Hum Comput Stud. 155, 102696 (2021)

    Article  Google Scholar 

  14. Carroll, J.: Completing design in use: closing the appropriation cycle. In: ECIS 2004 Proceedings, p. 44 (2004)

    Google Scholar 

  15. Chalmers, M.: Seamful design and ubicomp infrastructure. In: Proceedings of Ubicomp 2003 Workshop at the Crossroads: The Interaction of HCI and Systems Issues in Ubicomp, pp. 577–584 (2003)

    Google Scholar 

  16. Chen, H., Gomez, C., Huang, C.M., Unberath, M.: Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review. NPJ Digit. Med. 5(1), 156 (2022)

    Article  Google Scholar 

  17. Cornelissen, N.A.J., van Eerdt, R.J.M., Schraffenberger, H.K., Haselager, W.F.G.: Reflection machines: increasing meaningful human control over Decision Support Systems. Ethics Inf. Technol. 24 (2022). Article number: 19. https://doi.org/10.1007/s10676-022-09645-y

  18. Dai, X., Fishbach, A.: When waiting to choose increases patience. Organ. Behav. Hum. Decis. Process. 121(2), 256–266 (2013)

    Article  Google Scholar 

  19. Delacre, M., Lakens, D., Leys, C.: Why Psychologists Should By Default Use Welch’s t-test instead of Student’s t-test. Int. Rev. Soc. Psychol. 30(1), 92–101 (2017)

    Article  Google Scholar 

  20. Fonteyn, M.E., Kuipers, B., Grobe, S.J.: A description of think aloud method and protocol analysis. Qual. Health Res. 3(4), 430–441 (1993)

    Article  Google Scholar 

  21. Frischmann, B., Selinger, E.: Re-Engineering Humanity. Cambridge University Press, Cambridge (2018)

    Book  Google Scholar 

  22. Gallazzi, E., Famiglini, L., La Maida, G., Giorgi, P., Misaggi, B., Cabitza, F.: Coloured shadows: understanding the value of visual aided diagnosis through AI-generated saliency maps. In: Orthopaedic Proceedings, vol. 104. The British Editorial Society of Bone & Joint Surgery (2022)

    Google Scholar 

  23. Gray, C.M., Kou, Y., Battles, B., Hoggatt, J., Toombs, A.L.: The dark (patterns) side of UX design. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2018)

    Google Scholar 

  24. Gur, D., et al.: The “laboratory’’ effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 249(1), 47–53 (2008)

    Article  Google Scholar 

  25. Jaiswal, A., Arun, C.J., Varma, A.: Rebooting employees: upskilling for artificial intelligence in multinational corporations. Int. J. Hum. Resour. Manag. 33(6), 1179–1208 (2022)

    Article  Google Scholar 

  26. Karlsen, T.K., Oppen, M.: Professional knowledge and the limits of automation in administrations. In: Göranzon, B., Josefson, I. (eds.) Knowledge, Skill and Artificial Intelligence. HCS, pp. 139–149. Springer, London (1988). https://doi.org/10.1007/978-1-4471-1632-5_13

    Chapter  Google Scholar 

  27. Keane, M.: Analogical mechanisms. Artif. Intell. Rev. 2(4), 229–251 (1988). https://doi.org/10.1007/BF00138817

    Article  Google Scholar 

  28. Kelley, K., Preacher, K.J.: On effect size. Psychol. Methods 17(2), 137 (2012)

    Article  Google Scholar 

  29. Kenny, E.M., Ford, C., Quinn, M., Keane, M.T.: Explaining black-box classifiers using post-hoc explanations-by-example: the effect of explanations and error-rates in XAI user studies. Artif. Intell. 294, 103459 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  30. Krug, S.: Don’t Make Me Think!: A Common Sense Approach to Web Usability. Pearson Education India (2000)

    Google Scholar 

  31. Miller, T.: Explainable AI is dead, long live explainable AI! Hypothesis-driven decision support. arXiv preprint arXiv:2302.12389 (2023)

  32. Ohm, P., Frankle, J.: Desirable inefficiency. Fla. L. Rev. 70, 777 (2018)

    Google Scholar 

  33. Prabhudesai, S., Yang, L., Asthana, S., Huan, X., Liao, Q.V., Banovic, N.: Understanding uncertainty: how lay decision-makers perceive and interpret uncertainty in human-AI decision making. In: Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 379–396 (2023)

    Google Scholar 

  34. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)

    Google Scholar 

  35. Seo, K., Tang, J., Roll, I., Fels, S., Yoon, D.: The impact of artificial intelligence on learner-instructor interaction in online learning. Int. J. Educ. Technol. High. Educ. 18, 1–23 (2021)

    Article  Google Scholar 

  36. Shneiderman, B.: Human-Centered AI. Oxford University Press, Oxford (2022)

    Book  Google Scholar 

  37. Sunstein, C.R.: Sludge audits. Behav. Public Policy 6(4), 654–673 (2022)

    Article  Google Scholar 

  38. Verma, S., Dickerson, J., Hines, K.: Counterfactual explanations for machine learning: a review. arXiv preprint arXiv:2010.10596 (2020)

  39. Wagner, B.D., Robertson, C.E., Harris, J.K.: Application of two-part statistics for comparison of sequence variant counts. PLoS ONE 6(5), e20296 (2011)

    Article  Google Scholar 

  40. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)

    Google Scholar 

  41. Yau, K.L.A., et al.: Augmented intelligence: surveys of literature and expert opinion to understand relations between human intelligence and artificial intelligence. IEEE Access 9, 136744–136761 (2021)

    Article  Google Scholar 

  42. Zou, K.H., Fielding, J.R., Silverman, S.G., Tempany, C.M.: Hypothesis testing I: proportions. Radiology 226(3), 609–613 (2003)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Universitá degli Studi di Milano-Bicocca, Milan, Italy

    Federico Cabitza, Lorenzo Famiglini & Chiara Natali

  2. IRCCS Istituto Ortopedico Galeazzi Milano, Milan, Italy

    Federico Cabitza & Andrea Campagner

  3. UOC Patologia Vertebrale e Scoliosi, ASST G. Pini - CTO, Milan, Italy

    Valerio Caccavella & Enrico Gallazzi

Authors
  1. Federico Cabitza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrea Campagner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lorenzo Famiglini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chiara Natali
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Valerio Caccavella
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Enrico Gallazzi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Federico Cabitza .

Editor information

Editors and Affiliations

  1. University of Natural Resources and Life, Vienna, Austria

    Andreas Holzinger

  2. St. Pölten University of Applied Science, St. Pölten, Austria

    Peter Kieseberg

  3. University of Milano-Bicocca, Milan, Italy

    Federico Cabitza

  4. University of Milano-Bicocca, Milan, Italy

    Andrea Campagner

  5. TU Wien, Vienna, Austria

    A Min Tjoa

  6. SBA Research, Vienna, Austria

    Edgar Weippl

Rights and permissions

Reprints and permissions

Copyright information

© 2023 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cabitza, F., Campagner, A., Famiglini, L., Natali, C., Caccavella, V., Gallazzi, E. (2023). Let Me Think! Investigating the Effect of Explanations Feeding Doubts About the AI Advice. In: Holzinger, A., Kieseberg, P., Cabitza, F., Campagner, A., Tjoa, A.M., Weippl, E. (eds) Machine Learning and Knowledge Extraction. CD-MAKE 2023. Lecture Notes in Computer Science, vol 14065. Springer, Cham. https://doi.org/10.1007/978-3-031-40837-3_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-40837-3_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-40836-6

  • Online ISBN: 978-3-031-40837-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation