Skip to main content
SpringerLink
Log in

Color Shadows 2: Assessing the Impact of XAI on Diagnostic Decision-Making

  • Conference paper
  • First Online:
Explainable Artificial Intelligence (xAI 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1901))

Included in the following conference series:

  • 620 Accesses

Abstract

A comprehensive assessment of the impact of eXplainable AI (XAI) on diagnostic decision-making should adopt a socio-technical perspective. Our study focuses on Decision Support Systems (DSS) that provide explanations in the form of Activation Maps, assessing their impact in terms of automation bias and algorithmic aversion. Specifically, we focus on the XAI-assisted task of detecting thoraco-lumbar fractures from X-rays by radiologists, taking into account the complexity of the cases and the experience level of users. Our results show how XAI support has a clear and positive impact on diagnostic performance. By introducing the concepts of technology impact, reliance patterns, and the white box paradox, we highlight the importance of designing Human-AI Collaboration Protocols (HAI-CP) that are specific to the task at hand to optimize the integration of XAI into diagnostic decision-making.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.limesurvey.org/.

  2. 2.

    https://juxtapose.knightlab.com/.

  3. 3.

    The detailed analysis of each kind of XAI support is the subject of another work currently in preparation. In this paper we report on the longitudinal analysis of the impact of XAI without further distinguishing the type of output.

  4. 4.

    Both diagrams were generated via the tool available at https://haiiassassessment.pythonanywhere.com/.

  5. 5.

    Automation bias was calculated via the tool “DSS Quality Assessment" available at https://dss-quality-assessment.vercel.app/?step=4 following the metrics reported in [13].

References

  1. Liew, C.: The future of radiology augmented with artificial intelligence: a strategy for success. Eur. J. Radiol. 102, 152–156 (2018)

    Article  Google Scholar 

  2. Sogani, J., Allen Jr., B., Dreyer, K., McGinty, G.: Artificial intelligence in radiology: the ecosystem essential to improving patient care (2020)

    Google Scholar 

  3. Allen, B., Agarwal, S., Kalpathy-Cramer, J., Dreyer, K.: Democratizing AI. J. Am. Coll. Radiol. 16(7), 961–963 (2019)

    Article  Google Scholar 

  4. Quinn, T.P., Jacobs, S., Senadeera, M., Le, V., Coghlan, S.: The three ghosts of medical AI: can the black-box present deliver? Artif. Intell. Med. 124, 102158 (2022)

    Article  Google Scholar 

  5. Cabitza, F., et al.: Quod erat demonstrandum?-towards a typology of the concept of explanation for the design of explainable AI. Expert Syst. Appl. 213, 118888 (2023)

    Article  Google Scholar 

  6. Cabitza, F., et al.: Rams, hounds and white boxes: investigating human-AI collaboration protocols in medical diagnosis. Artif. Intell. Med. 138, 102506 (2023)

    Article  Google Scholar 

  7. Cabitza, F., Campagner, A., Natali, C., Parimbelli, E., Ronzio, L., Cameli, M.: Painting the black box white: experimental findings from applying XAI to an ECG reading setting. Mach. Learn. Knowl. Extr. 5(1), 269–286 (2023)

    Google Scholar 

  8. Bertrand, A., Belloum, R., Eagan, J.R., Maxwell, W.: How cognitive biases affect XAI-assisted decision-making: a systematic review. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 78–91 (2022)

    Google Scholar 

  9. Mumford, E.: A socio-technical approach to systems design. Requirements Eng. 5, 125–133 (2000)

    Article  Google Scholar 

  10. Delmas, P.D., et al.: Underdiagnosis of vertebral fractures is a worldwide problem: the impact study. J. Bone Mineral Res. 20(4), 557–563 (2005)

    Article  Google Scholar 

  11. Vilone, G., Longo, L.: Classification of explainable artificial intelligence methods through their output formats. Mach. Learn. Knowl. Extract. 3(3), 615–661 (2021)

    Article  Google Scholar 

  12. Skitka, L.J., Mosier, K.L., Burdick, M.: Does automation bias decision-making? Int. J. Hum.-Comput. Stud. 51(5), 991–1006 (1999)

    Article  Google Scholar 

  13. Cabitza, F., Campagner, A., Angius, R., Natali, C., Reverberi, C.: AI shall have no dominion: on how to measure technology dominance in AI-supported human decision-making. In CHI 2023: The Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2023)

    Google Scholar 

  14. Dietvorst, B.J., Simmons, J., Massey, C.: Understanding algorithm aversion: forecasters erroneously avoid algorithms after seeing them err. In: Academy of Management Proceedings, Briarcliff Manor, NY, USA, no. 1, p. 12227. Academy of Management (2014)

    Google Scholar 

  15. Jussupow, E., Benbasat, I., Heinzl, A.: Why are we averse towards algorithms? A comprehensive literature review on algorithm aversion. In: Twenty-Eighth European Conference on Information Systems (ECIS2020), Atlanta, GA USA, pp. 1–16. AIS (2020)

    Google Scholar 

  16. Sutton, S.G., Arnold, V., Holt, M.: An extension of the theory of technology dominance: understanding the underlying nature, causes and effects. Causes and effects (2022)

    Google Scholar 

  17. Cabitza, F.: Biases affecting human decision making in AI-supported second opinion settings. In: Torra, V., Narukawa, Y., Pasi, G., Viviani, M. (eds.) MDAI 2019. LNCS (LNAI), vol. 11676, pp. 283–294. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-26773-5_25

    Chapter  Google Scholar 

  18. Cabitza, F., Campagner, A., Famiglini, L., Gallazzi, E., La Maida, G.A.: Color shadows (part I): exploratory usability evaluation of activation maps in radiological machine learning. In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) CD-MAKE 2022. LNCS, vol. 13480, pp. 31–50. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-14463-9_3

    Chapter  Google Scholar 

  19. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  20. Friedman, C.P.: A “fundamental theorem’’ of biomedical informatics. J. Am. Med. Inform. Assoc. 16(2), 169–170 (2009)

    Article  Google Scholar 

  21. Kamar, E.: Directions in hybrid intelligence: complementing AI systems with human intelligence. In: IJCAI, pp. 4070–4073 (2016)

    Google Scholar 

  22. Gupta, P., Woolley, A.W.: Articulating the role of artificial intelligence in collective intelligence: a transactive systems framework. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Los Angeles, CA, USA, vol. 65, pp. 670–674. SAGE Publications (2021)

    Google Scholar 

  23. Miller, R.A., Masarie Jr., F.E.: The demise of the “Greek oracle” model for medical diagnostic systems. Methods Inf. Med. 29(01), 1–2 (1990)

    Google Scholar 

  24. Campagner, A., Sternini, F., Cabitza, F.: Decisions are not all equal. Introducing a utility metric based on case-wise raters’ perceptions. Comput. Methods Program. Biomed. 106930 (2022)

    Google Scholar 

  25. Dixon-Woods, M., Redwood, S., Leslie, M., Minion, J., Martin, G.P., Coleman, J.J.: Improving quality and safety of care using “technovigilance’’: an ethnographic case study of secondary use of data from an electronic prescribing and decision support system. Milbank Q. 91(3), 424–454 (2013)

    Article  Google Scholar 

  26. Cabitza, F., Zeitoun, J.-D.: The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence. Ann. Transl. Med. 7(8) (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Universitá degli Studi di Milano-Bicocca, Milan, Italy

    Chiara Natali, Lorenzo Famiglini, Andrea Campagner & Federico Cabitza

  2. IRCCS Istituto Ortopedico Galeazzi, Milan, Italy

    Andrea Campagner & Federico Cabitza

  3. Istituto Ortopedico Gaetano Pini—ASST Pini-CTO, Milan, Italy

    Giovanni Andrea La Maida & Enrico Gallazzi

Authors
  1. Chiara Natali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorenzo Famiglini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Campagner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Giovanni Andrea La Maida
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Enrico Gallazzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Federico Cabitza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chiara Natali .

Editor information

Editors and Affiliations

  1. Technological University Dublin, Dublin, Ireland

    Luca Longo

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Natali, C., Famiglini, L., Campagner, A., La Maida, G.A., Gallazzi, E., Cabitza, F. (2023). Color Shadows 2: Assessing the Impact of XAI on Diagnostic Decision-Making. In: Longo, L. (eds) Explainable Artificial Intelligence. xAI 2023. Communications in Computer and Information Science, vol 1901. Springer, Cham. https://doi.org/10.1007/978-3-031-44064-9_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-44064-9_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-44063-2

  • Online ISBN: 978-3-031-44064-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation