Skip to main content
SpringerLink
Log in

Explainable Artificial Intelligence in Healthcare: Opportunities, Gaps and Challenges and a Novel Way to Look at the Problem Space

  • Conference paper
  • First Online:
Intelligent Data Engineering and Automated Learning – IDEAL 2021 (IDEAL 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 13113))

Abstract

Explainable Artificial Intelligence (XAI) is rapidly becoming an emerging and fast-growing research field; however, its adoption in healthcare is still at the early stage despite the potential that XAI can bring to the application of AI in this industry. Many challenges remain to be solved, including setting standards for explanations, the degree of interaction between different stakeholders and the models, the implementation of quality and performance metrics, the agreement on standards for safety and accountability, its integration into clinical workflows, and IT infrastructure. This paper has two objectives. The first one is to present summarized outcomes of a literature survey and highlight the state-of-the-art for explainability including gaps, challenges, and opportunities for XAI in healthcare industry. For easier comprehension and onboarding to this research field we suggest a synthesized taxonomy for categorizing explainability methods. The second objective is to ask the question if applying a novel way of looking at explainability problem space, through a specific problem/domain lens, and automating that approach in an AutoML similar fashion, would help mitigate the challenges mentioned above. In the literature there is a tendency to look at the explainability of AI from model-first lens, which puts concrete problems and domains aside. For example, the explainability of a patient's survival model is treated the same as explaining a hospital cost procedure calculation. With a well-identified problem/domain that XAI should be applied to, the scope is clear and well-defined, enabling us to (semi-) automatically find suitable models, optimize their parameters and their explanations, metrics, stakeholders, safety/accountability level, and suggest means of their integration into clinical workflow .

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. World Health Organization (WHO). https://www.who.int/news/item/28-06-2021-who-issues-first-global-report-on-ai-in-health-and-six-guiding-principles-for-its-design-and-use. Accessed 14 July 2021

  2. Aurangzeb, A.M., Eckert, C., Teredesai, A.: Interpretable machine learning in healthcare. In: Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, pp. 559–560 (2018)

    Google Scholar 

  3. Pang, W., Markovic, M., Naja, I., Fung, C.P., Edwards, P.: On evidence capture for accountable AI systems. In: SICSA Workshop on eXplainable Artificial Intelligence (XAI) (2021)

    Google Scholar 

  4. Gunning, D., Aha, D.: Explainable artificial intelligence (XAI) program. AI Mag. 40(2), 44–58 (2019)

    Google Scholar 

  5. European Law General Data Protection Regulation. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A02016R0679-20160504&qid=1532348683434. Accessed 27 July 2021

  6. European Commission Artificial Intelligence Act. https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1623335154975&uri=CELEX%3A52021PC0206. Accessed 18 July 2021

  7. Dimensions query “Explainable AND Artificial Intelligence”. https://app.dimensions.ai/analytics/publication/overview/timeline?search_mode=content&search_text=explainable%20AND%20%22artificial%20intelligence%22&search_type=kws&search_field=full_search. Accessed 14 July 2021

  8. Dimensions query “Interpretable AND Artificial Intelligence”. https://app.dimensions.ai/analyics/publication/overview/timeline?search_mode=content&search_text=interpretable%20AND%20%22artificial%20intelligence%22&search_type=kws&search_field=full_search. Accessed 14 July 2021

  9. Das, A., Rad, P.: Opportunities and challenges in explainable artificial intelligence (XAI): a survey. ArXiv preprint arXiv:2006.11371 (2020)

  10. Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1(5), 206–215 (2019)

    Article  Google Scholar 

  11. Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., Kagal, L.: Explaining explanations: an overview of interpretability of machine learning. In: 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80–89, IEEE (2018)

    Google Scholar 

  12. Derek, D., Schulz, S., Besold, T.R.: What does explainable AI really mean? A new conceptualization of perspectives. ArXiv preprint arXiv:1710.00794 (2017)

  13. Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learning. ArXiv preprint arXiv:1702.08608 (2017)

  14. Molnar, C.: Interpretable Machine Learning, A Guide for Making Black Box Models Explainable. Leanpub, Monee, IL, USA (2020)

    Google Scholar 

  15. Ferreira, J.J., Monteiro, M.S.: What are people doing about XAI user experience? A survey on AI explainability research and practice. In: Marcus, A., Rosenzweig, E. (eds.) HCII 2020. LNCS, vol. 12201, pp. 56–73. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49760-6_4

    Chapter  Google Scholar 

  16. Tjoa, E., Guan, C.: A survey on explainable artificial intelligence (XAI): toward medical XAI. IEEE Trans. Neural Netw. Learn. Syst. 1–21 (2020)

    Google Scholar 

  17. Longo, L., Goebel, R., Lecue, F., Kieseberg, P., Holzinger, A.: Explainable artificial intelligence: concepts, applications, research challenges and visions. In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) CD-MAKE 2020. LNCS, vol. 12279, pp. 1–16. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-57321-8_1

    Chapter  Google Scholar 

  18. Adadi, A., Berrada, M.: Peeking inside the black box: a survey on explainable artificial intelligence (XAI). IEEE Access (6), 52138–52160 (2018)

    Google Scholar 

  19. Carvalho, D.V., Pereira, E.M.: Cardoso: machine learning interpretability: a survey on methods and metrics. Electronics 8(8), 832 (2019)

    Article  Google Scholar 

  20. Arrieta, A.B., et al.: Explainable Artificial Intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fus. 58, 82–115 (2020)

    Article  Google Scholar 

  21. Linardatos, P., Papastefanopoulos, V., Kotsiantis, S.: Explainable AI: a review of machine learning interpretability methods. Entropy 23(1), 18 (2021)

    Article  Google Scholar 

  22. Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)

    Google Scholar 

  23. Lundberg, S., Lee, S.I.: A unified approach to interpreting model predictions. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS), pp. 4765–4774 (2017)

    Google Scholar 

  24. Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 1189–1232 (2001)

    Google Scholar 

  25. Avanti, S., Greenside, P., Kundaje A.: Learning important features through propagating activation differences. In: International Conference on Machine Learning. PMLR, pp. 3145– 3153 (2017)

    Google Scholar 

  26. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626. IEEE (2017)

    Google Scholar 

  27. Hong, S.R., Hullman, J., Bertini, E.: Human factors in model interpretability: Industry practices, challenges, and needs. In: Proceedings of the ACM on Human-Computer Interaction 4 CSCW1, pp. 1–26 (2020)

    Google Scholar 

  28. Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., Wortman Vaughan, J.: Interpreting interpretability: understanding data scientists’ use of interpretability tools for machine learning. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2020)

    Google Scholar 

  29. Carrilo, A., Cantu, L.F., Noriega, A.: Individual explanations in machine learning models: a survey for pratictioners. arXiv preprint arXiv:2104.04144 (2021)

  30. Chen, C., Li, O., Tao, C., Barnett, A.J., Su, J., Rudin, C.: This looks like that: deep learning for interpretable image recognition. arXiv preprint arXiv:1806.10574 (2018)

  31. Singh, G., Yow, K.C.: These do not look like those: an interpretable deep learning model for image recognition. IEEE Access 9, 41482–41493 (2021)

    Article  Google Scholar 

  32. Eitel, F., et al.: Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional MRI using layer-wise relevance propagation. NeuroImage Clin. (24), 102003 (2019)

    Google Scholar 

  33. Royal College of Pathologists, Key Performance Indicators in Pathology. https://www.rcpath.org/uploads/assets/e7b7b680-a957-4f48-aa78e601e42816de/Key-Performance-Indicators-in-Pathology-Recommendations-from-the-Royal-College-of-Pathologists.pdf. Accessed 25 July 2021

  34. Floridi, L., Chiriatti, M.: GPT-3: Its nature, scope, limits, and consequences. Mind. Mach. 30(4), 681–694 (2020)

    Article  Google Scholar 

  35. Carlini, N., et al.: Extracting training data from large language models. arXiv preprint arXiv:2012.07805 (2020)

  36. Shaban-Nejad, A., Michalowski, M., Buckeridge, D.L.: Explainability and interpretability: keys to deep medicine. In: Shaban-Nejad, A., Michalowski, M., Buckeridge, D.L. (eds.) Explainable AI in Healthcare and Medicine. SCI, vol. 914, pp. 1–10. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-53352-6_1

    Chapter  Google Scholar 

  37. Harsha, N., Jenkins, S., Koch, P., Caruana R: Interpretml: a unified framework for machine learning interpretability. arXiv preprint arXiv:1909.09223 (2019)

  38. Matsoukas, Christos, M., Haslum, J.F., Söderberg, M., Smith, K.: Is it time to replace CNNs with transformers for medical images? arXiv preprint arXiv:2108.09038. Accepted at ICCV-2021: Workshop on Computer Vision for Automated Medical Diagno-sis (CVAMD) (2021)

  39. Wenqi, S., Tong, L., Zhu, Y., Wang, M.D.: COVID-19 automatic diagnosis with ra-diographic imaging: explainable attention transfer deep neural networks. IEEE J. Biomed. Health Inf. (25), 2376–2386 (2021)

    Google Scholar 

  40. Labati, R.D., Piuri, V., Scotti, F.: All-IDB: the acute lymphoblastic leukemia image database for image processing. In: 2011 18th IEEE International Conference on Image Processing, pp. 2045–2048. IEEE (2011)

    Google Scholar 

  41. Hutter, F., Kotthoff, L., Vanschoren, J.: Automated Machine Learning: Methods, Systems, Challenges. Springer, Heidelberg (2019). https://doi.org/10.1007/978-3-030-05318-5

Download references

Author information

Authors and Affiliations

  1. School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, UK

    Petra Korica, Neamat El Gayar & Wei Pang

Authors
  1. Petra Korica
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Neamat El Gayar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Petra Korica .

Editor information

Editors and Affiliations

  1. University of Manchester, Manchester, UK

    Hujun Yin

  2. Universidad Politecnica de Madrid, Madrid, Spain

    David Camacho

  3. University of Birmingham, Birmingham, UK

    Peter Tino

  4. University of Manchester, Manchester, UK

    Richard Allmendinger

  5. University of Huelva, Huelva, Spain

    Antonio J. Tallón-Ballesteros

  6. Southern University of Science and Technology, Shenzhen, China

    Ke Tang

  7. Yonsei University, Seoul, Korea (Republic of)

    Sung-Bae Cho

  8. University of Minho, Braga, Portugal

    Paulo Novais

  9. NOVA University of Lisbon, Lisbon, Portugal

    Susana Nascimento

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Korica, P., Gayar, N.E., Pang, W. (2021). Explainable Artificial Intelligence in Healthcare: Opportunities, Gaps and Challenges and a Novel Way to Look at the Problem Space. In: Yin, H., et al. Intelligent Data Engineering and Automated Learning – IDEAL 2021. IDEAL 2021. Lecture Notes in Computer Science(), vol 13113. Springer, Cham. https://doi.org/10.1007/978-3-030-91608-4_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-91608-4_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-91607-7

  • Online ISBN: 978-3-030-91608-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation