Concept attribution: Explaining CNN decisions to physicians

https://doi.org/10.1016/j.compbiomed.2020.103865Get rights and content
Under a Creative Commons license
open access

Highlights

  • Feature attribution explains CNNs in terms of the input pixels.

  • The abstraction of feature attribution to higher level impacting factors is hard.

  • Concept attribution explains CNNs with high-level concepts such as clinical factors.

  • Nuclei pleomorphism is shown as a relevant factor in breast tumor classification.

  • Concept attribution can match clinical expectations to the interpretability of CNNs.

Abstract

Deep learning explainability is often reached by gradient-based approaches that attribute the network output to perturbations of the input pixels. However, the relevance of input pixels may be difficult to relate to relevant image features in some applications, e.g. diagnostic measures in medical imaging. The framework described in this paper shifts the attribution focus from pixel values to user-defined concepts. By checking if certain diagnostic measures are present in the learned representations, experts can explain and entrust the network output. Being post-hoc, our method does not alter the network training and can be easily plugged into the latest state-of-the-art convolutional networks. This paper presents the main components of the framework for attribution to concepts, in addition to the introduction of a spatial pooling operation on top of the feature maps to obtain a solid interpretability analysis. Furthermore, regularized regression is analyzed as a solution to the regression overfitting in high-dimensionality latent spaces. The versatility of the proposed approach is shown by experiments on two medical applications, namely histopathology and retinopathy, and on one non-medical task, the task of handwritten digit classification. The obtained explanations are in line with clinicians’ guidelines and complementary to widely used visualization tools such as saliency maps.

Keywords

Machine learning
Interpretability
Biomedical imaging
Deep learning

Cited by (0)

Mara Graziani is a third-year PhD student with double affiliation at the computer science faculty at the University of Geneva and at the University of Applied Sciences of Western Switzerland (Hes-so). Her research aims at improving the interpretability of machine learning systems for healthcare. She was a visiting student at the Martinos Center, part of Harvard Medical School in Boston, MA, USA to focus on the interaction between clinicians and deep learning systems. From a background of IT Engineering, she was awarded the Engineering Department Award for completing the MPhil in Machine Learning, Speech and Language at the University of Cambridge (UK) in 2017.

Vincent Andrearczyk received a double Masters degree in electronics and signal processing from ENSEEIHT, France and Dublin City University, in 2012 and 2013 respectively. He completed his PhD degree on deep learning for texture and dynamic texture analysis at Dublin City University in 2017. He is currently a post-doctoral researcher at the University of Applied Sciences and Arts Western Switzerland with a research focus on deep learning for medical image analysis and texture feature extraction.

Prof. Stéphane Marchand-Maillet has obtained his PhD in Applied Mathematics at Imperial College (London, UK) in 1997. He is Associate Professor in the Department of Computer Science at University of Geneva since 2011. His research group (Viper) specializes in large-scale, high-dimensional distributed machine learning and information retrieval, mining and indexing, with applications to data modeling and prediction, including social network analysis. He has authored, co-authored or edited a number of publications on these topics. He and his group are part of several national and European and international projects in the domain. He is Senior PC Member of the International Joint Conference on AI (IJCAI, one of the oldest established conferences in AI). He was general co-chair of the International Conference of the ACM-SIG on Information Retrieval in 2010 and general co-chair of the 16th IEEE Conference in Business Informatics in 2014.

Henning Müller studied medical informatics at the University of Heidelberg, Germany, then worked at Daimler-Benz research in Portland, OR, USA. From 1998–2002 he worked on his PhD degree at the University of Geneva, Switzerland with a research stay at Monash University, Melbourne, Australia. Since 2002, Henning has been working for the medical informatics service at the University Hospital of Geneva. Since 2007, he has been a full professor at the HES-SO Valais and since 2011 he is responsible for the eHealth unit of the school. Since 2014, he is also professor at the medical faculty of the University of Geneva. In 2015/2016 he was on sabbatical at the Martinos Center, part of Harvard Medical School in Boston, MA, USA to focus on research activities. Henning is coordinator of the ExaMode EU project, was coordinator of the Khresmoi EU project, scientific coordinator of the VISCERAL EU project and is initiator of the ImageCLEF benchmark that has run medical tasks since 2004. He has authored over 500 scientific papers with more than 13,000 citations and is in the editorial board of several journals.

This document is the results of the research project funded by PROCESS in the EU H2020 program (grant agreement No. 777533).

1

Researcher.

2

Co-ordinator.