Abstract
Convolutional neural networks have made unprecedented breakthroughs in various tasks of computer vision. Due to its complex nonlinear model structure and the high latitude and complexity of data distribution, it has been criticized as an unexplained “black box”. Therefore, explaining the neural network model and uncovering the veil of the neural network have become the focus of attention. This paper starts with the term “interpretability”, summarizes the results of the interpretability of convolutional neural networks in the past three years (2016–2018), and analyses them with interpretable methods. Firstly, the concept of “interpretability” is introduced. Then the existing research achievements are classified and compared from four aspects, data characteristics and rule processing, model internal spatial analysis, interpretation and prediction, and model interpretation. Finally pointed out the possible research directions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gilpin, L.H., et al.: Explaining Explanations: An Overview of Interpretability of Machine Learning (2018)
WHI: 2017 Homepage. https://sites.google.com/view/whi2017/home
Lipton, Z.C.: The Mythos of Model Interpretability. Communications of the ACM (2016)
Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. J. Mach. Learn. Res. 3(6), 1157–1182 (2003)
Geras, K.J., et al.: High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks (2017)
Ling, J., et al.: Building data-driven models with microstructural images: generalization and interpretability. Mat. Discov. 10, 19–28 (2018). S235292451730042X
Chen, J., et al.: Learning to Explain: An Information-Theoretic Perspective on Model Interpretation (2018)
Freitas, Alex A.: Comprehensible classification models: a position paper. ACM Sigkdd Explor. Newsl. 15(1), 1–10 (2014)
Zilke, J.R.: DeepRED – Rule Extraction from Deep Neural Networks (2016)
Sato, M., Tsukimoto, H.: Rule extraction from neural networks via decision tree induction. In: International Joint Conference on Neural Networks (2001)
Wang, H.: ReNN: Rule-embedded Neural Networks (2018)
Erhan, D., et al.: Visualizing higher-layer features of a deep network. University of Montreal, 1341.3(1) (2009)
Nguyen, A., Jason Y., Jeff C.: Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks (2016). arXiv preprint arXiv:1602.03616
Zeiler, M.D., Fergus, R.: Visualizing and Understanding Convolutional Networks (2013)
Bolei, Z., et al.: Interpreting Deep Visual Representations via Network Dissection. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, p. 1 (2018)
https://cs.stanford.edu/people/karpathy/convnetjs//demo/classify2d.html
Zhou, B., et al.: Object detectors emerge in deep scene CNNS (2014). arXiv preprint arXiv:1412.6856
Yosinski, J., et al.: How transferable are features in deep neural networks?. Eprint Arxiv 27, 3320–3328 (2014)
Fong, R., Vedaldi, A.: Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks (2018)
Szegedy, C., et al.: Intriguing properties of neural networks. Computer Science (2013)
Kim, B., et al.: Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (2017)
Raghu, M., et al.: SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (2017)
Kindermans, P.J., et al.: Investigating the influence of noise and distractors on the interpretation of neural networks (2016)
Fong, R.C., Vedaldi, A.: Interpretable explanations of black boxes by meaningful perturbation. In: 2017 IEEE International Conference on Computer Vision (ICCV) IEEE Computer Society (2017)
Hara, S., et al.: Maximally Invariant Data Perturbation as Explanation (2018). arXiv preprint arXiv:1806.07004
Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you?: Explaining the predictions of any classifier. In: 22nd ACM SIGKDD International Conference ACM (2016)
Shrikumar, A., Greenside, P., Kundaje, A.: Learning Important Features Through Propagating Activation Differences (2017)
Sebastian, B., et al.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), e0130140 (2015)
Lundberg, S., Lee, S.I.: A Unified Approach to Interpreting Model Predictions (2017)
Sundararajan, M., Taly, A., Yan, Q.: Axiomatic Attribution for Deep Networks (2017)
Hendricks, L.A., et al.: Generating Visual Explanations. In: European Conference on Computer Vision (2016)
Hendricks, L.A., et al.: Generating Counterfactual Explanations with Natural Language (2018)
Mahendran, A., Vedaldi, A.: Understanding Deep Image Representations by Inverting Them (2014)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. Computer Science (2013)
Zeiler, M.D., et al.: Deconvolutional networks. Computer Vision & Pattern Recognition (2010)
Mahendran, A., Vedaldi, A.: Salient deconvolutional networks. In: European Conference on Computer Vision (2016)
Springenberg, J.T., et al.: Striving for simplicity: the all convolutional net (2014). arXiv preprint arXiv:1412.6806
Smilkov, D., et al.: Smoothgrad: removing noise by adding noise (2017). arXiv preprint arXiv:1706.03825
Zolna, K., Krzysztof, J.G., Kyunghyun, C.: Classifier-agnostic saliency map extraction (2018). arXiv preprint arXiv:1805.08249
Le, Q.V., et al.: Building high-level features using large scale unsupervised learning (2011). arXiv preprint arXiv:1112.6209
Yosinski, J., et al.: Understanding neural networks through deep visualization (2015). arXiv preprint arXiv:1506.06579
Dosovitskiy, A., Thomas B.: Inverting visual representations with convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
Zhou, B., et al.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
Selvaraju, R.R., et al.: Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (2016)
Oramas, J., Kaili W., Tinne, T.: Visual explanation by interpretation: improving visual feedback capabilities of deep neural networks (2017). arXiv preprint arXiv:1712.06302
Bau, D., et al.: Network dissection: quantifying interpretability of deep visual representations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
Zhang, Q., et al.: Interpreting CNN Knowledge via an Explanatory Graph (2017)
Zhang, Q., et al.: Interpreting CNNs via Decision Trees (2018)
Ba, L.J., Caruana, R.: Do deep nets really need to be deep?. In: International Conference on Neural Information Processing Systems, MIT Press (2014)
Abbasi-Asl, R., Yu, B.: Interpreting Convolutional Neural Networks Through Compression (2017)
Zhang, Q., Wu, Y.N., Zhu, S.-C.: Interpretable Convolutional Neural Networks (2018)
Wu, T., et al.: Towards Interpretable R-CNN by Unfolding Latent Structures (2017). arXiv preprint arXiv:1711.05226
Acknowledgment
This work was funded by Science and Technology Commission of Shanghai Municipality Program (No. 17411952800, No. 18441904500, 18DZ1113400) and Science and Technology Department of Hainan Province (No. ZDYF2018022).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Zhang, W., Cai, L., Chen, M., Wang, N. (2019). Progress in Interpretability Research of Convolutional Neural Networks. In: Yin, Y., Li, Y., Gao, H., Zhang, J. (eds) Mobile Computing, Applications, and Services. MobiCASE 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 290. Springer, Cham. https://doi.org/10.1007/978-3-030-28468-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-28468-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28467-1
Online ISBN: 978-3-030-28468-8
eBook Packages: Computer ScienceComputer Science (R0)