Abstract
Deep neural networks has been widely used in many fields, but there are growing concerns about its black-box nature. Previous interpretability studies provide four types of explanations including logical rules, revealing hidden semantics, sensitivity analysis, and providing examples as prototypes. In this paper, an interpretability method is proposed for revealing semantic representations at hidden layers of CNNs through lightweight annotation by occluding. First, visual semantic configurations are defined for a certain class. Then candidate filters whose activations are related to these specified visual semantics are probed by occluding. Finally, lightweight occlusion annotation and a scoring mechanism is used to screen out the filters that recognize these semantics. The method is applied to the datasets of mechanical equipment, animals and clothing images. The proposed method performs well in the experiments assessing interpretability qualitatively and quantitatively.
Q. Hong and Y. Wang—Contributed equally to this work. This work is supported by: National Defense Science and Technology Innovation Special Zone Project (No. 18-163-11-ZT-002-045-04); Engineering Research Center of State Financial Security, Ministry of Education, Central University of Finance and Economics, Beijing, 102206, China; Program for Innovation Research in Central University of Finance and Economics; National College Students’ Innovation and Entrepreneurship Training Program “Research on classification and interpretability of popular goods based on Neural Network”.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhang, Y., Tiňo, P., Leonardis, A., et al.: A survey on neural network interpretability. arXiv:2012.14261 (2020)
Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: high-precision model-agnostic explanations. In: AAAI Conference on Artificial Intelligence, vol. 32, no. 1, pp. 1527–1535 (2018)
Dhurandhar, A., Chen, P.Y., Luss, R., et al.: Explanations based on the missing: Towards contrastive explanations with pertinent negatives. arXiv:1802.07623 (2018)
Krishnan, R., Sivakumar, G., Bhattacharya, P.: Extracting decision trees from trained neural networks. Pattern Recogn. 32(12), 1999–2009 (1999)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visual-ising image classification models and saliency maps. arXiv:1312.6034 (2013)
Selvaraju, R.R., Cogswell, M., Das A., et al.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Fong, R.C., Vedaldi, A.: Interpretable explanations of black boxes by meaningful perturbation. In: IEEE International Conference on Computer Vision, pp. 3429–3437 (2017)
Li, O., Liu, H., Chen, C., et al.: Deep learning for case-based reasoning through prototypes: a neural network that explains its predictions. In: AAAI Conference on Artificial Intelligence, vol. 32, no. 1, pp. 3530–3537 (2018)
Chen, C., Li, O., Tao, C., et al.: This looks like that: deep learning for interpretable image recognition. arXiv:1806.10574 (2018)
Su, J., Vargas, D.V., Sakurai, K.: One pixel attack for fooling deep neural networks. IEEE Trans. Evol. Comput. 23(5), 828–841 (2019)
Bau, D., Zhou, B., Khosla, A., et al.: Network dissection: quantifying interpretability of deep visual representations. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6541–6549 (2017)
Erhan, D., Bengio, Y., Courville, A., et al.: Visualizing higher-layer features of a deep network. University of Montreal, Technical Report 1341 (2009)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional net-works. In: European Conference on Computer Vision, pp. 818–833 (2014)
Zhou, B., Khosla, A., Lapedriza, A., et al.: Object detectors emerge in deep scene CNNs. arXiv:1412.6856 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Hong, Q., Wang, Y., Li, H., Zhao, Y., Guo, W., Wang, X. (2021). Probing Filters to Interpret CNN Semantic Configurations by Occlusion. In: Zeng, J., Qin, P., Jing, W., Song, X., Lu, Z. (eds) Data Science. ICPCSEE 2021. Communications in Computer and Information Science, vol 1452. Springer, Singapore. https://doi.org/10.1007/978-981-16-5943-0_9
Download citation
DOI: https://doi.org/10.1007/978-981-16-5943-0_9
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-5942-3
Online ISBN: 978-981-16-5943-0
eBook Packages: Computer ScienceComputer Science (R0)