Abstract
Algorithms are powerful and necessary tools behind a large part of the information we use every day. However, they may introduce new sources of bias, discrimination and other unfair practices that affect people who are unaware of it. Greater algorithm transparency is indispensable to provide more credible and reliable services. Moreover, requiring developers to design transparent algorithm-driven applications allows them to keep the model accessible and human understandable, increasing the trust of end users. In this paper we present EBAnO, a new engine able to produce prediction-local explanations for a black-box model exploiting interpretable feature perturbations. EBAnO exploits the hypercolumns representation together with the cluster analysis to identify a set of interpretable features of images. Furthermore two indices have been proposed to measure the influence of input features on the final prediction made by a CNN model. EBAnO has been preliminary tested on a set of heterogeneous images. The results highlight the effectiveness of EBAnO in explaining the CNN classification through the evaluation of interpretable features influence.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Schmidhuber, J.: Deep learning in neural networks: an overview. CoRR abs/1404.7828 (2014). http://arxiv.org/abs/1404.7828
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, vol. 1, pp. 1097–1105. Curran Associates Inc., USA (2012). http://dl.acm.org/citation.cfm?id=2999134.2999257
Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning, ICML 2008, pp. 160–167. ACM, New York (2008). http://doi.acm.org/10.1145/1390156.1390177
Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)
Strannegård, C., Häggström, O., Wessberg, J., Balkenius, C.: Transparent neural networks. In: Bach, J., Goertzel, B., Iklé, M. (eds.) AGI 2012. LNCS (LNAI), vol. 7716, pp. 302–311. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35506-6_31
Zhang, Q., Wu, Y.N., Zhu, S.: Interpretable convolutional neural networks. CoRR abs/1710.00935 (2017). http://arxiv.org/abs/1710.00935
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Hariharan, B., Arbeláez, P.A., Girshick, R.B., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. CoRR abs/1411.5752 (2014). http://arxiv.org/abs/1411.5752
Juang, B.H., Rabiner, L.R.: The segmental k-means algorithm for estimating parameters of hidden Markov models. IEEE Trans. Acoust. Speech Signal Process. 38(9), 1639–1641 (1990)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014). http://arxiv.org/abs/1409.1556
Hariharan, B., Arbelaez, P., Girshick, R.B., Malik, J.: Simultaneous detection and segmentation. CoRR abs/1407.1808 (2014). http://arxiv.org/abs/1407.1808
Chollet, F., et al.: Keras (2015). https://github.com/fchollet/keras
Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Lepri, B., Staiano, J., Sangokoya, D., Letouzé, E., Oliver, N.: The tyranny of data? The bright and dark sides of data-driven decision-making for social good. CoRR abs/1612.00323 (2016). http://arxiv.org/abs/1612.00323
Diakopoulos, N.: Enabling accountability of algorithmic media: transparency as a constructive and critical lens. In: Cerquitelli, T., Quercia, D., Pasquale, F. (eds.) Transparent Data Mining for Big and Small Data. SBD, vol. 11, pp. 25–43. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54024-5_2
Datta, A., Sen, S., Zick, Y.: Algorithmic transparency via quantitative input influence: theory and experiments with learning systems. In: 2016 IEEE Symposium on Security and Privacy (SP), pp. 598–617, May 2016
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?": explaining the predictions of any classifier. CoRR abs/1602.04938 (2016). http://arxiv.org/abs/1602.04938
Alufaisan, Y., Kantarcioglu, M., Zhou, Y.: Detecting discrimination in a black-box classifier (2016)
Adler, P., et al.: Auditing black-box models for indirect influence. Knowl. Inf. Syst. 54, 1–28 (2017)
Fong, R., Vedaldi, A.: Interpretable explanations of black boxes by meaningful perturbation. CoRR abs/1704.03296 (2017). http://arxiv.org/abs/1704.03296
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. CoRR abs/1312.6034 (2013). http://arxiv.org/abs/1312.6034
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Ventura, F., Cerquitelli, T., Giacalone, F. (2018). Black-Box Model Explained Through an Assessment of Its Interpretable Features. In: Benczúr, A., et al. New Trends in Databases and Information Systems. ADBIS 2018. Communications in Computer and Information Science, vol 909. Springer, Cham. https://doi.org/10.1007/978-3-030-00063-9_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-00063-9_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00062-2
Online ISBN: 978-3-030-00063-9
eBook Packages: Computer ScienceComputer Science (R0)