Abstract
The present paper aims to propose a new type of model compression method to self-compress multi-layered neural networks into single-layered neural networks. Then, the compressed single-layered neural networks, by the help of mutual information control, can be easily and naturally interpreted with improved generalization. For interpreting the inference mechanism, we have proposed the self-compression method, applied directly to multi-layered neural networks, to produce compressed or collective weights. However, the interpretation of compressed networks tends to be instable or changeable due to the existence of multiple layers, and the need to simplify and stabilize the compressed networks and the corresponding weights has arisen. In this context, the present paper aims not to directly and immediately compress multi-layered networks but to compress multi-layered neural network into single neural networks, and then the single neural networks are forced to be simplified as much as possible by mutual information control, and they are further compressed into non-layered networks. The method was applied to two well-studied data sets: the teaching assistant evaluation data set and the inference of occupancy status. The results confirmed two important points. First, generalization performance could be improved by transferring information from multi-layered neural networks to single-layered networks. Second, compressed or collective connection weights were easily and naturally interpreted.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Andrews, R., Diederich, J., Tickle, A.B.: Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl. Based Syst. 8(6), 373–389 (1995)
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One 10(7), e0130140 (2015)
Benítez, J.M., Castro, J.L., Requena, I.: Are artificial neural networks black boxes? IEEE Trans. Neural Networks 8(5), 1156–1164 (1997)
Buciluǎ, C., Caruana, R., Niculescu-Mizil, A.: Model compression. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535–541. ACM (2006)
Candanedo, L.M., Feldheim, V.: Accurate occupancy detection of an office room from light, temperature, humidity and co2 measurements using statistical learning models. Energy Build. 112, 28–39 (2016)
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N.: Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1721–1730. ACM (2015)
Che, Z., Purushotham, S., Khemani, R., Liu, Y.: Interpretable deep models for icu outcome prediction. In: AMIA Annual Symposium Proceedings, vol. 2016, p. 371. American Medical Informatics Association (2016)
Cheng, Y., Wang, D., Zhou, P., Zhang, T.: A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 (2017)
Dietz, S., Anderson, D., Stern, N., Taylor, C., Zenghelis, D., et al.: Right for the right reasons. World Econ. 8(2), 229–258 (2007)
Goodman, B., Flaxman, S.: European union regulations on algorithmic decision-making and a “right to explanation”. arXiv preprint arXiv:1606.08813 (2016)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Kamimura, R.: Direct potentiality assimilation for improving multi-layered neural networks. In: Proceedings of the 2017 Federated Conference on Computer Science and Information Systems, pp. 19–23 (2017)
Kamimura, R.: Mutual information maximization for improving and interpreting multi-layered neural network. In: Proceedings of the 2017 IEEE Symposium Series on Computational Intelligence (SSCI) (SSCI 2017) (2017)
Karpathy, A., Johnson, J., Fei-Fei, L.: Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078 (2015)
Lim, T.S., Loh, W.Y., Shih, Y.S.: A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Mach. Learn. 40(3), 203–228 (2000)
Loh, W.Y., Shih, Y.S.: Split selection methods for classification trees. Statistica sinica, pp. 815–840 (1997)
Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you?: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144. ACM (2016)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365 (2017)
Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
Varshney, K.R., Alemzadeh, H.: On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big Data 5(3), 246–255 (2017)
Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., Lipson, H.: Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Kamimura, R. (2018). Information-Theoretic Self-compression of Multi-layered Neural Networks. In: Fagan, D., Martín-Vide, C., O'Neill, M., Vega-Rodríguez, M.A. (eds) Theory and Practice of Natural Computing. TPNC 2018. Lecture Notes in Computer Science(), vol 11324. Springer, Cham. https://doi.org/10.1007/978-3-030-04070-3_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-04070-3_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04069-7
Online ISBN: 978-3-030-04070-3
eBook Packages: Computer ScienceComputer Science (R0)