Abstract
The high computational complexity of deep neural networks makes them challenging to deploy in practical applications. Recent efforts mainly involve pruning and compression the weights of layers to reduce these costs, and use randomly initializing weights to fine-tune the pruned model. However, these approaches always lose important weights, resulting in the compressed model performing that is even worse than the original model. To address this problem, we propose a novel method replaced the traditional fine-tuning method with the knowledge distillation algorithm in this paper. Meanwhile, With the Resnet152 model, our method obtained the accuracy of 73.83% on CIFAR100 data and 22x compression, respectively, ResNet110 SVHN achieve 49x compression with 98.23% accuracy and all of which are preferable to the state-of-the-art.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)
Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or \(-\)1. arXiv preprint arXiv:1602.02830 (2016)
Denton, E.L., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. In: Advances in Neural Information Processing Systems, pp. 1269–1277 (2014)
Guo, Y., Yao, A., Chen, Y.: Dynamic network surgery for efficient DNNs. In: Advances in Neural Information Processing Systems, pp. 1379–1387 (2016)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Hou, L., Yao, Q., Kwok, J.T.: Loss-aware binarization of deep networks. arXiv preprint arXiv:1611.01600 (2016)
Howard, A.G., et al.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Huang, Z., Wang, N.: Data-driven sparse structure selection for deep neural networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 317–334. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01270-0_19
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \({<}\)0.5 mb model size. arXiv preprint arXiv:1602.07360 (2016)
Jacob, B., et al.: Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704–2713 (2018)
Lan, X., Zhu, X., Gong, S.: Knowledge distillation by on-the-fly native ensemble. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7528–7538. Curran Associates Inc. (2018)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
Zagoruyko, S., Komodakis, N.: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 (2016)
Zhang, X., Zou, J., Ming, X., He, K., Sun, J.: Efficient and accurate approximations of nonlinear convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1984–1992 (2015)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, C., Zhang, H., Chen, D. (2020). Knowledge Distillation Based on Pruned Model. In: Zheng, Z., Dai, HN., Tang, M., Chen, X. (eds) Blockchain and Trustworthy Systems. BlockSys 2019. Communications in Computer and Information Science, vol 1156. Springer, Singapore. https://doi.org/10.1007/978-981-15-2777-7_49
Download citation
DOI: https://doi.org/10.1007/978-981-15-2777-7_49
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-2776-0
Online ISBN: 978-981-15-2777-7
eBook Packages: Computer ScienceComputer Science (R0)