Abstract
To reduce the overhead of neural network training and inference, several techniques have been widely used to prune neural network models. Pruning algorithms can significantly reduce the number of parameters in the model, which in turn reduces the amount of computation required during model training and inference. Currently, the most popular pruning algorithm is the structured pruning algorithm, which prunes the model at the kernel level. Researchers usually use norm-based criteria to determine which kernels to prune. While this type of algorithms works well, there are some shortcomings. First, the effectiveness of the norm-based pruning algorithm lacks support from mathematical theories. Second, this pruning algorithm requires certain conditions to work well. To address these shortcomings, we propose a novel kernel pruning algorithm. Based on the observation that convolution kernels act as feature extractors, we design a functional similarity-based pruning algorithm as the criteria for selecting pruned kernels. Our experimental results show that when pruning ResNet with a high pruning ratio, this algorithm can obtain a sparse model with high accuracy. Moreover, when combined with the norm-based pruning algorithm, our functional similarity-based pruning algorithm can produce a more accurate model than either algorithm alone, even at the same pruning ratio.
X. Yi and B. Wang—Have contributed equally to this research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Diffenderfer, J., Kailkhura, B.: Multi-prize lottery ticket hypothesis: finding accurate binary neural networks by pruning a randomly weighted network. arXiv preprint arXiv:2103.09377 (2021)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems 28 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 (2018)
He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
Lin, M., Ji, R., Zhang, Y., Zhang, B., Wu, Y., Tian, Y.: Channel pruning via automatic structure search. arXiv preprint arXiv:2001.08565 (2020)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744 (2017)
Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270 (2018)
Luo, J.H., Wu, J., Lin, W.: ThiNet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017)
Malach, E., Yehudai, G., Shalev-Schwartz, S., Shamir, O.: Proving the lottery ticket hypothesis: pruning is all you need. In: International Conference on Machine Learning, pp. 6682–6691. PMLR (2020)
Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32 (2019)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Wang, Z., Li, C., Wang, X.: Convolutional neural network pruning with structural redundancy reduction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14913–14922 (2021)
Zhu, M., Gupta, S.: To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 (2017)
Zhuo, H., Qian, X., Fu, Y., Yang, H., Xue, X.: SCSP: spectral clustering filter pruning with soft self-adaption manners. arXiv preprint arXiv:1806.05320 (2018)
Acknowledgement
This work is supported in part by the National Key R/ &D Project No. 2021YFB0300300, the NSFC (62172430), the NSF of Hunan Province 2021JJ10052, the STIP of Hunan Province 2022RC3065, and the Key Laboratory of Advanced Microprocessor Chips and Systems.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yi, X. et al. (2024). A Hybrid Kernel Pruning Approach for Efficient and Accurate CNNs. In: Tari, Z., Li, K., Wu, H. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2023. Lecture Notes in Computer Science, vol 14493. Springer, Singapore. https://doi.org/10.1007/978-981-97-0862-8_3
Download citation
DOI: https://doi.org/10.1007/978-981-97-0862-8_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-0861-1
Online ISBN: 978-981-97-0862-8
eBook Packages: Computer ScienceComputer Science (R0)