Abstract
We present a new method for structured pruning of neural networks, based on the recently proposed neuron merging trick in which following a pruning operation, the weights of the next layer are suitably modified. By a rigorous mathematical analysis of the neuron merging technique we prove an upper bound on the reconstruction error. This bound defines a new objective function for pruning-and-merging. Our new optimal algorithm provably achieves the lowest objective cost among all possible prune-and-merge strategies. We also show empirically that nuclear norm regularization can be used to obtain even better pruning-and-merging accuracy; this finding is supported by our theoretical analysis.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
The bias also needs to be taken into account - see [9, Section 6.1] for details how to do that.
References
Church, R.L.: Beamr: an exact and approximate model for the p-median problem. Comput. Operat. Res. 35(2), 417–426 (2008)
Enderich, L., Timm, F., Burgard, W.: Holistic filter pruning for efficient deep neural networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2596–2605 (2021)
Hakimi, S.L.: Optimum distribution of switching centers in a communication network and some related graph theoretic problems. Oper. Res. 13(3), 462–475 (1965)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 (2018)
He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1389–1397 (2017)
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., Peste, A.: Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks. arXiv preprint arXiv:2102.00554 (2021)
Kim, W., Kim, S., Park, M., Jeon, G.: Neuron merging: compensating for pruned neurons. In: Advances in Neural Information Processing Systems, vol. 33 (2020)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Le, D.H., Hua, B.S.: Network pruning that matters: a case study on retraining variants. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=Cb54AMqHQFP
Li, T., Li, J., Liu, Z., Zhang, C.: Few sample knowledge distillation for efficient network compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14639–14647 (2020)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744 (2017)
Luo, J.H., Wu, J., Lin, W.: Thinet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017)
Mao, H., et al.: Exploring the granularity of sparsity in convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 13–20 (2017)
Nie, F., Huang, H., Ding, C.: Low-rank matrix recovery via efficient Schatten p-norm minimization. In: Twenty-sixth AAAI Conference on Artificial Intelligence (2012)
Recht, B., Xu, W., Hassibi, B.: Necessary and sufficient conditions for success of the nuclear norm heuristic for rank minimization. In: 2008 47th IEEE Conference on Decision and Control, pp. 3065–3070. IEEE (2008)
Reese, J.: Solution methods for the p-median problem: an annotated bibliography. Netw. Int. J. 48(3), 125–142 (2006)
Renda, A., Frankle, J., Carbin, M.: Comparing rewinding and fine-tuning in neural network pruning. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=S1gSj0NKvB
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Vadera, S., Ameen, S.: Methods for pruning deep neural networks. arXiv preprint arXiv:2011.00241 (2020)
You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. arXiv preprint arXiv:1909.08174 (2019)
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
Zhou, D., et al.: Go wide, then narrow: efficient training of deep thin networks. In: International Conference on Machine Learning, pp. 11546–11555. PMLR (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Goldberg, F., Lubarsky, Y., Gaissinski, A., Botchan, D., Kisilev, P. (2022). Pruning Neural Nets by Optimal Neuron Merging. In: El Yacoubi, M., Granger, E., Yuen, P.C., Pal, U., Vincent, N. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2022. Lecture Notes in Computer Science, vol 13363. Springer, Cham. https://doi.org/10.1007/978-3-031-09037-0_56
Download citation
DOI: https://doi.org/10.1007/978-3-031-09037-0_56
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09036-3
Online ISBN: 978-3-031-09037-0
eBook Packages: Computer ScienceComputer Science (R0)