Abstract
In this paper, we propose a filter pruning method, namely, Filter Pruning via Gradient Support Pursuit (FPGraSP), which can accelerate and compress very deep Convolutional Neural Networks effectively in an iterative way. Previous work reports that Gradient Support Pursuit (GraSP) is well employed for sparsity-constrained optimization in Machine Learning. We seek to develop a modification that GraSP can be applied to structured pruning in deep CNNs. Specifically, we select the filters with the maximum gradient values and merge their indices with the indices of the filters with the largest weights. We then update parameters over the above union. Finally, we utilize filter selection in a dynamic way to get the filters with the largest magnitude. Different from some previous methods which remove filters of smaller weights but neglect the influence of gradients, we exploit gradient information. Our experimental results on MNIST, CIFAR-10 and CIFAR-100 clearly demonstrate the efficiency of our FPGraSP algorithm. As an example, for pruning ResNet-56 on CIFAR-10, our FPGraSP without fine-tuning obtains 0.04\(\%\) accuracy drop, achieving 52.63\(\%\) FLOPs reduction.
Jue Wang is currently working toward the Master degree in the School of Automation, Nanjing University of Information Science and Technology.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alizadeh, M., Fernández-Marqués, J., Lane, N.D., Gal, Y.: An empirical study of binary neural networks’ optimisation (2018)
Bahmani, S., Raj, B., Boufounos, P.T.: Greedy sparsity-constrained optimization. J. Mach. Learn. Res. 14(Mar), 807–841 (2013)
Dong, X., Huang, J., Yang, Y., Yan, S.: More is less: a more complicated network with less inference complexity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5840–5848 (2017)
Guo, Y., Yao, A., Chen, Y.: Dynamic network surgery for efficient dnns. In: Advances In Neural Information Processing Systems, pp. 1379–1387 (2016)
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 (2018)
He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1389–1397 (2017)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Jin, X., Yuan, X., Feng, J., Yan, S.: Training skinny deep neural networks with iterative hard thresholding methods. arXiv preprint arXiv:1607.05423 (2016)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical repor, Citeseer (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
LeCun, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems, pp. 598–605 (1990)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744 (2017)
Luo, J.H., Wu, J., Lin, W.: Thinet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017)
Paszke, A., et al: Automatic differentiation in pytorch (2017)
Qian, N.: On the momentum term in gradient descent learning algorithms. Neural Netw. 12(1), 145–151 (1999)
Singh, B., Najibi, M., Davis, L.S.: Sniper: efficient multi-scale training. In: Advances in Neural Information Processing Systems, pp. 9310–9320 (2018)
Wang, W., Fu, C., Guo, J., Cai, D., He, X.: Cop: customized deep model compression via regularized correlation-based filter-level pruning. arXiv preprint arXiv:1906.10337 (2019)
Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems, pp. 2074–2082 (2016)
Ye, J., et al.: Learning compact recurrent neural networks with block-term tensor decomposition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9378–9387 (2018)
Zhuang, B., Shen, C., Tan, M., Liu, L., Reid, I.: Towards effective low-bitwidth convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7920–7928 (2018)
Zhuang, Z., et al.: Discrimination-aware channel pruning for deep neural networks. In: Advances in Neural Information Processing Systems, pp. 875–886 (2018)
Acknowledgements
This work was supported in part by National Major Project of China for New Generation of AI under Grant No. 2018AAA0100400 and in part by Natural Science Foundation of China (NSFC) under Grant No. 61876090 and No. 61936005.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, J., Ji, F., Yuan, XT. (2020). Pruning Deep Convolutional Neural Networks via Gradient Support Pursuit. In: Peng, Y., et al. Pattern Recognition and Computer Vision. PRCV 2020. Lecture Notes in Computer Science(), vol 12307. Springer, Cham. https://doi.org/10.1007/978-3-030-60636-7_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-60636-7_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60635-0
Online ISBN: 978-3-030-60636-7
eBook Packages: Computer ScienceComputer Science (R0)