Skip to main content
SpringerLink
Log in

Network pruning via probing the importance of filters

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Filter pruning is one of the most effective approaches to reduce the storage and computational cost of convolutional neural networks. How to measure the importance of each filter is the key problem for filter pruning. In this work, we propose a novel method that can evaluate the importance of each filter and gradually prunes those filters with small scores. Specifically, the importance is obtained via probing the effect of each filter on the task-related loss function by randomly pruning the original network. The smaller the effect on the task-related loss function, the lower the importance of the filter. It’s worth noting that our method is scale consistent across all layers without requiring layer-wise sensitivity analysis, which can be used to prune various networks, including ResNet and DenseNet. Extensive experiments demonstrate the outstanding performance of our method. For example, on ILSVRC-2012, our method can prune 42.74% floating point operations and 39.61% parameters of ResNet-50 with only 0.73% Top-1 accuracy loss and 0.37% Top-5 accuracy loss.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2015) Semantic image segmentation with deep convolutional nets and fully connected crfs. In: Int. conf. on learning representations (ICLR)

  2. Chen P, Liu S, Zhao H, Jia J (2021) Distilling knowledge via knowledge review. In: Proc. IEEE conf. computer vision and pattern recognition, pp 5008–5017

  3. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proc. IEEE conf. computer vision and pattern recognition, pp 1800–1807

  4. Chu X, Zhang B, Xu R, Li J (2019) Fairnas: rethinking evaluation fairness of weight sharing neural architecture search. arXiv preprint arXiv:1907.01845

  5. Courbariaux M, Bengio Y, David JP (2015) Binaryconnect: training deep neural networks with binary weights during propagations. Adv Neural Inf Process Syst 28:3123–3131

    Google Scholar 

  6. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: Proc. IEEE conf. computer vision and pattern recognition, pp 248–255

  7. Ding X, Ding G, Guo Y, Han J, Yan C (2019) Approximated oracle filter pruning for destructive cnn width optimization. In: Int. conf. on machine learning (ICML), pp 1607–1616

  8. Dong X, Yang Y (2019) Network pruning via transformable architecture search. Adv Neural Inf Process Syst 32:760–771

    Google Scholar 

  9. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proc. IEEE conf. computer vision and pattern recognition, pp 580–587

  10. Guo Y, Yao A, Chen Y (2016) Dynamic network surgery for efficient dnns. Adv Neural Inf Process Syst 29:1387–1395

    Google Scholar 

  11. Guo Z, Zhang X, Mu H, Heng W, Liu Z, Wei Y, Sun J (2020) Single path one-shot neural architecture search with uniform sampling. In: Proc. European conf. computer vision (ECCV), pp 544–560

  12. Han S, Pool J, Tran J, Dally WJ (2015) Learning both weights and connections for efficient neural networks. Adv Neural Inf Process Syst 28:1135–1143

    Google Scholar 

  13. Han S, Mao H, Dally WJ (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: Int. conf. on learning representations (ICLR)

  14. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc. IEEE conf. computer vision and pattern recognition, pp 770–778

  15. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: The IEEE int. conf. computer vision (ICCV), pp 1398–1406

  16. He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. In: Int. joint conf. artificial intelligence (IJCAI), pp 2234–2240

  17. He Y, Lin J, Liu Z, Wang H, Li LJ, Han S (2018) Amc: automl for model compression and acceleration on mobile devices. In: Proc. European conf. computer vision (ECCV), pp 784–800

  18. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proc. IEEE conf. computer vision and pattern recognition, pp 4340–4349

  19. He Y, Ding Y, Liu P, Zhu L, Zhang H, Yang Y (2020) Learning filter pruning criteria for deep convolutional neural networks acceleration. In: Proc. IEEE conf. computer vision and pattern recognition, pp –2018

  20. Hinton GE, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

  21. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proc. IEEE conf. computer vision and pattern recognition, pp 2261–2269

  22. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y (2017) Quantized neural networks: training neural networks with low precision weights and activations. J Mach Learn Res 18(1):6869–6898

    MathSciNet  MATH  Google Scholar 

  23. Hu H, Peng R, Tai YW, Tang CK (2016) Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250

  24. Jaderberg M, Vedaldi A, Zisserman A (2014) Speeding up convolutional neural networks with low rank expansions. In: British machine vision conference (BMVC)

  25. Krizhevsky A (2009) Learning multiple layers of features from tiny images. In: Technical report

  26. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  27. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. In: Int. conf. on learning representations (ICLR)

  28. Li B, Wu B, Su J, Wang G, Lin L (2020) Eagleeye: fast sub-net evaluation for efficient neural network pruning. In: Proc. European conf. computer vision (ECCV), pp 639–654

  29. Li Y, Gong R, Tan X, Yang Y, Hu P, Zhang Q, Yu F, Wang W, Gu S (2021) Brecq: pushing the limit of post-training quantization by block reconstruction. In: Int. conf. on learning representations (ICLR)

  30. Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann D (2019) Towards optimal structured cnn pruning via generative adversarial learning. In: Proc. IEEE conf. computer vision and pattern recognition, pp 2790–2799

  31. Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: Filter pruning using high-rank feature map. In: Proc. IEEE conf. computer vision and pattern recognition, pp 1529–1538

  32. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: The IEEE int. conf. computer vision (ICCV), pp 2755–2763

  33. Liu Z, Mu H, Zhang X, Guo Z, Yang X, Cheng KT, Sun J (2019) Metapruning: meta learning for automatic neural network channel pruning. In: The IEEE int. conf. computer vision (ICCV), pp 3296–3305

  34. Liu H, Simonyan K, Yang Y (2019) Darts: differentiable architecture search. In: Int. conf. on learning representations (ICLR)

  35. Luo JH, Wu J, Lin W (2017) Thinet: a filter level pruning method for deep neural network compression. In: The IEEE int. conf. computer vision (ICCV), pp 5068–5076

  36. Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2017) Pruning convolutional neural networks for resource efficient inference. In: Int. conf. on learning representations (ICLR)

  37. Ning X, Zhao T, Li W, Lei P, Wang Y, Yang H (2020) Dsa: more efficient budgeted pruning via differentiable sparsity allocation. In: Proc. European conf. computer vision (ECCV), pp 592–607

  38. Petsiuk V, Das A, Saenko K (2018) Rise: randomized input sampling for explanation of black-box models. In: British machine vision conference (BMVC), p 151

  39. Ren S, He K, Girshick R, Sun J (2017) Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  40. Shao M, Dai J, Kuang J, Meng D (2021) A dynamic CNN pruning method based on matrix similarity. Signal Image Video Process 15(2):381–389. https://doi.org/10.1007/s11760-020-01760-x

    Article  Google Scholar 

  41. Shao M, Dai J, Wang R, Zuo W (2021) CSHE: network pruning by using cluster similarity and matrix eigenvalues. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-021-01411-8

    Article  Google Scholar 

  42. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Int. conf. on learning representations (ICLR)

  43. Tang Y, Wang Y, Xu Y, Deng Y, Xu C, Tao D, Xu C (2021) Manifold regularized dynamic network pruning. In: Proc. IEEE conf. computer vision and pattern recognition, pp 5018–5028

  44. Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning structured sparsity in deep neural networks. Adv Neural Inf Process Syst 29:2074–2082

    Google Scholar 

  45. Yamamoto K (2021) Learnable companding quantization for accurate low-bit neural networks. In: Proc. IEEE conf. computer vision and pattern recognition, pp 5029–5038

  46. Ye J, Lu X, Lin ZL, Wang JZ (2018) Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. In: Int. conf. on learning representations (ICLR)

  47. You Z, Yan K, Ye J, Ma M, Wang P (2019) Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. Adv Neural Inf Process Syst 32:2133–2144

    Google Scholar 

  48. Yu J, Huang T (2019) Universally slimmable networks and improved training techniques. In: The ieee int. conf. computer vision (ICCV), pp 1803–1811

  49. Zhao C, Ni B, Zhang J, Zhao Q, Zhang W, Tian Q (2019) Variational convolutional neural network pruning. In: Proc. IEEE conf. computer vision and pattern recognition, pp 2780–2789

  50. Zhuang Z, Tan M, Zhuang B, Liu J, Guo Y, Wu Q, Huang J, Zhu J (2018) Discrimination-aware channel pruning for deep neural networks. Adv Neural Inf Process Syst 31:883–894

    Google Scholar 

Download references

Acknowledgements

The authors are very indebted to the anonymous referees for their critical comments and suggestions for the improvement of this paper. This work was supported by National Key Research and development Program of China (2021YFA1000102), and in part by the grants from the National Natural Science Foundation of China (nos. 61673396, 61976245, 61772344).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, China University of Petroleum, Qingdao, China

    Jiandong Kuang & Mingwen Shao

  2. College of Computer Science and Technology, Harbin Institute of Technology, Harbin, China

    Ran Wang & Wangmeng Zuo

  3. School of Information Science and Technology, Nantong University, Nantong, 226019, China

    Weiping Ding

Authors
  1. Jiandong Kuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingwen Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ran Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wangmeng Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weiping Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingwen Shao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kuang, J., Shao, M., Wang, R. et al. Network pruning via probing the importance of filters. Int. J. Mach. Learn. & Cyber. 13, 2403–2414 (2022). https://doi.org/10.1007/s13042-022-01530-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-022-01530-w

Keywords

Search

Navigation