Skip to main content
SpringerLink
Log in

Global balanced iterative pruning for efficient convolutional neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

With the increase of structure complexity, convolutional neural networks (CNNs) take a fair amount of computation cost. Meanwhile, existing research reveals the salient parameter redundancy in CNNs. The current pruning methods can compress CNNs with little performance drop, but when the pruning ratio increases, the accuracy loss is more serious and the compressing rates of parameters and floating-point operations (FLOPs) are unbalanced. Moreover, the existing iterative pruning methods are difficult to accurately identify and delete unimportant parameters due to the accuracy drop during pruning. We propose a novel global balanced iterative pruning method (GBIP) for CNNs. Firstly, a global equilibrium pruning strategy based on feature distribution is proposed. Then the intermediate and output features of original network are applied to guide the fine-tuning of pruned network. Moreover, we design a shallow fully-connected network to allow the output of two networks to play an adversarial game, thereby it can quickly recover the pruned accuracy among iterative pruning intervals. We conduct extensive experiments on the image classification tasks CIFAR-10, CIFAR-100, and ILSVRC-2012 to verify our pruning method can achieve efficient compression for CNNs even without accuracy loss. On the ILSVRC-2012, when removing 36.78% parameters and 45.55% FLOPs of ResNet-18, the Top-1 accuracy drop are only 0.66%. Our method is superior to some state-of-the-art pruning schemes in terms of compressing rate and accuracy. Moreover, we further demonstrate that GBIP has good generalization on the object detection task PASCAL VOC.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

All data generated or analysed during this study are included in this article.

References

  1. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  2. Han S, Pool J, Tran J, Dally WJ (2015) Learning both weights and connections for efficient neural networks. In: NIPS, vol 28

  3. Xu S, Chen H, Gong X, Liu K, Lü J, Zhang B (2021) Efficient structured pruning based on deep feature stabilization. Neural Comput Appl 33(13):7409–7420. https://doi.org/10.1007/s00521-021-05828-8

    Article  Google Scholar 

  4. Liu S, Ni’mah I, Menkovski V, Mocanu DC, Pechenizkiy M (2021) Efficient and effective training of sparse recurrent neural networks. Neural Comput Appl 33(15):9625–9636. https://doi.org/10.1007/s00521-021-05727-y

    Article  Google Scholar 

  5. Komodakis N, Zagoruyko S (2017) Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: ICLR

  6. Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann DS (2019) Towards optimal structured CNN pruning via generative adversarial learning. In: CVPR, pp 2790–2799. https://doi.org/10.1109/CVPR.2019.00290

  7. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR

  8. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  9. Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: CVPR, pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594

  10. Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images

  11. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein MS, Berg AC, Li F (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  12. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) SSD: Single Shot MultiBox Detector. In: ECCV, vol 9905, pp 21–37. https://doi.org/10.1007/978-3-319-46448-0_2

  13. Everingham M, Eslami SMA, Van Gool L, Williams CKI, Winn J, Zisserman A (2015) The PASCAL visual object classes challenge: a retrospective. Int J Comput Vis 111(1):98–136. https://doi.org/10.1007/s11263-014-0733-5

    Article  Google Scholar 

  14. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. In: ICLR

  15. Polyak A, Wolf L (2015) Channel-level acceleration of deep face representations. IEEE Access 3:2163–2175. https://doi.org/10.1109/ACCESS.2015.2494536

    Article  Google Scholar 

  16. He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. In: IJCAI, pp 2234–2240. https://doi.org/10.24963/ijcai.2018/309

  17. Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2017) Pruning convolutional neural networks for resource efficient inference. In: 5th International conference on learning representations, ICLR. https://openreview.net/forum?id=SJGCiw5gl

  18. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: ICCV, pp 2755–2763. https://doi.org/10.1109/ICCV.2017.298

  19. Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: filter pruning using high-rank feature map. In: CVPR, pp 1526–1535. https://doi.org/10.1109/CVPR42600.2020.00160

  20. Li H, Ma C, Xu W, Liu X (2020) Feature statistics guided efficient filter pruning. In: IJCAI, pp 2619–2625. https://doi.org/10.24963/ijcai.2020/363

  21. Tang Y, Wang Y, Xu Y, Deng Y, Xu C, Tao D, Xu C (2021) Manifold regularized dynamic network pruning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5018–5028

  22. Wu H, Tang Y, Zhang X (2021) A pruning method based on the measurement of feature extraction ability. Mach Vis Appl 32(1):1–11. https://doi.org/10.1007/s00138-020-01148-4

    Article  Google Scholar 

  23. Chin T-W, Ding R, Zhang C, Marculescu D (2020) Towards efficient model compression via learned global ranking. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1515–1525. https://doi.org/10.1109/CVPR42600.2020.00159

  24. Yu R, Li A, Chen C-F, Lai J-H, Morariu VI, Han X, Gao M, Lin C-Y, Davis LS (2018) Nisp: pruning networks using neuron importance score propagation. In: CVPR, pp 9194–9203. https://doi.org/10.1109/CVPR.2018.00958

  25. Lee N, Ajanthan T, Torr PHS (2019) Snip: single-shot network pruning based on connection sensitivity. In: ICLR

  26. Guo S, Wang Y, Li Q, Yan J (2020) DMCP: differentiable Markov channel pruning for neural networks. In: CVPR, pp 1536–1544. https://doi.org/10.1109/CVPR42600.2020.00161

  27. You Z, Yan K, Ye J, Ma M, Wang P (2019) Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. In: NeurIPS, pp 2130–2141

  28. Guo J, Ouyang W, Xu D (2020) Channel pruning guided by classification loss and feature importance. Proc. AAAI Conf. Artif. Intell. 34:10885–10892

    Google Scholar 

  29. He Y, Lin J, Liu Z, Wang H, Li L-J, Han S (2018) Amc: Automl for model compression and acceleration on mobile devices. In: ECCV, pp 815–832. https://doi.org/10.1007/978-3-030-01234-2_48

  30. Liu Z, Mu H, Zhang X, Guo Z, Yang X, Cheng K-T, Sun J (2019) Metapruning: meta learning for automatic neural network channel pruning. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 3296–3305

  31. Lin M, Ji R, Zhang Y, Zhang B, Wu Y, Tian Y (2020) Channel pruning via automatic structure search. In: IJCAI, pp 673–679. https://doi.org/10.24963/ijcai.2020/94

  32. Ding G, Zhang S, Jia Z, Zhong J, Han J (2020) Where to prune: using lstm to guide data-dependent soft pruning. IEEE Trans Image Process 30:293–304. https://doi.org/10.1109/TIP.2020.3035028

    Article  Google Scholar 

  33. Ba J, Caruana R (2014) Do deep nets really need to be deep? In: NeurIPS, pp 2654–2662

  34. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network

  35. Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2015) Fitnets: hints for thin deep nets. In: ICLR

  36. Liu Z, Sun M, Zhou T, Huang G, Darrell T (2019) Rethinking the value of network pruning. In: ICLR

  37. Zhuang Z, Tan M, Zhuang B, Liu J, Guo Y, Wu Q, Huang J, Zhu J (2018) Discrimination-aware channel pruning for deep neural networks. In: NeurIPS, pp 883–894

  38. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4340–4349. https://doi.org/10.1109/CVPR.2019.00447

  39. Ruan X, Liu Y, Yuan C, Li B, Hu W, Li Y, Maybank S (2020) Edp: an efficient decomposition and pruning scheme for convolutional neural network compression. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2020.3018177

    Article  Google Scholar 

  40. Li T, Wu B, Yang Y, Fan Y, Zhang Y, Liu W (2019) Compressing convolutional neural networks via factorized convolutional filters. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3977–3986. https://doi.org/10.1109/CVPR.2019.00410

  41. Peng H, Wu J, Chen S, Huang J (2019) Collaborative channel pruning for deep networks. In: International conference on machine learning, pp 5113–5122

  42. Molchanov P, Mallya A, Tyree S, Frosio I, Kautz J (2019) Importance estimation for neural network pruning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11264–11272. https://doi.org/10.1109/CVPR.2019.01152

  43. Gao S, Huang F, Cai W, Huang H (2021) Network pruning via performance maximization. In: IEEE conference on computer vision and pattern recognition, CVPR 2021, pp 9270–9280

  44. Zhao C, Ni B, Zhang J, Zhao Q, Zhang W, Tian Q (2019) Variational convolutional neural network pruning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2780–2789. https://doi.org/10.1109/CVPR.2019.00289

  45. Ding G, Zhang S, Jia Z, Zhong J, Han J (2021) Where to prune: using LSTM to guide data-dependent soft pruning. IEEE Trans Image Process 30:293–304. https://doi.org/10.1109/TIP.2020.3035028

    Article  Google Scholar 

  46. He Y, Ding Y, Liu P, Zhu L, Zhang H, Yang Y (2020) Learning filter pruning criteria for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2009–2018. https://doi.org/10.1109/CVPR42600.2020.00208

  47. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE international conference on computer vision, pp 1389–1397

  48. Gao S, Huang F, Pei J, Huang H (2020) Discrete model compression with resource constraint for deep neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1899–1908. https://doi.org/10.1109/CVPR42600.2020.00197

  49. Wang Z, Li C, Wang X (2021) Convolutional neural network pruning with structural redundancy reduction. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14913–14922

  50. Wang Z, Li C (2022) Channel pruning via lookahead search guided reinforcement learning. In: IEEE/CVF winter conference on applications of computer vision, WACV, pp 3513–3524. https://doi.org/10.1109/WACV51458.2022.00357

  51. Wang H, Qin C, Zhang Y, Fu Y (2021) Neural pruning via growing regularization. In: 9th International conference on learning representations, ICLR 2021. https://openreview.net/forum?id=o966_Is_nPA

  52. Singh P, Kadi VSR, Namboodiri VP (2020) FALF convnets: fatuous auxiliary loss based filter-pruning for efficient deep cnns. Image Vis Comput 93:103857. https://doi.org/10.1016/j.imavis.2019.103857

    Article  Google Scholar 

  53. Guan Y, Liu N, Zhao P, Che Z, Bian K, Wang Y, Tang J (2022) Dais: automatic channel pruning via differentiable annealing indicator search. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2022.3161284

    Article  Google Scholar 

  54. Dong X, Huang J, Yang Y, Yan S (2017) More is less: a more complicated network with less inference complexity. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5840–5848. https://doi.org/10.1109/CVPR.2017.205

  55. Ning X, Zhao T, Li W, Lei P, Wang Y, Yang H (2020) Dsa: more efficient budgeted pruning via differentiable sparsity allocation. In: ECCV 2020, pp 592–607. https://doi.org/10.1007/978-3-030-58580-8_35

  56. Liebenwein L, Baykal C, Lang H, Feldman D, Rus D (2020) Provable filter pruning for efficient neural networks. In: ICLR 2020

  57. Gao X, Zhao Y, Dudziak L, Mullins RD, Xu C (2019) Dynamic channel pruning: feature boosting and suppression. In: ICLR 2019

  58. Luo J, Zhang H, Zhou H, Xie C, Wu J, Lin W (2019) Thinet: pruning CNN filters for a thinner net. IEEE Trans Pattern Anal Mach Intell 41(10):2525–2538. https://doi.org/10.1109/TPAMI.2018.2858232

    Article  Google Scholar 

  59. Molchanov P, Mallya A, Tyree S, Frosio I, Kautz J (2019) Importance estimation for neural network pruning. In: CVPR, pp 11264–11272. https://doi.org/10.1109/CVPR.2019.01152

  60. O’Keeffe S, Villing R (2018) Evaluating extended pruning on object detection neural networks. In: 2018 29th Irish signals and systems conference (ISSC), pp 1–6. https://doi.org/10.1109/ISSC.2018.8585345

Download references

Acknowledgements

This work was supported in part by the Anhui Provincial Key Research and Development Program under Grant 202004a05020040, in part by the National Key Research and Development Program under Grant 2018YFC0604404, in part by Intelligent Network and New Energy Vehicle Special Project of Intelligent Manufacturing Institute of HFUT under Grant IMIWL2019003, and in part by Fundamental Research Funds for the Central Universities under Grant PA2021GDGP0061.

Author information

Authors and Affiliations

  1. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230009, China

    Jingfei Chang, Yang Lu, Ping Xue, Yiqun Xu & Zhen Wei

  2. Anhui Mine IOT and Security Monitoring Technology Key Laboratory, Hefei, 230088, China

    Yang Lu & Zhen Wei

  3. Engineering Research Center of Safety Critical Industrial Measurement and Control Technology, Hefei University of Technology, Ministry of Education, Hefei, 230009, China

    Yang Lu & Zhen Wei

Authors
  1. Jingfei Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ping Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yiqun Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhen Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yang Lu or Zhen Wei.

Ethics declarations

Conflict of interest

We declare that we have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chang, J., Lu, Y., Xue, P. et al. Global balanced iterative pruning for efficient convolutional neural networks. Neural Comput & Applic 34, 21119–21138 (2022). https://doi.org/10.1007/s00521-022-07594-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07594-7

Keywords

Search

Navigation