Skip to main content
SpringerLink
Log in

Group-based network pruning via nonlinear relationship between convolution filters

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Convolutional neural networks (CNNs) have many successful applications in various domains, but sometimes large computing resources are required. Therefore, pruning techniques are becoming increasingly popular for compressing and accelerating CNNs. Former commonly used group-based pruning methods address this issue using heuristic or linear grouping criteria (e.g., K-Means) without consideration of the high dimensionality and nonlinearity of convolution filters, which may lead to significant accuracy loss. In this paper, we propose a novel group-based pruning method, named kernel principal component analysis based group pruning (KPGP) method, which reduces the dimension of filters before grouping and uses nonlinear clustering criterion. Specifically, we use kernel principal component analysis (kernel-PCA) clustering to classify filters into groups, apply group pruning to each classified group, and reconstruct the pruned convolutional layers into group convolution structure. The proposed KPGP technique can maintain high performance according to its grouping criterion without the requirement of special hardware that many existing pruning methods do. Ablation study shows that the nonlinear clustering criterion in KPGP method is robust and effective. Moreover, we demonstrate the efficiency of KPGP method by applications to CIFAR-10 and ILSVRC-12, with negligible loss of accuracy, compared with state-of-the-art pruning methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. When 1/(1 − rℓ) < gℓ, the number of new input channels will be larger than the one before pruning. When 1/(1 − rℓ) = gℓ, the size of the new input is equal to the original one. In this case, the framework is similar to standard group convolution that both the numbers of input and output stay the same.

References

  1. Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional spaces. In: Proceedings of the 8th International Conference on Database Theory, ICDT ’01. Springer, Berlin, pp 420–434

  2. Cai Z, He X, Sun J, Vasconcelos N (2017) Deep learning with low precision by half-wave gaussian quantization. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5406–5414

  3. Chen G, Choi W, Yu X, Han TX, Chandraker MK (2017) Learning efficient object detection models with knowledge distillation. In: NIPS

  4. Chen Y, Fan H, Xu B, Yan Z, Kalantidis Y, Rohrbach MS, Yan S, Feng J (2020) Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In: 2019 IEEE/CVF International conference on computer vision (ICCV)

  5. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1800–1807

  6. Guo Q, Wu XJ, Kittler J, Feng Z (2020) Self-grouping convolutional neural networks. Neural Netw Official J Int Neural Netw Soc 132:491–505

    Article  Google Scholar 

  7. Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. In: NIPS

  8. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE Conference on computer vision & pattern recognition

  9. He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. In: Twenty-seventh international joint conference on artificial intelligence IJCAI-18

  10. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: 2019 IEEE/CVF Conference on computer vision and pattern recognition (CVPR)

  11. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. 2017 IEEE International Conference on Computer Vision (ICCV) pp 1398–1406

  12. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  13. Huang G, Liu S, van der Maaten L, Weinberger KQ (2018) Condensenet: An efficient densenet using learned group convolutions. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2752–2761

  14. Huang G, Liu Z, Weinberger KQ (2017) Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2261–2269

  15. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y (2016) Binarized neural networks. arXiv:1602.02505

  16. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y (2017) Quantized neural networks: Training neural networks with low precision weights and activations. J Mach Learn Res 18:187:1–187:30

    MathSciNet  MATH  Google Scholar 

  17. Iandola FN, Moskewicz MW, Ashraf K, Han S, Dally W, Keutzer K (2017) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 1mb model size. arXiv:1602.07360

  18. Ioannou YA, Robertson DP, Shotton J, Cipolla R, Criminisi A (2016) Training cnns with low-rank filters for efficient image classification. CoRR 1511.06744

  19. Jaderberg M, Vedaldi A, Zisserman A (2014) Speeding up convolutional neural networks with low rank expansions. arXiv:1405.3866

  20. Jordaan EM (2002) Development of robust inferential sensors : industrial application of support vector machines for regression / Annals of Operations Research

  21. Krizhevsky A, Hinton GE (2009) Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases 1(4)

  22. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25(2)

  23. LeCun Y, Denker JS, Solla SA (1990) Optimal brain damage. In: Touretzky D (ed) Advances in neural information processing systems (NIPS 1989), vol 2. Denver, Morgan Kaufman

  24. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. arXiv:1608.08710

  25. Li Y, Gu S, Zhang K, Gool L, Timofte R (2020) Dhp: Differentiable meta pruning via hypernetworks. arXiv:2003.13683

  26. Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann D (2019) Towards optimal structured cnn pruning via generative adversarial learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 2785–2794

  27. Liu B, Wang M, Foroosh H, Tappen M, Penksy M (2015) Sparse convolutional neural networks. In: 2015 IEEE Conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR.2015.7298681. IEEE Computer Society, Los Alamitos, pp 806–814

  28. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. 2017 IEEE International Conference on Computer Vision (ICCV) pp 2755–2763

  29. Liu Z, Sun M, Zhou T, Huang G, Darrell T (2019) Rethinking the value of network pruning. In: ICLR

  30. Luo JH, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression. 2017 IEEE International Conference on Computer Vision (ICCV) pp 5068–5076

  31. Mao H, Han S, Pool J, Li W, Liu X, Wang Y, Dally W (2017) Exploring the regularity of sparse structure in convolutional neural networks. arXiv:1705.08922

  32. Molchanov P, Mallya A, Tyree S, Frosio I, Kautz J (2019) Importance estimation for neural network pruning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 11:256–11:2 64

  33. Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2016) Pruning convolutional neural networks for resource efficient transfer learning. arXiv:1611.06440

  34. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in pytorch. In: NIPS-W

  35. Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: Imagenet classification using binary convolutional neural networks. arXiv:1603.05279

  36. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Li FF (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  37. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. CoRR 1409.1556

  38. Srivastava RK, Greff K, Schmidhuber J (2015) Training very deep networks. In: NIPS

  39. Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: a brief review. Comput Intell Neurosci 2018:1–13

    Google Scholar 

  40. Wang X, Kan M, Shan S, Chen X (2019) Fully learnable group convolution for acceleration of deep neural networks. In: 2019 IEEE/CVF Conference on computer vision and pattern recognition (CVPR), pp 9041–9050. https://doi.org/10.1109/CVPR.2019.00926

  41. Wang Y, Zhang X, Xie L, Zhou J, Su H, Zhang B, Hu X (2020) Pruning from scratch. Proc AAAI Conf Artif Intell 34:12,273–12,280

  42. Xie S, Girshick RB, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5987–5995

  43. Yim J, Joo D, Bae J, Kim J (2017) A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 7130–7138

  44. Yu N, Qiu S, Hu X, Li J (2017) Accelerating convolutional neural networks by group-wise 2d-filter pruning. In: 2017 International joint conference on neural networks (IJCNN), pp 2502–2509. https://doi.org/10.1109/IJCNN.2017.7966160

  45. Zhang T, Qi GJ, Xiao B, Wang J (2017) Interleaved group convolutions. 2017 IEEE International Conference on Computer Vision (ICCV), pp 4383–4392

  46. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: An extremely efficient convolutional neural network for mobile devices. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6848–6856

  47. Zhang X, Zou J, He K, Sun J (2016) Accelerating very deep convolutional networks for classification and detection. IEEE Trans Pattern Anal Mach Intell 38:1943–1955

    Article  Google Scholar 

  48. Zhang Z, Li J, Shao W, Peng Z, Zhang R, Wang X, Luo P (2019) Differentiable learning-to-group channels via groupable convolutional neural networks. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp 3541–3550

  49. Zhao R, xLuk W (2019) Efficient structured pruning and architecture searching for group convolution. In: 2019 IEEE/CVF International conference on computer vision workshop (ICCVW), pp 1961–1970. https://doi.org/10.1109/ICCVW.2019.00245

  50. Zhuo H, Qian X, Fu Y, Yang H, Xue X (2018) SCSP:, spectral clustering filter pruning with soft self-adaption manners. arXiv:1806.05320

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 11801409.

Author information

Authors and Affiliations

  1. Center for Combinatorics, Nankai University, Tianjin, China

    Guanqun Zhang

  2. Computer and Data Sciences, Case Western Reserve University, Cleveland, OH, USA

    Shuai Xu & Jing Li

  3. Center for Applied Mathematics, Tianjin University, Tianjin, China

    Alan J. X. Guo

Authors
  1. Guanqun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuai Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alan J. X. Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alan J. X. Guo.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, G., Xu, S., Li, J. et al. Group-based network pruning via nonlinear relationship between convolution filters. Appl Intell 52, 9274–9288 (2022). https://doi.org/10.1007/s10489-021-02907-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02907-0

Keywords

Search

Navigation