Skip to main content
SpringerLink
Log in

Efficient structured pruning based on deep feature stabilization

  • S.I. : DICTA 2019
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The application of convolutional neural networks (CNNs) in computer vision highly depends on the consumption of computation and memory resources, which affects its development on resource-limited devices. Accordingly, CNN compression has attracted increasing attention. In this paper, we propose an efficient end-to-end pruning method based on feature stabilization (EPFS), which is feasible to be implemented for structured pruning such as filter pruning and block pruning. For block pruning, we introduce a mask to scale the output of structures and the \(\ell _1\)-regularization term to sparsify the mask. For filter pruning, a novel \(\ell _2\)-regularization term is proposed to constraint the mask along with the \(\ell _1\)-regularization. Besides, we introduce the Center Loss to stabilize the deep feature and fast iterative shrinkage-thresholding algorithm (FISTA) to accelerate the convergence of mask. Extensive experiments demonstrate the superiority of our EPFS. On CIFAR-10, EPFS saves \(47.5\%\) FLOPs on VGGNet with \(1.17\%\) Top-1 accuracy increase. Furthermore, on ImageNet ILSVRC2012, EPFS reduces \(55.2\%\) FLOPs on ResNet-18 with o.nly \(1.63\%\) Top-1 accuracy decrease, which promotes the state-of-the-arts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. The number of floating-point operations.

References

  1. Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202

    Article  MathSciNet  Google Scholar 

  2. Denil M, Shakibi B, Dinh L, Ranzato M, De Freitas N (2013) Predicting parameters in deep learning. In: Advances in neural information processing systems (NIPS), pp 2148–2156

  3. Ding X, Ding G, Guo Y, Han J (2019) Centripetal sgd for pruning very deep convolutional networks with complicated structure. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4943–4953

  4. Dong X, Huang J, Yang Y, Yan S (2017) More is less: A more complicated network with less inference complexity. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 5840–5848

  5. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  6. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 580–587

  7. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: International conference on artificial intelligence and statistics (AISTATS), pp 315–323

  8. Goldstein T, Studer C, Baraniuk R (2014) A field guide to forward-backward splitting with a fasta implementation. arXiv preprint arXiv:1411.3406

  9. Gu J, Li C, Zhang B, Han J, Cao X, Liu J, Doermann D (2019) Projection convolutional neural networks for 1-bit cnns via discrete back propagation. In: AAAI conference on artificial intelligence (AAAI) vol 33, pp 8344–8351

  10. Gu J, Zhao J, Jiang X, Zhang B, Liu J, Guo G, Ji R (2019) Bayesian optimized 1-bit cnns. In: IEEE international conference on computer vision (ICCV), pp 4909–4917

  11. Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems (NIPS), pp 1135–1143

  12. Hassibi B, Stork DG, Wolff GJ (1993) Optimal brain surgeon and general network pruning. In: IEEE international conference on neural networks (ICNN), pp 293–299

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  14. He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. In: International joint conference on artificial intelligence (IJCAI), pp 2234–2240

  15. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4340–4349

  16. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: IEEE International conference on computer vision (ICCV), pp 1389–1397

  17. Howard A, Sandler M, Chu G, Chen LC, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V, et al (2019) Searching for mobilenetv3. In: IEEE international conference on computer vision (ICCV), pp 1314–1324

  18. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  19. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning (ICML), pp 448–456

  20. Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images. Tech. rep, Citeseer

  21. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems (NIPS), pp 1097–1105

  22. LeCun Y, Denker JS, Solla SA (1990) Optimal brain damage. In: Advances in neural information processing systems (NIPS), pp 598–605

  23. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. In: International conference on learning representations (ICLR), pp 1–13

  24. Lin S, Ji R, Chen C, Huang F (2017) Espace: Accelerating convolutional neural networks via eliminating spatial and channel redundancy. In: AAAI conference on artificial intelligence (AAAI), pp 1424–1430

  25. Lin S, Ji R, Li Y, Wu Y, Huang F, Zhang B (2018) Accelerating convolutional networks via global & dynamic filter pruning. In: International joint conference on artificial intelligence (IJCAI), pp 2425–2432

  26. Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann D (2019) Towards optimal structured cnn pruning via generative adversarial learning. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2790–2799

  27. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision (ECCV), pp 740–755

  28. Luan S, Chen C, Zhang B, Han J, Liu J (2018) Gabor convolutional networks. IEEE Trans Image Process 27(9):4357–4366

    Article  MathSciNet  Google Scholar 

  29. Luo JH, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression. In: IEEE international conference on computer vision (ICCV), pp 5058–5066

  30. Mathieu M, Henaff M, LeCun Y (2014) Fast training of convolutional networks through ffts. In: International conference on learning representations (ICLR), pp 1–9

  31. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, et al (2019) Pytorch: An imperative style, high-performance deep learning library. In: Advances in neural information processing systems (NIPS), pp 8026–8037

  32. Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: Imagenet classification using binary convolutional neural networks. In: European conference on computer vision (ECCV), pp 525–542

  33. Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2014) Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550

  34. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  35. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4510–4520

  36. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations (ICLR), pp 1–15

  37. Singh P, Verma VK, Rai P, Namboodiri V (2020) Leveraging filter correlations for deep model compression. In: The IEEE winter conference on applications of computer vision (WACV), pp 835–844

  38. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9

  39. Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B (Methodol) 58(1):267–288

    MathSciNet  MATH  Google Scholar 

  40. Wang X, Zhang B, Li C, Ji R, Han J, Cao X, Liu J (2018) Modulated convolutional networks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 840–848

  41. Wang Y, Xu C, You S, Tao D, Xu C (2016) Cnnpack: Packing convolutional neural networks in the frequency domain. In: Advances in neural information processing systems (NIPS), pp 253–261

  42. Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: European conference on computer vision (ECCV), pp 499–515

  43. Xu S, Chen H, Liu K, Lii J, Zhang B (2019) Efficient block pruning based on kernel and feature stablization. In: International conference on digital image computing: techniques and applications (DICTA), pp 1–6

  44. Yu R, Li A, Chen CF, Lai JH, Morariu VI, Han X, Gao M, Lin CY, Davis LS (2018) Nisp: Pruning networks using neuron importance score propagation. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 9194–9203

  45. Zhang Z, Saligrama V (2015) Rapid: Rapidly accelerated proximal gradient algorithms for convex minimization. In: International conference on acoustics, speech and signal processing (ICASSP), pp 3796–3800

  46. Zoph B, Le QV (2017) Neural architecture search with reinforcement learning. In: International conference on learning representations (ICLR), pp 1–16

Download references

Acknowledgements

Baochang Zhang is also with Shenzhen Academy of Aerospace Technology, Shenzhen, China, and he is the corresponding author. He is in part Supported by Shenzhen Science and Technology Program (No.KQTD2016112515134654). The work was supported by the Natural Science Foundation of China (62076016). Baochang Zhang is in part supported by Shenzhen Science and Technology Program (No.KQTD2016112515134654). This study was supported by Grant NO.2019JZZY011101 from the Key Research and Development Program of Shandong Province to Dianmin Sun.

Author information

Authors and Affiliations

  1. School of Automatic Science and Electrical Engineering, Beihang University, Beijing, 100191, China

    Sheng Xu, Hanlin Chen & Baochang Zhang

  2. Department of Computer Science and Engineering, University at Buffalo, Buffalo, 14260, NY, USA

    Xuan Gong

  3. School of Automatic Science and Electrical Engineering, State Key Laboratory of Software Development Environment Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University, Beijing, 100191, China

    Kexin Liu & Jinhu Lü

  4. Shenzhen Academy of Aerospace Technology, Shenzhen, 518057, China

    Baochang Zhang

Authors
  1. Sheng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanlin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuan Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kexin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jinhu Lü
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Baochang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Baochang Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, S., Chen, H., Gong, X. et al. Efficient structured pruning based on deep feature stabilization. Neural Comput & Applic 33, 7409–7420 (2021). https://doi.org/10.1007/s00521-021-05828-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-05828-8

Keywords

Search

Navigation