Skip to main content

Efficient Channel Pruning via Architecture-Guided Search Space Shrinking

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13534))

Included in the following conference series:

  • 3036 Accesses

Abstract

Recently, channel pruning methods search for the optimal channel numbers by training a weight-sharing network to evaluate architectures of subnetworks. However, the weight shared between subnetworks incurs severe evaluation bias and an accuracy drop. In this paper, we provide a comprehensive understanding of the search space’s impact on the evaluation by dissecting the training process of the weight-sharing network analytically. Specifically, it is proved that the sharing weights induce biased noise on gradients, whose magnitude is proportional to the search range of channel numbers and bias is relative to the average channel numbers of the search space. Motivated by the theoretical result, we design a channel pruning method by training a weight-sharing network with search space shrinking. The search space is iteratively shrunk guided by the optimal architecture searched in the weight-sharing network. The reduced search space boosts the accuracy of the evaluation and significantly cuts down the post-processing computation of finetuning. In the end, we demonstrate the superiority of our channel pruning method over state-of-the-art methods with experiments on ImageNet and COCO.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Lin, M., et al.: Hrank: filter pruning using high-rank feature map. In: 2020 IEEE/CVF Conference on Computer Vision, Pattern Recognition (CVPR) (2020)

    Google Scholar 

  2. He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: 2019 IEEE/CVF Conference on Computer Vision, Pattern Recognition (CVPR) (2019)

    Google Scholar 

  3. Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: 2019 IEEE/CVF Conference on Computer Vision, Pattern Recognition (CVPR) (2019)

    Google Scholar 

  4. Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.H.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems (2016)

    Google Scholar 

  5. You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2019)

    Google Scholar 

  6. Wang, J., et al.: Revisiting parameter sharing for automatic neural channel number search. In: Advances in Neural Information Processing Systems (2020)

    Google Scholar 

  7. Chen, Z., Niu, J., Xie, L., Liu, X., Wei, L., Tian, Q.: Network adjustment: channel search guided by flops utilization ratio. In: 2020 IEEE/CVF Conference on Computer Vision, Pattern Recognition (CVPR) (2020)

    Google Scholar 

  8. Liu, Z., et al.: Metapruning: meta learning for automatic neural network channel pruning. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  9. Li, B., Wu, B., Su, J., Wang, G.: EagleEye: fast sub-net evaluation for efficient neural network pruning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 639–654. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_38

    Chapter  Google Scholar 

  10. Yu, J., Huang, T.: Autoslim: towards oneshot architecture search for channel numbers. In: Advances in Neural Information Processing Systems (2019)

    Google Scholar 

  11. Li, X., et al.: Improving one-shot NAS by suppressing the posterior fading. In: 2020 IEEE/CVF Conference on Computer Vision, Pattern Recognition (CVPR) (2020)

    Google Scholar 

  12. Nayman, N., Noy, A., Ridnik, T., Friedman, I., Jin, R., Zelnik-Manor, L.: XNAS: neural architecture search with expert advice. In: Advances in Neural Information Processing Systems (2019)

    Google Scholar 

  13. Fang, M., Wang, Q., Zhong, Z.: Betanas: balanced training and selective drop for neural architecture search (2019)

    Google Scholar 

  14. Hu, Y., et al.: Angle-based search space shrinking for neural architecture search. In: European Conference on Computer Vision (2020)

    Google Scholar 

  15. Pourchot, A., Ducarouge, A., Sigaud, O.: To share or not to share: a comprehensive appraisal of weight-sharing (2020)

    Google Scholar 

  16. Ding, X., Ding, G., Guo, Y., Han, J., Yan, C.: Approximated oracle filter pruning for destructive CNN width optimization. In: International Conference on Machine Learning (2019)

    Google Scholar 

  17. Joo, D., Yi, E., Baek, S., Kim, J.: Linearly replaceable filters for deep network channel pruning. In: The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-2021) (2021)

    Google Scholar 

  18. Li, Y., et al.: Towards compact CNNs via collaborative compression. In: 2021 IEEE/CVF Conference on Computer Vision, Pattern Recognition (CVPR) (2021)

    Google Scholar 

  19. Guo, S., Wang, Y., Li, Q., Yan, J.: DMCP: differentiable Markov channel pruning for neural networks. In: 2020 IEEE/CVF Conference on Computer Vision, Pattern Recognition (CVPR) (2020)

    Google Scholar 

  20. Su, X., You, S., Wang, F., Qian, C., Zhang, C., Xu, C.: BCNet: searching for network width with bilaterally coupled network. In: 2021 IEEE/CVF Conference on Computer Vision, Pattern Recognition (CVPR) (2021)

    Google Scholar 

  21. Yu, J., Yang, L., Xu, N., Yang, J., Huang, T.S.: Slimmable neural networks. In: International Conference on Learning Representations (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zheyang Li .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 610 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, Z., Li, Z. (2022). Efficient Channel Pruning via Architecture-Guided Search Space Shrinking. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13534. Springer, Cham. https://doi.org/10.1007/978-3-031-18907-4_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-18907-4_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18906-7

  • Online ISBN: 978-3-031-18907-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics