Skip to main content
SpringerLink
Log in

A dynamic CNN pruning method based on matrix similarity

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Network pruning is one of the predominant approaches for deep model compression. Pruning large neural networks while maintaining their performance is often desirable because space and time complexity are reduced. Current pruning methods mainly focus on the importance of filters in the whole task. Different from previous methods, this paper focuses on the similarity between the filters or feature maps of the same layer. Firstly, cosine similarity is used as the matrix similarity measure to measure the similarity between channels, guiding the network to prune. Secondly, the proposed method is, respectively, applied to filters and feature maps pruning, and the pruning effects in different layers are summarized. Finally, we propose a method to set the pruning rate dynamically according to the situation of each layer. Our method obtains extremely sparse networks with virtually the same accuracy as the reference networks on the CIFAR-10 and ImageNet ILSVRC-12 classification tasks. On CIFAR-10, our network achieves the 52.70% compression ratio on ResNet-56 and increases only 0.13% on top-1 error.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Durand, T., Mehrasa, N., Mori, G.: Learning a deep convnet for multi-label classification with partial labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 647–657 (2019)

  2. Hu, M., Han, H., Shan, S., Chen, X.: Weakly supervised image classification through noise regularization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11517–11525 (2019)

  3. Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., et al.: Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 658–666 (2019)

  4. Li, P., Chen, X., Shen, S.: Stereo r-cnn based 3d object detection for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7644–7652 (2019)

  5. Zhang, P., Zhang, B., Chen, D., et al.: Cross-domain correspondence learning for exemplar-based image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5143–5153 (2020)

  6. Shaham, T.R., Dekel, T., Michaeli, T.: Learning a generative model from a single natural image. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4570–4580 (2019)

  7. Girdhar, R., Tran, D., Torresani, L., Ramanan, D.: Distinit: learning video representations without a single labeled video. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 852–861 (2019)

  8. Lin, T., Liu, X., Li, X., Ding, E., Wen, S.: Bmn: Boundary-matching network for temporal action proposal generation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3889–3898 (2019)

  9. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  10. Denil, M., Shakibi, B., Dinh, L., et al.: Predicting parameters in deep learning. In: Advances in Neural Information Processing Systems, pp. 2148–2156 (2013)

  11. Guo, Y., Yao, A., Chen, Y.: Dynamic network surgery for efficient dnns. In: Advances in Neural Information Processing Systems, pp. 1379–1387 (2019)

  12. Shang, W., Sohn, K., Almeida, D., et al.: Understanding and improving convolutional neural networks via concatenated rectified linear units. In: International Conference on Machine Learning, pp. 2217–2225 (2016)

  13. Zagoruyko, S., Omodakis, N.: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ArXiv preprint (2016)

  14. Lin, M., Ji, R., Wang, Y., et al.: HRank: filter pruning using high-rank feature map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529–1538 (2020)

  15. He, Y., Liu, P., Wang, Z., et al.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)

  16. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ArXiv preprint (2014)

  17. Szegedy, C., Liu, W., Jia, Y., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  18. Ayinde, B.O., Zurada, J.M.: Building efficient ConvNets using redundant feature pruning. In: CoRR arXiv:1802.07653 (2018)

  19. Pan, H., Diaa, B., Ahmet, E.C.: Computationally efficient wildfire detection method using a deep convolutional network pruned via Fourier analysis. Sensors 20(10), 2891 (2020)

    Article  Google Scholar 

  20. Srinivas, S., Babu, R.V.: Data-free parameter pruning for deep neural networks. ArXiv preprint (2015)

  21. You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 2130–2141 (2019)

  22. Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., et al.: Scnn: An accelerator for compressed-sparse convolutional neural networks. ACM SIGARCH Comput. Archit. News 45(2), 27–40 (2017)

    Article  Google Scholar 

  23. Ding, X., Ding, G., Guo, Y., Han, J., Yan, C.: Approximated oracle filter pruning for destructive cnn width optimization. ArXiv preprint (2019)

  24. Chen, Y.H., Krishna, T., Emer, J., Sze, E.: Eyeriss: an energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE J. Solid-State Circuits 52(1), 127–138 (2016)

    Article  Google Scholar 

  25. Paszke, P., Gross, S., Chintala, C., et al.: Automatic differentiation in PyTorch. In: Proceedings of Neural Information Processing Systems (2017)

  26. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. In: Handbook of Systemic Autoimmune Diseases (2009)

  27. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  28. Nesterov, Y.: A method of solving a convex programming problem with convergence rate o \((1/k^2)\). Sov. Math. Dokl (1983)

  29. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning fifilters for effificient ConvNets. ArXiv preprint (2017)

  30. Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January, pp. 2736–2744 (2017)

  31. He, Y., Kang, G., Don, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. ArXiv preprint (2018)

  32. Dong, X., Huang, J., Yang, Y., Yan, S.: More is less: A more complicated network with less inference complexity. In: Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January, pp. 5840–5848 (2017)

  33. He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January, pp. 1389–1397 (2017)

  34. Ye, J., Lu, X., Lin, Z., Wang, J.Z.: Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. ArXiv preprint (2018)

Download references

Acknowledgements

The authors are very indebted to the anonymous referees for their critical comments and suggestions for the improvement of this paper. This work was supported by the Grants from the National Natural Science Foundation of China (Nos. 61673396, 61976245) and the Fundamental Research Funds for the Central Universities (18CX02140A).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, China University of Petroleum, Qingdao, China

    Mingwen Shao, Junhui Dai & Jiandong Kuang

  2. School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, China

    Deyu Meng

Authors
  1. Mingwen Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junhui Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiandong Kuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deyu Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingwen Shao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, M., Dai, J., Kuang, J. et al. A dynamic CNN pruning method based on matrix similarity. SIViP 15, 381–389 (2021). https://doi.org/10.1007/s11760-020-01760-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-020-01760-x

Keywords

Search

Navigation