Skip to main content
SpringerLink
Log in

Kde-Entropy: preserve efficient filter

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

With the rapid development of convolutional neural networks (CNNs), higher accuracy is usually accompanied by huge parameters and calculations. In embedded devices with limited computing resources, huge network models are often difficult to deploy. It has been shown in previous researches that there are a large number of invalid filters in the CNNs network, and removing these invalid filters has little effect on network accuracy. Therefore, compressing CNNs by pruning these redundant filters can be done while maintaining accuracy. Model pruning is one of the main compression techniques for models. In the existing model pruning work, the filter with a small \({\ell _1}\)-norm value is pruned by calculating the \({\ell _1}\)-norm of the filter, but some researches have pointed out that the \({\ell _1}\)-norm is not always effective. This paper proposes a kernel density estimation-based entropy algorithm to prune filters with low information content. We experimentally demonstrate the effectiveness of the algorithm. Pruning VGG-16 model on CIFAR-10 datasets achieves an accuracy of 93.36% even after removing 91.97% of the original parameter count and 64.87% of the FLOPs. After pruning 75.84% of the network parameters and 52.35% of the FLOPs of the ResNet-110 model on the CIFAR-100 datasets, the accuracy remains almost the same.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

The data that support the findings of this study are openly available in [15] at https://www.cs.toronto.edu/kriz/cifar-10-python.tar.gz.

References

  1. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)

  2. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  3. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  4. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

  5. Lin, M., Ji, R., Li, S., Wang, Y., Wu, Y., Huang, F., Ye, Q.: Network pruning using adaptive exemplar filters. IEEE Trans. Neural Netw. Learn. Syst. 33(12), 7357–7366 (2021)

    Article  Google Scholar 

  6. Huang, Z., Wang, N.: Data-driven sparse structure selection for deep neural networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 304–320 (2018)

  7. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv:1608.08710 (2016)

  8. Lin, M., Ji, R., Zhang, Y., Zhang, B., Wu, Y., Tian, Y.: Channel pruning via automatic structure search. arXiv:2001.08565 (2020)

  9. Carreira-Perpinán, M.A., Idelbayev, Y.: “learning-compression” algorithms for neural net pruning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8532–8541 (2018)

  10. Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv:1510.00149 (2015)

  11. He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)

  12. Meng, F., Cheng, H., Li, K., Xu, Z., Ji, R., Sun, X., Lu, G.: Filter grafting for deep neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6599–6607 (2020)

  13. Silverman, B.W.: Density Estimation for Statistics and Data Analysis, vol. 26. CRC Press, Boca Raton (1986)

    Google Scholar 

  14. Luo, J.-H., Wu, J.: An entropy-based pruning method for CNN compression. arXiv:1706.05791 (2017)

  15. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

  16. LeCun, Y., Denker, J., Solla, S.: Optimal brain damage. Adv. Neural Inf. Process. Syst. 2 (1989)

  17. Hassibi, B., Stork, D.G., Wolff, G.J.: Optimal brain surgeon and general network pruning. In: IEEE International Conference on Neural Networks, pp. 293–299 (1993). IEEE

  18. Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., Xu, C.: Ghostnet: More features from cheap operations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1580–1589 (2020)

  19. Gao, H., Wang, Z., Ji, S.: Channelnets: Compact and efficient convolutional neural networks via channel-wise convolutions. Adv Neural Inf Process. Syst. 31 (2018)

  20. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (2017)

  21. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

  22. Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

  23. Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)

  24. Ma, N., Zhang, X., Zheng, H.-T., Sun, J.: Shufflenet v2: Practical guidelines for efficient CNN architecture design. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 116–131 (2018)

  25. Wu, J., Leng, C., Wang, Y., Hu, Q., Cheng, J.: Quantized convolutional neural networks for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4820–4828 (2016)

  26. Han, K., Wang, Y., Xu, Y., Xu, C., Wu, E., Xu, C.: Training binary neural networks through learning with noisy supervision. In: International Conference on Machine Learning, pp. 4017–4026 (2020). PMLR

  27. Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu, C., Xu, C., Tian, Q.: Data-free learning of student networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3514–3522 (2019)

  28. Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., Sugiyama, M.: Co-teaching: Robust training of deep neural networks with extremely noisy labels. Adv Neural Inf Process Syst 31 (2018)

  29. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv:1503.02531 (2015)

  30. Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I., Lempitsky, V.: Speeding-up convolutional neural networks using fine-tuned CP-decomposition. arXiv:1412.6553 (2014)

  31. Lin, S., Ji, R., Chen, C., Tao, D., Luo, J.: Holistic CNN compression via low-rank decomposition with knowledge transfer. IEEE Trans. Pattern Anal. Mach Intell. 41(12), 2889–2905 (2018)

    Article  Google Scholar 

  32. Fletcher, P.T., Venkatasubramanian, S., Joshi, S.: Robust statistics on Riemannian manifolds via the geometric median. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008). IEEE

  33. Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., Shao, L.: Hrank: Filter pruning using high-rank feature map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529–1538 (2020)

  34. Zagoruyko, S.: 92.45% on cifar-10 in torch. Torch Blog 11 (2015)

  35. Chen, Y., Wen, X., Zhang, Y., He, Q.: Fpc: Filter pruning via the contribution of output feature map for deep convolutional neural networks acceleration. Knowl-Based Syst. 238, 107876 (2022)

  36. Lin, S., Ji, R., Yan, C., Zhang, B., Cao, L., Ye, Q., Huang, F., Doermann, D.: Towards optimal structured cnn pruning via generative adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2790–2799 (2019)

  37. Zhao, C., Ni, B., Zhang, J., Zhao, Q., Zhang, W., Tian, Q.: Variational convolutional neural network pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2780–2789 (2019)

Download references

Funding

This research was funded by National Natural Science Foundation of China under Grant No: 12061066.

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, Northwest Normal University, PeiLi, Lanzhou, 730070, Gansu, China

    Long Wen, Yanjiao Yang, Bingyao Liu & Fengshun Zhang

Authors
  1. Long Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanjiao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bingyao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fengshun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LW and YY wrote the main manuscript text and BL and FZ prepared figures and tables. All authors reviewed the manuscript.

Corresponding author

Correspondence to Long Wen.

Ethics declarations

Conflict of interest

I (corresponding author) state that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 401 KB)

Supplementary file 2 (pdf 50 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, L., Yang, Y., Liu, B. et al. Kde-Entropy: preserve efficient filter. SIViP 18, 579–587 (2024). https://doi.org/10.1007/s11760-023-02763-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02763-0

Keywords

Search

Navigation