Skip to main content
Springer Nature Link
Log in

An effective two-stage channel pruning method based on two-dimensional information entropy

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Channel pruning can reduce the number of neural network parameters and computational cost by eliminating redundant channels, its main purpose is to adapt to resource constrained devices. Evaluation-based global pruning and network search-based pruning are two common methods of channel pruning. However, the network architecture pruned by the global mask is often not optimal, while the method that directly searches for the optimal architecture will introduce a large number of hyperparameters, which greatly increases the training cost. In this paper, we propose a novel Two-dimensional information Entropy based Channel Pruning method (TECP). The pruning process consists of two steps. First, a global mask pruning scheme is employed to obtained a pre-pruning model. Then, the two-dimensional information entropy is calculated by using feature maps of dense network to adjust the pre-pruning model adaptively to get a compact network. Moreover, the entropy values are used to determine the minimum number of reserved channels per layer based on to avoid the imbalance of network architecture and the layer collapse caused by global pruning. Extensive experiments with a variety of networks on several datasets clearly demonstrate the effectiveness of our proposed TECP method. For example, results show that on CIFAR-10, the compressed model achieves comparable accuracy to the original model, but with a significantly lower number of parameters (44.29% for ResNet-20 and 46.79% for VGG-16). This is beneficial for industrial deployment. And experimental results also show that TECP method obtain the better performance compared with state-of-the-art method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

No data was used for the research described in the article.

References

  1. Masana M, Liu X, Twardowski B et al (2023) Class-incremental learning: Survey and performance evaluation on image classification. IEEE Trans Pattern Anal Mach Intell 45:5513–5533. https://doi.org/10.1109/TPAMI.2022.3213473

    Article  Google Scholar 

  2. Ahmad HM, Rahimi A (2022) Deep learning methods for object detection in smart manufacturing: A survey. J Manuf Syst 64:181–196. https://doi.org/10.1016/j.jmsy.2022.06.011

    Article  Google Scholar 

  3. Aljabri M, AlGhamdi M (2022) A review on the use of deep learning for medical images segmentation. Neurocomputing 506:311–335. https://doi.org/10.1016/j.neucom.2022.07.070

    Article  Google Scholar 

  4. Paymode AS, Malode VB (2022) Transfer learning for multi-crop leaf disease image classification using convolutional neural network vgg. Artif Intell Agric 6:23–33. https://doi.org/10.1016/j.aiia.2021.12.002

    Article  Google Scholar 

  5. Hadipour-Rokni R, Askari Asli-Ardeh E, Jahanbakhshi A et al (2023) Intelligent detection of citrus fruit pests using machine vision system and convolutional neural network through transfer learning technique. Comput Biol Med 155:106611. https://doi.org/10.1016/j.compbiomed.2023.106611

    Article  Google Scholar 

  6. Mohanta BK, Jena D, Satapathy U et al (2020) Survey on iot security: Challenges and solution using machine learning, artificial intelligence and blockchain technology. Internet of Things 11:100227. https://doi.org/10.1016/j.iot.2020.100227

    Article  Google Scholar 

  7. Yang J, Wang Y, Zhao H et al (2022) Mobilenet and knowledge distillation-based automatic scenario recognition method in vehicle-to-vehicle systems. IEEE Trans Veh Technol 71:11006–11016. https://doi.org/10.1109/TVT.2022.3184994

    Article  Google Scholar 

  8. Yang H, Liu J, Mei G et al (2023) Research on real-time detection method of rail corrugation based on improved shufflenet v2. Eng Appl Artif Intell 126:106825. https://doi.org/10.1016/j.engappai.2023.106825

    Article  Google Scholar 

  9. Liang T, Glossner J, Wang L et al (2021) Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing 461:370–403. https://doi.org/10.1016/j.neucom.2021.07.045

    Article  Google Scholar 

  10. Shuvo MMH, Islam SK, Cheng J et al (2023) Efficient acceleration of deep learning inference on resource-constrained edge devices: A review. Proc IEEE 111:42–91. https://doi.org/10.1109/JPROC.2022.3226481

    Article  Google Scholar 

  11. Zhao R, Gui G, Xue Z et al (2022) A novel intrusion detection method based on lightweight neural network for internet of things. IEEE Internet of Things Journal 9:9960–9972. https://doi.org/10.1109/JIOT.2021.3119055

    Article  Google Scholar 

  12. Chang J, Lu Y, Xue P et al (2022) Automatic channel pruning via clustering and swarm intelligence optimization for cnn. Appl Intell 52:17751–17771. https://doi.org/10.1007/s10489-022-03508-1

    Article  Google Scholar 

  13. Peng J, Sun W, Li HC et al (2022) Low-rank and sparse representation for hyperspectral image processing: A review. IEEE Geosci Remote Sens Mag 10:10–43. https://doi.org/10.1109/MGRS.2021.3075491

    Article  Google Scholar 

  14. Chu Y, Li P, Bai Y et al (2022) Group channel pruning and spatial attention distilling for object detection. Appl Intell 52:16246–16264. https://doi.org/10.1007/s10489-022-03293-x

    Article  Google Scholar 

  15. Abdar M, Pourpanah F, Hussain S et al (2021) A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Inf Fusion 76:243–297. https://doi.org/10.1016/j.inffus.2021.05.008

    Article  Google Scholar 

  16. Tanaka H, Kunin D, Yamins DL, et al (2020) Pruning neural networks without any data by iteratively conserving synaptic flow. In: Advances in neural information processing systems, vol 33. Curran Associates, Inc., pp 6377–6389. https://proceedings.neurips.cc/paper_files/paper/2020/file/46a4378f835dc8040c8057beb6a2da52-Paper.pdf

  17. Yeom SK, Seegerer P, Lapuschkin S et al (2021) Pruning by explaining: A novel criterion for deep neural network pruning. Pattern Recognit 115:107899. https://doi.org/10.1016/j.patcog.2021.107899

    Article  Google Scholar 

  18. Tessier H, Gripon V, Léonardon M et al (2022) Rethinking weight decay for efficient neural network pruning. J Imaging 8:64. https://doi.org/10.3390/jimaging8030064

    Article  Google Scholar 

  19. Wimmer P, Mehnert J, Condurache A (2022) Interspace pruning: Using adaptive filter representations to improve training of sparse cnns. In: 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 12517–12527 https://doi.org/10.1109/CVPR52688.2022.01220

  20. Abdelfattah A, Costa T, Dongarra J et al (2021) A set of batched basic linear algebra subprograms and lapack routines. ACM Trans Math Softw 47. https://doi.org/10.1145/3431921

  21. Zhang Y, Freris NM (2023) Adaptive filter pruning via sensitivity feedback. IEEE Transactions on Neural Networks and Learning Systems, pp 1–13. https://doi.org/10.1109/TNNLS.2023.3246263

  22. He Y, Liu P, Wang Z, et al (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 4335–4344 https://doi.org/10.1109/CVPR.2019.00447

  23. Zhu J, Pei J (2022) Progressive kernel pruning cnn compression method with an adjustable input channel. Appl Intell 52:1–22. https://doi.org/10.1007/s10489-021-02932-z

    Article  Google Scholar 

  24. Yang W, Xiao Y (2022) Structured pruning via feature channels similarity and mutual learning for convolutional neural network compression. Appl Intell 52:14560–14570. https://doi.org/10.1007/s10489-022-03403-9

    Article  Google Scholar 

  25. Wang Z, Li F, Shi G et al (2020) Network pruning using sparse learning and genetic algorithm. Neurocomputing 404:247–256. https://doi.org/10.1016/j.neucom.2020.03.082

    Article  Google Scholar 

  26. Xie Y, Chen H, Ma Y et al (2022) Automated design of cnn architecture based on efficient evolutionary search. Neurocomputing 491:160–171. https://doi.org/10.1016/j.neucom.2022.03.046

    Article  Google Scholar 

  27. He Y, Lin J, Liu Z, et al (2018) Amc: Automl for model compression and acceleration on mobile devices. In: Ferrari V, Hebert M, Sminchisescu C, et al (eds) Computer Vision – ECCV 2018. Springer International Publishing, Cham, pp 815–832 https://doi.org/10.1007/978-3-030-01234-2_48

  28. Guo S, Wang Y, Li Q, et al (2020) Dmcp: Differentiable markov channel pruning for neural networks. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1536–1544 https://doi.org/10.1109/CVPR42600.2020.00161

  29. Wang L, Huang W, Zhang M et al (2022) Pruning graph neural networks by evaluating edge properties. Knowl-Based Syst 256:109847. https://doi.org/10.1016/j.knosys.2022.109847

    Article  Google Scholar 

  30. Medhat S, Abdel-Galil H, Aboutabl AE, et al (2023) Iterative magnitude pruning-based light-version of alexnet for skin cancer classification. Neural Computing and Applications, pp 1–16. https://doi.org/10.1007/s00521-023-09111-w

  31. Hou Z, Qin M, Sun F, et al (2022) Chex: Channel exploration for cnn model compression. In: 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 12277–12288 https://doi.org/10.1109/CVPR52688.2022.01197

  32. Zhang Z, Qi F, Liu Z et al (2021) Know what you don’t need: Single-shot meta-pruning for attention heads. AI Open 2:36–42. https://doi.org/10.1016/j.aiopen.2021.05.003

    Article  Google Scholar 

  33. Hayashi T, Cimr D, Studnička F et al (2024) Distance-based one-class time-series classification approach using local cluster balance. Expert Syst Appl 235:121201. https://doi.org/10.1016/j.eswa.2023.121201

    Article  Google Scholar 

  34. Luo JH, Wu J (2020) Neural network pruning with residual-connections and limited-data. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1455–1464 https://doi.org/10.1109/CVPR42600.2020.00153

  35. Tofigh S, Ahmad MO, Swamy M (2022) A low-complexity modified thinet algorithm for pruning convolutional neural networks. IEEE Signal Proc Lett 29:1012–1016. https://doi.org/10.1109/LSP.2022.3164328

    Article  Google Scholar 

  36. He Y, Dong X, Kang G et al (2020) Asymptotic soft filter pruning for deep convolutional neural networks. IEEE Trans Cyberne 50:3594–3604. https://doi.org/10.1109/TCYB.2019.2933477

    Article  Google Scholar 

  37. Huang GB, Mattar M, Berg T, et al (2008) Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In: Workshop on faces in’Real-Life’Images: detection, alignment, and recognition

  38. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 815–823 https://doi.org/10.1109/CVPR.2015.7298682

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China(NSFC) under Grant no.61973009.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China

    Yifei Xu, Jinfu Yang, Runshi Wang & Haoqing Li

  2. Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology, Beijing, 100124, China

    Jinfu Yang

Authors
  1. Yifei Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jinfu Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Runshi Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Haoqing Li
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Yifei Xu: Investigation, Methodology, Writing- original draft. Jinfu Yang: Supervision, Writing - review & editing, Funding acquisition. Runshi Wang: Resources, Writing - review & editing. Haoqing Li: Validation, Writing - review & editing.

Corresponding author

Correspondence to Jinfu Yang.

Ethics declarations

Competing flicts of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, Y., Yang, J., Wang, R. et al. An effective two-stage channel pruning method based on two-dimensional information entropy. Appl Intell 54, 8491–8504 (2024). https://doi.org/10.1007/s10489-024-05615-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05615-7

Keywords