Skip to main content
Springer Nature Link
Log in

DAAR: Dual attention cooperative adaptive pruning rate by data-driven for filter pruning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Model compression can address the limitations of deep learning in resource-constrained situations by reducing the computational and storage requirements of the model. Structured pruning has emerged as an important compression technique because of its operational flexibility and effectiveness. However, the existing structural pruning methods have two limitations: 1) They use a single measurement to identify the importance of the filters in all the layers, resulting in a loss of spatial information in the shallow layers. 2) The pruning rate is highly dependent on manual interference, which is highly subjective. In this paper, a filter pruning method called dual attention cooperative adaptive pruning rate (DAAR) is proposed. Specifically, a dual attention module that combines spatial attention and channel attention is proposed to measure the effectiveness of the filters. Spatial attention is used in the shallow layers, and channel attention is used in the deep layers. This allows the filter measurements to consider spatial information effectively. An adaptive pruning rate adjustment strategy is also used to eliminate manual subjectivity, achieving precision pruning of each convolutional layer. The experimental results on various datasets and networks demonstrate that the DAAR method achieves improved model performance after pruning. For example, in the CIFAR10 dataset, the precision increases from 93.5% to 93.75% after removing the floating point operations (FLOPs) of 84.1%, outperforming the state-of-the-art pruning methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The data will be made available upon reasonable request.

References

  1. Han S, Mao H, Dally WJ (2015) Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

  2. Minaee S, Boykov Y, Porikli F, Plaza A, Kehtarnavaz N, Terzopoulos D (2021) Image segmentation using deep learning: A survey. IEEE Trans Pattern Anal Mach Intell 44(7):3523–3542

    Google Scholar 

  3. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788

  4. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  5. Luo J-H, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066

  6. Goh H-A, Ho C-K, Abas FS (2023) Front-end deep learning web apps development and deployment: a review. Appl Intell 53(12):15923–15945

    Article  Google Scholar 

  7. Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems 28

  8. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y (2016) Binarized neural networks. Advances in Neural Information Processing Systems 29

  9. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

  10. Zhu, J, Pei J (2022) Progressive kernel pruning cnn compression method with an adjustable input channel. Applied Intelligence 1–22

  11. Zhang Q, Zhang M, Chen T, Sun Z, Ma Y, Yu B (2019) Recent advances in convolutional neural network acceleration. Neurocomputing 323:37–51

    Article  MATH  Google Scholar 

  12. He Y, Xiao L (2023) Structured pruning for deep convolutional neural networks: A survey. arXiv preprint arXiv:2303.00566

  13. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340–4349

  14. Wang X, Zheng Z, He Y, Yan F, Zeng Z, Yang, Y (2023) Progressive local filter pruning for image retrieval acceleration. IEEE Transactions on Multimedia

  15. Gkrispanis K, Gkalelis N, Mezaris V (2024) Filter-pruning of lightweight face detectors using a geometric median criterion. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pp. 280–289

  16. Kumar A, Shaikh AM, Li Y, Bilal H, Yin B (2021) Pruning filters with l1-norm and capped l1-norm for cnn compression. Appl Intell 51:1152–1160

    Article  MATH  Google Scholar 

  17. Li G, Li R, Li T, Shen C, Zou X, Wang J, Wang C, Li N (2024) Sfp: Similarity-based filter pruning for deep neural networks. Inf Sci 689:121418

    Article  MATH  Google Scholar 

  18. Zheng Y, Sun P, Ren Q, Xu W, Zhu D (2024) A novel and efficient model pruning method for deep convolutional neural networks by evaluating the direct and indirect effects of filters. Neurocomputing 569:127124

    Article  Google Scholar 

  19. Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: Filter pruning using high-rank feature map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529–1538

  20. Guo M-H, Xu T-X, Liu J-J, Liu Z-N, Jiang P-T, Mu T-J, Zhang S-H, Martin RR, Cheng M-M, Hu S-M (2022) Attention mechanisms in computer vision: A survey. Computational visual media. 8(3):331–368

    Article  MATH  Google Scholar 

  21. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826

  22. Jaderberg M, Simonyan K, Zisserman A et al (2015) Spatial transformer networks. Advances in neural information processing systems. 28

  23. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141

  24. Wang X-J, Yao W, Fu H (2019) A convolutional neural network pruning method based on attention mechanism. In: SEKE, pp. 343–452

  25. Lu W, Jiang Y, Jing P, Chu J, Fan F (2023) A novel channel pruning approach based on local attention and global ranking for cnn model compression. In: 2023 IEEE International Conference on Multimedia and Expo (ICME), pp. 1433–1438. IEEE

  26. Zhao K, Jain A, Zhao M (2023) Automatic attention pruning: Improving and automating model pruning using attentions. In: International Conference on Artificial Intelligence and Statistics, pp. 10470–10486. PMLR

  27. Yamamoto K, Maeno K (2018) Pcas: Pruning channels with attention statistics for deep network compression. arXiv preprint arXiv:1806.05382

  28. Chen Y, Wu G, Shuai M, Lou S, Zhang Y, An Z (2024) Fpar: filter pruning via attention and rank enhancement for deep convolutional neural networks acceleration. International Journal of Machine Learning and Cybernetics, 1–13

  29. Luo J-H, Wu J (2020) Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recogn 107:107461

    Article  MATH  Google Scholar 

  30. Cheng H, Wang Z, Ma L, Wei Z, Alsaadi FE, Liu X (2023) Differentiable channel pruning guided via attention mechanism: a novel neural network pruning approach. Complex & Intelligent Systems. 9(5):5611–5624

    Article  MATH  Google Scholar 

  31. Xue Y, Yao W, Peng S, Yao S (2024) Automatic filter pruning algorithm for image classification. Appl Intell 54(1):216–230

    Article  MATH  Google Scholar 

  32. He Y, Lin J, Liu Z, Wang H, Li L-J, Han S (2018) Amc: Automl for model compression and acceleration on mobile devices. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 784–800

  33. Liu Y, Guo Y, Guo J, Jiang L, Chen J (2021) Conditional automated channel pruning for deep neural networks. IEEE Signal Process Lett 28:1275–1279

    Article  MATH  Google Scholar 

  34. Liu Z, Mu H, Zhang X, Guo Z, Yang X, Cheng K-T, Sun J (2019) Metapruning: Meta learning for automatic neural network channel pruning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3296–3305

  35. Chang J, Lu Y, Xue P, Xu Y, Wei Z (2022) Automatic channel pruning via clustering and swarm intelligence optimization for cnn. Appl Intell 52(15):17751–17771

    Article  Google Scholar 

  36. Chen S, Zhao Q (2018) Shallowing deep networks: Layer-wise pruning based on feature representations. IEEE Trans Pattern Anal Mach Intell 41(12):3048–3056

    Article  MathSciNet  MATH  Google Scholar 

  37. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744

  38. Alvarez JM, Salzmann M (2017) Compression-aware training of deep networks. Advances in neural information processing systems. 30

  39. Jiang C, Li G, Qian C, Tang K (2018) Efficient dnn neuron pruning by minimizing layer-wise nonlinear reconstruction error. IJCAI 2018:2–2

    MATH  Google Scholar 

  40. Zhang D, Wang H, Figueiredo M, Balzano L (2018) Learning to share: Simultaneous parameter tying and sparsification in deep learning. In: International Conference on Learning Representations

  41. He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866

  42. Zhuang Z, Tan M, Zhuang B, Liu J, Guo Y, Wu Q, Huang J, Zhu, J (2018) Discrimination-aware channel pruning for deep neural networks. Advances in neural information processing systems. 31

  43. Li Y, Lin S, Zhang B, Liu J, Doermann D, Wu Y, Huang F, Ji R (2019) Exploiting kernel sparsity and entropy for interpretable cnn compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2800–2809

  44. Zhang H, Liu L, Zhou H, Hou W, Sun H, Zheng N (2021) Akecp: Adaptive knowledge extraction from feature maps for fast and efficient channel pruning. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 648–657

  45. Feng K-Y, Fei X, Gong M, Qin A, Li H, Wu Y (2022) An automatically layer-wise searching strategy for channel pruning based on task-driven sparsity optimization. IEEE Trans Circuits Syst Video Technol 32(9):5790–5802

  46. Liu Y, Wu D, Zhou W, Fan K, Zhou Z (2023) Eacp: An effective automatic channel pruning for neural networks. Neurocomputing 526:131–142

  47. Lin S, Ji R, Li Y, Deng C, Li X (2019) Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems. 31(2):574–588

    Article  MathSciNet  MATH  Google Scholar 

  48. Ding X, Ding G, Guo Y, Han J, Yan C (2019) Approximated oracle filter pruning for destructive cnn width optimization. In: International Conference on Machine Learning, pp. 1607–1616. PMLR

  49. Ruan X, Liu Y, Yuan C, Li B, Hu W, Li Y, Maybank S (2020) Edp: An efficient decomposition and pruning scheme for convolutional neural network compression. IEEE Transactions on Neural Networks and Learning Systems. 32(10):4499–4513

    Article  MATH  Google Scholar 

  50. Lin M, Cao L, Li S, Ye Q, Tian Y, Liu J, Tian Q, Ji R (2021) Filter sketch for network pruning. IEEE Transactions on Neural Networks and Learning Systems. 33(12):7091–7100

    Article  MATH  Google Scholar 

  51. Hou Z, Qin M, Sun F, Ma X, Yuan K, Xu Y, Chen Y-K, Jin R, Xie Y, Kung S-Y (2022) Chex: Channel exploration for cnn model compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12287–12298

  52. Gao S, Huang F, Pei J, Huang H (2020) Discrete model compression with resource constraint for deep neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1899–1908

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China (Grant No. 62071303, 62201355), Guangdong Basic and Applied Basic Research Foundation (2024A1515010977), Shenzhen Science and Technology Projection (JCYJ20220531102407018), Guangdong Provincial Key Laboratory (Grant 2023B1212060076).

Author information

Authors and Affiliations

  1. The Guangdong Key Laboratory of Intelligent Information Processing, College of Electronics and Information Engineering, Shenzhen University, Shenzhen, 518060, China

    Suyun Lian, Yang Zhao & Jihong Pei

  2. College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China

    Yang Zhao

Authors
  1. Suyun Lian
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yang Zhao
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jihong Pei
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Suyun Lian: design, implementation, formal analysis and writing. Zhao Yang: guidance, review and editing. Jihong Pei: validation.

Corresponding author

Correspondence to Jihong Pei.

Ethics declarations

Ethical and Informed Consent for Data Used:

No ethical approval or informed consent was necessary for this study, as the data were already publicly available and did not involve human or animal subjects.

Competing Interests:

The paper is original in terms of its contents and is not under consideration for publication in any other journals/proceedings. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lian, S., Zhao, Y. & Pei, J. DAAR: Dual attention cooperative adaptive pruning rate by data-driven for filter pruning. Appl Intell 55, 402 (2025). https://doi.org/10.1007/s10489-025-06332-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-025-06332-5

Keywords