Skip to main content
Springer Nature Link
Log in

Multiple sparse spaces network pruning via a joint similarity criterion

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

In this paper, a simple and effective neural network pruning framework is proposed to solve the problems of low model acceleration efficiency and inaccurate identification of pruning channels in conventional methods. Therefore, this paper first proposes a multi-sparse space network pruning scheme, which reduces the impact of pruning on network performance by defining the pruning task as an optimisation task in two different sparse spaces to gradually remove redundant parameters from the network. In this paper, we focus on the distribution characteristics of network weights in different sparse spaces, and we show that a decision method combining distance and direction information between weights can better locate the redundant information in the network. Experimental results and analysis have shown that the method can effectively prune neural networks, obtaining better results at higher compression and acceleration rates compared to other state-of-the-art methods. For example, on CIFAR-10, it reduces FLOPs by 67.5% and 64.2% for ResNet56 and ResNet110, respectively, while improving accuracy by 0.10% and 0.55%, respectively. On the CIFAR-100 dataset, the FLOPs for ResNet32 were reduced by 40.3%, while the accuracy was improved by 0.06%. On the STL-10 dataset, it was able to reduce the FLOPs of the ResNet18 model by 71.5% and gain an accuracy improvement of 0.59%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Since our research is in progress, the code cannot be shared at this time. However, the datasets used are openly accessible.

References

  1. Szegedy C, Ioffe S, Vanhoucke V, Alemi A (2016) Inception-v4, inception-ResNet and the impact of residual connections on learning. In: Proceedings of the AAAI conference on artificial intelligencee, vol 31

  2. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  3. Chen B, Zhao T, Liu J, Lin L (2021) Multipath feature recalibration densenet for image classification. Int J Mach Learn Cybern 12(3):651–660. https://doi.org/10.1007/s13042-020-01194-4

    Article  Google Scholar 

  4. Liu B, Zhou Y, Sun W (2020) Character-level text classification via convolutional neural network and gated recurrent unit. Int J Mach Learn Cybern 11(8):1939–1949. https://doi.org/10.1007/s13042-020-01084-9

    Article  Google Scholar 

  5. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788. https://doi.org/10.1109/CVPR.2016.91

  6. Zhou T, Li Z, Zhang C (2019) Enhance the recognition ability to occlusions and small objects with robust faster r-cnn. Int J Mach Learn Cybern 10(11):3155–3166. https://doi.org/10.1007/s13042-019-01006-4

    Article  Google Scholar 

  7. Lian G, Wang Y, Qin H, Chen G (2022) Towards unified on-road object detection and depth estimation from a single image. Int J Mach Learn Cybern 13(5):1231–1241. https://doi.org/10.1007/s13042-021-01444-z

    Article  Google Scholar 

  8. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965

  9. Hu T, Yang M, Yang W, Li A (2019) An end-to-end differential network learning method for semantic segmentation. Int J Mach Learn Cybern 10(7):1909–1924. https://doi.org/10.1007/s13042-018-0899-1

    Article  Google Scholar 

  10. Aslan S, Ciocca G, Mazzini D, Schettini R (2020) Benchmarking algorithms for food localization and semantic segmentation. Int J Mach Learn Cybern 11(12):2827–2847. https://doi.org/10.1007/s13042-020-01153-z

    Article  Google Scholar 

  11. Han S, Mao H, Dally W.J (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. In: Proceedings of International Conference of Learning Representation (ICLR)

  12. Polino A, Pascanu R, Alistarh D (2018) Model compression via distillation and quantization. arXiv:1802.05668

  13. Idelbayev Y, Carreira-Perpinan MA (2020) Low-rank compression of neural nets: learning the rank of each layer. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8046–8056. IEEE Computer Society, Los Alamitos, CA, USA . https://doi.org/10.1109/CVPR42600.2020.00807

  14. Qiu Q, Cheng X, Calderbank R, Sapiro G (2018) DCFNet: deep neural network with decomposed convolutional filters. In: Dy J, Krause, A (eds) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 4198–4207 . https://proceedings.mlr.press/v80/qiu18a.html

  15. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1398–1406 . https://doi.org/10.1109/ICCV.2017.155

  16. Carreira-Perpinan MA, Idelbayev Y (2018) "Learning-compression" algorithms for neural net pruning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8532–8541. https://doi.org/10.1109/CVPR.2018.00890

  17. Shao M, Dai J, Wang R, Kuang J, Zuo W (2022) Cshe: network pruning by using cluster similarity and matrix eigenvalues. Int J Mach Learn Cybern 13(2):371–382. https://doi.org/10.1007/s13042-021-01411-8

    Article  Google Scholar 

  18. Kuang J, Shao M, Wang R, Zuo W, Ding W (2022) Network pruning via probing the importance of filters. Int J Mach Learn Cybern. https://doi.org/10.1007/s13042-022-01530-w

    Article  Google Scholar 

  19. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv:1503.02531

  20. Yim J, Joo D, Bae J, Kim J (2017) A gift from knowledge distillation: fast optimization, network minimization and transfer learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7130–7138 . https://doi.org/10.1109/CVPR.2017.754

  21. Han K, Wang Y, Tian Q, Guo J, Xu C, Xu C (2020) Ghostnet: more features from cheap operations. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1577–1586. https://doi.org/10.1109/CVPR42600.2020.00165

  22. Howard A, Sandler M, Chen B, Wang W, Chen L.-C, Tan M, Chu G, Vasudevan V, Zhu Y, Pang R, Adam H, Le Q (2019) Searching for mobilenetv3. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1314–1324. https://doi.org/10.1109/ICCV.2019.00140

  23. Srinivas S, Babu RV (2015) Data-free parameter pruning for deep neural networks. arXiv:1507.06149

  24. Han S, Pool J, Tran J, Dally WJ (2015) Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143. http://papers.nips.cc/paper/5784-learning-both-weights-and-connections-for-efficient-neural-network

  25. Ullrich K, Meeds E, Welling M (2017) Soft weight-sharing for neural network compression. In: Proceedings of International Conference of Learning Representation (ICLR).

  26. He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. In: Proceedings of International Joint Conferences on Artificial Intelligence, pp 2234–2240

  27. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4335–4344. https://doi.org/10.1109/CVPR.2019.00447

  28. Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: Filter pruning using high-rank feature map. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1526–1535. IEEE Computer Society, Los Alamitos, CA, USA. https://doi.org/10.1109/CVPR42600.2020.00160

  29. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2018) Pruning filters for efficient ConvNets. In: Proceedings of International Conference of Learning Representation (ICLR)

  30. Guo Y, Yao A, Chen Y (2016) Dynamic network surgery for efficient dnns. In: Advances in Neural Information Processing Systems, pp. 1379-1387.

  31. Park J, Li S, Wen W, Tang PTP, Li H, Chen Y, Dubey P (2017) Faster CNNs with direct sparse convolutions and guided pruning

  32. Han S, Liu X, Mao H, Pu J, Pedram A, Horowitz MA, Dally WJ (2016) Eie: efficient inference engine on compressed deep neural network. In: 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 243–254. IEEE Computer Society, Los Alamitos, CA, USA . https://doi.org/10.1109/ISCA.2016.30

  33. Huang G, Liu S, Maaten Lvd, Weinberger KQ (2018) Condensenet: an efficient densenet using learned group convolutions. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2752–2761. https://doi.org/10.1109/CVPR.2018.00291

  34. Li Y, Lin S, Zhang B, Liu J, Doermann D, Wu Y, Huang F, Ji R (2019) Exploiting kernel sparsity and entropy for interpretable cnn compression. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2795–2804 . https://doi.org/10.1109/CVPR.2019.00291

  35. Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning structured sparsity in deep neural networks. In: Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R (eds) Advances in Neural Information Processing Systems, vol. 29. https://proceedings.neurips.cc/paper/2016/file/41bfd20a38bb1b0bec75acf0845530a7-Paper.pdf

  36. Wang X, Zheng Z, He Y, Yan F, Zeng Z, Yang Y (2022) Soft person reidentification network pruning via blockwise adjacent filter decaying. IEEE Transact Cybern 52(12):13293–13307. https://doi.org/10.1109/TCYB.2021.3130047

    Article  Google Scholar 

  37. Wang X, Zheng Z, He Y, Yan F, Zeng Z, Yang Y (2023) Progressive local filter pruning for image retrieval acceleration. IEEE Transact Multimedia. https://doi.org/10.1109/TMM.2023.3256092

    Article  Google Scholar 

  38. You Z, Yan K, Ye J, Ma M, Wang P (2019) Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In: Wallach H, Larochelle H, Beygelzimer A, d’ Alché-Buc F, Fox E, Garnett R (eds) Advances in Neural Information Processing Systems, vol. 32. https://proceedings.neurips.cc/paper/2019/file/b51a15f382ac914391a58850ab343b00-Paper.pdf

  39. Gao S, Huang F, Pei J, Huang H (2020) Discrete model compression with resource constraint for deep neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  40. Herrmann C, Bowen RS, Zabih R (2020) Channel selection using gumbel softmax. In: European Conference on Computer Vision, pp. 241–257 . Springer

  41. Gao S, Huang F, Cai W, Huang H (2021) Network pruning via performance maximization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9270–9280

  42. Samriya JK, Tiwari R, Cheng X, Singh RK, Shankar A, Kumar M (2022) Network intrusion detection using aco-dnn model with dvfs based energy optimization in cloud framework. Sustain Comput 35:100746

    Google Scholar 

  43. Ikram ST, Priya V, Anbarasu B, Cheng X, Ghalib MR, Shankar A (2022) Prediction of iiot traffic using a modified whale optimization approach integrated with random forest classifier. J Supercomput 78(8):10725–10756

    Article  Google Scholar 

  44. Lin C, Zhong Z, Wei W, Yan J (2018) Synaptic strength for convolutional neural network. In: advances in neural information processing systems, pp. 10169–10178

  45. Lebedev V, Lempitsky V (2016) Fast convnets using group-wise brain damage. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2554–2564. https://doi.org/10.1109/CVPR.2016.280

  46. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2755–2763. https://doi.org/10.1109/ICCV.2017.298

  47. Luo J.-H, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5068–5076. https://doi.org/10.1109/ICCV.2017.541

  48. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. 2015 International conference on learning representations (ICLR) arXiv: 1409.1556

  49. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. https://doi.org/10.1109/CVPR.2016.90

  50. Krizhevsky A (2009) Learning multiple layers of features from tiny images. Technical report

  51. Darlow LN, Crowley EJ, Antoniou A, Storkey AJ (2018) CINIC-10 is not ImageNet or CIFAR-10

  52. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115(3):211–252

    Article  MathSciNet  Google Scholar 

  53. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in pytorch. In: advances in neural information processing systems

  54. Dong X, Huang J, Yang Y, Yan S (2017) More is less: a more complicated network with less inference complexity. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1895–1903. https://doi.org/10.1109/CVPR.2017.205

  55. Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann D (2019) Towards optimal structured cnn pruning via generative adversarial learning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2785–2794. https://doi.org/10.1109/CVPR.2019.00290

  56. Ning X, Zhao T, Li W, Lei P, Wang Y, Yang H (2020) Dsa: More efficient budgeted pruning via differentiable sparsity allocation. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer Vision - ECCV 2020. Springer, Cham, pp 592–607

    Chapter  Google Scholar 

  57. He Y, Dong X, Kang G, Fu Y, Yan C, Yang Y (2020) Asymptotic soft filter pruning for deep convolutional neural networks. IEEE Transact Cybern 50(8):3594–3604. https://doi.org/10.1109/TCYB.2019.2933477

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Natural Science Foundation of Hebei Province of China (Grant No.F2020203003).

Author information

Authors and Affiliations

  1. School of Electrical Engineering, Yanshan University, Qinhuangdao, 066001, China

    Guoqiang Li, Anbang Chen & Bowen Liu

  2. Key Lab of Industrial Computer Control Engineering of Hebei Province, Yanshan University, Qinhuangdao, 066001, China

    Guoqiang Li, Anbang Chen & Bowen Liu

Authors
  1. Guoqiang Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Anbang Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Bowen Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

GL: supervision, reviewing and editing, validation, project administration. AC: conceptualization, methodology, software, writing, original draft preparation. BL: investigation, data curation, editing

Corresponding author

Correspondence to Guoqiang Li.

Ethics declarations

Conflict of interest

The authors declared that they have no conflicts of interest in this work. We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, G., Chen, A. & Liu, B. Multiple sparse spaces network pruning via a joint similarity criterion. Int. J. Mach. Learn. & Cyber. 14, 4079–4099 (2023). https://doi.org/10.1007/s13042-023-01882-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01882-x

Keywords