Skip to main content
SpringerLink
Log in

Convolutional neural network pruning based on misclassification cost

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In a convolutional neural network (CNN), overparameterization increases the risk of overfitting, decelerates the inference, and impedes edge computing. To resolve these challenges, one possible solution is to prune CNN parameters. The essence of pruning is to identify and eliminate unimportant filters, which should yield the highest speed increase and the lowest accuracy loss. In contrast with other pruning methods and in conformity with the real-world, this paper does not evaluate the accuracy of a CNN as its overall performance but analyzes different misclassification costs. This modification accelerates the pruning process and improves the prune ratio. The proposed algorithm determines the expected specificity/sensitivity for each class and finds the smallest CNN that is consistent with them. The layer-wise relevance propagation is employed to measure the contribution of each filter to every class discrimination. The importance of each filter is determined by integrating its local (usefulness in its layer) and global (contribution to the network output) usefulness. Since the proposed algorithm frequently fluctuates between pruning and recovery, further fine-tuning is unnecessary. According to simulation results, the proposed algorithm was efficient in both pruning a CNN and attaining the desired sensitivity/specificity of classes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and materials

The source of all datasets and models used in this study is given in the manuscript.

References

  1. Lin CH, Yu CC, Wang TY et al (2020) Classification of the tree for aerial image using a deep convolution neural network and visual feature clustering. J Supercomput 76:2503–2517. https://doi.org/10.1007/s11227-019-03012-3

    Article  Google Scholar 

  2. Acharya S, Alsadoon A, Prasad PWC et al (2020) Deep convolutional network for breast cancer classification: enhanced loss function (ELF). J Supercomput 76:8548–8565. https://doi.org/10.1007/s11227-020-03157-6

    Article  Google Scholar 

  3. Zhang J, Zhang Z, Ji X et al (2022) Deep learning convolutional neural network in diagnosis of serous effusion in patients with malignant tumor by tomography. J Supercomput 78:4449–4466. https://doi.org/10.1007/s11227-021-04051-5

    Article  Google Scholar 

  4. Alajlan AM, Almasri MM (2022) Automatic Lane marking prediction using convolutional neural network and S-shaped binary butterfly optimization. J Supercomput 78:3715–3745. https://doi.org/10.1007/s11227-021-03988-x

    Article  Google Scholar 

  5. Zaghari N, Fathy M, Jameii SM et al (2021) The improvement in obstacle detection in autonomous vehicles using YOLO non-maximum suppression fuzzy algorithm. J Supercomput 77:13421–13446. https://doi.org/10.1007/s11227-021-03813-5

    Article  Google Scholar 

  6. Boukerche A, Hou Z (2021) Object detection using deep learning methods in traffic scenarios. ACM Comput Surv. https://doi.org/10.1145/3434398

    Article  Google Scholar 

  7. Jiang H, Diao Z, Yao YD (2022) Deep learning techniques for tumor segmentation: a review. J Supercomput 78:1807–1851. https://doi.org/10.1007/s11227-021-03901-6

    Article  Google Scholar 

  8. Alam MGR, Khan AM, Shejuty MF et al (2023) Ejection fraction estimation using deep semantic segmentation neural network. J Supercomput 79:27–50. https://doi.org/10.1007/s11227-022-04642-w

    Article  Google Scholar 

  9. Zhang C, Bengio S, Hardt M, Recht B, Vinyals O (2021) Understanding deep learning (still) requires rethinking generalization. Commun ACM 64:107–115. https://doi.org/10.1145/3446776

    Article  Google Scholar 

  10. Lin SB, Wang K, Wang Y, Zhou DX (2022) Universal consistency of deep convolutional neural networks. IEEE Trans Inf Theory 68:4610–4617. https://doi.org/10.1109/TIT.2022.3151753

    Article  MathSciNet  MATH  Google Scholar 

  11. Lin SB (2019) Generalization and expressivity for deep nets. IEEE Trans Neural Netw Learn Syst 30:1392–1406. https://doi.org/10.1109/TNNLS.2018.2868980

    Article  MathSciNet  Google Scholar 

  12. Denton E, Zaremba W, Bruna J, LeCun Y, Fergus R (2014) Exploiting linear structure within convolutional networks for efficient evaluation. In: Proceedings of the 27th International Conference on Neural Information Processing Systems—Volume 1. MIT Press, Cambridge, MA, USA, pp 1269–1277

  13. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012) Improving neural networks by preventing co-adaptation of feature detectors. https://arxiv.org/abs/1207.0580

  14. Wang Z, Li C, Wang X (2021) Convolutional neural network pruning with structural redundancy reduction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 14908–14917. https://doi.org/10.1109/CVPR46437.2021.01467

  15. Mondal M, Das B, Roy SD, Singh P, Lall B, Joshi SD (2022) Adaptive CNN filter pruning using global importance metric. Comput Vis Image Underst 222:103511. https://doi.org/10.1016/J.CVIU.2022.103511

    Article  Google Scholar 

  16. Pei S, Wu Y, Guo J, Qiu M (2022) Neural network pruning by recurrent weights for finance market. ACM Trans Internet Technol 22:1–23. https://doi.org/10.1145/3433547

    Article  Google Scholar 

  17. Sarvani CH, Ghorai M, Dubey SR, Basha SHS (2022) HRel: filter pruning based on high relevance between activation maps and class labels. Neural Netw 147:186–197. https://doi.org/10.1016/J.NEUNET.2021.12.017

    Article  Google Scholar 

  18. Soltani M, Wu S, Ding J, Ravier R, Tarokh V (2020) On the information of feature maps and pruning of deep neural networks. In: Proceedings—International Conference on Pattern Recognition, pp 6988–6995. https://doi.org/10.1109/ICPR48806.2021.9412579

  19. Wang J, Jiang T, Cui Z, Cao Z (2021) Filter pruning with a feature map entropy importance criterion for convolution neural networks compressing. Neurocomputing 461:41–54. https://doi.org/10.1016/J.NEUCOM.2021.07.034

    Article  Google Scholar 

  20. Ruan X, Liu Y, Yuan C, Li B, Hu W, Li Y, Maybank S (2021) EDP: an efficient decomposition and pruning scheme for convolutional neural network compression. IEEE Trans Neural Networks Learn Syst 32:4499–4513. https://doi.org/10.1109/TNNLS.2020.3018177

    Article  Google Scholar 

  21. Pattanayak S, Nag S, Mittal S (2021) Curating: a multi-objective-based pruning technique for CNNs. J Syst Archit 116:102031. https://doi.org/10.1016/J.SYSARC.2021.102031

    Article  Google Scholar 

  22. Liu X, Li B, Chen Z, Yuan Y (2021) Exploring gradient flow based saliency for DNN model compression. In: Proceedings of the 29th ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, pp 3238–3246

  23. Molchanov P, Mallya A, Tyree S, Frosio I, Kautz J (2019) Importance estimation for neural network pruning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 11256–11264. https://doi.org/10.1109/CVPR.2019.01152

  24. Liu J, Zhuang B, Zhuang Z, Guo Y, Huang J, Zhu J, Tan M (2022) Discrimination-aware network pruning for deep model compression. IEEE Trans Pattern Anal Mach Intell 44:4035–4051. https://doi.org/10.1109/TPAMI.2021.3066410

    Article  Google Scholar 

  25. Wang B, Ma F, Ge L, Ma H, Wang H, Mohamed MA (2021) Icing-EdgeNet: a pruning lightweight edge intelligent method of discriminative driving channel for ice thickness of transmission lines. IEEE Trans Instrum Meas 70:1–12. https://doi.org/10.1109/TIM.2020.3018831

    Article  Google Scholar 

  26. Hou Z, Kung SY (2020) A discriminant information approach to deep neural network pruning. Proceedings—International Conference on Pattern Recognition, pp 9553–9560. https://doi.org/10.1109/ICPR48806.2021.9412693

  27. Xu T, Wu Y, Zheng X, Xi T, Zhang G, Ding E, Chao F, Ji R (2021) CDP: towards optimal filter pruning via class-wise discriminative power. In: Proceedings of the 29th ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, pp 5491–5500

  28. Saraee E, Jalal M, Betke M (2020) Visual complexity analysis using deep intermediate-layer features. Comput Vis Image Underst 195:102949. https://doi.org/10.1016/J.CVIU.2020.102949

    Article  Google Scholar 

  29. Luo JH, Zhang H, Zhou HY, Xie CW, Wu J, Lin W (2019) ThiNet: pruning CNN filters for a thinner net. IEEE Trans Pattern Anal Mach Intell 41:2525–2538. https://doi.org/10.1109/TPAMI.2018.2858232

    Article  Google Scholar 

  30. Guo J, Ouyang W, Xu D (2020) Channel pruning guided by classification loss and feature importance. In: 34th {AAAI} Conference on Artificial Intelligence, {AAAI} 2020, pp 10885–10892

  31. Yu R, Li A, Chen CF, Lai JH, Morariu VI, Han X, Gao M, Lin CY, Davis LS (2018) NISP: pruning networks using neuron importance score propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 9194–9203. https://doi.org/10.1109/CVPR.2018.00958

  32. Oh J, Kim H, Baik S, Hong C, Mu Lee K (2022) Batch normalization tells you which filter is important. In: Proceeding—2022 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, pp 3351–3360. https://doi.org/10.1109/WACV51458.2022.00341

  33. Zhang Y, Yuan Y, Wang Q (2022) ACP: adaptive channel pruning for efficient neural networks. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, pp 4488–4492, https://doi.org/10.1109/icassp43922.2022.9747839

  34. Yin S, Li C, Meng F, Tan W, Bao Y, Liang Y, Liu W (2022) Exploring structural sparsity in neural image compression, pp 471–475. https://doi.org/10.1109/ICIP46576.2022.9897508

  35. Zhao C, Ni B, Zhang J, Zhao Q, Zhang W, Tian Q (2019) Variational convolutional neural network pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2775–2784. https://doi.org/10.1109/CVPR.2019.00289

  36. Si Y, Guo W (2020) Application of a Taylor expansion criterion-based pruning convolutional network for bearing intelligent diagnosis. In: 2020 Global Reliability and Prognostics and Health Management (PHM-Shanghai). https://doi.org/10.1109/PHM-SHANGHAI49105.2020.9280998

  37. You Z, Yan K, Ye J, Ma M, Wang P (2019) Gate Decorator: global filter pruning method for accelerating deep convolutional neural networks. In: Wallach H, Larochelle H, Beygelzimer A, Alché-Buc F, Fox E, Garnett R (eds) Advances in neural information processing systems. Curran Associates

    Google Scholar 

  38. Yu C, Wang J, Chen Y, Qin X (2019) Transfer channel pruning for compressing deep domain adaptation models. Int J Mach Learn Cybern 1011(10):3129–3144. https://doi.org/10.1007/S13042-019-01004-6

    Article  Google Scholar 

  39. Huang Z, Li L, Sun H (2020) Global biased pruning considering layer contribution. IEEE Access 8:173521–173529. https://doi.org/10.1109/ACCESS.2020.3025130

    Article  Google Scholar 

  40. Chen Z, Xu TB, Du C, Liu CL, He H (2021) Dynamical channel pruning by conditional accuracy change for deep neural networks. IEEE Trans Neural Netw Learn Syst 32:799–813. https://doi.org/10.1109/TNNLS.2020.2979517

    Article  Google Scholar 

  41. Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning with class imbalance. J Big Data 6:1–54. https://doi.org/10.1186/S40537-019-0192-5/TABLES/18

    Article  Google Scholar 

  42. Liu Y, Lu H, Yan K, Xia H, An C (2016) Applying cost-sensitive extreme learning machine and dissimilarity integration to gene expression data classification. Comput Intell Neurosci. https://doi.org/10.1155/2016/8056253

    Article  Google Scholar 

  43. Lei K, Mardani M, Pauly JM, Vasanawala SS (2021) Wasserstein GANs for MR imaging: from paired to unpaired training. IEEE Trans Med Imaging 40:105–115. https://doi.org/10.1109/TMI.2020.3022968

    Article  Google Scholar 

  44. Lei K, Syed AB, Zhu X, Pauly JM, Vasanawala SS (2022) Artifact- and content-specific quality assessment for MRI with image rulers. Med Image Anal 77:102344. https://doi.org/10.1016/J.MEDIA.2021.102344

    Article  Google Scholar 

  45. Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One 10:e0130140. https://doi.org/10.1371/JOURNAL.PONE.0130140

    Article  Google Scholar 

  46. Yeom SK, Seegerer P, Lapuschkin S, Binder A, Wiedemann S, Müller KR, Samek W (2021) Pruning by explaining: a novel criterion for deep neural network pruning. Pattern Recognit 115:107899. https://doi.org/10.1016/J.PATCOG.2021.107899

    Article  Google Scholar 

  47. Hägele M, Seegerer P, Lapuschkin S, Bockmayr M, Samek W, Klauschen F, Müller KR, Binder A (2020) Resolving challenges in deep learning-based analyses of histopathological images using explanation methods. Sci Rep 101(10):1–12. https://doi.org/10.1038/s41598-020-62724-2

    Article  Google Scholar 

  48. Seegerer P, Binder A, Saitenmacher R, Bockmayr M, Alber M, Jurmeister P, Klauschen F, Müller KR (2020) Interpretable deep neural network to predict estrogen receptor status from Haematoxylin-Eosin images. In: Artificial Intelligence and Machine Learning for Digital Pathology: State-of-the-Art and Future Challenges, pp 16–37. https://doi.org/10.1007/978-3-030-50402-1_2

  49. Simonyan K, Vedaldi A, Zisserman A (2014) Deep inside convolutional networks: visualising image classification models and saliency maps. In: 2nd International Conference on Learning Representations, ICLR 2014—Workshop Track Proceedings. http://arxiv.org/abs/1312.6034

  50. Li C, Zia MZ, Tran QH, Yu X, Hager GD, Chandraker M (2019) Deep supervision with intermediate concepts. IEEE Trans Pattern Anal Mach Intell 41:1828–1843. https://doi.org/10.1109/TPAMI.2018.2863285

    Article  Google Scholar 

  51. Gonzalez-Garcia A, Modolo D, Ferrari V (2018) Do semantic parts emerge in convolutional neural networks? Int J Comput Vis 126:476–494. https://doi.org/10.1007/S11263-017-1048-0/FIGURES/15

    Article  MathSciNet  Google Scholar 

  52. Fan FL, Xiong J, Li M, Wang G (2021) On interpretability of artificial neural networks: a survey. IEEE Trans Radiat Plasma Med Sci 5:741–760. https://doi.org/10.1109/TRPMS.2021.3066428

    Article  Google Scholar 

  53. Yang C, Shen Y, Zhou B (2021) Semantic hierarchy emerges in deep generative representations for scene synthesis. Int J Comput Vis 1295(129):1451–1466. https://doi.org/10.1007/S11263-020-01429-5

    Article  Google Scholar 

  54. Hu J, Cao L, Tong T, Ye Q, Zhang S, Li K, Huang F, Shao L, Ji R (2022) Architecture disentanglement for deep neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 652–661. https://doi.org/10.1109/ICCV48922.2021.00071

  55. Zhou B, Bau D, Oliva A, Torralba A (2019) Interpreting deep visual representations via network dissection. IEEE Trans Pattern Anal Mach Intell 41:2131–2145. https://doi.org/10.1109/TPAMI.2018.2858759

    Article  Google Scholar 

  56. Sun Y, Yen GG, Yi Z (2019) Evolving unsupervised deep neural networks for learning meaningful representations. IEEE Trans Evol Comput 23:89–103. https://doi.org/10.1109/TEVC.2018.2808689

    Article  Google Scholar 

  57. Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, pp 499–515. https://doi.org/10.1007/978-3-319-46478-31/TABLES/4

  58. Peng H, Yu S (2021) Beyond softmax loss: intra-concentration and inter-separability loss for classification. Neurocomputing 438:155–164. https://doi.org/10.1016/J.NEUCOM.2020.11.030

    Article  Google Scholar 

  59. Singh P, Verma VK, Rai P, Namboodiri VP (2020) Acceleration of deep convolutional neural networks using adaptive filter pruning. IEEE J Sel Top Signal Process 14:838–847. https://doi.org/10.1109/JSTSP.2020.2992390

    Article  Google Scholar 

  60. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network, pp 1–9. http://arxiv.org/abs/1503.02531

  61. Prakosa SW, Leu JS, Chen ZH (2021) Improving the accuracy of pruned network using knowledge distillation. Pattern Anal Appl 24(2):819–830. https://doi.org/10.1007/s10044-020-00940-2

    Article  Google Scholar 

  62. Wei Z, Li H, Zhang X (2022) Model compression by iterative pruning with knowledge distillation and its application to speech enhancement. In: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2022-September, pp 941–945. https://doi.org/10.21437/INTERSPEECH.2022-619

  63. Xie H, Jiang W, Luo H, Yu H (2021) Model compression via pruning and knowledge distillation for person re-identification. J Amb Intell Hum Comput 12(2):2149–2161. https://doi.org/10.1007/S12652-020-02312-4/METRICS

    Article  Google Scholar 

  64. Aghli N, Ribeiro E (2021) Combining weight pruning and knowledge distillation for CNN compression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp 3185–3192. https://doi.org/10.1109/CVPRW53098.2021.00356

  65. Kim J, Chang S, Kwak N (2021) PQK: model compression via pruning, quantization, and knowledge distillation. In: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, pp 1863–1867. https://doi.org/10.21437/INTERSPEECH.2021-248

  66. Yim J, Joo D, Bae J, Kim J (2017) A gift from knowledge distillation: fast optimization, network minimization and transfer learning. In: Proceedings—30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January, pp 7130–7138. https://doi.org/10.1109/CVPR.2017.754

  67. Chen Y, Wang N, Zhang Z (2018) DarkRank: accelerating deep metric learning via cross sample similarities transfer. In: 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp 2852–2859. https://doi.org/10.1609/aaai.v32i1.11783

  68. Zhao K, Chen Y, Zhao M (2023) A contrastive knowledge transfer framework for model compression and transfer learning. In: ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 1–5. https://doi.org/10.1109/ICASSP49357.2023.10095744

  69. Park W, Kim D, Lu Y, Cho M (2019) Relational knowledge distillation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June, pp. 3962–3971. https://doi.org/10.1109/CVPR.2019.00409

  70. Li C, Cheng G, Wang G, Zhou P, Han J (2023) Instance-aware distillation for efficient object detection in remote sensing images. In: IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2023.3238801

  71. Zhou ZJ, Zhou Y, Jiang Z, Men A, Wang H (2022) An efficient method for model pruning using knowledge distillation with few samples. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal—Proceedings, 2022-May, pp 2515–2519. https://doi.org/10.1109/ICASSP43922.2022.9746024

  72. Su J, Fraser NJ, Gambardella G, Blott M, Durelli G, Thomas DB, Leong PHW, Cheung PYK (2018) Accuracy to throughput trade-offs for reduced precision neural networks on reconfigurable logic. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10824 LNCS, pp 29–42. https://doi.org/10.1007/978-3-319-78890-6_3/COVER

  73. Jacob B, Kligys S, Chen B, Zhu M, Tang M, Howard A, Adam H, Kalenichenko D (2018) Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 2704–2713. https://doi.org/10.1109/CVPR.2018.00286

  74. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y (2016) Binarized neural networks. Adv Neural Inf Process Syst, 29(4). https://github.com/itayhubara/BinaryNet

  75. Marinó GC, Petrini A, Malchiodi D, Frasca M (2023) Deep neural networks compression: a comparative survey and choice recommendations. Neurocomputing 520:152–170. https://doi.org/10.1016/J.NEUCOM.2022.11.072

    Article  Google Scholar 

  76. Gholami A, Kim S, Dong Z, Yao Z, Mahoney MW, Keutzer K (2022) A survey of quantization methods for efficient neural network inference. In: Low-Power Computer Vision, pp 291–326. https://doi.org/10.1201/9781003162810-13

  77. Han S, Mao H, Dally WJ (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: 4th International Conference on Learning Representations, ICLR 2016—Conference Track Proceedings. http://arxiv.org/abs/1510.00149

  78. Son S, Nah S, Lee KM (2018) Clustering convolutional kernels to compress deep neural networks. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11212 LNCS, pp 225–240. https://doi.org/10.1007/978-3-030-01237-3_14

  79. Ma N, Zhang X, Zheng HT, Sun J (2018) Shufflenet V2: practical guidelines for efficient cnn architecture design. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11218 LNCS, pp 122–138. https://doi.org/10.1007/978-3-030-01264-9_8

  80. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 4510–4520. https://doi.org/10.1109/CVPR.2018.00474

  81. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. http://arxiv.org/abs/1602.07360

  82. Nousi P, Tzelepi M, Passalis N, Tefas A (2022) Lightweight deep learning. Deep learning for robot perception and cognition. Elsevier, pp 131–164

    Chapter  Google Scholar 

  83. LeCun Y, Denker J, Solla S (1989) Optimal brain damage. In: Touretzky D (ed) Advances in neural information processing systems. Morgan-Kaufmann

    Google Scholar 

  84. Vadera S, Ameen S (2022) Methods for pruning deep neural networks. IEEE Access 10:63280–63300. https://doi.org/10.1109/ACCESS.2022.3182659

    Article  Google Scholar 

  85. Choudhary T, Mishra V, Goswami A, Sarangapani J (2020) A comprehensive survey on model compression and acceleration. Artif Intell Rev 537(53):5113–5155. https://doi.org/10.1007/S10462-020-09816-7

    Article  Google Scholar 

  86. Chen X, Mao J, Xie J (2021) Comparison Analysis for Pruning Algorithms of Neural Networks. In: 2021 2nd International Conference on Computer Engineering and Intelligent Control (ICCEIC), pp 50–56. https://doi.org/10.1109/ICCEIC54227.2021.00018

  87. Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: filter pruning using high-Rank feature map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1526–1535. https://doi.org/10.1109/CVPR42600.2020.00160

  88. Xie X, Zhang H, Wang J, Chang Q, Wang J, Pal NR (2020) Learning optimized structure of neural networks by hidden node pruning with L1 regularization. IEEE Trans Cybern 50:1333–1346. https://doi.org/10.1109/TCYB.2019.2950105

    Article  Google Scholar 

  89. Koneru BNG, Vasudevan V (2019) Sparse artificial neural networks using a novel smoothed LASSO penalization. IEEE Trans Circuits Syst II Express Briefs 66:848–852. https://doi.org/10.1109/TCSII.2019.2908729

    Article  Google Scholar 

  90. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient ConvNets. In: Proceedings of the 5th International Conference on Learning Representations (ICLR). Toulon, France

  91. Morcos AS, Barrett DGT, Rabinowitz NC, Botvinick M (2018) On the importance of single directions for generalization. In: 6th International Conference in Learning Representation ICLR 2018—Conference Track Proceeding. https://doi.org/10.48550/arxiv.1803.06959

  92. Ukita J (2020) Causal importance of low-level feature selectivity for generalization in image recognition. Neural Netw 125:185–193. https://doi.org/10.1016/J.NEUNET.2020.02.009

    Article  Google Scholar 

  93. Shi J, Xu J, Tasaka K, Chen Z (2021) SASL: saliency-adaptive sparsity learning for neural network acceleration. IEEE Trans Circuits Syst Video Technol 31:2008–2019. https://doi.org/10.1109/TCSVT.2020.3013170

    Article  Google Scholar 

  94. Yang HM, Zhang XY, Yin F, Liu CL (2018) Robust classification with convolutional prototype learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3474–3482. https://doi.org/10.1109/CVPR.2018.00366

  95. Son S, Nah S, Lee KM (2018) Clustering convolutional kernels to compress deep neural networks. In: Proceedings of the European Conference on Computer Vision, pp 225–240. https://doi.org/10.1007/978-3-030-01237-3_14

  96. Zhou Z, Zhou W, Li H, Hong R (2018) Online filter clustering and pruning for efficient convnets. In: 2018 25th IEEE International Conference on Image Processing, pp 11–15. https://doi.org/10.1109/ICIP.2018.8451123

  97. Vasilis Pikoulis E, Mavrokefalidis C, Lalos AS (2020) A new clustering-based technique for the acceleration of deep convolutional networks. In: Proceedings—19th IEEE International Conference in Machine Learning Application ICMLA 2020, pp 1432–1439. https://doi.org/10.1109/ICMLA51294.2020.00222

  98. Entezari R, Saukh O (2020) Class-dependent pruning of deep neural networks. In: 2020 IEEE Second Workshop on Machine Learning on Edge in Sensor Systems (SenSys-ML), Sydney, NSW, Australia, 2020, pp 13–18, https://doi.org/10.1109/SenSysML50931.2020.00010

  99. Tripathy BK, Reddy Maddikunta PK, Pham Q-V, Gadekallu TR, Dev K, Pandya S, ElHalawany BM (2022) Harris Hawk optimization: a survey on variants and applications. Comput Intell Neurosci 2022:1–20. https://doi.org/10.1155/2022/2218594

    Article  Google Scholar 

  100. Gadekallu TR, Srivastava G, Liyanage M, Iyyaparaja M, Chowdhary CL, Koppu S, Maddikunta PKR (2022) Hand gesture recognition based on a Harris Hawks optimized convolution neural network. Comput Electr Eng 100:107836. https://doi.org/10.1016/J.COMPELECENG.2022.107836

    Article  Google Scholar 

  101. Han S, Pool J, Tran J, Dally WJ (2015) Learning both weights and connections for efficient neural networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems—Volume 1. MIT Press, Cambridge, MA, USA, pp 1135–1143

  102. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1398–1406. https://doi.org/10.1109/ICCV.2017.155

  103. Zhang X, Zou J, He K, Sun J (2016) Accelerating very deep convolutional networks for classification and detection. IEEE Trans Pattern Anal Mach Intell 38:1943–1955. https://doi.org/10.1109/TPAMI.2015.2502579

    Article  Google Scholar 

  104. Shi R, Niu L, Zhou R (2022) Sparse CapsNet with explicit regularizer. Pattern Recognit 124:108486. https://doi.org/10.1016/J.PATCOG.2021.108486

    Article  Google Scholar 

  105. Von Luxburg U (2007) A tutorial on spectral clustering. Stat Comput 174(17):395–416. https://doi.org/10.1007/S11222-007-9033-Z

    Article  MathSciNet  Google Scholar 

  106. Rosasco L, Belkin M, De VE (2010) On learning with integral operators. J Mach Learn Res 11:905–934

    MathSciNet  MATH  Google Scholar 

  107. He Y, Dong X, Kang G, Fu Y, Yan C, Yang Y (2020) Asymptotic soft filter pruning for deep convolutional neural networks. IEEE Trans Cybern 50:3594–3604. https://doi.org/10.1109/TCYB.2019.2933477

    Article  Google Scholar 

  108. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in PyTorch. In: NIPS-W

  109. Liu S, Deng W (2016) Very deep convolutional neural network-based image classification using small training sample size. In: Proceedings—3rd IAPR Asian Conference on Pattern Recognition, ACPR 2015, pp 730–734. https://doi.org/10.1109/ACPR.2015.7486599

  110. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  111. Krizhevsky A, Sutskever I, Hinton GE (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60(6):84–90. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  112. Liu W, Wen Y, Yu Z, Yang M (2016) Large-margin Softmax loss for convolutional neural networks. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning—Volume 4, JMLR.org, pp 507–516

  113. Krizhevsky A, Krizhevsky A (2009) Learning multiple layers of features from tiny images. University of Toronto, Toronto, Canada

  114. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM (2017) ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings—30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January, pp 3462–3471. https://doi.org/10.1109/CVPR.2017.369

  115. Cohen JP, Morrison P, Dao L, Roth K, Duong TQ, Ghassemi M (2020) COVID-19 image data collection: prospective predictions are the future. J Mach Learn Biomed Imaging 2020: 2–3. https://arxiv.org/abs/2006.11988v3

  116. Pan H, Chao Z, Qian J, Zhuang B, Wang S, Xiao J (2021) Network pruning using linear dependency analysis on feature maps. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 1720–1724. https://doi.org/10.1109/ICASSP39728.2021.9414393

  117. Huang Z, Wang N (2018) Data-driven sparse structure selection for deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp 317–334. https://doi.org/10.1007/978-3-030-01270-0_19

  118. Qi C, Shen S, Li R, Zhao Z, Liu Q, Liang J, Zhang H (2021) An efficient pruning scheme of deep neural networks for Internet of Things applications. EURASIP J Adv Signal Process 1:1–21. https://doi.org/10.1186/S13634-021-00744-4

    Article  Google Scholar 

  119. Berthelier A, Yan Y, Chateau T, Blanc C, Duffner S, Garcia C (2021) Learning sparse filters in deep convolutional neural networks with A l 1 /l 2 pseudo-norm. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12661 LNCS, pp 662–676. https://doi.org/10.1007/978-3-030-68763-2_50

  120. Li Y, Lin S, Zhang B, Liu J, Doermann D, Wu Y, Huang F, Ji R (2019) Exploiting kernel sparsity and entropy for interpretable CNN compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2795–2804. https://doi.org/10.1109/CVPR.2019.00291

  121. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June, pp 4335–4344. https://doi.org/10.1109/CVPR.2019.00447

  122. Lin M, Cao L, Zhang Y, Shao L, Lin CW, Ji R (2022) Pruning networks with cross-layer ranking & k-Reciprocal nearest filters. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2022.3156047

    Article  Google Scholar 

  123. Phan HH, Vu NS (2019) Information theory-based pruning for CNN compression and its application to image classification and action recognition. In: 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2019. https://doi.org/10.1109/AVSS.2019.8909826

  124. Wu T, Li X, Zhou D, Li N, Shi J (2021) Differential evolution based layer-wise weight pruning for compressing deep neural networks. Sensors 21:880. https://doi.org/10.3390/S21030880

    Article  Google Scholar 

  125. Sui X, Lv Q, Zhi L, Zhu B, Yang Y, Zhang Y, Tan Z (2023) A hardware-friendly high-precision CNN pruning method and its FPGA implementation. Sensors 23(2):824. https://doi.org/10.3390/s23020824

    Article  Google Scholar 

  126. Parambath SAP, Usunier N, Grandvalet Y (2014) Optimizing F-measures by cost-sensitive classification. In: Advances in Neural Information Processing Systems 27

  127. Elkan C (2001) The foundations of cost-sensitive learning. In: Proceedings of the Seventeenth International Conference on Artificial Intelligence, 4–10 August 2001; Seattle, 1

  128. Janisch J, Pevný T, Lisý V (2017) Classification with costly features using deep reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, 33. https://doi.org/10.1609/aaai.v33i01.33013959

  129. Melo F (2013) Area under the ROC Curve. In: Encyclopedia of Systems Biology, pp 38–39. https://doi.org/10.1007/978-1-4419-9863-7_209

  130. Adams NM, Hand DJ (1999) Comparing classifiers when the misallocation costs are uncertain. Pattern Recogn 32(7):1139–1147. https://doi.org/10.1016/S0031-3203(98)00154-X

    Article  Google Scholar 

  131. Hand DJ, Till RJ (2001) A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach Learn 45(2):171–186. https://doi.org/10.1023/A:1010920819831/METRICS

    Article  MATH  Google Scholar 

  132. Ringrose TJ, Hand DJ (1997) Construction and assessment of classification rules. Biometrics 53(3):1181. https://doi.org/10.2307/2533581

    Article  Google Scholar 

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran

    Saeed Ahmadluei & Behrooz Masoumi

  2. Electrical Engineering Department, Amirkabir University of Technology, Hafez Ave., Tehran, 15916-34311, Iran

    Karim Faez

Authors
  1. Saeed Ahmadluei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Karim Faez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Behrooz Masoumi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by SA, KF and BM. The first draft of the manuscript was written by SA and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Karim Faez.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmadluei, S., Faez, K. & Masoumi, B. Convolutional neural network pruning based on misclassification cost. J Supercomput 79, 21185–21234 (2023). https://doi.org/10.1007/s11227-023-05487-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-023-05487-7

Keywords

Search

Navigation