Abstract
Dynamic pruning is an effective technique that has been widely applied to deep neural network compression. However, many existing dynamic methods involve network modification and time-consuming retraining. In this paper, we propose a dynamic region pruning method to skip partial regions in CNNs by considering the characteristic of pooling (max- or min-pooling). Specifically, kernels in a filter are ordered according to their \(\ell _{1}\)-norm so that a smaller left partial sum could be obtained, leading to more opportunities for pruning. Without fine-tuning or network retraining, scale factors approximating the left partial sum are efficiently and thoroughly explored by model inferencing on the validation dataset. The experimental results on the testing dataset show that we could achieve up to 43.6% operation reduction while keeping a negligible accuracy loss.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ehteshami Bejnordi, A., Krestel, R.: Dynamic channel and layer gating in convolutional neural networks. In: Schmid, U., Klügl, F., Wolter, D. (eds.) KI 2020. LNCS (LNAI), vol. 12325, pp. 33–45. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58285-2_3
Chen, A., Smith, G.: (2017). https://github.com/aaron-xichen/pytorch-playground
Gao, X., Zhao, Y., Dudziak, Ł., Mullins, R., Zhong Xu, C.: Dynamic channel pruning: feature boosting and suppression. In: International Conference on Learning Representations (2019)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Hua, W., Zhou, Y., De Sa, C., Zhang, Z., Suh, G.E.: Channel gating neural networks. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 1886–1896 (2019)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012)
Lin, J., Rao, Y., Lu, J., Zhou, J.: Runtime neural pruning. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 2178–2188 (2017)
Liu, C., Wang, Y., Han, K., Xu, C., Xu, C.: Learning instance-wise sparsity for accelerating deep models. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 3001–3007. International Joint Conferences on Artificial Intelligence Organization (2019)
Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: NIPS (2017)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR (2015)
Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)
Wang, Y., et al.: Dual dynamic inference: enabling more efficient, adaptive, and controllable deep inference. IEEE J. Sel. Top. Signal Process. 14(4), 623–633 (2020)
Wu, Z., et al.: Blockdrop: dynamic inference paths in residual networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8817–8826 (2018)
Xia, W., Yin, H., Dai, X., Jha, N.K.: Fully dynamic inference with deep neural networks. IEEE Trans. Emerg. Top. Comput. 01, 1 (2021)
Yang, B., Bender, G., Le, Q.V., Ngiam, J.: CondConv: conditionally parameterized convolutions for efficient inference. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 1307–1318 (2019)
Yuan, Z., Wu, B., Sun, G., Liang, Z., Zhao, S., Bi, W.: S2DNAS: transforming static CNN model for dynamic inference via neural architecture search. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020, Part II. LNCS, vol. 12347, pp. 175–192. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_11
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 61804017 and in part by the National Key Research and Development Project of China (No. 2020AAA0104603).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Y., Liu, D., Xing, Y. (2021). Dynamic Convolution Pruning Using Pooling Characteristic in Convolution Neural Networks. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Communications in Computer and Information Science, vol 1516. Springer, Cham. https://doi.org/10.1007/978-3-030-92307-5_65
Download citation
DOI: https://doi.org/10.1007/978-3-030-92307-5_65
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92306-8
Online ISBN: 978-3-030-92307-5
eBook Packages: Computer ScienceComputer Science (R0)