ABSTRACT
Weight pruning is a well-known technique used for network compression. In contrast to filter pruning, weight pruning produces higher compression ratios as it is more fine-grained. However, pruning individual weights results in broken kernels, which cannot be directly accelerated on general platforms, leading to hardware compatibility issues. To address this issue, we propose Shift Pruning (SP), a novel weight pruning method that is compatible with general platforms. SP converts spatial convolutions into regular 1 X 1 convolutions and shift operations, which are simply memory movements that do not require additional FLOPs or parameters. Specifically, we decompose the original K X K convolution into parallel branches of shift-convolution operations and devise the Differentiable Shift Operator (DSO), an approximation form of the actual shift operation, to automatically learn the crucial directions for adequate spatial interactions with the designed shift-related loss function. A regularization term is proposed to prevent redundant shifting, which is beneficial for low-resolution situations. To further improve inference efficacy, we develop a post-training transformation that can construct a more compact model. The introduced channel-wise slimming allows SP to prune in a hybrid-structural manner, catering for both hardware compatibility and a high compression ratio. Extensive experiments on the CIFAR-10 and ImageNet datasets demonstrate that our proposed method achieves superior performance in both accuracy and FLOPs reduction compared to other state-of-the-art techniques. For instance, on ImageNet, we can reduce 48.8% of total FLOPs on ResNet-34 with only 0.22% Top-1 accuracy drop.
- Manoj Alwani, Vashisht Madhavan, and Yang Wang. 2021. DECORE: Deep Compression with Reinforcement Learning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 12339--12349.Google Scholar
- Manoj Alwani, Vashisht Madhavan, and Yang Wang. 2022. DECORE: Deep Compression with Reinforcement Learning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022), 12339--12349.Google Scholar
- Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to 1 or -1. arXiv: Learning (2016).Google Scholar
- Emily L. Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014. Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. ArXiv, Vol. abs/1404.0736 (2014).Google Scholar
- Xiaohan Ding, Guiguang Ding, Yuchen Guo, and J. Han. 2019. Centripetal SGD for Pruning Very Deep Convolutional Networks With Complicated Structure. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 4938--4948.Google Scholar
- Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. 2020. ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2020), 4490--4500.Google Scholar
- Xiaohan Ding, X. Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. 2021. RepVGG: Making VGG-style ConvNets Great Again. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 13728--13737.Google Scholar
- Xuanyi Dong and Yi Yang. 2019. Network Pruning via Transformable Architecture Search. ArXiv, Vol. abs/1905.09717 (2019).Google Scholar
- Sara Elkerdawy, Mostafa Elhoushi, Hong Zhang, and Nilanjan Ray. 2022. Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022), 12444--12453.Google Scholar
- Lukas Enderich, Fabian Timm, and Wolfram Burgard. 2021. Holistic Filter Pruning for Efficient Deep Neural Networks. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2021), 2595--2604.Google Scholar
- Jonathan Frankle and Michael Carbin. 2019. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. arXiv: Learning (2019).Google Scholar
- Shangqian Gao, Feihu Huang, Weidong (Tom) Cai, and Heng Huang. 2021. Network Pruning via Performance Maximization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 9266--9276.Google ScholarCross Ref
- Shangqian Gao, Feihu Huang, Jian Pei, and Heng Huang. 2020. Discrete Model Compression With Resource Constraint for Deep Neural Networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), 1896--1905.Google Scholar
- Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. 2020. DMCP: Differentiable Markov Channel Pruning for Neural Networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), 1536--1544.Google Scholar
- Yi Guo, Huan Yuan, Jianchao Tan, Zhangyang Wang, Sen Yang, and Ji Liu. 2021. GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021), 5219--5230.Google ScholarCross Ref
- Ghouthi Boukli Hacene, C. Lassance, Vincent Gripon, Matthieu Courbariaux, and Yoshua Bengio. 2021. Attention Based Pruning for Shift Networks. 2020 25th International Conference on Pattern Recognition (ICPR) (2021), 4054--4061.Google Scholar
- Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. arXiv: Computer Vision and Pattern Recognition (2016).Google Scholar
- Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 770--778.Google Scholar
- Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. 2020a. Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), 2006--2015.Google Scholar
- Yang He, Xuanyi Dong, Guoliang Kang, Yanwei Fu, Chenggang Clarence Yan, and Yi Yang. 2020b. Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks. IEEE Transactions on Cybernetics, Vol. 50 (2020), 3594--3604.Google ScholarCross Ref
- Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. 2018. Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks. ArXiv, Vol. abs/1808.06866 (2018).Google Scholar
- Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. 2019. Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 4335--4344.Google ScholarCross Ref
- Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yingzhi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, and S. Y. Kung. 2022. CHEX: CHannel EXploration for CNN Model Compression. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022), 12277--12288.Google Scholar
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4700--4708.Google ScholarCross Ref
- Eric Jang, Shixiang Shane Gu, and Ben Poole. 2017. Categorical Reparameterization with Gumbel-Softmax. ArXiv, Vol. abs/1611.01144 (2017).Google Scholar
- Yunho Jeon and Junmo Kim. 2018. Constructing Fast Network through Deconstruction of Convolution. In NeurIPS.Google Scholar
- Francisco Erivaldo Fernandes Junior and Gary G. Yen. 2021. Pruning Deep Convolutional Neural Networks Architectures with Evolution Strategy. Inf. Sci., Vol. 552 (2021), 29--47.Google ScholarCross Ref
- Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images.Google Scholar
- Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2017. Pruning Filters for Efficient ConvNets. ArXiv, Vol. abs/1608.08710 (2017).Google Scholar
- Yuchao Li, Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang, Jincheng Ma, Qi Tian, and Rongrong Ji. 2021. Towards Compact CNNs via Collaborative Compression. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 6434--6443.Google ScholarCross Ref
- Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. 2020. Provable Filter Pruning for Efficient Neural Networks. ArXiv, Vol. abs/1911.07412 (2020).Google Scholar
- Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. 2020a. HRank: Filter Pruning Using High-Rank Feature Map. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), 1526--1535.Google Scholar
- Mingbao Lin, Rongrong Ji, Yu xin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. 2020b. Channel Pruning via Automatic Structure Search. ArXiv, Vol. abs/2001.08565 (2020).Google Scholar
- Shaohui Lin, R. Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David S. Doermann. 2019. Towards Optimal Structured CNN Pruning via Generative Adversarial Learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 2785--2794.Google ScholarCross Ref
- Jing Liu, Bohan Zhuang, Zhuangwei Zhuang, Yong Guo, Junzhou Huang, Jin-Hui Zhu, and Mingkui Tan. 2022. Discrimination-Aware Network Pruning for Deep Model Compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 44 (2022), 4035--4051.Google ScholarDigital Library
- Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learning Efficient Convolutional Networks through Network Slimming. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 2755--2763.Google ScholarCross Ref
- Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. 2019. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference on computer vision. 3296--3305.Google ScholarCross Ref
- Zili Liu, Peisong Wang, and Zaixing Li. 2021. More-Similar-Less-Important: Filter Pruning VIA Kmeans Clustering. 2021 IEEE International Conference on Multimedia and Expo (ICME) (2021), 1--6.Google Scholar
- Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiao-Wei Guo, Guangming Lu, and Xing Sun. 2020. Pruning Filter in Filter. ArXiv, Vol. abs/2009.14410 (2020).Google Scholar
- Ying Nie, Kai Han, Zhenhua Liu, An Xiao, Yiping Deng, Chunjing Xu, and Yunhe Wang. 2021. GhostSR: Learning Ghost Features for Efficient Image Super-Resolution. ArXiv, Vol. abs/2101.08525 (2021).Google Scholar
- Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. 2020. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III. Springer, 592--607.Google Scholar
- Tao Niu, Yinglei Teng, Panpan Zou, and Yiding Liu. 2023. Lossless Filter Pruning via Adaptive Clustering for Convolutional Neural Networks. https://openreview.net/forum?id=Pi5LI8sJYYzGoogle Scholar
- Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb. 2021. SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning. ArXiv, Vol. abs/2110.11395 (2021).Google Scholar
- Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 779--788.Google ScholarCross Ref
- Xiaofeng Ruan, Yufan Liu, Bing Li, Chunfen Yuan, and Weiming Hu. 2021. DPFPS: Dynamic and Progressive Filter Pruning for Compressing Convolutional Neural Networks from Scratch. In AAAI.Google Scholar
- Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, Vol. 115 (2015), 211--252.Google ScholarDigital Library
- Haopu Shang, Jialong Wu, Wenjing Hong, and Chaojun Qian. 2022. Neural Network Pruning by Cooperative Coevolution. In International Joint Conference on Artificial Intelligence.Google ScholarCross Ref
- Evan Shelhamer, Jonathan Long, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015), 3431--3440.Google Scholar
- Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR, Vol. abs/1409.1556 (2015).Google Scholar
- Yang Sui, Miao Yin, Yi Xie, Huy Phan, S. Zonouz, and Bo Yuan. 2021. CHIP: CHannel Independence-based Pruning for Compact Neural Networks. In Neural Information Processing Systems.Google Scholar
- Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. 2020. SCOP: Scientific Control for Reliable Neural Network Pruning. ArXiv, Vol. abs/2010.10732 (2020).Google Scholar
- Chaoqi Wang, ChaoQi Wang, Guodong Zhang, and Roger Baker Grosse. 2020. Picking Winning Tickets Before Training by Preserving Gradient Flow. ArXiv, Vol. abs/2002.07376 (2020).Google Scholar
- Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. 2017. Residual Attention Network for Image Classification. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 6450--6458.Google Scholar
- Bichen Wu, Alvin Wan, Xiangyu Yue, Peter H. Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph E. Gonzalez, and Kurt Keutzer. 2018. Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 9127--9135.Google ScholarCross Ref
- Xia Xiao, Zigeng Wang, and Sanguthevar Rajasekaran. 2019. AutoPrune: Automatic Network Pruning by Regularizing Auxiliary Parameters. In NeurIPS.Google Scholar
- Tianshuo Xu, Yuhang Wu, Xiawu Zheng, Teng Xi, Gang Zhang, Errui Ding, Fei Chao, and Rongrong Ji. 2021. CDP: towards optimal filter pruning via class-wise discriminative power. In Proceedings of the 29th ACM International Conference on Multimedia. 5491--5500.Google ScholarDigital Library
- Haoran You, Chaojian Li, Pengfei Xu, Y. Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, and Richard Baraniuk. 2020. Drawing early-bird tickets: Towards more efficient training of deep networks. ArXiv, Vol. abs/1909.11957 (2020).Google Scholar
- Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. 2019. Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks. ArXiv, Vol. abs/1909.08174 (2019).Google Scholar
- Haonan Zhang, Longjun Liu, Hengyi Zhou, Wenxuan Hou, Hongbin Sun, and Nanning Zheng. 2021. Akecp: Adaptive knowledge extraction from feature maps for fast and efficient channel pruning. In Proceedings of the 29th ACM International Conference on Multimedia. 648--657.Google ScholarDigital Library
- Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. 2018. A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers. In ECCV.Google Scholar
- Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian. 2019. Variational Convolutional Neural Network Pruning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 2775--2784.Google Scholar
- Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. 2019. Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask. In NeurIPS.Google Scholar
- Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. 2020. Neuron-level Structured Pruning using Polarization Regularizer. In NeurIPS.Google Scholar
Index Terms
- Shift Pruning: Equivalent Weight Pruning for CNN via Differentiable Shift Operator
Recommendations
Stochastic weight pruning and the role of regularization in shaping network structure
Graphical abstractDisplay Omitted
Highlights- WtoNP – a general stochastic approach for neural network pruning.
- WtoNP ...
AbstractThe pressing need to reduce the capacity of deep neural networks has stimulated the development of network dilution methods and their analysis. In this study we present a framework for neural network pruning by sampling from a ...
Weight-Dependent Gates for Differentiable Neural Network Pruning
Computer Vision – ECCV 2020 WorkshopsAbstractIn this paper, we propose a simple and effective network pruning framework, which introduces novel weight-dependent gates to prune filter adaptively. We argue that the pruning decision should depend on the convolutional weights, in other words, it ...
Structured Pruning with Automatic Pruning Rate Derivation for Image Processing Neural Networks
ISMSI '22: Proceedings of the 2022 6th International Conference on Intelligent Systems, Metaheuristics & Swarm IntelligenceStructured pruning has been proposed for network model compression. Because most of existing structured pruning methods assign pruning rate manually, finding appropriate pruning rate to suppress the degradation of pruned model accuracy is difficult. ...
Comments