Linear Substitution Pruning: Consider All Filters Together
Authors: 
Kai Liu, Cong LiuAuthors Info & Claims
Kai Liu, Cong LiuAuthors Info & ClaimsPages 68 - 75
Abstract
Filter (neuron) pruning is a neural network compression approach. In previous work, the importance of each filter is generally considered individually or pairwisely. This paper shows that considering the linear relation among all filters can help us prune more efficiently. Based on this intuition, we propose a new filter pruning method, named Linear Substitution Pruning (LSP). Similar to LSP, we also propose a model compensation method, called Linear Substitution Compensation (LSC), which restores the model performance after pruning by using all remaining filters to compensate for the error caused by pruning. The experiments show that our method outperforms the state-of-the-art filter pruning methods. LSP achieves a reduction of 61.04% in flops on ResNet110 with an increase of 0.96% in top-1 accuracy at the same time, and it achieves a reduction of 51.84% in flops on ResNet50 with also an increase of 0.05% in top-1 accuracy.
References
[1]
Christophe Couvreur and Yoram Bresler. 2000. On the optimality of the backward greedy algorithm for the subset selection problem. SIAM J. Matrix Anal. Appl. 21, 3 (2000), 797–808.
[2]
Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. 2020. Lossless CNN Channel Pruning via Decoupling Remembering and Forgetting. arXiv preprint arXiv:2007.03260(2020).
[3]
Ahmed K Farahat, Ahmed Elgohary, Ali Ghodsi, and Mohamed S Kamel. 2015. Greedy column subset selection for large-scale data sets. Knowledge and Information Systems 45, 1 (2015), 1–34.
[4]
Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635(2018).
[5]
Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. 2021. Network Pruning via Performance Maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9270–9280.
[6]
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. 2020. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1539–1547.
[7]
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. 2020. Learning filter pruning criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2009–2018.
[8]
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. 2019. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4340–4349.
[9]
Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision. 1389–1397.
[10]
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. 2016. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250(2016).
[11]
Harshil Jain, Akshat Agarwal, Kumar Shridhar, and Denis Kleyko. 2020. End to end binarized neural networks for text classification. arXiv preprint arXiv:2010.05223(2020).
[12]
Donggyu Joo, Eojindl Yi, Sunghyun Baek, and Junmo Kim. 2021. Linearly Replaceable Filters for Deep Network Channel Pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 8021–8029.
[13]
Woojeong Kim, Suhyun Kim, Mincheol Park, and Geonseok Jeon. 2020. Neuron Merging: Compensating for Pruned Neurons. arXiv preprint arXiv:2010.13160(2020).
[14]
Tamara G Kolda and Brett W Bader. 2009. Tensor decompositions and applications. SIAM review 51, 3 (2009), 455–500.
[15]
Alex Krizhevsky, Geoffrey Hinton, 2009. Learning multiple layers of features from tiny images. (2009).
[16]
Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. Advances in neural information processing systems 2 (1989), 598–605.
[17]
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. 2020. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In European Conference on Computer Vision. Springer, 639–654.
[18]
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710(2016).
[19]
Hang Li, Chen Ma, Wei Xu, and Xue Liu. 2020. Feature Statistics Guided Efficient Filter Pruning.
[20]
Yawei Li, Shuhang Gu, Luc Van Gool, and Radu Timofte. 2019. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5623–5632.
[21]
Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. 2020. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8018–8027.
[22]
Yawei Li, Shuhang Gu, Kai Zhang, Luc Van Gool, and Radu Timofte. 2020. DHP: Differentiable meta pruning via hypernetworks. arXiv preprint arXiv:2003.13683(2020).
[23]
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. 2020. HRank: Filter Pruning using High-Rank Feature Map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1529–1538.
[24]
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2117–2125.
[25]
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. 2019. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3296–3305.
[26]
Jian-Hao Luo and Jianxin Wu. 2020. Neural Network Pruning with Residual-Connections and Limited-Data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1458–1467.
[27]
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. 2017. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision. 5058–5066.
[28]
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 5191–5198.
[29]
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE International Conference on Computer Vision. 1325–1334.
[30]
Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. 2020. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. arXiv preprint arXiv:2004.02164(2020).
[31]
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550(2014).
[32]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211–252. https://doi.org/10.1007/s11263-015-0816-y
[33]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556(2014).
[34]
Xiu Su, Shan You, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. 2021. BCNet: Searching for Network Width with Bilaterally Coupled Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2175–2184.
[35]
Mingxing Tan and Quoc V Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946(2019).
[36]
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. 2020. SCOP: Scientific Control for Reliable Neural Network Pruning.
[37]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008.
[38]
Chaoqi Wang, Guodong Zhang, and Roger Grosse. 2020. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376(2020).
[39]
Zi Wang, Chengcheng Li, and Xiangyang Wang. 2021. Convolutional neural network pruning with structural redundancy reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14913–14922.
[40]
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. 2015. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence 38, 10(2015), 1943–1955.
[41]
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. 2020. Neuron-level Structured Pruning using Polarization Regularizer.
- Linear Substitution Pruning: Consider All Filters Together
Recommendations
A novel and efficient model pruning method for deep convolutional neural networks by evaluating the direct and indirect effects of filters
AbstractDeploying deep convolutional neural networks (DCNNs) on devices with low memory resources or in applications with strict latency requirements remains a challenge. The weight-based filter pruning is an effective technique that has been widely ...
Studying the plasticity in deep convolutional neural networks using random pruning
Recently, there has been a lot of work on pruning filters from deep convolutional neural networks (CNNs) with the intention of reducing computations. The key idea is to rank the filters based on a certain criterion (say, $$l_1$$l1-norm, average ...
Filter pruning via annealing decaying for deep convolutional neural networks acceleration
AbstractFilter pruning has been used on a large scale to compress and accelerate convolutional neural networks. The goal of filter pruning is to find the optimal network substructure from the unpruned network. Most previous pruning methods directly remove ...
Comments
Information & Contributors
Information
Published In
March 2022
809 pages
ISBN:9781450396110
DOI:10.1145/3532213
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 July 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- NSFC
Conference
ICCAI '22
ICCAI '22: 2022 8th International Conference on Computing and Artificial Intelligence
March 18 - 21, 2022
Tianjin, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 69Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)1
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format