skip to main content
10.1145/3503161.3547965acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Bayesian based Re-parameterization for DNN Model Pruning

Published: 10 October 2022 Publication History

Abstract

Filter pruning, as an effective strategy to obtain efficient compact structures from over-parametric deep neural networks(DNN), has attracted a lot of attention. Previous pruning methods select channels for pruning by developing different criteria, yet little attention has been devoted to whether these criteria can represent correlations between channels. Meanwhile, most existing methods generally ignore the parameters being pruned and only perform additional training on the retained network to reduce accuracy loss. In this paper, we present a novel perspective of re-parametric pruning by Bayesian estimation. First, we estimate the probability distribution of different channels based on Bayesian estimation and indicate the importance of the channels by the discrepancy in the distribution before and after channel pruning. Second, to minimize the variation in distribution after pruning, we re-parameterize the pruned network based on the probability distribution to pursue optimal pruning. We evaluate our approach on popular datasets with some typical network architectures, and comprehensive experimental results validate that this method illustrates better performance compared to the state-of-the-art approaches.

References

[1]
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2019. Oncefor-All: Train One Network and Specialize it for Efficient Deployment.
[2]
Han Cai, Ligeng Zhu, and Song Han. 2018. Proxylessnas: Direct neural architecture search on target task and hardware. (2018).
[3]
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. 2021. Distilling Knowledge via Knowledge Review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5008--5017.
[4]
Ting-Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. 2020. Towards Efficient Model Compression via Learned Global Ranking. 1518--1528.
[5]
Xuanyi Dong and Yi Yang. 2019. Network pruning via transformable architecture search. 759--770.
[6]
Richard M Dudley. 2010. Sample functions of the Gaussian process. Selected Works of RM Dudley (2010), 187--224.
[7]
R. Eldan and O. Shamir. 2015. The Power of Depth for Feedforward Neural Networks. Computer Science (2015).
[8]
Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. 2021. Network Pruning via Performance Maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9270--9280.
[9]
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. 2020. DMCP: Differentiable Markov Channel Pruning for Neural Networks. 1539--1547.
[10]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, and Ieee. 2016. Deep Residual Learning for Image Recognition (IEEE Conference on Computer Vision and Pattern Recognition). 770--778. https://doi.org/10.1109/cvpr.2016.90
[11]
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. 2020. Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration. 2009--2018.
[12]
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. 2018. Soft filter pruning for accelerating deep convolutional neural networks.
[13]
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. 2019. Filter pruning via geometric median for deep convolutional neural networks acceleration. 4340-- 4349.
[14]
Tao Huang, Weisheng Dong, Jinshan Liu, Fangfang Wu, Guang ming Shi, and Xin Li. 2020. Accelerating Convolutional Neural Network via Structured Gaussian Scale Mixture Models: a Joint Grouping and Pruning Approach. (2020).
[15]
Alex Krizhevsky, I. Sutskever, and G. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks.
[16]
Ya Le and Xuan Yang. 2015. Tiny imagenet visual recognition challenge. CS 231N 7 (2015), 7.
[17]
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang. 2020. Blockwisely Supervised Neural Architecture Search with Knowledge Distillation. arXiv:1911.13053 [cs.CV]
[18]
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016).
[19]
Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. 2020. Group sparsity: The hinge between filter pruning and decomposition for network compression. 8018--8027.
[20]
Yuchao Li, Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang, Jincheng Ma, Qi Tian, and Rongrong Ji. 2021. Towards Compact CNNs via Collaborative Compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6438--6447.
[21]
Zhihang Li, Teng Xi, Jiankang Deng, Gang Zhang, Shengzhao Wen, and Ran He. 2020. Gp-nas: Gaussian process based neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11933-- 11942.
[22]
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. 2020. HRank: Filter Pruning using High-Rank Feature Map. 1529--1538.
[23]
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. (2018).
[24]
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. 2019. Metapruning: Meta learning for automatic neural network channel pruning. 3296--3305.
[25]
Christos Louizos, Karen Ullrich, and Max Welling. 2017. Bayesian compression for deep learning. Advances in neural information processing systems 30 (2017).
[26]
Xiaotong Lu, Han Huang, Weisheng Dong, Xin Li, and Guangming Shi. 2020. Beyond network pruning: a joint search-and-training approach.
[27]
Chao Ma, Jia-Bin Huang, Xiaokang Yang, Ming-Hsuan Yang, and Ieee. 2015. Hierarchical Convolutional Features for Visual Tracking (IEEE International Conference on Computer Vision). 3074--3082. https://doi.org/10.1109/iccv.2015.352
[28]
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. 2017. Structured bayesian pruning via log-normal multiplicative noise. Advances in Neural Information Processing Systems 30 (2017).
[29]
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. 2018. Efficient neural architecture search via parameter sharing. (2018).
[30]
Carl Edward Rasmussen. 2003. Gaussian processes for machine learning.
[31]
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2019. Regularized evolution for image classifier architecture search, Vol. 33. 4780--4789.
[32]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2017. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. 39, 6 (2017), 1137--1149. <GotoISI>://WOS:000401091200007
[33]
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and LiangChieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. 4510-- 4520.
[34]
Zi Wang, Chengcheng Li, and Xiangyang Wang. 2021. Convolutional neural network pruning with structural redundancy reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14913--14922.
[35]
Xin Yuan, Liangliang Ren, Jiwen Lu, and Jie Zhou. 2019. Enhanced bayesian compression via deep reinforcement learning. 6946--6955.
[36]
Yuefu Zhou, Ya Zhang, Yanfeng Wang, and Qi Tian. 2019. Accelerate cnn via recursive bayesian pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3306--3315.
[37]
Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. (2016)

Cited By

View all
  • (2024)HINER: Neural Representation for Hyperspectral ImageProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681643(9837-9846)Online publication date: 28-Oct-2024
  • (2024)All-in-One Hardware-Oriented Model Compression for Efficient Multi-Hardware DeploymentIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.343462634:12(12345-12359)Online publication date: Dec-2024
  • (2023)MIEP: Channel Pruning with Multi-granular Importance Estimation for Object DetectionProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612563(2908-2917)Online publication date: 26-Oct-2023

Index Terms

  1. Bayesian based Re-parameterization for DNN Model Pruning

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '22: Proceedings of the 30th ACM International Conference on Multimedia
    October 2022
    7537 pages
    ISBN:9781450392037
    DOI:10.1145/3503161
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 October 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. bayesian
    2. channel pruning
    3. model compression

    Qualifiers

    • Research-article

    Funding Sources

    • National Key RD Program of China
    • Natural Science Foundation of China

    Conference

    MM '22
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)37
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)HINER: Neural Representation for Hyperspectral ImageProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681643(9837-9846)Online publication date: 28-Oct-2024
    • (2024)All-in-One Hardware-Oriented Model Compression for Efficient Multi-Hardware DeploymentIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.343462634:12(12345-12359)Online publication date: Dec-2024
    • (2023)MIEP: Channel Pruning with Multi-granular Importance Estimation for Object DetectionProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612563(2908-2917)Online publication date: 26-Oct-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media