skip to main content
10.1145/3383972.3384008acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicmlcConference Proceedingsconference-collections
research-article

An Attentive Pruning Method for Edge Computing

Published: 26 May 2020 Publication History

Abstract

Nowadays, the power of an edge computing hardware is usually the bottleneck of AI application. Most of the embedding devices in factory have no GPU, some of them even using Intel i3 or ARM chips. Model compression is very convenient and effective way to handle this problem. But it still has two major limitations, the arbitrariness in pruning ratio setup and inevitable accuracy drop. In this paper, we propose an adaptive network channel pruning method without a priori knowledge of pruning ratio. This method is based on the attentive weights from modified SE block, establishes a detectable and trainable mask learning module from the original to-be-prune network. Moreover we make innovative modifications on SE block, to enhance the sparsity of attentive weights. Extensive experiments afterwards indicate that our method can not only accelerate model inference process or equivalently decrease model footprint, but also get better performance in test set. Even a tiny network like Yolo cifar-10 with 15 layers can be pruned about 50% FLOPs without accuracy decrease using proposed method.

References

[1]
A. Krizhevsky, I. Sutskever, and G. E. Hinton."ImageNet Classification with Deep Convolutional Neural Networks." Neural Information Processing Systems, vol. 141, no. 5, 2012, pp. 1097--1105.
[2]
J. Long, E. Shelhamer, and T. Darrell."Fully Convolutional Networks for Semantic Segmentation." ArXiv Preprint ArXiv:1605.06211, 2014.
[3]
S. Ren, K. He, R. Girshick, et al. "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, 2017, pp. 1137--1149.
[4]
A. Toshev and C. Szegedy. "DeepPose: Human Pose Estimation via Deep Neural Networks." CVPR '14 Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1653--1660.
[5]
Redmon J, Farhadi A. Toshev. "DeepPose: Human Pose Estimation via Deep Neural Networks." CVPR '14 Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1653--1660.
[6]
Z. Bo, X. Wu, J. S. Feng, et al. "Diversified Visual Attention Networks for Fine-Grained Object Classification." IEEE Transactions on Multimedia, vol. 19, no. 6, 2017, pp. 1245--1256.
[7]
W Fei, M. Q. Jiang, Q Chen, el al. "Residual Attention Network for Image Classification." 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6450--6458.
[8]
J. Hu, S Li, S Albanie, et al. "Squeeze-and-Excitation Networks." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132--7141.
[9]
Y. L. Cun, J. S. Denker, S. A. Solla. "Optimal Brain Damage." Advances in Neural Information Processing Systems 2, vol. 2, 1989, pp. 598--605.
[10]
B. Hassibi. "Second Order Derivatives for Network Pruning: Optimal Brain Surgeon." Advances in Neural Information Processing Systems 5, 1992, pp. 164--171.
[11]
C. Louizos, M. Welling and D. Kingma, "Learning Sparse Neural Networks through L_0 Regularization." ICLR 2018: International Conference on Learning Representations 2018, 2018.
[12]
L. Theis, I. Korshunova and A. Tejani, et.al., "Faster Gaze Prediction with Dense Networks and Fisher Pruning." ArXiv Preprint ArXiv:1801.05787, 2018. S.
[13]
S. Lin, R. ji, Y. Li, et.al., "Accelerating Convolutional Networks via Global & Dynamic Filter Pruning." IJCAI 2018: 27th International Joint Conference on Artificial Intelligence, 2018, pp. 2425--2432.
[14]
P. Molchanov, S. Tyree, T. Karras, et.al., "Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning," 2016.
[15]
J. Redmon. "Darknet: Open Source Neural Network in C," \url{http:/pjreddie.com/darknet/}.2013-2016
[16]
X. Glorot, Y. Bengio. "Understanding the difficulty of training deep feedforward neural networks," Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 2010, pp. 249--256.

Cited By

View all
  • (2023)Toward Designing an Attentive Deep Trajectory Predictor Based on Bluetooth Low Energy Signal2023 57th Annual Conference on Information Sciences and Systems (CISS)10.1109/CISS56502.2023.10089682(1-6)Online publication date: 22-Mar-2023

Index Terms

  1. An Attentive Pruning Method for Edge Computing

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICMLC '20: Proceedings of the 2020 12th International Conference on Machine Learning and Computing
    February 2020
    607 pages
    ISBN:9781450376426
    DOI:10.1145/3383972
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    In-Cooperation

    • Shenzhen University: Shenzhen University

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 May 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Improved SE block
    2. Keep Accuracy
    3. Model Compress

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    ICMLC 2020

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)17
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 20 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Toward Designing an Attentive Deep Trajectory Predictor Based on Bluetooth Low Energy Signal2023 57th Annual Conference on Information Sciences and Systems (CISS)10.1109/CISS56502.2023.10089682(1-6)Online publication date: 22-Mar-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media