Zhiyu Zhu
Minhui Xue
ABSTRACT
The rapid development of deep learning has demonstrated its potential for deployment in many intelligent service systems. However, some issues such as optimisation (e.g., how to reduce the deployment resources costs and further improve the detection speed), especially in scenarios where limited resources are available, remain challenging to address. In this paper, we aim to delve into the principles of deep neural networks, focusing on the importance of network neurons. The goal is to identify the neurons that exert minimal impact on model performances, thereby aiding in the process of model pruning. In this work, we have thoroughly considered the deep learning model pruning process with and without fine-tuning step, ensuring the model performance consistency. To achieve our objectives, we propose a methodology that employs adversarial attack methods to explore deep neural network parameters. This approach is combined with an innovative attribution algorithm to analyse the level of network neurons involvement. In our experiments, our approach can effectively quantify the importance of network neuron. We extend the evaluation through comprehensive experiments conducted on a range of datasets, including CIFAR-10, CIFAR-100 and Caltech101. The results demonstrate that, our method have consistently achieved the state-of-the-art performance over many existing methods. We anticipate that this work will help to reduce the heavy training and inference cost of deep neural network models where a lightweight deep learning enhanced service and system is possible. The source code is open source at https://github.com/LMBTough/FVW.
- A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis et al., "Deep learning for computer vision: A brief review," Computational intelligence and neuroscience, vol. 2018, 2018.Google Scholar
- A. Osipov, E. Pleshakova, S. Gataullin, S. Korchagin, M. Ivanov, A. Finogeev, and V. Yadav, "Deep learning method for recognition and classification of images from video recorders in difficult weather conditions," Sustainability, vol. 14, no. 4, p. 2420, 2022.Google ScholarCross Ref
- S. Mehtab and J. Sen, "Analysis and forecasting of financial time series using cnn and lstm-based deep learning models," in Advances in Distributed Computing and Machine Learning: Proceedings of ICADCML 2021. Springer, 2022, pp. 405--423.Google Scholar
- M. E. Alzahrani, T. H. Aldhyani, S. N. Alsubari, M. M. Althobaiti, and A. Fahad, "Developing an intelligent system with deep learning algorithms for sentiment analysis of e-commerce product reviews," Computational Intelligence and Neuro-science, vol. 2022, 2022.Google Scholar
- S. Han, J. Pool, J. Tran, and W. Dally, "Learning both weights and connections for efficient neural network," in Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds., vol. 28. Curran Associates, Inc., 2015.Google Scholar
- M. Uzair and N. Jamil, "Effects of hidden layers on the efficiency of neural networks," in 2020 IEEE 23rd international multitopic conference (INMIC). IEEE, 2020, pp. 1--6.Google Scholar
- Y. Y. Huang and W. Y. Wang, "Deep residual learning for weakly-supervised relation extraction," arXiv preprint arXiv:1707.08866, 2017.Google Scholar
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.Google Scholar
- M. H. Zhu and S. Gupta, "To prune, or not to prune: Exploring the efficacy of pruning for model compression," 2018. [Online]. Available: https://openreview.net/forum"id=S1lN69AT-Google Scholar
- P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz, "Importance estimation for neural network pruning," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 11 264--11 272.Google Scholar
- H. Hu, R. Peng, Y.-W. Tai, and C.-K. Tang, "Network trimming: A data-driven neuron pruning approach towards efficient deep architectures," arXiv preprint arXiv:1607.03250, 2016.Google Scholar
- T.-J. Yang, Y.-H. Chen, and V. Sze, "Designing energy-efficient convolutional neural networks using energy-aware pruning," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5687--5695.Google Scholar
- Y. LeCun, J. Denker, and S. Solla, "Optimal brain damage," Advances in neural information processing systems, vol. 2, 1989.Google Scholar
- M. Mondal, B. Das, S. D. Roy, P. Singh, B. Lall, and S. D. Joshi, "Adaptive cnn filter pruning using global importance metric," Computer Vision and Image Understanding, vol. 222, p. 103511, 2022.Google ScholarDigital Library
- S. Anwar, K. Hwang, and W. Sung, "Structured pruning of deep convolutional neural networks," ACM Journal on Emerging Technologies in Computing Systems (JETC), vol. 13, no. 3, pp. 1--18, 2017.Google ScholarDigital Library
- C. Zhao, B. Ni, J. Zhang, Q. Zhao, W. Zhang, and Q. Tian, "Variational convolutional neural network pruning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2780--2789.Google Scholar
- Z. Wang, C. Li, and X. Wang, "Convolutional neural network pruning with structural redundancy reduction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14 913--14 922.Google Scholar
- M. Sundararajan, A. Taly, and Q. Yan, "Axiomatic attribution for deep networks," in International conference on machine learning. PMLR, 2017, pp. 3319--3328.Google Scholar
- A. Patra and J. A. Noble, "Incremental learning of fetal heart anatomies using interpretable saliency maps," in Medical Image Understanding and Analysis: 23rd Conference, MIUA 2019, Liverpool, UK, July 24--26, 2019, Proceedings 23. Springer, 2020, pp. 129--141.Google Scholar
- Z. Wang, M. Fredrikson, and A. Datta, "Robust models are more interpretable because attributions look normal," arXiv preprint arXiv:2103.11257, 2021.Google Scholar
- I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," arXiv preprint arXiv:1412.6572, 2014.Google Scholar
- A. Kurakin, I. J. Goodfellow, and S. Bengio, "Adversarial examples in the physical world," in Artificial intelligence safety and security. Chapman and Hall/CRC, 2018, pp. 99--112.Google Scholar
- Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, "Boosting adversarial attacks with momentum," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9185--9193.Google Scholar
- J. Lin, C. Song, K. He, L. Wang, and J. E. Hopcroft, "Nesterov accelerated gradient and scale invariance for adversarial attacks," arXiv preprint arXiv:1908.06281, 2019.Google Scholar
- A. Krizhevsky, V. Nair, and G. Hinton, "The cifar-10 dataset," online: http://www.cs. toronto.edu/kriz/cifar. html, vol. 55, no. 5, 2014.Google Scholar
- --, "Cifar-10 and cifar-100 datasets," URl: https://www. cs. toronto. edu/kriz/cifar.html, vol. 6, no. 1, p. 1, 2009.Google Scholar
- F.-F. Li, M. Andreeto, M. Ranzato, and P. Perona, "Caltech 101," 4 2022.Google Scholar
- K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770--778.Google Scholar
- H. Wang, C. Qin, Y. Zhang, and Y. Fu, "Neural pruning via growing regularization," arXiv preprint arXiv:2012.09243, 2020.Google Scholar
- G. Retsinas, A. Elafrou, G. Goumas, and P. Maragos, "Weight pruning via adaptive sparsity loss," arXiv preprint arXiv:2006.02768, 2020.Google Scholar
Index Terms
- FVW: Finding Valuable Weight on Deep Neural Network for Model Pruning
Recommendations
Facial Expression Intensity Estimation using Deep Convolutional Neural Network
ACIT '21: Proceedings of the the 8th International Virtual Conference on Applied Computing & Information TechnologyFacial expression recognition has been applied in various fields, but it is not enough to express complex emotions. It is required to detect the intensity of facial expression because the intensity of facial expression varies depending on the intensity ...
Detect and Remove Watermark in Deep Neural Networks via Generative Adversarial Networks
Information SecurityAbstractDeep neural networks (DNN) have achieved remarkable performance in various fields. However, training a DNN model from scratch requires expensive computing resources and a lot of training data, which are difficult to obtain for most individual ...
A hybrid adversarial training for deep learning model and denoising network resistant to adversarial examples
AbstractDeep neural networks (DNNs) are vulnerable to adversarial attacks that generate adversarial examples by adding small perturbations to the clean images. To combat adversarial attacks, the two main defense methods used are denoising and adversarial ...
Comments