Chongyang Liu
ABSTRACT
The large number of parameters in convolutional neural network (CNN) makes it a computationally intensive and storage-intensive network model. Although the effect of CNN is prominent in various identification and classification tasks, it is difficult to deploy on embedded devices because the model is too large. In order to solve this problem, an improved scheme for pruning operations in compression methods is proposed. First, the distribution of network connection is analyzed so as to determine the pruning threshold initially; then, using the pruning method to delete connections whose weights are less than the threshold, make the network quickly reach the limit of pruning but maintain accuracy. The verification experiment was performed on the Lenet-5 network which trained on the MINST data set and Lenet-5 was compressed 10.56 times without loss of accuracy.
- A. Krizhevsky, I. Sutskever, and G. E Hinton,"Imagenet classification with deepconvolutional neural networks". In Advances in neural information processing systems, pp.1097--1105, 2012. Google Scholar
Digital Library
- D. Amodei, R. Anubhai, E. Battenberg, et al.,"Deep speech 2: End-to-end speech recognition in english and mandarin". arXiv preprint arXiv:1512.02595, 2015.Google Scholar
- M. Luong, H. Pham, and C. D Manning. "Effective approaches to attention-based neural machine translation". arXiv preprint arXiv:1508.04025, 2015.Google Scholar
- M. Horowitz. "Energy table for 45nm process", Stanford VLSI wiki.Google Scholar
- S. Han, H. Mao, W. J. Dally, "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding", ICLR'16.Google Scholar
- F. Iandola, S. Han, M. Moskewicz, et al. "SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size". 2016.Google Scholar
- M. Rastegari, V. Ordonez, J. Redmon, et al. "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks". European Conference on Computer Vision, pp. 525--542, 2016.Google ScholarCross Ref
- A. G Howard, M. Zhu, B. Chen, et al. "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications". arXiv preprint arXiv:1704.0486, 2017.Google Scholar
- Z. Xiang, Z. Xin, L. Meng, et al. "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices". arXiv preprint arXiv:1707.01083, 2017.Google Scholar
- S. Han, J. Pool, J.Tran, and W. Dally. "Learning both weights and connections for efficient neural network". In Advances in Neural Information Processing Systems, pp.1135--1143, 2015. Google ScholarDigital Library
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based learningapplied to document recognition". Proceedings of the IEEE, vol.86, pp.2278--2324, 1998.Google ScholarCross Ref
- https://stacks.stanford.edu/file/druid:qf934gh3708/EFFICIENT%20METHODS%20AND%20HARDWARE%20FOR%20DEEP%20LEARNING-augmented.pdfGoogle Scholar
- K. Simonyan and A. Zisserman. "Very deep convolutional networks for large-scale image recognition". arXiv preprint arXiv:1409.1556, 2014.Google Scholar
- C. Szegedy, W. Liu, Y. Jia, et al. "Going deeper with convolutions".In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1--9, 2015.Google ScholarCross Ref
- K. He, X. Zhang, S. Ren, and J. Sun. "Deep residual learning for imagerecognition". arXiv preprint arXiv:1512.03385, 2015.Google Scholar
- Y. Han, T. Jiang T, Y. Ma, et al."Compression of deep neural networks", Application Research of Computers, vol. 35, pp. 1--7, 2017.Google Scholar
- Y. Jia, et al. "Caffe: Convolutional architecture for fast feature embedding". arXiv preprint arXiv:1408.5093, 2014.Google Scholar
Index Terms
- Improvement of Pruning Method for Convolution Neural Network Compression
Recommendations
Pruning algorithm of convolutional neural network based on optimal threshold
ICMAI '20: Proceedings of the 2020 5th International Conference on Mathematics and Artificial IntelligenceIn the process of pruning, in order to automatically obtain an optimal pruning threshold that can balance the maximum sparse rate and the minimum error. This paper proposes a convolutional neural network pruning algorithm based on the optimal threshold. ...
Deep neural network compression through interpretability-based filter pruning
Highlights- Filters are visualized by the activation maximization to explain functions of filters.
AbstractThis paper proposes a method to compress deep neural networks (DNNs) based on interpretability. For a trained DNN model, the activation maximization technique is first used to visualize every filter of the DNN model. Then, a single-...
Entropy-based pruning method for convolutional neural networks
Various compression approaches including pruning techniques have been developed to lighten the computational complexity of neural networks. Most pruning techniques determine the threshold of pruning weights or input features based on statistical ...
Comments