Abstract
The deployment of robust neural network based models on low-cost devices touches the problem with hardware constraints like limited memory footprint and computing power. This work presents a general method for a rapid reduction of parameters (80–90%) in a trained (DNN or LSTM) network by removing its redundant synapses, while the classification accuracy is not significantly hurt. The massive reduction of parameters leads to a notable decrease of the model’s size and the actual prediction time of on-board classifiers. We show the pruning results on a simple speech recognition task, however, the method is applicable to any classification data.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Depending on the implementation rows/columns might correspond to layer inputs/outputs or outputs/inputs.
References
Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., Feng, J.: Dual path networks. arXiv preprint arXiv:1707.01629 (2017)
Xiong, W., Wu, L., Alleva, F., Droppo, J., Huang, X., Stolcke, A.: The Microsoft 2017 conversational speech recognition system. CoRR,abs/1708.06073 (2017)
Chen, G., Parada, C., Sainath, T.N.: Query-by-example keyword spotting using long short-term memory networks. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2015). ISBN 978-1-4673-6997-8
Zhang, Y., Suda, N., Lai, L., Chandra, V.: Hello edge: keyword spotting on microcontrollers. arXiv arXiv:1711.07128v3 (2018)
Warden, P.: speech commands: a public dataset for single-word speech recognition (2017). http://download.tensorflow.org/data/speech_commands_v0.01.tar.gz
Reed, R.: Pruning algorithms - a survey. IEEE Trans. Neural Netw. 4, 740–747 (1993)
Bulín, M.: Optimization of neural network. Master thesis. University of West Bohemia. Univerzitní 8, 30100 Pilsen, Czech Republic (2017)
Mozer, M., Smolensky, P.: Skeletonization: a technique for trimming the fat from a network via relevance assessment. University of Colorado, Boulder, Department of Computer Science (1989)
LeCun, Y., Denker J.S., Solla, S.: Optimal brain damage. In: Advances in Neural Information Processing Systems, pp. 598–605 (1990)
Karnin, E.D.: A simple procedure for pruning back-propagation trained neural networks. IEEE Trans. Neural Netw. 1, 239–242 (1990)
Kaggle Inc.: TensorFlow speech recognition challenge (2017). https://www.kaggle.com/c/tensorflow-speech-recognition-challenge
Chollet, F., et al.: Keras (2015). https://keras.io
Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). tensorflow.org
Acknowledgments
This research was supported by the Ministry of Education, Youth and Sports of the Czech Republic project No. LO1506.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Bulín, M., Šmídl, L., Švec, J. (2018). Towards Network Simplification for Low-Cost Devices by Removing Synapses. In: Karpov, A., Jokisch, O., Potapova, R. (eds) Speech and Computer. SPECOM 2018. Lecture Notes in Computer Science(), vol 11096. Springer, Cham. https://doi.org/10.1007/978-3-319-99579-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-99579-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99578-6
Online ISBN: 978-3-319-99579-3
eBook Packages: Computer ScienceComputer Science (R0)