Abstract
Convolutional Neural Networks (CNNs) is now widely utilized in computer vision applications, including image classification, object detection and segmentation. However, high memory complexity and computation intensive have limited the deployment on low power embedded devices. We propose a method to train convolutional neural networks with low bitwidth by performing weight normalization. By normalization, the distribution of the weight can be narrowed, which enables the low bitwidth network to achieve a good trade-off between range and precision. Moreover, adding a scaling factor to the weight solves the problem of inadequate expressiveness at low bits, which further improves the performance of classification. The experiments on various datasets show that our method can achieve comparable prediction accuracy as that of full-precision models. To emphasize, the proposed scheme can quantize the network of AlexNet to 3-bit fixed point on ImageNet, and the accuracy of top-1 drop only by 1%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Denil, M., Shakibi, B., Dinh, L., de Freitas, N., et al.: Predicting parameters in deep learning. In: Advances in Neural Information Processing Systems, pp. 2148–2156 (2013)
Tai, C., Xiao, T., Zhang, Y., Wang, X., et al.: Convolutional neural networks with low-rank regularization. In: Proceedings of International Conference on Learning, Representations (2016)
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)
Molchanov, D., Ashukha, A., Vetrov, D.: Variational dropout sparsifies deep neural networks. In: International Conference on Machine Learning, pp. 2498–2507 (2017)
Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems, pp. 3123–3131 (2015)
Courbariaux, M., Bengio, Y.: Binarynet: training deep neural networks with weights and activations constrained to +1 or –1. CoRR (2016)
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: Xnor-net: Imagenet classification using binary convolutional neural networks. arXiv preprint arXiv:1603.05279 (2016)
Buciluǎ, C., Caruana, R., Niculescu-Mizil, A.: Model compression. In: Proceedings of the 12th ACM/SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2006, pp. 535–541. New York, NY, USA (2006)
Sau, B.B., Balasubramanian, V.N.: Deep Model Compression: Distilling Knowledge from Noisy Teachers (2016)
Hu, Q., Wang, P., Cheng, J.: From Hashing to CNNs: Training Binary Weight Networks via Hashing (2018)
Li, F., Liu, B.: Ternary weight networks. arXiv preprint arXiv:1605.04711 (2016)
Zhu, C., Han, S., Mao, H., Dally, W.J.: Trained ternary quantization. In: Proceedings of International Conference on Learning Representations (2017)
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., Zou, Y.: Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 (2016)
Gysel, P.: Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks (2016)
Zhuang, B., Shen, C., Tan, M., Liu, L., Reid, I.: Towards Effective Low-bitwidth Convolutional Neural Networks. arXiv preprint arXiv:1711.00205 (2017)
Acknowledgment
This work was supported by NNSF of China Grants No. 61574013, 61532005.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Fan, H., An, J., Wang, D. (2019). Training Low Bitwidth Model with Weight Normalization for Convolutional Neural Networks. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2019. Lecture Notes in Computer Science(), vol 11857. Springer, Cham. https://doi.org/10.1007/978-3-030-31654-9_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-31654-9_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-31653-2
Online ISBN: 978-3-030-31654-9
eBook Packages: Computer ScienceComputer Science (R0)