Abstract
Deep neural network is a state-of-the-art technology for achieving high accuracy in various machine learning tasks. Since the available computing power and memory footprint are restricted in embedded computing, precision quantization of numerical representations, such as fixed-point, binary, and logarithmic, are commonly used for higher computing efficiency. The main problem of quantization is accuracy degradation due to its lower numerical representation. There is generally a trade-off between numerical precision and accuracy. In this paper, we propose a quantization-error-aware training method to attain higher accuracy in quantized neural networks. Our approach appends an additional regularization term that is based on quantization errors of weights to the loss function. We evaluate the accuracy by using MNIST and CIFAR-10. The evaluation results show that the proposed approach achieves higher accuracy than the standard approach with quantized forwarding.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ando, K., Orimo, K., Ueyoshi, K., Yonekawa, H., Sato, S., Nakahara, H., Ikebe, M., Asai, T., Takamaeda-Yamazaki, S., Kuroda, T., Motomura, M.: BRein memory: a 13-layer 4.2 K neuron/0.8 M synapse binary/ternary reconfigurable in-memory deep neural network accelerator in 65 nm CMOS. In: 2017 IEEE Symposium on VLSI Circuits (VLSI-Circuits), pp. C24–C25, Kyoto, Japan (2017)
Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or \(-1\). ArXiv e-prints arXiv:1602.02830, February 2016
Gysel, P., Motamedi, M., Ghiasi, S.: Hardware-oriented approximation of convolutional neural networks. CoRR abs/1604.03168 (2016). http://arxiv.org/abs/1604.03168
Hou, L., Yao, Q., Kwok, J.T.: Loss-aware binarization of deep networks. CoRR abs/1611.01600 (2016). http://arxiv.org/abs/1611.01600
Janocha, K., Czarnecki, W.M.: On loss functions for deep neural networks in classification. CoRR abs/1702.05659 (2017). http://arxiv.org/abs/1702.05659
Johnson, M., Schuster, M., Le, Q.V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., Hughes, M., Dean, J.: Google’s multilingual neural machine translation system: enabling zero-shot translation. ArXiv e-prints, Nov 2016
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014). http://arxiv.org/abs/1412.6980
Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (Canadian institute for advanced research). http://www.cs.toronto.edu/ kriz/cifar.html
LeCun, Y., Bengio, Y., Hinton, G.: Artificial intelligence: deep neural reasoning. Nature 538, 467–468 (2016)
LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010). http://yann.lecun.com/exdb/mnist/
Lin, D.D., Talathi, S.S., Annapureddy, V.S.: Fixed point quantization of deep convolutional networks. CoRR abs/1511.06393 (2015). http://arxiv.org/abs/1511.06393
Miyashita, D., Lee, E.H., Murmann, B.: Convolutional neural networks using logarithmic data representation. CoRR abs/1603.01025 (2016). http://arxiv.org/abs/1603.01025
Shin, D., Lee, J., Lee, J., Yoo, H.J.: 14.2 DNPU: an 8.1TOPS/W reconfigurable CNN-RNN processor for general-purpose deep neural networks. In: 2017 IEEE International Solid-State Circuits Conference (ISSCC), pp. 240–241, February 2017
Acknowledgment
This work is supported in part by JST ACCEL and Technova.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Hirose, K. et al. (2017). Quantization Error-Based Regularization in Neural Networks. In: Bramer, M., Petridis, M. (eds) Artificial Intelligence XXXIV. SGAI 2017. Lecture Notes in Computer Science(), vol 10630. Springer, Cham. https://doi.org/10.1007/978-3-319-71078-5_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-71078-5_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-71077-8
Online ISBN: 978-3-319-71078-5
eBook Packages: Computer ScienceComputer Science (R0)