Hardware-efficient stochastic rounding unit design for DNN training: late breaking results
Authors: 
Sung-En Chang, Geng Yuan, Alec Lu, Mengshu Sun, Yanyu Li, Xiaolong Ma, Zhengang Li, Yanyue Xie, Minghai Qin, Xue Lin, 
Zhenman Fang, Yanzhi WangAuthors Info & Claims
Sung-En Chang, Geng Yuan, Alec Lu, Mengshu Sun, Yanyu Li, Xiaolong Ma, Zhengang Li, Yanyue Xie, Minghai Qin, Xue Lin, Zhenman Fang, Yanzhi WangAuthors Info & ClaimsPages 1396 - 1397
Abstract
Stochastic rounding is crucial in the training of low-bit deep neural networks (DNNs) to achieve high accuracy. Unfortunately, prior studies require a large number of high-precision stochastic rounding units (SRUs) to guarantee the low-bit DNN accuracy, which involves considerable hardware overhead. In this paper, we propose an automated framework to explore hardware-efficient low-bit SRUs (ESRUs) that can still generate high-quality random numbers to guarantee the accuracy of low-bit DNN training. Experimental results using state-of-the-art DNN models demonstrate that, compared to the prior 24-bit SRU with 24-bit pseudo random number generator (PRNG), our 8-bit with 3-bit PRNG reduces the SRU resource usage by 9.75× while achieving a higher accuracy.
References
[1]
Courbariaux et al. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. In (NeurIPS).
[2]
Cheng Luo et al. 2020. Towards efficient deep neural network training by FPGA-based batch-level parallelism. Journal of Semiconductors (2020).
[3]
Fengfu Li et al. 2016. Ternary weight networks. arXiv preprint arXiv:1605.04711 (2016).
[4]
Roth Jr et al. 2016. Digital systems design using VHDL. Cengage Learning.
[5]
Sung-En Chang et al. 2021. Mix and match: A novel fpga-centric deep neural network quantization framework. In 2021 HPCA. IEEE.
[6]
Sung-En Chang et al. 2021. RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions. In Proceedings of ICCV. 5251--5260.
[7]
Naigang et al. Wang. 2018. Training deep neural networks with 8-bit floating point numbers. In Proceedings of the NIPS. 7686--7695.
[8]
Yukuan et al. Yang. 2020. Training high-performance and large-scale deep neural networks with full 8-bit integers. Neural Networks (2020).
[9]
Kang et al. Zhao. 2021. Distribution Adaptive INT8 Quantization for Training CNNs. In Proceedings of the AAAI.
[10]
Shuchang et al. Zhou. 2016. Dorefa-net: Training low bitwidth Convolutional Neural Networks with low bitwidth gradients. arXiv:1606.06160 (2016).
[11]
Feng et al. Zhu. 2020. Towards unified int8 training for convolutional neural network. In Proceedings of CVPR. 1969--1979.
Recommendations
You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding
Computer Vision – ECCV 2022AbstractStochastic rounding is a critical technique used in low-precision deep neural networks (DNNs) training to ensure good model accuracy. However, it requires a large number of random numbers generated on the fly. This is not a trivial task on the ...
Decimal Floating-Point Adder and Multifunction Unit with Injection-Based Rounding
ARITH '07: Proceedings of the 18th IEEE Symposium on Computer ArithmeticShrinking feature sizes gives more headroom for designers to extend the functionality of microprocessors. The IEEE 754R working group has revised the IEEE 754-1985 Standard for Binary Floating-Point Arithmetic to include specifications for decimal ...
Hardware Designs for Binary Integer Decimal-Based Rounding
Decimal floating-point (DFP) arithmetic is becoming increasingly important and specifications for it are included in the revised IEEE 754 standard for floating-point arithmetic (IEEE 754-2008). The binary encoding of DFP numbers specified in IEEE 754-...
Comments
Information & Contributors
Information
Published In
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 23 August 2022
Check for updates
Qualifiers
- Research-article
Conference
DAC '22
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,770 of 5,499 submissions, 32%
Upcoming Conference
DAC '25
- Sponsor:
- sigda
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 243Total Downloads
- Downloads (Last 12 months)44
- Downloads (Last 6 weeks)4
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in