research-article

Softmax Regression Design for Stochastic Computing Based Deep Convolutional Neural Networks

Authors:

Jeffrey Draper,

Yanzhi WangAuthors Info & Claims

GLSVLSI '17: Proceedings of the Great Lakes Symposium on VLSI 2017

Pages 467 - 470

https://doi.org/10.1145/3060403.3060467

Published: 10 May 2017 Publication History

Abstract

Recently, Deep Convolutional Neural Networks (DCNNs) have made tremendous advances, achieving close to or even better accuracy than human-level perception in various tasks. Stochastic Computing (SC), as an alternate to the conventional binary computing paradigm, has the potential to enable massively parallel and highly scalable hardware implementations of DCNNs. In this paper, we design and optimize the SC based Softmax Regression function. Experiment results show that compared with a binary SR, the proposed SC-SR under longer bit stream can reach the same level of accuracy with the improvement of 295X, 62X, 2617X in terms of power, area and energy, respectively. Binary SR is suggested for future DCNNs with short bit stream length input whereas SC-SR is recommended for longer bit stream.

References

[1]

Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," phNature, vol. 521, no. 7553, pp. 436--444, 2015.

[2]

K. Kim, J. Kim, J. Yu, J. Seo, J. Lee, and K. Choi, "Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks," in Proceedings of the 53rd Annual Design Automation Conference. ACM, 2016, p. 124.

Digital Library

[3]

J. Li, A. Ren, Z. Li, C. Ding, B. Yuan, Q. Qiu, and Y. Wang, "Towards acceleration of deep convolutional neural networks using stochastic computing," in Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2017.

[4]

Y. Xue, J. Li, S. Nazarian, and P. Bogdan, "Fundamental challenges towards making iot a reachable reality: A model-centric investigation," ACM Transactions on Design Automation of Electronic Systems (TODAES), 2017.

Digital Library

[5]

J. Li and J. Draper, "Accelerating soft-error-rate (ser) estimation in the presence of single event transients," in Proceedings of the 53rd Annual Design Automation Conference. ACM, 2016, p. 55.

Digital Library

[6]

B. D. Brown and H. C. Card, "Stochastic neural computation. i. computational elements," IEEE Transactions on computers, vol. 50, no. 9, pp. 891--905, 2001.

Digital Library

[7]

J. Li and J. Draper, "Joint soft-error-rate (ser) estimation for combinational logic and sequential elements," in VLSI (ISVLSI), 2016 IEEE Computer Society Annual Symposium on, 2016, pp. 737--742.

[8]

Z. Li, A. Ren, J. Li, Q. Qiu, Y. Wang, and B. Yuan, "Dscnn: Hardware-oriented optimization for stochastic computing based deep convolutional neural networks," in Proceeding of IEEE International Conference on Computer Design, 2016.

[9]

A. Ren, J. Li, Z. Li, C. Ding, X. Qian, Q. Qiu, B. Yuan, and Y. Wang, "Sc-dcnn: Highly-scalable deep convolutional neural network using stochastic computing," arXiv preprint arXiv:1611.05939, 2016.

Digital Library

[10]

Z. Li, A. Ren, J. Li, Q. Qiu, B. Yuan, J. Draper, and Y. Wang, "Structural design optimization for deep convolutional neural networks using stochastic computing."

[11]

J. Li, Z. Yuan, Z. Li, C. Ding, A. Ren, Q. Qiu, J. Draper, and Y. Wang, "Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks," in Neural Networks (IJCNN), 2017 International Joint Conference on. IEEE, 2017.

[12]

K., J. Lee, and K. Choi, "Approximate de-randomizer for stochastic circuits," phProc. ISOCC, 2015.

[13]

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278--2324, 1998.

[14]

Nangate 45nm Open Library, Nangate Inc., 2009. {Online}. Available: http://www.nangate.com/

[15]

A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in neural information processing systems, 2012.

Digital Library

[16]

L. Deng, "The mnist database of handwritten digit images for machine learning research," IEEE Signal Processing Magazine, vol. 29, no. 6.

Cited By

Xie THu YWei RLi MWang YWang RHuang R(2024)ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546691(1-6)Online publication date: 25-Mar-2024
https://doi.org/10.23919/DATE58400.2024.10546691
Zhu XLi NWang Y(2022)Software change‐proneness prediction based on deep learningJournal of Software: Evolution and Process10.1002/smr.243434:4Online publication date: 5-Apr-2022
https://dl.acm.org/doi/10.1002/smr.2434
Uma JVivekanandan PShankar S(2022)Optimized intellectual resource scheduling using deep reinforcement Q‐learning in cloud computingTransactions on Emerging Telecommunications Technologies10.1002/ett.446333:5Online publication date: 27-May-2022
https://dl.acm.org/doi/10.1002/ett.4463
Show More Cited By

Index Terms

Softmax Regression Design for Stochastic Computing Based Deep Convolutional Neural Networks

Recommendations

Normalization and dropout for stochastic computing-based deep convolutional neural networks
Abstract
Recently, Deep Convolutional Neural Network (DCNN) has been recognized as the most effective model for pattern recognition and classification tasks. With the fast growing Internet of Things (IoTs) and wearable devices, it becomes attractive to ...
Highlights
- A novel and efficient stochastic computing based normalization design is proposed, which is comprised of summation and square, activation, and division units.
- Dropout technique is integrated to the stochastic computing based deep ...
Convergence of deep convolutional neural networks
Abstract
Convergence of deep neural networks as the depth of the networks tends to infinity is fundamental in building the mathematical foundation for deep learning. In a previous study, we investigated this question for deep networks with the Rectified ...
Towards dropout training for convolutional neural networks

Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in convolutional and pooling layers is still not clear. This paper ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GLSVLSI '17: Proceedings of the Great Lakes Symposium on VLSI 2017

May 2017

516 pages

ISBN:9781450349727

DOI:10.1145/3060403

General Chairs:
Laleh Behjat
University of Calgary, Canada
,
Jie Han
University of Alberta, Canada
,
Program Chairs:
Miroslav N. Velev
Aries Design Automation, USA
,
Deming Chen
University of Illinois, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGDA: ACM Special Interest Group on Design Automation
IEEE CEDA
IEEE CASS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GLSVLSI '17

Sponsor:

SIGDA

GLSVLSI '17: Great Lakes Symposium on VLSI 2017

May 10 - 12, 2017

Alberta, Banff, Canada

Acceptance Rates

GLSVLSI '17 Paper Acceptance Rate 48 of 197 submissions, 24%;

Overall Acceptance Rate 312 of 1,156 submissions, 27%

Upcoming Conference

GLSVLSI '25

Sponsor:
sigda

Great Lakes Symposium on VLSI 2025

June 30 - July 2, 2025

New Orleans , LA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
319
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)2

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xie THu YWei RLi MWang YWang RHuang R(2024)ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546691(1-6)Online publication date: 25-Mar-2024
https://doi.org/10.23919/DATE58400.2024.10546691
Zhu XLi NWang Y(2022)Software change‐proneness prediction based on deep learningJournal of Software: Evolution and Process10.1002/smr.243434:4Online publication date: 5-Apr-2022
https://dl.acm.org/doi/10.1002/smr.2434
Uma JVivekanandan PShankar S(2022)Optimized intellectual resource scheduling using deep reinforcement Q‐learning in cloud computingTransactions on Emerging Telecommunications Technologies10.1002/ett.446333:5Online publication date: 27-May-2022
https://dl.acm.org/doi/10.1002/ett.4463
Ghasemshirazi SShirvani GRaisi S(2021)MC-GAN: A Reimplementation of GameGAN With a Gaming Perspective2021 7th International Conference on Web Research (ICWR)10.1109/ICWR51868.2021.9443138(323-328)Online publication date: 19-May-2021
https://doi.org/10.1109/ICWR51868.2021.9443138
Gao YLiu WLombardi F(2020)Design and Implementation of an Approximate Softmax Layer for Deep Neural Networks2020 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS45731.2020.9180870(1-5)Online publication date: Oct-2020
https://doi.org/10.1109/ISCAS45731.2020.9180870
Arora BChaudhary DSatsangi MYadav MSingh LSudhish P(2020)Agribot: A Natural Language Generative Neural Networks Engine for Agricultural Applications2020 International Conference on Contemporary Computing and Applications (IC3A)10.1109/IC3A48958.2020.233263(28-33)Online publication date: Feb-2020
https://doi.org/10.1109/IC3A48958.2020.233263
Zhu RQiu TWang JSui SLi YFeng MMa HQu S(2020)Multiplexing the aperture of a metasurface: inverse design via deep-learning-forward genetic algorithmJournal of Physics D: Applied Physics10.1088/1361-6463/aba64f53:45(455002)Online publication date: 11-Aug-2020
https://doi.org/10.1088/1361-6463/aba64f
Wang YYu P(2019)Offline Handwritten New Tai Lue Characters Recognition Using CNN-SVM2019 IEEE 2nd International Conference on Electronic Information and Communication Technology (ICEICT)10.1109/ICEICT.2019.8846292(636-639)Online publication date: Jan-2019
https://doi.org/10.1109/ICEICT.2019.8846292
Li JYuan ZLi ZRen ADing CDraper JNazarian SQiu QYuan BWang Y(2019)Normalization and dropout for stochastic computing-based deep convolutional neural networksIntegration, the VLSI Journal10.1016/j.vlsi.2017.11.00265:C(395-403)Online publication date: 1-Mar-2019
https://dl.acm.org/doi/10.1016/j.vlsi.2017.11.002
Wang YZhang HLi YXie H(2019)Simplified Representation Learning Model Based on Parameter-Sharing for Knowledge Graph CompletionInformation Retrieval10.1007/978-3-030-31624-2_6(67-78)Online publication date: 20-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-31624-2_6
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten