skip to main content
10.1145/3060403.3060467acmconferencesArticle/Chapter ViewAbstractPublication PagesglsvlsiConference Proceedingsconference-collections
research-article

Softmax Regression Design for Stochastic Computing Based Deep Convolutional Neural Networks

Published: 10 May 2017 Publication History

Abstract

Recently, Deep Convolutional Neural Networks (DCNNs) have made tremendous advances, achieving close to or even better accuracy than human-level perception in various tasks. Stochastic Computing (SC), as an alternate to the conventional binary computing paradigm, has the potential to enable massively parallel and highly scalable hardware implementations of DCNNs. In this paper, we design and optimize the SC based Softmax Regression function. Experiment results show that compared with a binary SR, the proposed SC-SR under longer bit stream can reach the same level of accuracy with the improvement of 295X, 62X, 2617X in terms of power, area and energy, respectively. Binary SR is suggested for future DCNNs with short bit stream length input whereas SC-SR is recommended for longer bit stream.

References

[1]
Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," phNature, vol. 521, no. 7553, pp. 436--444, 2015.
[2]
K. Kim, J. Kim, J. Yu, J. Seo, J. Lee, and K. Choi, "Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks," in Proceedings of the 53rd Annual Design Automation Conference. ACM, 2016, p. 124.
[3]
J. Li, A. Ren, Z. Li, C. Ding, B. Yuan, Q. Qiu, and Y. Wang, "Towards acceleration of deep convolutional neural networks using stochastic computing," in Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2017.
[4]
Y. Xue, J. Li, S. Nazarian, and P. Bogdan, "Fundamental challenges towards making iot a reachable reality: A model-centric investigation," ACM Transactions on Design Automation of Electronic Systems (TODAES), 2017.
[5]
J. Li and J. Draper, "Accelerating soft-error-rate (ser) estimation in the presence of single event transients," in Proceedings of the 53rd Annual Design Automation Conference. ACM, 2016, p. 55.
[6]
B. D. Brown and H. C. Card, "Stochastic neural computation. i. computational elements," IEEE Transactions on computers, vol. 50, no. 9, pp. 891--905, 2001.
[7]
J. Li and J. Draper, "Joint soft-error-rate (ser) estimation for combinational logic and sequential elements," in VLSI (ISVLSI), 2016 IEEE Computer Society Annual Symposium on, 2016, pp. 737--742.
[8]
Z. Li, A. Ren, J. Li, Q. Qiu, Y. Wang, and B. Yuan, "Dscnn: Hardware-oriented optimization for stochastic computing based deep convolutional neural networks," in Proceeding of IEEE International Conference on Computer Design, 2016.
[9]
A. Ren, J. Li, Z. Li, C. Ding, X. Qian, Q. Qiu, B. Yuan, and Y. Wang, "Sc-dcnn: Highly-scalable deep convolutional neural network using stochastic computing," arXiv preprint arXiv:1611.05939, 2016.
[10]
Z. Li, A. Ren, J. Li, Q. Qiu, B. Yuan, J. Draper, and Y. Wang, "Structural design optimization for deep convolutional neural networks using stochastic computing."
[11]
J. Li, Z. Yuan, Z. Li, C. Ding, A. Ren, Q. Qiu, J. Draper, and Y. Wang, "Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks," in Neural Networks (IJCNN), 2017 International Joint Conference on. IEEE, 2017.
[12]
K., J. Lee, and K. Choi, "Approximate de-randomizer for stochastic circuits," phProc. ISOCC, 2015.
[13]
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278--2324, 1998.
[14]
Nangate 45nm Open Library, Nangate Inc., 2009. {Online}. Available: http://www.nangate.com/
[15]
A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in neural information processing systems, 2012.
[16]
L. Deng, "The mnist database of handwritten digit images for machine learning research," IEEE Signal Processing Magazine, vol. 29, no. 6.

Cited By

View all
  • (2024)ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546691(1-6)Online publication date: 25-Mar-2024
  • (2022)Software change‐proneness prediction based on deep learningJournal of Software: Evolution and Process10.1002/smr.243434:4Online publication date: 5-Apr-2022
  • (2022)Optimized intellectual resource scheduling using deep reinforcement Q‐learning in cloud computingTransactions on Emerging Telecommunications Technologies10.1002/ett.446333:5Online publication date: 27-May-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GLSVLSI '17: Proceedings of the Great Lakes Symposium on VLSI 2017
May 2017
516 pages
ISBN:9781450349727
DOI:10.1145/3060403
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 May 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. deep convolutional neural networks
  2. deep learning
  3. softmax regression
  4. stochastic computing

Qualifiers

  • Research-article

Conference

GLSVLSI '17
Sponsor:
GLSVLSI '17: Great Lakes Symposium on VLSI 2017
May 10 - 12, 2017
Alberta, Banff, Canada

Acceptance Rates

GLSVLSI '17 Paper Acceptance Rate 48 of 197 submissions, 24%;
Overall Acceptance Rate 312 of 1,156 submissions, 27%

Upcoming Conference

GLSVLSI '25
Great Lakes Symposium on VLSI 2025
June 30 - July 2, 2025
New Orleans , LA , USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)24
  • Downloads (Last 6 weeks)2
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546691(1-6)Online publication date: 25-Mar-2024
  • (2022)Software change‐proneness prediction based on deep learningJournal of Software: Evolution and Process10.1002/smr.243434:4Online publication date: 5-Apr-2022
  • (2022)Optimized intellectual resource scheduling using deep reinforcement Q‐learning in cloud computingTransactions on Emerging Telecommunications Technologies10.1002/ett.446333:5Online publication date: 27-May-2022
  • (2021)MC-GAN: A Reimplementation of GameGAN With a Gaming Perspective2021 7th International Conference on Web Research (ICWR)10.1109/ICWR51868.2021.9443138(323-328)Online publication date: 19-May-2021
  • (2020)Design and Implementation of an Approximate Softmax Layer for Deep Neural Networks2020 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS45731.2020.9180870(1-5)Online publication date: Oct-2020
  • (2020)Agribot: A Natural Language Generative Neural Networks Engine for Agricultural Applications2020 International Conference on Contemporary Computing and Applications (IC3A)10.1109/IC3A48958.2020.233263(28-33)Online publication date: Feb-2020
  • (2020)Multiplexing the aperture of a metasurface: inverse design via deep-learning-forward genetic algorithmJournal of Physics D: Applied Physics10.1088/1361-6463/aba64f53:45(455002)Online publication date: 11-Aug-2020
  • (2019)Offline Handwritten New Tai Lue Characters Recognition Using CNN-SVM2019 IEEE 2nd International Conference on Electronic Information and Communication Technology (ICEICT)10.1109/ICEICT.2019.8846292(636-639)Online publication date: Jan-2019
  • (2019)Normalization and dropout for stochastic computing-based deep convolutional neural networksIntegration, the VLSI Journal10.1016/j.vlsi.2017.11.00265:C(395-403)Online publication date: 1-Mar-2019
  • (2019)Simplified Representation Learning Model Based on Parameter-Sharing for Knowledge Graph CompletionInformation Retrieval10.1007/978-3-030-31624-2_6(67-78)Online publication date: 20-Sep-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media