Skip to main content
SpringerLink
Log in

A Weight Importance Analysis Technique for Area- and Power-Efficient Binary Weight Neural Network Processor Design

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Recently, the binary weight neural network (BWNN) processor design has attracted lots of attention due to its low computational complexity and memory demands. For the design of BWNN processor, emerging memory technologies such as RRAM can be used to replace conventional SRAM to save area and accessing power. However, RRAM is prone to bit errors, leading to reduced classification accuracy. To combine BWNN and RRAM to reduce the area overhead and power consumption while maintaining a high classification accuracy is a significant research challenge. In this work, we propose an automatic weight importance analysis technique and a mixed weight storage scheme to address the above-mentioned issue. For demonstration, we applied the proposed techniques to two typical BWNNs. The experimental results show that more than 78% (40%) area saving and 57% (30%) power saving can be achieved with less than 1% accuracy loss. The proposed techniques are applicable in resource- and power-constrained neural network processor design and show significant potentials for AI-based Internet-of-Things (IoT) devices that usually have low computational and storage resources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Tian Y, Luo P, Wang X, Tang X. Pedestrian detection aided by deep learning semantic tasks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015. pp. 5079–87.

  2. Dominguez-Sanchez A, Cazorla M, Orts-Escolano S. Pedestrian movement direction recognition using convolutional neural networks. IEEE Trans Intell Transp Syst. 2017;18:3540–8.

    Article  Google Scholar 

  3. Jiang W, Wang W. Face detection and recognition for home service robots with end-to-end deep neural networks. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017. pp. 2232–6.

  4. Liu X, Kawanishi T, Wu X, Kashino K. Scene text recognition with high performance CNN classifier and efficient word inference. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2016. pp. 1322–6.

  5. Han S, Mao H, Dally WJ. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. International Conference on Learning Representations (ICLR). 2016.

  6. Hashemi S, Anthony N, Tann H, Bahar RI, Reda S. Understanding the impact of precision quantization on the accuracy and energy of neural networks. Design, Automation Test in Europe Conference Exhibition (DATE), 2017. 2017. pp. 1474–9.

  7. Krizhevsky A, Sutskever I, Hinton GE. ImageNet Classification with Deep Convolutional Neural Networks. 25th International Conference on Neural Information Processing Systems. 2012. pp. 1097–1105.

  8. Hong S, Lee I, Park Y. Optimizing a FPGA-based neural accelerator for small IoT devices. International Conference on Electronics, Information, and Communication (ICEIC). 2018. pp. 1–2.

  9. Hong S, Park Y. A FPGA-based neural accelerator for small IoT devices. 2017 International SoC Design Conference (ISOCC). 2017. pp. 294–5.

  10. Yushuang Y, Qingqi P. A robust deep-neural-network-based compressed model for mobile device assisted by edge server. IEEE Access. 2019;7:179104–17.

    Article  Google Scholar 

  11. Kailun W, Yiwen G, Changshui Z. Compressing deep neural networks with sparse matrix factorization. IEEE Transactions on Neural Networks and Learning Systems. 2019.

  12. Courbariaux M, Bengio Y, David J-P. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. 28th International Conference on Neural Information Processing Systems. 2015. pp. 3123–3131.

  13. Courbariaux M, Hubara I, Soudry D, El-Yaniv R, Bengio Y. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. 2016.

  14. Deng J, Dong W, Socher R, Li L-J, Kai Li, Li Fei-Fei. ImageNet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition. 2009. pp. 248–55.

  15. Rastegari M, Ordonez V, Redmon J, Farhadi A. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. Computer Vision – ECCV 2016. 2016. pp. 525–42.

  16. Park S, Sheri A, Kim J, Noh J, Jang J, Jeon M, et al. Neuromorphic speech systems using advanced ReRAM-based synapse. IEEE International Electron Devices Meeting. 2013. pp. 25.6.1–25.6.4.

  17. Eryilmaz SB, Kuzum D, Yu S, Wong H-SP. Device and system level design considerations for analog-non-volatile-memory based neuromorphic architectures. IEEE International Electron Devices Meeting (IEDM). 2015. pp. 4.1.1–4.1.4.

  18. Chen P-Y, Lin B, Wang I-T, Hou T-H, Ye J, Vrudhula S, et al. Mitigating effects of non-ideal synaptic device characteristics for on-chip learning. IEEE/ACM International Conference on Computer-Aided Design (ICCAD). 2015. pp. 194–9.

  19. Yu S, Chen P-Y, Cao Y, Xia L, Wang Y, Wu H. Scaling-up resistive synaptic arrays for neuro-inspired architecture: Challenges and prospect. IEEE International Electron Devices Meeting (IEDM). 2015. pp. 17.3.1–17.3.4.

  20. Yu S. Resistive Random Access Memory (RRAM). Morgan & Claypool. 2016.

  21. Chen P-Y, Yu S. Partition SRAM and RRAM based synaptic arrays for neuro-inspired computing. IEEE International Symposium on Circuits and Systems (ISCAS). 2016. pp. 2310–3.

  22. Xu X, Lv H, Liu H, Zhang M, Wang G, Long S, et al. Investigation of the forming program failture in 1T1R structure. 12th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT). 2014. pp. 1–3.

  23. Jana D, Dutta M, Samanta S, Maikap S. RRAM characteristics using a new Cr/GdOx/TiN structure. Nanoscale Res Lett. 2014;9:680.

    Article  Google Scholar 

  24. Chen C-Y, Shih H-C, Wu C-W, Lin C-H, Chiu P-F, Sheu S-S, Chen FT. RRAM defect modeling and failure analysis based on March test and a novel squeeze-search scheme. IEEE Trans Comput. 2015;64:180–90.

    Article  MathSciNet  Google Scholar 

  25. Liu C, Hu M, Strachan JP, Li H. Rescuing memristor-based neuromorphic design with high defects. 54th ACM/EDAC/IEEE Design Automation Conference (DAC). 2017. pp. 1–6.

  26. Shih H-C, Chen C-Y, Wu C-W, Lin C-H, Sheu S-S. Training-based forming process for RRAM yield improvement. 29th VLSI Test Symposium. 2011. pp. 146–51.

  27. Hamdioui S, Taouil M, Haron NZ. Testing open defects in memristor-based memories. IEEE Trans Comput. 2015;64:247–59.

    Article  MathSciNet  Google Scholar 

  28. Li P, Xu D. Optimal operation of microgrid based on improved binary particle swarm optimization algorithm with double-structure coding. International Conference on Power System Technology. 2014. pp. 3141–6.

  29. Guangyou Y. A Modified Particle Swarm Optimizer Algorithm. 8th International Conference on Electronic Measurement and Instruments. 2007. pp. 2–675–2–679.

  30. Holland JH, Holland P of P and of EE and CSJH, Holland SL in HRM. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. MIT Press; 1992.

  31. Chibante R. Simulated Annealing: Theory with Applications. BoD – Books on Demand; 2010.

Download references

Funding

This work was jointly funded by National Key R&D Program of China (No. 2019YFB2204500), NSAF (No. U2030204) and National Natural Science Foundation of China (No. 62074026 & No. 61871096).

Author information

Authors and Affiliations

  1. School of Information and Communication Engineering, University of Electronic Science and Technology of China, West Hi-Tech Zone, Chengdu, 611731, China

    Yin Wang, Yuxiang Xie, Jiayan Gan, Liang Chang & Jun Zhou

  2. College of Engineering, Mathematics and Physical Sciences, University of Exeter, North Park Road, Exeter, UK

    Chunbo Luo

Authors
  1. Yin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuxiang Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiayan Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liang Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chunbo Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jun Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Zhou.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Xie, Y., Gan, J. et al. A Weight Importance Analysis Technique for Area- and Power-Efficient Binary Weight Neural Network Processor Design. Cogn Comput 13, 179–188 (2021). https://doi.org/10.1007/s12559-020-09794-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-020-09794-6

Keywords

Search

Navigation