Elsevier

Neurocomputing

Volume 498, 7 August 2022, Pages 1-13
Neurocomputing

Novel, parallel and differential synaptic architecture based on NAND flash memory for high-density and highly-reliable binary neural networks

https://doi.org/10.1016/j.neucom.2022.05.030Get rights and content

Abstract

A novel synaptic architecture based on a NAND flash memory structure is proposed as a high-density synapse capable of exclusive NOR (XNOR) operation for binary neural networks (BNNs). For the first time, a 4T2S-based synaptic architecture with a complementary input voltage that implements an equivalent bitwise XNOR operation is proposed. Two adjacent NAND flash strings connected with the word lines are used as one synaptic string with four input transistors connected to the bit-line. By changing the threshold voltage of the NAND flash cells and input voltages in a complementary fashion, the XNOR operation is successfully demonstrated. The large on/off current ratio (∼7 × 105) of the NAND flash cells and differential sensing scheme can implement high-density and highly reliable binary neural networks without error correcting codes (ECC), which can reduce the burden of the complementary metal–oxide–semiconductor (CMOS) overhead. Despite the string structure of NAND flash memory, the parallel read scheme significantly reduces the read-out latency when compared to the sequential read scheme. In addition, we show that with only 1 erase/program pulse, sufficiently low bit-error rate (7.6 × 10−9 %) is achieved without the conventional incremental step pulse programming (ISPP) scheme. Finally, the estimated synapse density of V-NAND flash memory with 128 stacks is ∼103 times that of the 2T2R synapse in resistive random access memory (RRAM).

Introduction

Recently, deep neural networks (DNNs) have shown remarkable performance for various intelligent tasks, such as computer vision, natural language processing, and speech recognition [1], [2], [3]. However, recent high-performance DNNs require a large network size and a huge volume of parameters, which need very fast graphics processing units (GPUs), enormous memory-storage, and large computational power [4], [5]. Therefore, it is difficult to run DNNs on mobile edge devices [6], [7]. Binary neural networks (BNNs) significantly reduce the computing overhead and memory footprint by binarizing the weights and inputs to 1 or −1 [6], [7], [8], [9], [10], [11], [12], [13], [14]. Instead of a high-precision floating-point multiplication and accumulation (MAC) operation, BNNs perform an exclusive NOR (XNOR) operation and bit-counting, which provides a promising solution to the implementation of hardware neural networks (HNNs) [7], [13]. In addition, recent studies have shown that BNNs could achieve a satisfying classification accuracy on representative image datasets, such as Modified National Institute of Standards and Technology (MNIST), Canadian Institute For Advanced Research (CIFAR-10), and ImageNet [11], [12], [13].

In previous research, two RRAMs with select transistors (2T2R) were mainly used as a synapse to perform an XNOR operation [13], [14]. However, memristors require further research for large-scale integration in terms of cell characteristics variation, reliability, and integration of selectors [15]. In addition, the sensing margin of the current sense amplifier (CSA) sensing the current of a synapse is small due to the small on/off current ratio of the memristors [13], [16]. The recent state-of-the-art DNN algorithms typically require an enormous parameter size. To accommodate this, NAND flash memory can be used, which has a great advantage in cell density and a large storage capacity per chip. NAND flash memory technology is well known as one of the most competitive solutions for immense data storage. In addition, NAND flash memory has been demonstrated as a technologically mature and cost-competitive technology among the various nonvolatile memory technologies [17], [18], [19]. However, it is difficult to apply NAND flash memory that is composed of cell strings to neural networks due to the specificity of the string structure.

In a previous study, our group has reported a neural network that uses NAND flash memory cells as analog synapse devices with multilevel states [20]. In this study, for the first time, we propose a 4T2S-based (four transistors and two NAND strings) synaptic architecture based on the NAND flash memory structure as a high-density synaptic string capable of XNOR operations with a large resistance ratio (∼7 × 105). The large resistance ratio of the NAND flash cells results in a sufficiently low bit-error rate and is beneficial for the large sensing margin of the current sense amplifier. We propose a differential sensing scheme merged with the NAND flash string structure, which further reduces the bit-error rate. The sufficiently low bit-error rate can implement highly reliable BNNs without an error correcting code (ECC), which requires large decoding circuits, time, and energy overhead. In addition, we propose a parallel read scheme to resolve the difficulty of applying NAND flash memory to neural networks because of the specificity of the string structure. It significantly reduces the read-out latency when compared to the sequential read scheme. Furthermore, we show that only one erase and program pulse can implement a sufficiently low bit-error rate without conventional incremental step pulse programming (ISPP). Therefore, time and energy can be greatly reduced when compared to those of the ISPP method. We also demonstrate that programming the NAND cells with a moderate VPGM can achieve not only a low-bit error rate but also a high endurance without the ISPP scheme. Finally, it is shown that the estimated synapse density of vertical NAND (V-NAND) flash memory with 128 stacks is ∼103 times that of the 2T2R-based synapse in RRAMs.

Section snippets

4T2S synaptic string structure

Fig. 1 shows a 2T2S (two transistors and two NAND strings) synaptic string structure for XNOR operations with a fixed reference current. Fig. 1 (a) shows the synapse string connected with the sense amplifier and (b) shows a current sense amplifier circuit. A synaptic device consists of two NAND cells whose complementary state defines the synaptic weight. As shown in Fig. 1 (a), a weight of +1 can be defined as the state where the right cell has a high threshold voltage (Vt,high) and the left

Measurement results of NAND cells as binary synapse

2-D NAND flash memory in this work was fabricated in the industry using 26 nm technology. Fig. 7 shows the layout of the NAND cell strings. One cell string consists of 64 cells including a drain select line (DSL) transistor, a source select line (SSL) transistor, and two dummy cells. The channel length and width of the cells are 26 and 20 nm, respectively. On the other hand, the channel length and width of the select line transistors are 130 and 20 nm, respectively. The channel length of the

Conclusion

A novel synaptic architecture based on a technologically mature NAND flash memory structure and its operation scheme has been proposed for high-density and highly reliable binary neural networks. For the first time, we proposed a 4T2S-based synaptic architecture with complementary input voltages that implements an equivalent bitwise XNOR operation. Thanks to the small bit-error rate (4.2 × 10−8 %) of the NAND flash cells, a highly reliable binary neural network could be achieved. In addition, a

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research was supported by National R&D Program through the National Research Foundation of Korea (NRF) funded by Ministry of Science and ICT (2021M3F3A2A02037889) and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2021.

Sung-Tae Lee received the B.S. and the Ph.D. degrees in electrical and computer engineering from Seoul National University (SNU), Seoul, Korea, in 2016 and 2021, respectively. He worked as a Postdoctoral Researcher at Georgia Institute of Technology, USA, from 2021 to 2022. He joined the Department of Electronic Engineering, Gachon University, Republic of Korea, in 2022, where he is currently working as an Assistant Professor. His current research interests include neuromorphic system and its

References (33)

  • Y. LeCun et al.

    Deep learning

    Nature

    (2015)
  • J. Fan

    Human tracking using convolutional neural networks

    IEEE Trans. Neural Networks

    (2010)
  • A. Karpathy et al.

    Deep visual-semantic alignments for generating image descriptions

  • K. He et al.

    Deep Residual Learning for Image Recognition

  • J. Gu et al.

    Implementation and evaluation of deep neural networks (DNN) on mainstream heterogeneous system

  • R. Andri et al.

    YodaNN: An Architecture for Ultralow Power Binary-Weight CNN Acceleration

    IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.

    (2017)
  • S. Yu

    Neuro-inspired computing with emerging nonvolatile memorys

    Proc. IEEE

    (2018)
  • M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, Y. Bengio, Binarized Neural Networks: Training Deep Neural Networks...
  • S. Yu et al.

    Binary neural network with 16 Mb RRAM macro chip for classification and online training

  • T. Tang et al.

    Binary convolutional neural network on RRAM

  • L. Jiang et al.

    XNOR-POP: A processing-in-Memory Architecture for Binary Convolutional Neural Networks in Wide-IO2 DRAMs

  • I. Hubara et al.

    Binarized Neural Networks

  • X. Sun et al.

    XNOR-RRAM: A scalable and parallel resistive synaptic architecture for binary neural networks

  • M. Bocquet et al.

    In-Memory and Error-Immune Differential RRAM Implementation of Binarized Deep Neural Networks

  • J. Woo et al.

    Design considerations of selector device in cross-point RRAM array for neuromorphic computing

  • C.P. Lo

    Embedded 2Mb ReRAM macro with 2.6 ns read access time using dynamic-trip-point-mismatch sampling current-mode sense amplifier for IoE applications

  • Cited by (0)

    Sung-Tae Lee received the B.S. and the Ph.D. degrees in electrical and computer engineering from Seoul National University (SNU), Seoul, Korea, in 2016 and 2021, respectively. He worked as a Postdoctoral Researcher at Georgia Institute of Technology, USA, from 2021 to 2022. He joined the Department of Electronic Engineering, Gachon University, Republic of Korea, in 2022, where he is currently working as an Assistant Professor. His current research interests include neuromorphic system and its application in computing.

    Hyeongsu Kim received the B.S. degree in electrical engineering from Seoul National University (SNU), Seoul, South Korea, in 2018, where he is currently pursuing the M.S. degree with the Department of Electrical and Computer Engineering. His-current research interests include neuromorphic system and its application in computing.

    Honam Yoo recieved the B.S ans M.S. degrees in Physics and Astronomy from Seoul National University (SNU), Seoul, South Korea, in 2004 and 2009, respectively. He is currently pursuing the Ph.D. degree with the Department of Electrical and Computer Engineering, Seoul National University (SNU), Seoul, South Korea. His current research interests include neuromorphic system and neural networks.

    Dongseok Kwon received the B.S. degree in electrical engineering from Pohang University of Science and Technology (POSTECH), Pohang, South Korea, in 2017. He is currently pursuing the M.S. degree with the Department of Electrical and Computer Engineering, Seoul National University (SNU), Seoul, South Korea. His current research interests include neuromorphic system and its application in computing.

    Jong-Ho Lee (F’16) received the Ph.D. degree from Seoul National University (SNU), Seoul, in 1993, in electronic engineering. He was a Post-Doctoral Fellow with the Massachusetts Institute of Technology, Cambridge, MA, USA, from 1998 to 1999. He has been a Professor with the School of Electrical and Computer Engineering, SNU, since 2009.

    View full text