Novel, parallel and differential synaptic architecture based on NAND flash memory for high-density and highly-reliable binary neural networks
Introduction
Recently, deep neural networks (DNNs) have shown remarkable performance for various intelligent tasks, such as computer vision, natural language processing, and speech recognition [1], [2], [3]. However, recent high-performance DNNs require a large network size and a huge volume of parameters, which need very fast graphics processing units (GPUs), enormous memory-storage, and large computational power [4], [5]. Therefore, it is difficult to run DNNs on mobile edge devices [6], [7]. Binary neural networks (BNNs) significantly reduce the computing overhead and memory footprint by binarizing the weights and inputs to 1 or −1 [6], [7], [8], [9], [10], [11], [12], [13], [14]. Instead of a high-precision floating-point multiplication and accumulation (MAC) operation, BNNs perform an exclusive NOR (XNOR) operation and bit-counting, which provides a promising solution to the implementation of hardware neural networks (HNNs) [7], [13]. In addition, recent studies have shown that BNNs could achieve a satisfying classification accuracy on representative image datasets, such as Modified National Institute of Standards and Technology (MNIST), Canadian Institute For Advanced Research (CIFAR-10), and ImageNet [11], [12], [13].
In previous research, two RRAMs with select transistors (2T2R) were mainly used as a synapse to perform an XNOR operation [13], [14]. However, memristors require further research for large-scale integration in terms of cell characteristics variation, reliability, and integration of selectors [15]. In addition, the sensing margin of the current sense amplifier (CSA) sensing the current of a synapse is small due to the small on/off current ratio of the memristors [13], [16]. The recent state-of-the-art DNN algorithms typically require an enormous parameter size. To accommodate this, NAND flash memory can be used, which has a great advantage in cell density and a large storage capacity per chip. NAND flash memory technology is well known as one of the most competitive solutions for immense data storage. In addition, NAND flash memory has been demonstrated as a technologically mature and cost-competitive technology among the various nonvolatile memory technologies [17], [18], [19]. However, it is difficult to apply NAND flash memory that is composed of cell strings to neural networks due to the specificity of the string structure.
In a previous study, our group has reported a neural network that uses NAND flash memory cells as analog synapse devices with multilevel states [20]. In this study, for the first time, we propose a 4T2S-based (four transistors and two NAND strings) synaptic architecture based on the NAND flash memory structure as a high-density synaptic string capable of XNOR operations with a large resistance ratio (∼7 × 105). The large resistance ratio of the NAND flash cells results in a sufficiently low bit-error rate and is beneficial for the large sensing margin of the current sense amplifier. We propose a differential sensing scheme merged with the NAND flash string structure, which further reduces the bit-error rate. The sufficiently low bit-error rate can implement highly reliable BNNs without an error correcting code (ECC), which requires large decoding circuits, time, and energy overhead. In addition, we propose a parallel read scheme to resolve the difficulty of applying NAND flash memory to neural networks because of the specificity of the string structure. It significantly reduces the read-out latency when compared to the sequential read scheme. Furthermore, we show that only one erase and program pulse can implement a sufficiently low bit-error rate without conventional incremental step pulse programming (ISPP). Therefore, time and energy can be greatly reduced when compared to those of the ISPP method. We also demonstrate that programming the NAND cells with a moderate VPGM can achieve not only a low-bit error rate but also a high endurance without the ISPP scheme. Finally, it is shown that the estimated synapse density of vertical NAND (V-NAND) flash memory with 128 stacks is ∼103 times that of the 2T2R-based synapse in RRAMs.
Section snippets
4T2S synaptic string structure
Fig. 1 shows a 2T2S (two transistors and two NAND strings) synaptic string structure for XNOR operations with a fixed reference current. Fig. 1 (a) shows the synapse string connected with the sense amplifier and (b) shows a current sense amplifier circuit. A synaptic device consists of two NAND cells whose complementary state defines the synaptic weight. As shown in Fig. 1 (a), a weight of +1 can be defined as the state where the right cell has a high threshold voltage (Vt,high) and the left
Measurement results of NAND cells as binary synapse
2-D NAND flash memory in this work was fabricated in the industry using 26 nm technology. Fig. 7 shows the layout of the NAND cell strings. One cell string consists of 64 cells including a drain select line (DSL) transistor, a source select line (SSL) transistor, and two dummy cells. The channel length and width of the cells are 26 and 20 nm, respectively. On the other hand, the channel length and width of the select line transistors are 130 and 20 nm, respectively. The channel length of the
Conclusion
A novel synaptic architecture based on a technologically mature NAND flash memory structure and its operation scheme has been proposed for high-density and highly reliable binary neural networks. For the first time, we proposed a 4T2S-based synaptic architecture with complementary input voltages that implements an equivalent bitwise XNOR operation. Thanks to the small bit-error rate (4.2 × 10−8 %) of the NAND flash cells, a highly reliable binary neural network could be achieved. In addition, a
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This research was supported by National R&D Program through the National Research Foundation of Korea (NRF) funded by Ministry of Science and ICT (2021M3F3A2A02037889) and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2021.
Sung-Tae Lee received the B.S. and the Ph.D. degrees in electrical and computer engineering from Seoul National University (SNU), Seoul, Korea, in 2016 and 2021, respectively. He worked as a Postdoctoral Researcher at Georgia Institute of Technology, USA, from 2021 to 2022. He joined the Department of Electronic Engineering, Gachon University, Republic of Korea, in 2022, where he is currently working as an Assistant Professor. His current research interests include neuromorphic system and its
References (33)
- et al.
Deep learning
Nature
(2015) Human tracking using convolutional neural networks
IEEE Trans. Neural Networks
(2010)- et al.
Deep visual-semantic alignments for generating image descriptions
- et al.
Deep Residual Learning for Image Recognition
- et al.
Implementation and evaluation of deep neural networks (DNN) on mainstream heterogeneous system
- et al.
YodaNN: An Architecture for Ultralow Power Binary-Weight CNN Acceleration
IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.
(2017) Neuro-inspired computing with emerging nonvolatile memorys
Proc. IEEE
(2018)- M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, Y. Bengio, Binarized Neural Networks: Training Deep Neural Networks...
- et al.
Binary neural network with 16 Mb RRAM macro chip for classification and online training
- et al.
Binary convolutional neural network on RRAM
XNOR-POP: A processing-in-Memory Architecture for Binary Convolutional Neural Networks in Wide-IO2 DRAMs
Binarized Neural Networks
XNOR-RRAM: A scalable and parallel resistive synaptic architecture for binary neural networks
In-Memory and Error-Immune Differential RRAM Implementation of Binarized Deep Neural Networks
Design considerations of selector device in cross-point RRAM array for neuromorphic computing
Embedded 2Mb ReRAM macro with 2.6 ns read access time using dynamic-trip-point-mismatch sampling current-mode sense amplifier for IoE applications
Cited by (0)
Sung-Tae Lee received the B.S. and the Ph.D. degrees in electrical and computer engineering from Seoul National University (SNU), Seoul, Korea, in 2016 and 2021, respectively. He worked as a Postdoctoral Researcher at Georgia Institute of Technology, USA, from 2021 to 2022. He joined the Department of Electronic Engineering, Gachon University, Republic of Korea, in 2022, where he is currently working as an Assistant Professor. His current research interests include neuromorphic system and its application in computing.
Hyeongsu Kim received the B.S. degree in electrical engineering from Seoul National University (SNU), Seoul, South Korea, in 2018, where he is currently pursuing the M.S. degree with the Department of Electrical and Computer Engineering. His-current research interests include neuromorphic system and its application in computing.
Honam Yoo recieved the B.S ans M.S. degrees in Physics and Astronomy from Seoul National University (SNU), Seoul, South Korea, in 2004 and 2009, respectively. He is currently pursuing the Ph.D. degree with the Department of Electrical and Computer Engineering, Seoul National University (SNU), Seoul, South Korea. His current research interests include neuromorphic system and neural networks.
Dongseok Kwon received the B.S. degree in electrical engineering from Pohang University of Science and Technology (POSTECH), Pohang, South Korea, in 2017. He is currently pursuing the M.S. degree with the Department of Electrical and Computer Engineering, Seoul National University (SNU), Seoul, South Korea. His current research interests include neuromorphic system and its application in computing.
Jong-Ho Lee (F’16) received the Ph.D. degree from Seoul National University (SNU), Seoul, in 1993, in electronic engineering. He was a Post-Doctoral Fellow with the Massachusetts Institute of Technology, Cambridge, MA, USA, from 1998 to 1999. He has been a Professor with the School of Electrical and Computer Engineering, SNU, since 2009.