Stable and compact design of Memristive GoogLeNet Neural Network

doi:10.1016/j.neucom.2021.01.122

Neurocomputing

Volume 441, 21 June 2021, Pages 52-63

https://doi.org/10.1016/j.neucom.2021.01.122 Get rights and content

Abstract

According to the requirements of edge intelligence for circuit volume, power consumption and computing performance, a Memristive GoogLeNet Neural Network (MGNN) circuit is designed using memristor which is a new device integrating storage and computing as the basic circuit element. This circuit adopts $1 \times 1$ convolution and multi-scale convolution feature fusion to reduce the number of layers required by the network while ensuring the recognition accuracy of circuit. In order to reduce the size of the memristor crossbars in the circuit, we design word-line pruning and bit-line pruning methods of Memristive Convolution (MC) layers. We also use the parameter distribution of the memristive neural network to further reduce the size of memristor crossbars. The Memristive Batch Normalization (MBN) layer and Memristive Dropout (MD) are merged into front MC layers according mathematical analysis for cutting the number of network layers and decreasing the power consumption of the circuit. We also design the channel optimization and layer optimization methods of MC layers which greatly reduce the negative effect of multi-state conductance of memristors on the accuracy, improve the stability of the circuit, and reduce the circuit volume and power consumption. Experiments show that this circuit can get 89.83% accuracy on the CIFAR-10 data set, and the power consumption of a single neuron is only $1.3 μ W$ . When the number of memristor multi-state conductance is $2^{4} = 16$ , the accuracy of the MGNN circuit close to float MGNN can still be obtained.

Section snippets

Preface

In recent years, benefiting from technology progresses in algorithms, computing power, and data sets, deep learning has been developed tremendously in various fields such as security, e-commerce, manufacturing, agriculture, and smart home, and improved the efficiency of human production and life greatly [1]. Current intelligent applications based on deep learning usually rely on cloud data centers with powerful computing capabilities because of a lot of requirement of calculations [2], [3].

Related work

Memristor with the characteristics of variable resistance, low non-volatile power consumption and high integration density has very good application prospects in the fields of storage, artificial neural network and logic computing.

The development of neuromorphic computing circuits based on memristors can be divided into three stages: The first stage is the development of a single device. In 2008, Professor Stan William of Hewlett–Packard Lab produced a memristor in the laboratory at the first

Model of memristor

Academician Leon Chua (UC Berkeley) first proposed the concept of memristors in 1976 [13]. Since Hewlett–Packard Labs proposed the $TiO 2$ physical realization and mathematical model of the memristor in 2008 [46], [47], the research on the memristor has become a research hot-spot in academia and industry. Research teams all over the world are trying to use various materials to prepare new devices with memristor characteristics. With the production of memristors with different materials and

Overview of MGNN

Deep learning networks such as AlexNet and Visual Geometry Group (VGG) obtain better recognition results from the perspective of increasing the depth of the network. However, the increase in the number of layers will bring about problems such as over-fitting, gradient disappearance, and gradient explosion. GoogLeNet improves the training effect from the perspective of using computing resources more efficiently and extracting more features under the same amount of calculation. Since GoogLeNet

Optimization of Batch Normalization layer

Memristive Batch Normalization layers usually follow the memristive convolution layer. MBN layers are used to speed up training process, reduce over-fitting, and make the network in-sensitive to conductance initialization [52]. MBN layers try to normalize the output of each memristive convolution layer to data with mean 0 and variance 1. The output of the kth MBN layer can be defined as ${\hat{x}}_{bn}^{(k)} = \frac{x_{bn}^{(k)} - E [x_{bn}^{(k)}]}{\sqrt{V [x_{bn}^{(k)}]}} .$ In Eq. (13), the features $x_{bn}^{(k)}$ extracted by the previous memristive

Experiment overview

Firstly, we established the MGNN model using the tensorflow framework on the traditional Graphics Processing Unit (GPU) server. Then we used the CIFAR-10 data set to train the model. In the training process, $L_{2}$ regularization constraints are used for pruning the model. The trained parameters are imported into the MGNN circuit model established by MATLAB simulink, and the circuit model is used to analyze the image recognition accuracy, required memristor crossbars, power consumption and the

Conclusion

In this article, a new type of passive device with integrated storage and calculation name as memristor is used to design a compact and stable MGNN circuit, which adopts $1 \times 1$ convolution and multi-scale feature fusion in structure to reduce the number of memristive neural network layers and maintain the recognition accuracy of the circuit. In order to get a more compact circuit, this article designs word-line pruning and bit-line pruning methods for the memristive convolution layer, which

CRediT authorship contribution statement

Huanhuan Ran: Writing - original draft, Resources. Shiping Wen: Conceptualization. Kaibo Shi: Investigation. Tingwen Huang: Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Huanhuan Ran is with the Key Laboratory of Electronic Thin Films and Integrated Devices, University of Electronic Science and Technology of China, Chengdu?Sichuan, 610054, China.

References (55)

A. Yousefpour et al.
All one needs to know about fog computing and related edge computing paradigms: A complete survey
Journal of Systems Architecture
(2019)
J. Chen et al.
An efficient memristor-based circuit implementation of squeeze-and-excitation fully convolutional neural networks
IEEE Transactions on Neural Networks and Learning Systems
(2020)
Y. Wang et al.
Event-based sliding-mode synchronization of delayed memristive neural networks via continuous/periodic sampling algorithm
Applied Mathematics and Computation
(2020)
S. Wang et al.
Event-triggered synchronization of multiple memristive neural networks with cyber-physical attacks
Information Sciences
(2020)
Y. Cao et al.
Global exponential synchronization of delayed memristive neural networks with reaction-diffusion terms
Neural Networks
(2020)
M. Di Marco et al.
Memristor standard cellular neural networks computing in the flux–charge domain
Neural Networks
(2017)
S. Wen et al.
Memristive lstm network for sentiment analysis
IEEE Transactions on Systems, Man, and Cybernetics: Systems
(2019)
O. Krestinskaya et al.
Neuromemristive circuits for edge computing: A review
IEEE Transactions on Neural Networks and Learning Systems
(2020)
H. Khelifi et al.
Bringing deep learning at the edge of information-centric internet of things
IEEE Communications Letters
(2018)
J. Zhu
A comprehensive review on emerging artificial neuromorphic devices
Applied Physics Reviews
(2020)

J. Park, S. Samarakoon, M. Bennis, M. Debbah, Wireless network intelligence at the...

Z. Zhou et al.

Secure and efficient vehicle-to-grid energy trading in cyber physical systems: Integration of blockchain and edge computing

IEEE Transactions on Systems, Man, and Cybernetics: Systems

(2020)

S. Wen et al.

Observer-based adaptive control for multiagent systems with unknown parameters under attacks

IEEE Transactions on Neural Networks and Learning Systems

(2021)

Z. Zhou et al.

Edge intelligence: Paving the last mile of artificial intelligence with edge computing

Proceedings of the IEEE

(2019)

J. Chen et al.

Deep learning with edge computing: A review

Proceedings of the IEEE

(2019)

K. Roy, A. Jaiswal, P. Panda, Towards spike-based machine intelligence with neuromorphic computing, Nature 575...

Z. Yu, A.M. Abdulghani, A. Zahid, H. Heidari, M.A. Imran, Q.H. Abbasi, An overview of neuromorphic computing for...

C.D. Schuman, T.E. Potok, R.M. Patton, J.D. Birdwell, M.E. Dean, G.S. Rose, J.S. Plank, A survey of neuromorphic...

L.O. Chua et al.

Memristive devices and systems

Proceedings of the IEEE

(1976)

S.H. Jo, T. Chang, I. Ebong, B.B. Bhadviya, P. Mazumder, W. Lu, Nanoscale memristor device as synapse in neuromorphic...

S.-Y. Sun et al.

Cascaded architecture for memristor crossbar array based larger-scale neuromorphic computing

IEEE Access

(2019)

S.-Y. Sun et al.

Cascaded neural network for memristor based neuromorphic computing

S.H. Jo et al.

Nanoscale memristor device as synapse in neuromorphic systems

Nano Letters

(2010)

P. Chi et al.

Prime: A novel processing-in-memory architecture for neural network computation in reram-based main memory

ACM SIGARCH Computer Architecture News

(2016)

S. Wen et al.

CKFO: Convolutional kernel first operated algorithm with applications in memristor-based convolutional neural networks

IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

(2020)

H. Ran et al.

Memristor-based edge computing of blaze block for image recognition

IEEE Transactions on Neural Networks and Learning Systems

(2020)

I. Kataeva et al.

Efficient training algorithms for neural networks based on memristive crossbar circuits

Cited by (22)

A short-term PV resource assessment method with parallel DenseNet201 and BiLSTM under multiple data features
2024, Energy Reports
Recently, renewable energy has been widely deployed. Among all renewable energy sources, photovoltaic power is widely deployed in the grid. Because of the instability of PV power generation, PV grid connection affects electrical grid stability and reliability. An accurate assessment of PV resources is therefore essential to the robustness and reliability of the power system. To increase the precision of PV resource assessment, this study proposes a short-term PV resource assessment method with parallel DenseNet201 and BiLSTM (PDBL). This study predicts the power generation for 24 hours on day 15 with irradiation intensity, full-field power, and temperature for each hour of 14 days at the PV farm. In this study, irradiation intensity data, full-field power data, and field temperature data are fed into BiLSTM, BiLSTM, and DenseNet201 to predict PV farm power respectively; results of three networks are optimized with a deep fully connected layer network to obtain prediction results with higher accuracy. Experiments on the same data set show that the root mean square error (RMSE) of the PDBL short-term PV resource assessment approach is 7.5278, which is 15.8143 smaller than the RMSE of BiLSTM.
A Hybrid Weight Quantization Strategy for Memristive Neural Networks
2023, Neurocomputing
Due to the ability to store data and process information, the memristor-based neuromorphic system has attracted extensive attention. Its efficient parallel computing approach allows it to implement neural networks in hardware. However, due to the limitation of the range of memristor conductance, it is difficult to represent high-precision weights in memristive neural network. During the off-chip learning, it is crucial to find an efficient weight quantization scheme and map it to the memristor array. Therefore, a hybrid weight quantization strategy that combines uniform and non-uniform quantization is proposed to overcome these problems. Specifically, the curve fitting of pulse modulation for tantalum oxide-based memristor is carried out, and the mapping rules of weights are proposed to simplify the process of reading verification. Furthermore, the hybrid quantization strategy is proposed and applied to a multilayer perceptron and a convolutional neural network, respectively. The effectiveness and robustness of the hybrid quantization scheme are verified in the MNIST dataset. Experiments show that the proposed hybrid quantization scheme can achieve 99.26% accuracy at 4 bits and tolerate 20% noise interference. The simulation results in this paper also provide an effective solution for the hardware implementation of memristive neural networks.
Experimental demonstration of SnO₂ nanofiber-based memristors and their data-driven modeling for nanoelectronic applications
2023, Chip
This paper demonstrated the fabrication, characterization, data-driven modeling, and practical application of a 1D SnO₂ nanofiber-based memristor, in which a 1D SnO₂ active layer was sandwiched between silver (Ag) and aluminum (Al) electrodes. This device yielded a very high R_OFF : R_ON of ∼10⁴ (I_ON : I_OFF of ∼10⁵) with an excellent activation slope of 10 mV/dec, low set voltage of V_SET ∼ 1.14 V and good repeatability. This paper physically explained the conduction mechanism in the layered SnO₂ nanofiber-based memristor. The conductive network was composed of nanofibers that play a vital role in the memristive action, since more conductive paths could facilitate the hopping of electron carriers. Energy band structures experimentally extracted with the adoption of ultraviolet photoelectron spectroscopy strongly support the claims reported in this paper. An machine learning (ML)–assisted, data-driven model of the fabricated memristor was also developed employing different popular algorithms such as polynomial regression, support vector regression, k nearest neighbors, and artificial neural network (ANN) to model the data of the fabricated device. We have proposed two types of ANN models (type I and type II) algorithms, illustrated with a detailed flowchart, to model the fabricated memristor. Benchmarking with standard ML techniques shows that the type II ANN algorithm provides the best mean absolute percentage error of 0.0175 with a 98% R² score. The proposed data-driven model was further validated with the characterization results of similar new memristors fabricated adopting the same fabrication recipe, which gave satisfactory predictions. Lastly, the ANN type II model was applied to design and implement simple AND & OR logic functionalities adopting the fabricated memristors with expected, near-ideal characteristics.
Inception-embedded attention memory fully-connected network for short-term wind power prediction
2023, Applied Soft Computing
With the increasing demand for energy in the world today, wind energy has turned out to be an attractive alternative to traditional fossil energy sources because of the characteristics of being clean, non-polluting, and easily accessible. Reliably predicting wind power is vital to improving energy utilization and ensuring the stability of power system operation. However, because of the uncertainty and instability of wind energy, accurately predicting wind power is still challenging. Therefore, this study proposes an Inception-embedded attention memory fully-connected network short-term wind power prediction model, incorporating improved attention mechanisms. As a result, the Inception-embedded attention memory fully-connected network can give reliable wind power predictions. This study utilizes a dataset of about 400 days from Natal and compares the Inception-embedded attention memory fully-connected network with 23 algorithms including EffiientNet, NasNet, and ResNet. The comparison results show that the Inception-embedded attention memory fully-connected network obtains reliable wind power prediction one day ahead and outperforms all other compared algorithms by more than 40% in all evaluation metrics.
Cancelable ECG biometric based on combination of deep transfer learning with DNA and amino acid approaches for human authentication
2022, Information Sciences
Citation Excerpt :
These models have already learned to extract informative and powerful ECG features. There are several available pre-trained networks trained such as resnet [39], googlenet [40], xception [41] and vgg [42]. In this study, we worked on vgg-16 pre-train model [42].
Recently, electrocardiogram (ECG) signals have received a high level of attention as a physiological signal in the field of biometrics. It has presented great possibilities for its strength against counterfeit. However, the ECG feature templates are irreplaceable, and a compromised template implies a permanent loss of identity. Therefore, several studies have been introduced biometric template protection techniques such as cancelable techniques to protect the original template in case it is stolen or lost. In this research, a cancelable ECG approach is proposed to protect the ECG feature template for human authentication. In our system, we first employed some image processing techniques for preprocessing the input ECG signals. Then, a deep transfer learning approach is employed to extract the deep ECG features. Later, the proposed cancelable approach based on DNA and amino acid is applied to protect the deep feature templates. Lastly, a Support Vector Machine (SVM) is employed for authentication. Extensive experiments on two commonly used datasets coupled with comprehensive theoretical analysis demonstrate the highest accuracy of the proposed system and the strong resilience of the system to various security and privacy attacks. Results show that the proposed cancelable method meets all requirements of cancelable biometrics such as irreversibility, revocability, and unlinkability.
Automated Pallet Racking Examination in Edge Platform Based on MobileNetV2: Towards Smart Manufacturing
2024, Journal of Grid Computing

View all citing articles on Scopus

Huanhuan Ran is with the Key Laboratory of Electronic Thin Films and Integrated Devices, University of Electronic Science and Technology of China, Chengdu?Sichuan, 610054, China.

Shipping Wen is with Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Ultimo, NSW 2007, Australia.

Kaibo Shi is with School of Information Science and Engineering, Chengdu University, Chengdu, Sichuan, China.

Tingwen Huang is with Science Program, Texas A $&$ M University at Qatar, 23874, Doha, Qatar.

View full text

Stable and compact design of Memristive GoogLeNet Neural Network

Abstract

Section snippets

Preface

Related work

Model of memristor

Overview of MGNN

Optimization of Batch Normalization layer

Experiment overview

Conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Journal of Systems Architecture

IEEE Transactions on Neural Networks and Learning Systems

Applied Mathematics and Computation

Information Sciences

Neural Networks

Neural Networks

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Neuromemristive circuits for edge computing: A review

IEEE Transactions on Neural Networks and Learning Systems

Bringing deep learning at the edge of information-centric internet of things

IEEE Communications Letters

A comprehensive review on emerging artificial neuromorphic devices

Applied Physics Reviews

Secure and efficient vehicle-to-grid energy trading in cyber physical systems: Integration of blockchain and edge computing

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Observer-based adaptive control for multiagent systems with unknown parameters under attacks

IEEE Transactions on Neural Networks and Learning Systems

Edge intelligence: Paving the last mile of artificial intelligence with edge computing

Proceedings of the IEEE

Deep learning with edge computing: A review

Proceedings of the IEEE

Memristive devices and systems

Proceedings of the IEEE

Cascaded architecture for memristor crossbar array based larger-scale neuromorphic computing

IEEE Access

Cascaded neural network for memristor based neuromorphic computing

Nanoscale memristor device as synapse in neuromorphic systems

Nano Letters

Prime: A novel processing-in-memory architecture for neural network computation in reram-based main memory

ACM SIGARCH Computer Architecture News

CKFO: Convolutional kernel first operated algorithm with applications in memristor-based convolutional neural networks

IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

Memristor-based edge computing of blaze block for image recognition

IEEE Transactions on Neural Networks and Learning Systems

Efficient training algorithms for neural networks based on memristive crossbar circuits