LSTM-SNP: A long short-term memory model inspired from spiking neural P systems☆
Introduction
Spiking neural P (SNP) systems [1] are a class of distributed and parallel computing models that are abstracted by applying the mechanisms of spiking neurons [2], [3], [4], [5], [6]. From a topological perspective, an SNP system can be expressed by a directed graph, where the nodes denote the neurons and the arcs indicate the synapses between these neurons. Each neuron in an SNP system has two components: data and spiking rules. The data reflect the internal state, and the spiking rules describe the system?s dynamic behaviour. The working mechanism of spiking rules involves two processes: spike consumption and spike generation. In addition to the two characteristics, other biological mechanisms have been introduced in the past, such as astrocytes [7], anti-spike [8], multiple channels [9], polarisation [10], structural plasticity [11], [12], delay on synapses [13], dynamic threshold mechanism [14], nonlinear coupled mechanism [15], inhibitory rules [16], dendrite [17], plasmids [18], autapses [19], communication on request [20], and numerical SNP systems [21]. Currently, nonlinear SNP (NSNP) systems [22], which can be considered as a nonlinear version of SNP systems, are being investigated. In addition to the two aforementioned characteristics, NSNP systems have a nonlinear mechanism: the consumed spikes are a nonlinear function of the state in the neuron, and the generated spikes are nonlinear functions of the state. Therefore, NSNP systems can capture the nonlinear characteristics of a complex system. The focus of this study is the utilisation of NSNP systems to develop a novel long short-term memory (LSTM) model for processing sequential data, such as time series forecasting problems.
As time series forecasting problems are utilised to conduct case studies to evaluate the proposed model, we briefly review the existing work on time series forecasting. A time series is generally time-ordered data points. In a time series forecasting task, a model attempts to predict future points based on previous observations. In recent years, many prediction models have been proposed for time series forecasting tasks.
The ARIMA model is a widely used linear time series forecasting model that integrates an autoregressive (AR) model and a moving average (MA) model. ARIMA can achieve good accuracy when forecasting a relatively stationary time series. However, ARIMA fails to model nonlinear and non-stationary time series because most time series in real-life applications have complex nonlinear relationships. To address this limitation, some improved models based on the multi-scale transform have been developed [23], [24].
Artificial neural networks (ANNs) are the most widely used models to model nonlinear and non-stationary time series, including deep learning (DL)-based models. ANNs have two advantages over ARIMA and the improved models: (i) ANNs have the significant ability to process complex nonlinear input–output relationships. (ii) ANNs are adaptively data-driven, indicating that they can be established adaptively according to the features in the time series data. In recent years, some ANN-based and DL-based prediction models have been developed for time series forecasting. Geva [25] developed a multiscale neural network architecture for time series prediction. Liu et al. [26] proposed a DL-based prediction model that integrates convolutional neural networks (CNNs) and fuzzy cognitive maps (FCM). Bai et al. [27] developed a prediction model based on convolutional and recurrent networks. In addition, Jang [28] proposed an adaptive-network-based fuzzy inference model that can be used for time series forecasting tasks.
Recurrent neural networks (RNNs) are particularly suitable for processing sequential data, including time series forecasting. Some RNN-like architectures have been developed, such as LSTM [29] and gated recurrent units (GRUs) [30], [31].
LSTM is a variant of RNNs [29], [30]. The LSTM model has a hidden state, and three gates are considered to control the transmission of information in the neuron. The three gates are the forget gate, input gate, and output gate. As mentioned earlier, each spiking neuron in SNP systems has an internal state and two spiking mechanisms (spike consumption and spike generation). Motivated by the three characteristics of spiking neurons, we propose a parameterised NSNP system that consists of only one nonlinear spiking neuron. The parameterised NSNP system has nonlinear spiking mechanisms (nonlinear spike consumption and generation) and nonlinear gate functions (reset, consumption, and generation gates). Moreover, the parameterised NSNP system is trainable; hence, it can be trained according to the data. Based on the parameterised NSNP system, a novel variant of the LSTM model is developed, called the LSTM-SNP model. Therefore, the LSTM-SNP model is a novel recurrent-type model based on the nonlinear mechanisms of the spiking neurons in NSNP systems.
The contributions of this study can be summarised as follows.
- (1)
We propose a parameterised NSNP system based on the nonlinear mechanisms of the spiking neurons in NSNP systems.
- (2)
Based on the parameterised NSNP system, we develop a recurrent-type model, called the LSTM-SNP model. The LSTM-SNP model can be regarded as a variant of LSTM; however, the LSTM-SNP model has different gate functions, state equations, and input–output equations.
The remainder of this paper is organised as follows. Section 2 describes SNP systems and details the proposed LSTM-SNP model and its implementation. The experimental results of time series forecasting and a discussion are presented in Section 3. Finally, the conclusions and directions for future work are provided in Section 4.
Section snippets
NSNP systems
SNP systems are a class of neural-like membrane computing models that are abstracted by applying the mechanisms of spiking neurons, and NSNP systems are nonlinear versions of SNP systems. The formal definitions of SNP systems and NSNP systems can be found in [1], [22], respectively. To process the time series, we extend the original NSNP system as an extended NSNP system. Note that in the extended NSNP system, and are not required to be .
Definition 1 An extended NSNP system of degree
Dataset
To evaluate the proposed LSTM-SNP model, a time series forecasting problem is used as a case study. Five real-world time series are used as benchmark datasets in the experiments: Mackey–Glass time series (MG), S&P 500 stock index (SP500), Monthly milk production per cow (Milk), Monthly Lake Erie levels (Lake), and Monthly closings of the Dow-Jones industrial index (Closing).
A summary of the time series is provided in Table 2. For each time series, the features are normalised to range [−1, 1].
Performance measure
To
Conclusions and future work
In this study, we proposed a parameterised NSNP system that has only one nonlinear neuron. The parameterised NSNP system differs from existing NSNP systems in two aspects: the parameterised mechanism and nonlinear gate functions. Based on the parameterised NSNP system, we developed a novel variant of the LSTM model, known as the LSTM-SNP model. Compared with LSTM and GRU, the following differences are observed: (i) the three gate functions in the LSTM-SNP model are inspired by the nonlinear
CRediT authorship contribution statement
Qian Liu: Conceptualization, Software, Experiments, Writing. Lifan Long: Conceptualization, Software, Experiments. Qian Yang: Conceptualization, Software, Experiments. Hong Peng: Conceptualization, Writing. Jun Wang: Conceptualization, Writing. Xiaohui Luo: Software, Experiments.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
The authors thank the anonymous reviewers for providing very insightful and constructive suggestions, which have greatly help improve the presentation of this paper.
References (37)
- et al.
Spiking neural P systems with multiple channels
Neural Netw.
(2017) - et al.
Dynamic threshold neural P systems
Knowl.-Based Syst.
(2019) - et al.
Dendrite P systems
Neural Netw.
(2020) - et al.
Neural-like P systems with plasmids
Inform. and Comput.
(2021) - et al.
Spiking neural P systems with autapses
Inform. Sci.
(2021) - et al.
CNN-FCM: Systems modeling promotes stability of deep learning in time series prediction
Knowl.-Based Syst.
(2020) - et al.
A robust time series prediction method based on empirical mode decomposition and high-order fuzzy cognitive maps
Knowl.-Based Syst.
(2020) - et al.
Time series forecasting based on echo state network and empirical wavelet transformation
Appl. Soft Comput.
(2021) - et al.
A hybrid ETS-ANN model for time series forecasting
Eng. Appl. Artif. Intell.
(2017) - et al.
Long-term prediction of time series based on stepwise linear division algorithm and time-variant zonary fuzzy information granules
Internat. J. Approx. Reason.
(2019)
Time series forecasting based on kernel mapping and high-order fuzzy cognitive maps
Knowl.-Based Syst.
Spiking neural P systems
Fund. Inform.
The Oxford Handbook of Membrane Computing
A return to stochasticity and probability in spiking neural P systems
J. Membrane Comput.
A formal framework for spiking neural P systems
J. Membrane Comput.
Spiking neural P systems: matrix representation and formal verification
J. Membrane Comput.
A complete arithmetic calculator constructed from spiking neural P systems and its application to information fusion
Int. J. Neural Syst.
Spiking neural P systems with astrocyte-like control
J. UCS
Cited by (43)
Deep synergetic spiking neural P systems for the overall survival time prediction of glioblastoma patients
2024, Expert Systems with ApplicationsNSNP-DFER: A Nonlinear Spiking Neural P Network for Dynamic Facial Expression Recognition
2024, Computers and Electrical EngineeringTime series classification models based on nonlinear spiking neural P systems
2024, Engineering Applications of Artificial IntelligenceSentence-level sentiment classification based on multi-attention bidirectional gated spiking neural P systems
2024, Applied Soft ComputingSequence recommendation using multi-level self-attention network with gated spiking neural P systems
2024, Information Sciences