Elsevier

Knowledge-Based Systems

Volume 235, 10 January 2022, 107656
Knowledge-Based Systems

LSTM-SNP: A long short-term memory model inspired from spiking neural P systems

https://doi.org/10.1016/j.knosys.2021.107656Get rights and content

Highlights

  • We propose a parameterised nonlinear SNP system.

  • We propose a variant of RNN, called LSTM-SNP model.

  • We develop a new prediction model for time series.

Abstract

Spiking neural P (SNP) systems are a class of neural-like membrane computing models that are abstracted by applying the mechanisms of spiking neurons. In SNP systems, each spiking neuron has three characteristics: (i) internal state, (ii) spike consumption, and (iii) spike generation. These three characteristics are used to form a parameterised nonlinear SNP system, which has a nonlinear spiking mechanism, three nonlinear gate functions, and trainable parameters. Based on the parameterised nonlinear SNP system, we develop a novel variant of long short-term memory (LSTM), called the LSTM-SNP model. LSTM-SNP is a recurrent-type model that can process sequential data. Time series forecasting problems are used to conduct a case study. Five benchmark time series are used to evaluate the proposed LSTM-SNP model and compare seven state-of-the-art prediction models and five baseline prediction models. The comparison results show the effectiveness of the proposed LSTM-SNP model for time series forecasting.

Introduction

Spiking neural P (SNP) systems [1] are a class of distributed and parallel computing models that are abstracted by applying the mechanisms of spiking neurons [2], [3], [4], [5], [6]. From a topological perspective, an SNP system can be expressed by a directed graph, where the nodes denote the neurons and the arcs indicate the synapses between these neurons. Each neuron in an SNP system has two components: data and spiking rules. The data reflect the internal state, and the spiking rules describe the system?s dynamic behaviour. The working mechanism of spiking rules involves two processes: spike consumption and spike generation. In addition to the two characteristics, other biological mechanisms have been introduced in the past, such as astrocytes [7], anti-spike [8], multiple channels [9], polarisation [10], structural plasticity [11], [12], delay on synapses [13], dynamic threshold mechanism [14], nonlinear coupled mechanism [15], inhibitory rules [16], dendrite [17], plasmids [18], autapses [19], communication on request [20], and numerical SNP systems [21]. Currently, nonlinear SNP (NSNP) systems [22], which can be considered as a nonlinear version of SNP systems, are being investigated. In addition to the two aforementioned characteristics, NSNP systems have a nonlinear mechanism: the consumed spikes are a nonlinear function of the state in the neuron, and the generated spikes are nonlinear functions of the state. Therefore, NSNP systems can capture the nonlinear characteristics of a complex system. The focus of this study is the utilisation of NSNP systems to develop a novel long short-term memory (LSTM) model for processing sequential data, such as time series forecasting problems.

As time series forecasting problems are utilised to conduct case studies to evaluate the proposed model, we briefly review the existing work on time series forecasting. A time series is generally time-ordered data points. In a time series forecasting task, a model attempts to predict future points based on previous observations. In recent years, many prediction models have been proposed for time series forecasting tasks.

The ARIMA model is a widely used linear time series forecasting model that integrates an autoregressive (AR) model and a moving average (MA) model. ARIMA can achieve good accuracy when forecasting a relatively stationary time series. However, ARIMA fails to model nonlinear and non-stationary time series because most time series in real-life applications have complex nonlinear relationships. To address this limitation, some improved models based on the multi-scale transform have been developed [23], [24].

Artificial neural networks (ANNs) are the most widely used models to model nonlinear and non-stationary time series, including deep learning (DL)-based models. ANNs have two advantages over ARIMA and the improved models: (i) ANNs have the significant ability to process complex nonlinear input–output relationships. (ii) ANNs are adaptively data-driven, indicating that they can be established adaptively according to the features in the time series data. In recent years, some ANN-based and DL-based prediction models have been developed for time series forecasting. Geva [25] developed a multiscale neural network architecture for time series prediction. Liu et al. [26] proposed a DL-based prediction model that integrates convolutional neural networks (CNNs) and fuzzy cognitive maps (FCM). Bai et al. [27] developed a prediction model based on convolutional and recurrent networks. In addition, Jang [28] proposed an adaptive-network-based fuzzy inference model that can be used for time series forecasting tasks.

Recurrent neural networks (RNNs) are particularly suitable for processing sequential data, including time series forecasting. Some RNN-like architectures have been developed, such as LSTM [29] and gated recurrent units (GRUs) [30], [31].

LSTM is a variant of RNNs [29], [30]. The LSTM model has a hidden state, and three gates are considered to control the transmission of information in the neuron. The three gates are the forget gate, input gate, and output gate. As mentioned earlier, each spiking neuron in SNP systems has an internal state and two spiking mechanisms (spike consumption and spike generation). Motivated by the three characteristics of spiking neurons, we propose a parameterised NSNP system that consists of only one nonlinear spiking neuron. The parameterised NSNP system has nonlinear spiking mechanisms (nonlinear spike consumption and generation) and nonlinear gate functions (reset, consumption, and generation gates). Moreover, the parameterised NSNP system is trainable; hence, it can be trained according to the data. Based on the parameterised NSNP system, a novel variant of the LSTM model is developed, called the LSTM-SNP model. Therefore, the LSTM-SNP model is a novel recurrent-type model based on the nonlinear mechanisms of the spiking neurons in NSNP systems.

The contributions of this study can be summarised as follows.

  • (1)

    We propose a parameterised NSNP system based on the nonlinear mechanisms of the spiking neurons in NSNP systems.

  • (2)

    Based on the parameterised NSNP system, we develop a recurrent-type model, called the LSTM-SNP model. The LSTM-SNP model can be regarded as a variant of LSTM; however, the LSTM-SNP model has different gate functions, state equations, and input–output equations.

The remainder of this paper is organised as follows. Section 2 describes SNP systems and details the proposed LSTM-SNP model and its implementation. The experimental results of time series forecasting and a discussion are presented in Section 3. Finally, the conclusions and directions for future work are provided in Section 4.

Section snippets

NSNP systems

SNP systems are a class of neural-like membrane computing models that are abstracted by applying the mechanisms of spiking neurons, and NSNP systems are nonlinear versions of SNP systems. The formal definitions of SNP systems and NSNP systems can be found in [1], [22], respectively. To process the time series, we extend the original NSNP system as an extended NSNP system. Note that in the extended NSNP system, f(u) and g(u) are not required to be 0g(u)f(u).

Definition 1

An extended NSNP system of degree m1

Dataset

To evaluate the proposed LSTM-SNP model, a time series forecasting problem is used as a case study. Five real-world time series are used as benchmark datasets in the experiments: Mackey–Glass time series (MG), S&P 500 stock index (SP500), Monthly milk production per cow (Milk), Monthly Lake Erie levels (Lake), and Monthly closings of the Dow-Jones industrial index (Closing).

A summary of the time series is provided in Table 2. For each time series, the features are normalised to range [−1, 1].

Performance measure

To

Conclusions and future work

In this study, we proposed a parameterised NSNP system that has only one nonlinear neuron. The parameterised NSNP system differs from existing NSNP systems in two aspects: the parameterised mechanism and nonlinear gate functions. Based on the parameterised NSNP system, we developed a novel variant of the LSTM model, known as the LSTM-SNP model. Compared with LSTM and GRU, the following differences are observed: (i) the three gate functions in the LSTM-SNP model are inspired by the nonlinear

CRediT authorship contribution statement

Qian Liu: Conceptualization, Software, Experiments, Writing. Lifan Long: Conceptualization, Software, Experiments. Qian Yang: Conceptualization, Software, Experiments. Hong Peng: Conceptualization, Writing. Jun Wang: Conceptualization, Writing. Xiaohui Luo: Software, Experiments.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The authors thank the anonymous reviewers for providing very insightful and constructive suggestions, which have greatly help improve the presentation of this paper.

References (37)

  • YuanK. et al.

    Time series forecasting based on kernel mapping and high-order fuzzy cognitive maps

    Knowl.-Based Syst.

    (2020)
  • IonescuM. et al.

    Spiking neural P systems

    Fund. Inform.

    (2006)
  • Gh. PǎunM. et al.

    The Oxford Handbook of Membrane Computing

    (2010)
  • LazoP.P.L. et al.

    A return to stochasticity and probability in spiking neural P systems

    J. Membrane Comput.

    (2021)
  • VerlanS. et al.

    A formal framework for spiking neural P systems

    J. Membrane Comput.

    (2020)
  • GheorgheM. et al.

    Spiking neural P systems: matrix representation and formal verification

    J. Membrane Comput.

    (2021)
  • ZhangG. et al.

    A complete arithmetic calculator constructed from spiking neural P systems and its application to information fusion

    Int. J. Neural Syst.

    (2021)
  • PǎunG.

    Spiking neural P systems with astrocyte-like control

    J. UCS

    (2007)
  • Cited by (43)

    • Time series classification models based on nonlinear spiking neural P systems

      2024, Engineering Applications of Artificial Intelligence
    View all citing articles on Scopus

    This work was partially supported by the National Natural Science Foundation of China (No. 62076206 and No. 62176216), China.

    View full text