Predicting chaotic time series and replicating chaotic attractors based on two novel echo state network models

Abstract

Neural network is the inevitable outcome of the rapid development of artificial intelligence. Based on the idea of homotopy and combined activation function, two novel echo state network (ESN) models are proposed. Compared with several activation functions commonly used in the neural network, the proposed models provide intuitive but effective approaches to chaotic forecasting, and the prediction accuracy is higher. Secondly, for the Mackey-Glass (MG) time series and Rössler attractor, the prediction errors and prediction step sizes of the two novel models are superior to many pre-existing ESN models, which demonstrates the merit of the proposed models. Moreover, several parameters play key roles in network training, such as spectral radius, sparse degree etc., and their effects on network performance are analyzed. Notably, it is also investigated that the trained network can replicate Rössler chaotic attractor well. At the end of the results, the parameters of the proposed models are optimized, and the relatively optimized parameters are obtained after a large number of data experiments.

Introduction

As an important role of nonlinear science research, chaos phenomenon has extremely important implications and application value. Chaotic systems exhibit very complex and interesting dynamical behaviors, such as chaotic attractors, Hopf branches, period-doubling branches and homoclinic orbits. Although chaos has been known for nearly half a century, it is a pity that there is still no universally accepted and strict mathematical definition of chaos until today. Nevertheless, the general characteristics of chaos have been summarized: the sensitive dependence on initial value, long-term unpredictability, Lyapunov index greater than 0 etc. The Lorenz system [1] is the first physical and mathematical model of chaos in the history of chaos evolution, which is an important milestone. Over the years, some more complex and diverse chaotic systems have been artificially constructed, like Rössler system [2], Chua system [3], Chen system [4].

With the rapid development of artificial intelligence today, the application of artificial intelligence to deal with chaos related problems has become a topic of great concern in dynamics. The problems associated with chaos are prediction of chaotic time series, reconstruction of chaotic attractor, chaos control. The use of artificial intelligence tools for example neural networks is one of the alternatives in this field that is increasingly used by researchers in different engineering disciplines. Artificial neural network (ANN) is utilized to simulate the structure and function of biological neural network [5]. Owing to the fast-growing contemporary neurobiology, mathematics, physics, and computer science, a large number of artificial neural network models have been proposed, and the research has gradually become a focus [6], [7], [8], such as the McCullochPitts neuron model [9], linear threshold function neuron model [10], [11], backpropagation neural network model [12], [13], [14], recurrent neural network (RNN) model [15], [16], [17].

The reason why RNN is more suitable for human brain simulation is that RNN has a better ability to describe the dynamic properties of the system, compared with the feed forward neural network [18]. Under normal circumstances, RNNs can infinitely approach any complex nonlinear dynamic system. These advantages make RNN widely used in prediction [19], nonlinear system identification [20], and adaptive filtering [21]. Nonetheless, some inevitable problems still exist in RNNs, such as, the network structure and training algorithm are complex, the computation is large, the convergence speed is slow, and the error gradient may disappear or produce distortion with the increase of training steps. Some improvements have been achieved in recent years [22], [23], [24]. Gradient vanishing phenomenon is common but difficult to overcome in neural network training. Nonetheless, with the further development of artificial intelligence, experts have proposed some solutions to the gradient vanishing problem. In 1998, Hochreiter theoretically analyzed the gradient vanishing problem of recurrent neural networks, but the improved algorithm also has some drawbacks (e.g. practicable only for discrete problems) [25]. Hinton mentioned in 2006 that in order to improve the problem of gradient vanishing, unsupervised layer-by-layer training is adopted. After the pre-training layer-by-layer is complete, the entire network is “fine-tuned”. This method has certain benefits, but it is not used much now [26]. In recent years, a new multilayer neural network was proposed that adds linear neurons, which is easier to train than traditional MLP networks and reduces the effects of gradient vanishing [27]. Recurrent identity network (RIN), which allows a plain recurrent network to overcome the gradient vanishing problem while training very deep models without the use of gates, was proposed in 2019. On multiple sequence modeling benchmarks, the RIN model demonstrates competitive performance and converge faster in all tasks [28].

As mentioned above, gradient disappearance brings indelible problems to the training of neural networks. However, Jaeger proposed ESN in 2001 [29], and the training process of this model is greatly different from that of traditional RNN. Training ESN model does not need to use gradient to update parameters, but use the state variables sampled during network training and linear output to get the optimal parameters. So ESN model, in a sense, avoids the problem of local minima and gradient disappearance. The merits of ESN are that the process of training considerably reduced the amount of computation, overcame the problem of memory fading, and also solved the structural defects of the traditional RNN. The most significant difference between ESN and traditional RNNs lies in the structure of reservoir layer. In 2002, ESN and liquid state machine (LSM) are collectively referred to as reservoir computing [30]. The weights of each layer in traditional RNNs are computed by the gradient descent algorithm. While in ESN, only the weights between reservoir layer and output layer need to be trained, which greatly reduces the calculation amount and time of the traditional RNN and makes the calculation efficiency higher. Gradually, based on these unique advantages, ESN has attracted increasing attention in the field of neural networks [31], [32]. Not long after ESN model was proposed, it has been employed to chaotic time series prediction. For the reconstruction attractor, notably, in 2017, Pathak, Lu et al. used ESN methods to replicate spatiotemporal chaotic attractors and calculate the Lyapunov exponents from data [33]. The paper caused a stir in the field of chaos and was immediately reported by the major media. For the time series data of actual production and life, the novel ESN were proposed, the calculation amount are greatly reduced and the prediction accuracy was improved for the task of wind speed and wind direction forecasting [34]. In addition, ESN has also been applied in the field of handwritten digit recognition [35].

In this paper, two novel ESN models are proposed based on homotopy and combination activation function, and successfully applied to chaotic time series. The rest of this paper is organized as follows. In Section 2, the Leakage Integral Echo State Network (LIESN), the proposed Homotopy Activation Function Leakage Integral Echo State Network (HAF-LIESN) and Combinatorial Activation Function Leakage Integral Echo State Network (CAF-LIESN) models are briefly introduced. Moreover, how to train the output weight is given by a detailed derivation process. We apply the proposed models to the prediction of MG chaotic time series and Rössler chaotic time series in Section 3. Meanwhile, the evaluate methods are mean squared error (MSE), root mean squared error (RMSE) and measures average error (MAE). Several parameters play key roles in network training, such as spectral radius, sparse degree etc, and Section 3 also investigates their effects on network performance. What is more, the Rössler chaotic attractor is replicated. In view of the two models proposed in this paper, the parameter optimization of the model is discussed in Section 3.3, which leads to better network performance. The final section summarizes our contributions and gives some suggestions for future works.