Prediction for noisy nonlinear time series by echo state network based on dual estimation

Abstract

When using echo state networks (ESNs) to establish a regression model for noisy nonlinear time series, only the output uncertainty was usually concerned in some literature. However, the unconsidered internal states uncertainty is actually important as well. In this study, an improved ESN model with noise addition is proposed, in which the additive noises describe the internal state uncertainty and the output uncertainty. In terms of the parameters determination of this prediction model, a nonlinear/linear dual estimation consisting of a nonlinear Kalman filter and a linear one is proposed to perform the supervised learning. For verifying the effectiveness of the proposed method, the noisy Mackey Glass time series and the generation flow of blast furnace gas (BFG) in steel industry practice are both employed. The experimental results demonstrate that the proposed method is effective and robust for noisy nonlinear time series prediction.

Introduction

Theoretically, recurrent neural networks are universal approximators, and as such, have an excellent ability of approximating any nonlinear mapping to any degree of accuracy [1]. Echo state network, a kind of recurrent neural network, exhibits good performance for the prediction of nonlinear or non-Gaussian dynamic system [2]. The dynamic reservoir of ESN, instead of the hidden layer of generic neural network, involves a large number of sparsely connected neurons that shows a sound memory characteristic; furthermore, only the output connections need to be determined during the learning process, which simplifies the establishment of the network. Recently, such network had been successfully applied to time series prediction [3], [4], short-term load forecasting [5], signal processing [6], [7] and automatic control [8].

On the other hand, ESNs fundamentally suffer from two basic limitations despite its well popularity. First, the ill-conditioned solution associated with linear regression or recursive least squares method could hardly be avoided in the learning process, and such solution might deviate from the real system. As mentioned in [9], the large output weights impaired the generalization capability of the model that means the model input slightly deviated from the training data such that the relatively poor results would occur. In addition, the model with large weights might lead to the lack of stability when ESN features the output feedback. The second issue is the unsatisfactory low performance when the sample uncertainties involved. Since the core of ESN lies in the dynamic reservoir driven by the external input, the update of internal states often accompany with the uncertainty from input. Thus, an inaccurate target value might be obtained. In practice, the uncertainty in real-world dynamic system is fairly prevalent; for example, consider the situation where the various noises are often introduced into an industrial system by sensors. The low accuracy of ESNs in industrial time series prediction was addressed in [10], and the similar demonstrations had been reported in other fields such as sea clutter prediction [11], nonlinear system modeling [12], and nonlinear filtering [13].

To avoid the ill-condition, the eigenvalue spread of the correlation matrix of reservoir activation signals was proposed in [9], where the ill-condition phenomenon was prevented by adding noise to the reservoir during training so as to promote the stability and robustness of the network. However, this method failed to avoid the ill-condition thoroughly and the additive noise might impair the prediction accuracy. Subsequently, the singular value decomposition [10], the biologically motivated learning method [12] and the swarm intelligence optimization [13] were adopted to train the network. All of these methods avoided the ill-condition and improved the accuracy for noiseless time series more or less, but as for the noisy nonlinear series, the quality of these methods had not been proved or demonstrated. In [10] presented, the empirical mode decomposition was used for noise reduction; yet, the noise-reduced sample still comprised of the uncertainties. In [14] reported, the Bayesian method that focused on the distribution of output weights was employed to perform the network training. Although the prediction accuracy with the Bayesian based ESN was improved thanks to the considered output uncertainty, the uncertainty of the internal states resulted from the intrinsic sample was ignored. In addition, a kind of decoupled ESN based on the thought of multiple reservoirs was proposed in [11] and [15], in which those methods were somewhat suitable for complex problems, but the internal states uncertainty was also not mentioned.

In this study, an improved ESN model with noise addition is proposed to predict the noisy nonlinear time series, in which the uncertainties from internal states and outputs are meanwhile considered in accordance with the industrial practice. For the optimal model parameters, a nonlinear/linear dual estimation method is designed, in which a nonlinear Kalman filter serves to estimate the internal states, and a linear one estimates the output weights. The contribution of this paper lies in the following three aspects. First, the ill-conditioned solution could be avoided by using the dual estimation for the parameters determination. Second, the implementation process of the proposed method is much easier than that of the classical dual estimation based RNN. Finally, due to the consideration of internal states uncertainty, the accuracy and the robustness of the model are greatly enhanced. To demonstrate the accuracy and the robustness of the proposed method, the standard and noisy Mackey-Glass time series are first employed as the testing examples. And, the proposed method is further adopted to predict the generation flow of blast furnace gas in steel industry. The experimental results indicate the proposed method presents a good performance for the prediction of noisy industrial time series.

The rest of the paper is organized as follows. Section 2 presents an improved ESN model with additive noise for prediction and elaborates how to determine the parameters of this model. In Section 3, several types of nonlinear Kalman filters are introduced to estimate the internal states of the established model. And, the corresponding output weights estimation is described in Section 4. Section 5 carries out the two classes of simulation experiments to verify the effectiveness of the proposed method. Finally, we summarize the paper and give the future work in Section 6.