Time series prediction with input noise based on the ESN and the EM and its industrial applications

Highlights

• An ESN model with input noise is proposed for noisy time series prediction.

• The EM algorithm is designed to update all uncertain parameters iteratively.

• The experimental results show good performance of the proposed method.

Abstract

Industrial time series data usually have a high noise. In this paper, an echo state network (ESN) model with input noise is proposed to address the problem of predicting time series with noise. In the ESN, the introduction of the input noise makes it difficult to accurately estimate the non-linear states of the dynamical reservoir, therefore, in this study, the states are approximated by linearizing it through an extended Kalman filter (EKF). For the learning of the model parameters, the expectation maximization algorithm (EM) is used to iteratively update all the uncertain parameters to construct the prediction intervals, where the state estimation is performed using a forward back algorithm. To verify the effectiveness of the proposed method, two benchmark data sets and three real gas data sets from steel enterprises are used in this paper. Experimental results show that the prediction accuracy of the proposed method is better than that of the existing methods.

Introduction

Industrial enterprises generally rely on supervisory control and data acquisition (SCADA) systems to collect real-time data. These industrial data contain huge information about the production processes. Accurate prediction of industrial time series data can provide guidance for the subsequent optimization and scheduling for the industrial processes. Due to the non-linear, unstable and strongly coupled nature of the actual production process, it is difficult to model mechanisms with high reliability (Zhou, Yang, & Gui, 2015). As a result, a class of data-based modeling methods has recently emerged. Data-based modeling methods model the relationship between input and output variables by using data.

The machine learning models are widely used for time series prediction, such as the relevance vector machine (RVM) (Chen, Wang, Zhao, & Wang, 2019), the least square support vector machine (LS-SVM) (Guo, He, Jiang, Chu, Malekian, & Li, 2018), the Gaussian process regression (GPR) (Zhang et al., 2019), and the neural network (Al-Saati, Omran, Salman, Al-Saati, & Hashim, 2021). To deal with the nonlinearity of the time series, many model were developed, such as the self-organizing pipelined recurrent wavelet neural network (Su, Yang, & Qiao, 2023), the REMD-MMLP with temporal-window (Yang, & Fan, 2022), the compete ensemble empirical mode decomposition with adaptive noise (Feng, et al., 2022), and the double multiplicative neuron (DMN) model with modified particle swarm optimization (Pan, et al., 2021). Particularly, the recurrent neural network (RNN) showed better performance in nonlinear time series prediction due to its excellent nonlinear approximation ability (Wei et al., 2022). However, its application is greatly limited due to its characteristics of large amount of calculation, slow solution speed, and local optima. ESN, firstly proposed by Jaeger (Jaeger, 2001), is a new type of RNN (Yang et al., 2018, Xu et al., 2019). Compared with the traditional RNN, the ESN overcame the disadvantage of local optimum by simplifying the network design and training process.

Due to the complex production environment, it is inevitable to collect noisy industrial data, which will cause uncertainty in the time series prediction model (Thiagarajan and Madden, 2008, Lu et al., 2017). For the ESN, the output uncertainty was usually concerned in some literature (Li et al., 2012, Han et al., 2020). However, it is not enough to only consider the output uncertainty. Since the ESN is a three-layer neural network, the internal state uncertainty and the input uncertainty should be also considered. Recently, Sheng et al. (Sheng, Zhao, & Ying, 2012) reported an improved ESN model with the internal state noise and output noise, where a nonlinear/linear dual estimation consisting of a nonlinear Kalman filter and a linear one was used to perform the supervised learning. The experimental results showed that the proposed method was effective and robust for noisy and chaotic time series prediction. For time series prediction, the input and output are both constructed from the same series, thus, the input and output noise should be considered simultaneously when training a prediction model.

At present, the input noise was considered in many time series prediction models. McHutchon and Rasmussen (McHutchon, & Rasmussen, 2011) reported a Gaussian process regression model (NIGP) that takes into account input noise, where local linearization is performed at each input point to approximate the posterior probability distribution of the GP. Lázaro-Gredilla and Titsias (Lázaro-Gredilla & Titsias, 2011) proposed a variational heteroscedastic Gaussian process regression (VHGPR) model, where the approximate posterior probability distributions of model parameters were calculated based on the heteroscedasticity assumption. Chen et al. (Chen et al., 2020) proposed a relevance vector machine model with uncertain inputs (RVMUI). In this work, a Gaussian approximation of the marginal likelihood was reported and the posterior distribution over the model weights was approximated with the Markov chain Monte Carlo (MCMC) algorithm. However, there are few methods related to the ESN considering input noise. Given that the ESN model exhibits very good performance on modeling the nonlinearity of time series, it is necessary to develop an input noisy version of ESN for time series prediction.

Although the output uncertainty and the internal state uncertainty are considered in the existing ESN-based models, the input uncertainty is still inevitable in an ESN. In this study, an ESN model with uncertain input (ESNUI) is proposed for time series prediction, where the internal state uncertainty and the output uncertainty are also considered. The contributions are list as follows.

• Since the introduction of input noise into the ESN makes it difficult to accurately estimate the non-linear state of the dynamic repository, the state of the dynamic repository is linearized and a Gaussian approximation is made to the input variables after the non-linear mapping.

• For learning model parameters, the Expectation-Maximization (EM) algorithm is employed to update all uncertain parameters iteratively. The forward–backward algorithm is used to predict the state of the dynamical reservoir in each iteration process.

• To verify the effectiveness of this method, two standard data sets and three industrial data sets are employed respectively. The experimental results show that the proposed method performs better than the compared methods.

This paper is structured as follows. Section 2 provides the preliminaries of the ESN. Section 3 describes the proposed ESNUI model and its derivation in detail. Section 4 shows the experimental analysis of the approach. Finally, Section 5 draws a concise conclusion.