Gradient-based iterative identification method for multivariate equation-error autoregressive moving average systems using the decomposition technique

doi:10.1016/j.jfranklin.2018.12.002

Journal of the Franklin Institute

Volume 356, Issue 3, February 2019, Pages 1658-1676

https://doi.org/10.1016/j.jfranklin.2018.12.002 Get rights and content

Abstract

This paper studies the parameter estimation problems of multivariate equation-error autoregressive moving average systems. Firstly, a gradient-based iterative algorithm is presented as a comparison. In order to improve the computational efficiency and the parameter estimation accuracy, a decomposition-based gradient iterative algorithm is presented by using the decomposition technique. The key is to transform an original system into two subsystems and to estimate the parameters of each subsystem, respectively. Compared with the gradient-based iterative algorithm, the decomposition-based algorithm requires less computational efforts, and the simulation results indicate that this algorithm is effective.

Introduction

System identification is the foundation of constructing the mathematical models of systems from input–output data [1], [2], and parameter estimation is essential for system identification [3], [4]. So exploring new parameter estimation methods is an eternal theme of system identification [5], [6], and many identification methods have been applied in different fields, such as system control [7], information filtering [8] and signal processing [9]. As a consequence of the wide variety of applications, some identification methods have been reported in the literature. For example, Chaudhary et al. exploited a strength of momentum least mean square algorithm for nonlinear system identification problems represented with Hammerstein model [10], [11]. Yu and Verhaegen studied the identification of a network comprised of interconnected clusters of linear time-invariant systems and a subspace identification method was proposed for identifying each single cluster using only local input and output data [12]. Gan et al. presented a variable projection approach for estimation of radial basis function network-based autoregressive with exogenous inputs models by using the orthogonal projection [13].

Parameter estimation methods can be applied to many areas [14], [15], [16], [17]. Since multivariable systems widely exist in practical control systems, especially in modern large-scale industrial systems, it has drawn a great deal of attention of many researchers in multivariable system identification. At the same time, a series of methods have been proposed for multivariable systems, including gradient-based methods [18], the least squares methods [19], [20]. Different from the signal-input signal-output systems, multivariable systems has the characteristics of complex structure, multiple variables and non-linearity [21]. In addition, the interaction between multiple inputs and outputs of multivariate systems results in the complexity and coupling of the structures [22]. An effective method to identify multivariate systems is to use the decomposition technique [23]. The decomposition can transform a large-scale system into several small subsystems with fewer variables, thus the dimensions of the involved covariance matrices in each subsystem become smaller compared with the original system. By means of the decomposition technique, the computational burden of identification algorithms can be effectively reduced, and the computational efficiency can also be improved. Recently, Ma et al. proposed a recursive least squares identification method for multivariate pseudolinear systems using the decomposition technique and the multi-innovation theory [24]. Ding developed a hierarchial gradient based iterative algorithm and a hierarchical least squares based iterative algorithm for the identification problem of multi-input-output-error autoregressive systems by using the hierarchial identification principle [25]. The mentioned algorithms can give more accurate parameter estimates compared with the recursive or iterative based algorithms. However, in order to obtain convergent and consistent estimates, these methods are inevitably dependent upon the accuracy of the noise model.

In recent years, the iterative identification methods play an important role in the field of system identification because it can make full use of all input–output data. These methods have been successfully applied to state space systems [26], linear systems and nonlinear systems [27]. The iterative identification algorithms can be derived by means of defining and minimizing an output error criterion function, and many iterative identification algorithms were reported for different systems. For example, Xu and Ding proposed the Newton iterative algorithm and the gradient iterative algorithm for the parameter estimation problems of dynamical systems by means of the impulse response measurement data [28]. Li and Liu used the filtering technique and iterative methods to study the parameter estimation problems of a bilinear system with autoregressive noise [29]. Liu and Alleyne presented an iterative learning identification approach for the parameter estimates of linear time-varying systems [30]. However, these methods are usually for single-input single-output systems. The issue of how to efficiently apply the iterative identification methods and the decomposition technique for multivariate equation-error autoregressive moving average systems is a potential area of research. There are the motivation factors for the authors to investigate in exploring the identification problem of multivariate systems.

Different from the previous work in [31] which researches the recursive least squares identification methods, this paper uses the gradient-based iterative identification methods based on the decomposition technique. Compared with the recursive algorithm, the iterative algorithm uses a batch of data to refresh parameter estimation and improves identification accuracy. The key of the decomposition technique is to transform a multivariate equation-error autoregressive moving average system into two subsystems, where one contains a system model parameter vector and a measured information matrix, and the other contains a noise model parameter matrix and an unmeasured information vector, and then to estimate each subsystem by using the negative gradient search. The main contributions of this paper are as follows.

•
A decomposition-based gradient iterative (D-GI) algorithm is developed for the multivariate equation-error autoregressive moving average system based on the decomposition technique.
•
The D-GI identification method can effectively reduce the computational burden compared with the gradient-based iterative (GI) algorithm.
•
The proposed D-GI algorithm can generate more accurate parameter estimates and has higher computational efficiency than the GI algorithm.

The proposed algorithms in this paper can combine the multi-innovation method [32], the neural network methods [33] and the kernel methods [34], [35] to study parameter identification of different systems [36], [37].

The rest of this paper is organized as follows. Section 2 describes an identification model of multivariate equation-error autoregressive moving average systems. Section 3 introduces the GI algorithm for comparisons. Section 4 presents a D-GI algorithm. Section 5 gives the convergence analysis of the two proposed algorithms. Section 6 provides a simulation example to show the effective of the proposed algorithms in this paper. Finally, we offer some concluding remarks in Section 7.

Section snippets

System description and identification model

Let us introduce some notations first.

Symbols	Meaning
$1_{n}$ :	An $n$ -dimensional column vector whose entries are all 1.
$I$ or $I_{n}$ :	The identity matrix of appropriate sizes or $n \times n$ .
$tr [X]$ :	The trace of the square matrix $X$ .
$X^{T}$ :	The transpose of the vector or matrix $X$ .
$\| X \|$ :	The determinant of the square matrix $X$ : $\| X \| : = \det [X]$ .
${∥ X ∥}^{2}$ :	The norm of a matrix $X$ : ${∥ X ∥}^{2} : = tr [X X^{T}]$
$A = : X$ :	$X$ is defined by $A$ .
$X : = A$ :	$X$ is defined by $A$ .
${\hat{θ}}_{k}$ :	The estimate of $θ$ at iteration $k$ .

Consider the following multivariate equation-error autoregressive

The gradient-based iterative algorithm

For the purpose of showing the advantage of the proposed algorithm, we give a gradient-based iterative (GI) algorithm for comparison in this section. Assume that L is the data length. According to the identification model in Eq. (7), consider the input–output data from $t = 1$ to $t = L,$ and define a static criterion function $J (ϑ) : = \frac{1}{2} \sum_{t = 1}^{L} {∥ y (t) - Γ (t) ϑ ∥}^{2} .$ Define the stacked output vector Y(L) and the stacked information matrix Ψ(L) as $\begin{matrix} Y (L) : = [\begin{matrix} y (1) \\ y (2) \\ ⋮ \\ y (L) \end{matrix}] \in R^{m L}, Ψ (L) : = [\begin{matrix} Γ (1) \\ Γ (2) \\ ⋮ \\ Γ (L) \end{matrix}] \in R^{(m L) \times n_{0}} . \end{matrix}$ From Eq. (8),

The decomposition-based gradient iterative algorithm

Although the GI algorithm can improve the parameter estimation accuracy, the disadvantage is that it needs heavy computational load for large-scale systems. In order to improve the computational efficiency of the GI algorithm, we derive a decomposition-based gradient iterative (D-GI) algorithm in this section.

Introduce two intermediate variables: $\begin{matrix} y_{1} (t) & : = & y (t) - Φ_{n} (t) θ_{n}, \end{matrix}$ $\begin{matrix} y_{2} (t) & : = & y (t) - Φ (t) θ . \end{matrix}$ According to the hierarchical identification principle, the system in Eq. (6) can be decomposed into the

Convergence analysis

The convergence analysis of the two proposed algorithms is simply illustrated as follows.

Let $\hat{θ},$ ${\hat{Φ}}_{n} (t)$ and ${\hat{θ}}_{n}$ represent the estimates of θ, Φ_n(t) and θ_n, respectively. The output prediction is given by $\hat{y} (t) = Φ (t) \hat{θ} + {\hat{Φ}}_{n} (t) {\hat{θ}}_{n} .$ Define the prediction error optimization criterion $J (\hat{ϑ}) = \frac{1}{L} \sum_{t = 1}^{L} {∥ \hat{y} (t) - y (t) ∥}^{2} .$ According to the methods in [50], suppose that the information vectors Φ(t) and ${\hat{Φ}}_{n} (t)$ are persistently exciting. Obviously, the independent and identically distributed noise assumption on the

Example

Consider the following multivariate equation-error autoregressive moving average system: $\begin{matrix} y (t) & = & Φ (t) θ + \frac{D (z)}{C (z)} v (t), \\ Φ (t) & = & [\begin{matrix} - y_{1} (t - 1) & y_{1} (t - 2) \sin (y_{2} (t - 2)) & y_{2} (t - 1) & y_{2} (t - 1) u_{1} (t - 2) & u_{1} (t - 2) u_{2} (t - 2) \\ - y_{1} (t - 1) & y_{1} (t - 2) \sin (t / π) & y_{2} (t - 1) & y_{1} (t - 2) u_{2} (t - 2) & \sin (u_{2} (t - 2)) \end{matrix}] \in R^{2 \times 5}, \\ C (z) & = & 1 + c_{1} z^{- 1} + c_{2} z^{- 2} = 1 + 0.51 z^{- 1} - 0.14 z^{- 2}, \\ D (z) & = & 1 + d_{1} z^{- 1} + d_{2} z^{- 2} = 1 + 0.37 z^{- 1} + 0.25 z^{- 2}, \\ θ & = & {[θ_{1}, θ_{2}, θ_{3}, θ_{4}, θ_{5}]}^{T} = {[0.85, 0.45, 1.00, 0.23, - 0.30]}^{T}, \\ θ_{n} & = & {[c_{1}, c_{2}, d_{1}, d_{2}]}^{T} = {[0.51, - 0.14, 0.37, 0.25]}^{T}, \\ ϑ & = & {[θ^{T}, θ_{n}^{T}]}^{T} . \end{matrix}$

In simulation, the inputs u₁(t) and u₂(t) are taken as two independent

Conclusions

This paper discusses the parameter estimation problems of multivariate equation-error autoregressive moving average systems and derives a D-GI algorithm based on the decomposition technique. Compared with the GI algorithm, the proposed D-GI algorithm requires less computational effort, and has higher accuracy than the GI algorithm. The simulation example is given to show the effectiveness of the D-GI algorithm. In addition, the proposed algorithms can be combined with other identification

References (58)

S. Zubair et al.
Momentum fractional LMS for power signal parameter estimation
Signal Process.
(2018)
N.I. Chaudhary et al.
Design of normalized fractional adaptive algorithms for parameter estimation of control autoregressive autoregressive systems
Appl. Math. Model.
(2018)
J. Na et al.
Robust adaptive parameter estimation of sinusoidal signals
Automatica
(2015)
N.I. Chaudhary et al.
A new computing approach for power signal modeling using fractional adaptive algorithms
ISA Trans.
(2017)
Y. Cao et al.
Parallel processing algorithm for railway signal fault diagnosis data based on cloud computing
Future Gener. Comput. Syst.
(2018)
F. Ding et al.
A hierarchical least squares identification algorithm for Hammerstein nonlinear systems using the key term separation
J. Frankl. Inst.
(2018)
X. Zhang et al.
Combined state and parameter estimation for a bilinear state space system with moving average noise
J. Frankl. Inst.
(2018)
M.H. Li et al.
The least squares based iterative algorithms for parameter estimation of a bilinear system with autoregressive noise using the data filtering technique
Signal Process.
(2018)
F. Ding
Several multi-innovation identification methods
Digital Signal Process.
(2010)
F.Z. Geng et al.
An optimal reproducing kernel method for linear nonlocal boundary value problems
Appl. Math. Lett.
(2018)

Y.Z. Zhang et al.

Optimization of information interaction protocols in cooperative vehicle-infrastructure systems

Chin. J. Electron.

(2018)

Cited by (34)

Hierarchical iterative identification of output nonlinear Box-Jenkins Wiener model with ARMA noise
2023, ISA Transactions
This paper focuses on development of a three-stage iterative identification algorithm for parameter estimation of a Wiener model with autoregressive moving average (ARMA) noise entering between linear and nonlinear part. The proposed three-stage algorithm is developed based on generalized extended gradient iterative (GEGI) algorithm to increase the convergence rate at a low number of iterations. To increase the convergence rate, a second algorithm is derived based on generalized extended least squares algorithm (GELSI). The proposed 3-stage generalized extended least square iterative (3S-GELSI) algorithm will also decrease the computational burden compared to the single-stage algorithm. Finally, to show that the discussed methods can identify the considered system effectively, simulation results of three examples are provided. The results demonstrate that the 3-stage GEGI (3S-GEGI) algorithm outperforms its basic single-stage counterpart in terms of convergence rate. Moreover, the proposed 3S-GELSI algorithm exhibits even greater superiority in terms of convergence rate compared to other existing methods. Additionally, it will be demonstrated that the computational burden decreases in the 3S-GELSI algorithm, as indicated by the reduced number of total flops.
A new multivariate equation-error autoregressive moving average system with conditional heteroscedastic noise: Maximum likelihood identification
2021, Digital Signal Processing: A Review Journal
Citation Excerpt :
In the past decade, the MIMO system identification researches have mainly concentrated on improving parameter estimation accuracy and computational efficiency. For example, hierarchical least-squares identification [7], adaptive gradient-based iterative algorithm [8] or decomposition-based techniques [9–11] are used to transform a large-scale system into several small subsystems with fewer variables. However, the proposed MIMO models are not very realistic and fail to appropriately reflect the characteristics of actual MIMO systems.
The aim of this paper is to deal with the problem of conditional heteroscedastic noise in multivariate systems. In this regard, a new multivariate equation-error system with colored noise is introduced, in which the noise conditional variance varies with time and the noise exhibits a GARCH process. Thus, in this our new approach, both the time dependency and probability distribution of noise samples shall be identified. Based on the maximum likelihood principle and orthogonality of parameters, the problem of multivariate system identification is reduced into two separate maximum likelihood problems. An iterative algorithm is proposed, which can efficiently estimate the problem parameters based on gradient reduction and nonlinear optimization. Besides, a bootstrap algorithm is applied for assessing the accuracy of estimators. Also, some asymptotic result of proposed method in large sample size is provided. According to bootstrap simulation results, the proposed algorithm can effectively estimate the parameters of the system with conditional heteroscedastic noise and provides more accurate parameter estimates with a much lower confidence interval, compared to the least square method.
Robust adaptive identification for sandwich systems with unknown time-delay
2020, ISA Transactions
A discrete time, robust adaptive estimator is developed to identify the time-delay and sandwich systems parameters. To obtain explicitly the expression of delay parameter, the observation and augmented data are reconstructed. By using the filter operator and some auxiliary vectors, the parameter identification error vector is derived. Based on the parameter identification error term and initial estimation error term, a novel criterion function is invented. In comparison to the common criterion function, a classy estimation property is provided based on the criterion function in that paper because the identification error term can lift estimation accuracy, and the initial estimation error term speeds up the convergence. Under the persistent excitation condition, convergence of the developed estimator is analyzed. The availability and superiority of proposed identification scheme are verified by both numerical simulation and a turntable servo system.
Robust hierarchical identification of Wiener systems in the presence of dynamic disturbances
2020, Journal of the Franklin Institute
This paper is concerned with robust identification of Wiener systems in the presence of dynamic disturbances and stochastic noises. Since conventional statistical method cannot eliminate the dynamic disturbance. To solve this problem, the dynamic disturbance signal is modelled as a self-excitation time-varying parameters to be estimated by tracking strategy. Using the multi-innovation and auxiliary model scheme, a hierarchical recursive least squares algorithm with forgetting factors is designed in discrete-time domain. To improve the parameter estimation accuracy and decrease the error variance, the multi-innovation strategy is used to estimate the time-invariant system parameters. The dynamic disturbance still uses the single innovation method for quick tracking. The estimation error upper bound with finite sample data and asymptotic convergence properties are analyzed. Some guidelines are suggested to help the choice of multi-innovation length. The proposed parameterized identification facilitates controller design and system performance analysis. The effectiveness and superiority of the proposed algorithm are confirmed by utilizing theoretical analysis and numerical examples.
Generalization of the gradient method with fractional order gradient direction
2020, Journal of the Franklin Institute
Citation Excerpt :
Gradient method, as a classical tool in optimization theory, because of its intuitive principle, simple structure and easy implementation, has attracted much attention and developed rapidly since its inception. Now it prevalently used in many fields, such as automatic control [1], system identification [2], machine learning [3], and image denoising [4]. Besides, many effective algorithms are based on it or inspired by it [5].
Fractional calculus is an efficient tool, which has the potential to improve the performance of gradient methods. However, when the first order gradient direction is generalized by fractional order gradient one, the corresponding algorithm converges to the fractional extreme point of the target function which is not equal to the real extreme point. This drawback critically hampers the application of this method. To solve such a convergence problem, the current paper analyzes the specific reasons and proposes three possible solutions. Considering the long memory characteristics of fractional derivative, short memory principle is a prior choice. Apart from the truncation of memory length, two new methods are developed to reach the convergence. The former is the truncation of the infinite series, and the latter is the modification of the constant fractional order. Finally, six illustrative examples are performed to illustrate the effectiveness and practicability of proposed methods.
Fitting the exponential autoregressive model through recursive search
2019, Journal of the Franklin Institute
Citation Excerpt :
This paper aims to derive new recursive identification algorithms for the ExpAR model. The gradient-based identification is an essential parameter estimation method [34–36]. Recently, Wang and Ding derived a gradient-based iterative algorithm for nonlinear systems with moving average noise [37].
This paper focuses on the recursive parameter estimation methods for the exponential autoregressive (ExpAR) model. Applying the negative gradient search and introducing a forgetting factor, a stochastic gradient and a forgetting factor stochastic gradient algorithms are presented. In order to improve the parameter estimation accuracy and the convergence rate, the multi-innovation identification theory is employed to derive a forgetting factor multi-innovation stochastic gradient algorithm. A simulation example is provided to test the effectiveness of the proposed algorithms.

View all citing articles on Scopus

^☆: This work was supported by the National Natural Science Foundation of China (No. 61873111), and the 111 Project (B12018) and the National First-Class Discipline Program of Light Industry Technology and Engineering (LITE2018-26).

View full text

Gradient-based iterative identification method for multivariate equation-error autoregressive moving average systems using the decomposition technique☆

Abstract

Introduction

Section snippets

System description and identification model

The gradient-based iterative algorithm

The decomposition-based gradient iterative algorithm

Convergence analysis

Example

Conclusions

Signal Process.

Appl. Math. Model.

Automatica

ISA Trans.

Future Gener. Comput. Syst.

J. Frankl. Inst.

J. Frankl. Inst.

Signal Process.

Digital Signal Process.

Appl. Math. Lett.

Appl. Math. Lett.

Automatica

J. Math. Anal. Appl.

J. Comput. Appl. Math.

Insur. Math. Econ.

Signal Process.

IEEE Trans. Cybern.

Subspace identification of local systems in one-dimensional homogeneous networks

IEEE Trans. Control Syst. Technol.

Iterative parameter estimation for signal models based on measured data

Circuits Syst. Signal Process.

State filtering-based least squares parameter estimation for bilinear systems using the hierarchical identification principle

IET Control Theory Appl.

Improving transient performance of adaptive control via a modified reference model and novel adaptation

Int. J. Robust Nonlinear Control

Adaptive gradient-based iterative algorithm for multivariate controlled autoregressive moving average systems using the data filtering technique

Complexity

Design of momentum LMS adaptive strategy for parameter estimation of hammerstein controlled autoregressive systems

Neural Comput. Appl.

Modified volterra LMS algorithm to fractional order for identification of hammerstein nonlinear system

IET Signal Process.

Subspace identification of distributed clusters of homogeneous systems

IEEE Trans. Autom. Control

A variable projection approach for efficient estimation of RBF-ARX model

IEEE Trans. Cybern.

Standard analysis for transfer delay in CTCS-3

Chin. J. Electron.

Performance evaluation with improved receiver design for asynchronous coordinated multipoint transmissions

Chin. J. Electron.

Optimization of information interaction protocols in cooperative vehicle-infrastructure systems

Chin. J. Electron.