Maximum likelihood recursive least squares estimation for multivariate equation-error ARMA systems

doi:10.1016/j.jfranklin.2018.07.041

Journal of the Franklin Institute

Volume 355, Issue 15, October 2018, Pages 7609-7625

https://doi.org/10.1016/j.jfranklin.2018.07.041 Get rights and content

Abstract

This paper focuses on the parameter estimation problems of multivariate equation-error systems. A recursive generalized extended least squares algorithm is presented as a comparison. Based on the maximum likelihood principle and the coupling identification concept, the multivariate equation-error system is decomposed into several regressive identification models, each of which has only a parameter vector, and a coupled subsystem maximum likelihood recursive least squares identification algorithm is developed for estimating the parameter vectors of these submodels. The simulation example shows that the proposed algorithm is effective and has high estimation accuracy.

Introduction

For a long time, system identification and model parameter estimation have been applied in control and signal processing [1], [2], [3]. Most control laws are designed based on the mathematical models [4], [5], [6]. Parameter estimation is used in system and signal modeling [7], [8], [9]. Due to the complex structures, input and output variables, uncertain interference signals and high dimension of multivariate systems, they have become a hot issue [10], [11], [12]. For multivariate systems, Wang et al. provided a hierarchical extended stochastic gradient algorithm for nonlinear multi-input multi-output Hammerstein systems [13]; Ma et al. presented a decomposition-based recursive generalized least squares algorithm for multivariate pseudo-linear autoregressive systems [14].

The parameter estimation methods have wide applications in many areas and play an important role in system identification [15], [16] and signal processing [17], [18], [19]. Different from the least squares [20], [21], [22] and the stochastic gradient methods [23], [24], [25], the maximum likelihood identification method needs to construct a likelihood function with respect to the observed data and the unknown parameters, and the parameter estimation can be obtained by maximizing the likelihood function [26], [27]. Chen et al. derived a filtering based maximum likelihood multi-innovation extended gradient algorithm for controlled autoregressive autoregressive moving average systems based on the maximum likelihood principle [28]. Li et al. proposed a maximum likelihood least squares based iterative algorithm for identifying the parameters of bilinear systems with colored noises [29]. Wang et al. extended the maximum likelihood estimation to multivariable controlled autoregressive moving average like systems in colored noise environment and proposed a recursive maximum likelihood identification identification algorithm [30].

The coupling identification concept is useful for simplifying the parameter estimation of the coupled parameter multivariable systems and can be applied other areas [31], [32], [33]. It is based on the coupled relationship of the parameter estimates between the subsystems of a multivariable system [31]. Recently, a coupled least squares algorithm has been proposed for multiple linear regression systems [32]. Huang et al. derived the coupled stochastic gradient parameter estimation algorithms by using the auxiliary model identification idea and the coupling identification concept [34].

This paper studies the parameter estimation for the multivariate equation-error systems. The main contributions of this paper are as follows.

•
The system is decomposed into several subsystems, each of which has only a parameter vector.
•
Based on the maximum likelihood principle and the coupling identification concept, a coupled subsystem maximum likelihood recursive least squares identification algorithm is derived.
•
The average value of the parameter estimates of these subsystems is taken to improve the accuracy of parameter estimation. Moreover, a recursive generalized extended least squares algorithm is derived as a comparison.

Briefly, the rest of this paper is organized as follows. Section 2 gives the identification model of a multivariate equation-error system. Section 3 presents a coupled subsystem maximum likelihood recursive least squares algorithm for the multivariate equation-error system. As a comparison, a recursive generalized extended least squares algorithm is given in Section 4. The numerical example is provided in Section 5 to show the effectiveness of the proposed algorithm. Finally, Section 6 offers some concluding remarks.

Section snippets

System description

Let us introduce some notation. The symbol I_m is an m × m identity matrix; 1_n is an n-dimensional column vector whose elements are 1; the superscript T denotes the matrix transpose; z represents unit forward shift operator: $z x (t) = x (t + 1)$ and $z^{- 1} x (t) = x (t - 1)$ .

Consider the following multivariate equation-error system: $A (z) y (t) = Φ_{s} (t) θ_{s} + \frac{D (z)}{C (z)} v (t),$ where $y (t) : = [y_{1} (t), y_{2} (t), \dots, y_{m} (t)]^{T} \in R^{m}$ is the system output vector, $Φ_{s} (t) \in R^{m \times n}$ is the measured information matrix consisting of the input-output data, $θ_{s} \in R^{n}$

The coupled subsystem maximum likelihood recursive least squares algorithm

In order to distinguish the estimation of $θ$ in each subsystem, we mark $θ_{j}$ as $θ$ in the jth subsystem. For a given data set of measurements $ϕ_{j L} : = [ϕ_{j} (1), ϕ_{j} (2), \dots, ϕ_{j} (L)]$ and $y_{j L} : = [y_{j} (1), y_{j} (2), \dots, y_{j} (L)]$ . Assume that $L_{j} (y_{j L} | ϕ_{j L}, θ_{j})$ $(j = 1, 2, \dots, m)$ is the likelihood function. The maximum likelihood estimate is obtained by maximizing the likelihood function, ${\hat{θ}}_{ML} = \underset{θ_{j}}{argmax} L_{j} (y_{j L} | ϕ_{j L}, θ_{j}), j = 1, 2, \dots, m .$ The likelihood function $L_{j} (y_{j L} | ϕ_{j L}, θ_{j})$ can be expressed as $\begin{matrix} L_{j} (y_{j L} | ϕ_{j L}, θ_{j}) & = & p (y_{j L} | ϕ_{j L}, θ_{j}) \\ = & p [y_{j} (L) | y_{j, L - 1}, ϕ_{j, L - 1}, θ_{j}] p [ \end{matrix}$

The recursive generalized extended least squares algorithm

As a comparison, we simply derive the recursive generalized extended least squares algorithm for the multivariate equation-error system. From Eq. (1), we can get the following equation for the multivariate equation-error system $y (t) + \sum_{i = 1}^{n_{a}} a_{i} y (t - i) = Φ_{s} (t) θ_{s} - \sum_{i = 1}^{n_{c}} c_{i} w (t - i) + \sum_{i = 1}^{n_{d}} d_{i} v (t - i) + v (t) .$ Transform the above equation into the following identification model $y (t) = φ (t) θ + v (t),$ where $\begin{matrix} ϑ & : = & [θ_{s}^{T}, a^{T}]^{T} \in R^{n + n_{a}}, \\ θ & : = & [ϑ^{T}, c^{T}, d^{T}]^{T} \in R^{n_{0}}, \\ ϕ (t) & : = & [Φ_{s} (t), - y (t - 1), - y (t - 2), \dots, - y (t - n_{a})] \in R^{m \times (n + n_{a})}, \\ ψ (t) & : = & [- w (t - 1), - w (t - 2), \dots, - w (t \end{matrix}$

Example

Consider the following multivariate equation-error model: $A (z) y (t) = Φ_{s} (t) θ_{s} + \frac{D (z)}{C (z)} v (t),$ $y (t) = [\begin{matrix} y_{1} (t) \\ y_{2} (t) \end{matrix}], Φ_{s} (t) = [\begin{matrix} Φ_{s 1} (t) \\ Φ_{s 2} (t) \end{matrix}], v (t) = [\begin{matrix} v_{1} (t) \\ v_{2} (t) \end{matrix}],$ $\begin{matrix} A (z) & = & 1 + 0.23 z^{- 1} + 0.90 z^{- 2}, C (z) = 1 + 0.62 z^{- 1}, \\ D (z) & = & 1 - 0.36 z^{- 1}, θ_{s} = [\begin{matrix} - 0.85 \\ 0.60 \end{matrix}] . \end{matrix}$

The parameter vectors to be identified are $\begin{matrix} a & = & [0.23, 0.90]^{T}, c = 0.62, d = - 0.36, \\ θ & = & [θ_{s}^{T}, a^{T}, c, d]^{T} = [- 0.85, 0.60, 0.23, 0.90, 0.62, - 0.36]^{T} . \end{matrix}$

In simulation, $y (t) = [\begin{matrix} y_{1} (t) \\ y_{2} (t) \end{matrix}] \in R^{2}$ is the output vector, $Φ_{s} (t)$ is a 2 × 2 matrix sequence, $v (t) = [\begin{matrix} v_{1} (t) \\ v_{2} (t) \end{matrix}] \in R^{2}$ is a white noise vector with zero mean, $σ_{1}^{2}$ and $σ_{2}$

Conclusions

The coupled subsystem maximum likelihood recursive least squares estimation algorithm is presented for multivariate equation-error systems by using the maximum likelihood principle and the coupling identification concept. The proposed algorithm is compared with the existing recursive generalized extended least squares algorithm. The simulation results indicate that the coupled subsystem maximum likelihood recursive least squares estimation algorithm is effective and has high estimation

References (56)

NaJ. et al.
Improving transient performance of adaptive control via a modified reference model and novel adaptation
Int. J. Robust Nonlinear Control
(2017)
NaJ. et al.
Robust adaptive parameter estimation of sinusoidal signals
Automatica
(2015)
GanM. et al.
A variable projection approach for efficient estimation of RBF-ARX model
IEEE Trans. Cybernet.
(2015)
GanM. et al.
On some separated algorithms for separable nonlinear squares problems
IEEE Trans. Cybernet.
(2018)
NaJ. et al.
Robust adaptive finite-time parameter estimation and control for robotic systems
Int. J. Robust Nonlinear Control
(2015)
XuL.
The damping iterative parameter identification method for dynamical systems based on the sine signal measurement
Signal Process.
(2016)
XuL. et al.
Hierarchical parameter estimation for the frequency response based on the dynamical window data
Int. J. Control Automat. Syst.
(2018)
ChenM.T. et al.
Iterative identification algorithms for bilinear-in-parameter systems with autoregressive moving average noise
J. Frankl. Inst.
(2017)
ZhangX. et al.
Combined state and parameter estimation for a bilinear state space system with moving average noise
J. Frankl. Inst.
(2018)
DingJ.L.
Recursive and iterative least squares parameter estimation algorithms for multiple-input-output-error systems with autoregressive noise
Circuits Syst. Signal Process.
(2018)

Cited by (11)

A new multivariate equation-error autoregressive moving average system with conditional heteroscedastic noise: Maximum likelihood identification
2021, Digital Signal Processing: A Review Journal
Citation Excerpt :
For the future research, the proposed algorithm can be integrated with other multivariate systems with different noise structures [7,12] or identification methods [10,36,37].
The aim of this paper is to deal with the problem of conditional heteroscedastic noise in multivariate systems. In this regard, a new multivariate equation-error system with colored noise is introduced, in which the noise conditional variance varies with time and the noise exhibits a GARCH process. Thus, in this our new approach, both the time dependency and probability distribution of noise samples shall be identified. Based on the maximum likelihood principle and orthogonality of parameters, the problem of multivariate system identification is reduced into two separate maximum likelihood problems. An iterative algorithm is proposed, which can efficiently estimate the problem parameters based on gradient reduction and nonlinear optimization. Besides, a bootstrap algorithm is applied for assessing the accuracy of estimators. Also, some asymptotic result of proposed method in large sample size is provided. According to bootstrap simulation results, the proposed algorithm can effectively estimate the parameters of the system with conditional heteroscedastic noise and provides more accurate parameter estimates with a much lower confidence interval, compared to the least square method.
Decomposition-based maximum likelihood gradient iterative algorithm for multivariate systems with colored noise
2024, International Journal of Robust and Nonlinear Control
A novel compositional data model for predicting the energy consumption structures of Europe, Japan, and China
2023, Environment, Development and Sustainability
Multi-innovation gradient estimation for multivariate systems with colored noise based on the maximum likelihood principle
2022, Research Square
Parameter and time delay estimation algorithm based on gradient pursuit for multi-input C-ARMA systems
2022, Kongzhi yu Juece/Control and Decision
Partially Coupled Stochastic Gradient Estimation for Multivariate Equation-Error Systems
2022, Mathematics

View all citing articles on Scopus

^☆: This work was supported by the Jiangsu Province Industry University Prospective Joint Research Project (BY2015019-29), the Fundamental Research Funds for the Central Universities (JUSRP51733B), the Graduate Education Innovation Program of Jiangsu Province (No. KYCX17\_1458), the Taishan Scholar Project Fund of Shandong Province of China (No. ts20130939), the 111 Project (B12018) and the National First-Class Discipline Program of Light Industry Technology and Engineering (LITE2018-026).

View full text

Maximum likelihood recursive least squares estimation for multivariate equation-error ARMA systems☆

Abstract

Introduction

Section snippets

System description

The coupled subsystem maximum likelihood recursive least squares algorithm

The recursive generalized extended least squares algorithm

Example

Conclusions

Improving transient performance of adaptive control via a modified reference model and novel adaptation

Int. J. Robust Nonlinear Control

Robust adaptive parameter estimation of sinusoidal signals

Automatica

A variable projection approach for efficient estimation of RBF-ARX model

IEEE Trans. Cybernet.

On some separated algorithms for separable nonlinear squares problems

IEEE Trans. Cybernet.

Robust adaptive finite-time parameter estimation and control for robotic systems

Int. J. Robust Nonlinear Control

The damping iterative parameter identification method for dynamical systems based on the sine signal measurement

Signal Process.

Hierarchical parameter estimation for the frequency response based on the dynamical window data

Int. J. Control Automat. Syst.

Iterative identification algorithms for bilinear-in-parameter systems with autoregressive moving average noise

J. Frankl. Inst.

Combined state and parameter estimation for a bilinear state space system with moving average noise

J. Frankl. Inst.

Recursive and iterative least squares parameter estimation algorithms for multiple-input-output-error systems with autoregressive noise

Circuits Syst. Signal Process.

Parameter identification for pseudo-linear systems with ARMA noise using the filtering technique

IET Control Theory Appl.

Recursive parameter estimation algorithm for multivariate output-error systems

J. Frankl. Inst.

Recasted models based hierarchical extended stochastic gradient method for MIMO nonlinear systems

IET Control Theory Appl.

Decomposition-based recursive least squares identification methods for multivariate pseudolinear systems using the multi-innovation

Int. J. Syst. Sci.

A recursive identification algorithm for Wiener nonlinear systems with linear state-space subsystem

Circuits Syst. Signal Process.

Auxiliary model based least squares iterative algorithms for parameter estimation of bilinear systems using interval-varying measurements

IEEE Access.

Some new results of designing an IIR filter with colored noise for signal processing

Digital Signal Process.

Parameter estimation for control systems based on impulse responses

Int. J. Control, Autom. Syst.

The parameter estimation algorithms based on the dynamical response measurement data

Adv. Mech. Eng.

A recursive least squares parameter estimation algorithm for output nonlinear autoregressive systems using the input-output data filtering

J. Frankl. Inst.

Recursive parameter identification of the dynamical models for bilinear state space systems

Nonlinear Dyn.

A hierarchical least squares identification algorithm for Hammerstein nonlinear systems using the key term separation

J. Frankl. Inst.

Hierarchical stochastic gradient algorithm and its performance analysis for a class of bilinear-in-parameter systems

Circuits Syst. Signal Process.

Adaptive gradient-based iterative algorithm for multivariate controlled autoregressive moving average systems using the data filtering technique

Complexity

Least squares based iterative parameter estimation algorithm for stochastic dynamical systems with ARMA noise using the model equivalence

Int. J. Control, Autom. Syst.

Maximum likelihood estimation method for dual-rate Hammerstein systems

Int. J. Control, Autom. Syst.

Tutorial on maximum likelihood estimation

J. Math. Psychol.

Data filtering based multi-innovation extended gradient method for controlled autoregressive autoregressive moving average systems using the maximum likelihood principle

Math. Comput. Simul.