Balanced simplicity–accuracy neural network model families for system identification

Abstract

Nonlinear system identification tends to provide highly accurate models these last decades; however, the user remains interested in finding a good balance between high-accuracy models and moderate complexity. In this paper, four balanced accuracy–complexity identification model families are proposed. These models are derived, by selecting different combinations of activation functions in a dedicated neural network design presented in our previous work (Romero-Ugalde et al. in Neurocomputing 101:170–180. doi:10.1016/j.neucom.2012.08.013, 2013). The neural network, based on a recurrent three-layer architecture, helps to reduce the number of parameters of the model after the training phase without any loss of estimation accuracy. Even if this reduction is achieved by a convenient choice of the activation functions and the initial conditions of the synaptic weights, it nevertheless leads to a wide range of models among the most encountered in the literature. To validate the proposed approach, three different systems are identified: The first one corresponds to the unavoidable Wiener–Hammerstein system proposed in SYSID2009 as a benchmark; the second system is a flexible robot arm; and the third system corresponds to an acoustic duct.

Appendices

Appendix 1: Training

The adaptation laws of the synaptic weights, derived according to the steepest descent gradient, are as follows:

1.1 FF1 adaptation algorithms

$$\begin{aligned} X(k+1)=X(k)+\eta e(k) T \end{aligned}$$

(18)

$$\begin{aligned} Z_b(k+1)=Z_b(k)+\eta e(k) X r_b \end{aligned}$$

(19)

$$\begin{aligned} Z_a(k+1)=Z_a(k)+ \eta e(k) X r_a \end{aligned}$$

(20)

$$\begin{aligned} Z_h(k+1)=Z_h(k)+ \eta e(k) X \end{aligned}$$

(21)

$$\begin{aligned} V_{b_{i}}(k+1)=V_{b_{i}}(k)+\eta e(k) X Z_b \tanh (J_{u}W_{b_{i}}) \end{aligned}$$

(22)

$$\begin{aligned} V_{a_{i}}(k+1)=V_{a_{i}}(k)+\eta e(k) X Z_a \tanh (J_{\hat{y}}W_{a_{i}}) \end{aligned}$$

(23)

$$\begin{aligned} W_{b_{i}}(k+1)=W_{b_{i}}(k)+\eta e(k) X Z_b V_{b_i} {\mathrm{sech}}^2(J_{u}W_{b_i}) J_u \end{aligned}$$

(24)

$$\begin{aligned} W_{a_{i}}(k+1)=W_{a_{i}}(k)+\eta e(k) X Z_a V_{a_i} {\mathrm{sech}}^2(J_{\hat{y}}W_{a_i}) J_{\hat{y}} \end{aligned}$$

(25)

1.2 FF2 adaptation algorithms

$$\begin{aligned} X(k+1)=X(k)+\eta e(k) T \end{aligned}$$

(26)

$$\begin{aligned} Z_b(k+1)=Z_b(k)+\eta e(k) X \tanh (r_b) \end{aligned}$$

(27)

$$\begin{aligned} Z_a(k+1)=Z_a(k)+\eta e(k) X \tanh (r_a) \end{aligned}$$

(28)

$$\begin{aligned} Z_h(k+1)=Z_h(k)+ \eta e(k) X \end{aligned}$$

(29)

$$\begin{aligned} V_{b_{i}}(k+1)=V_{b_{i}}(k)+\eta e(k) X Z_b {\mathrm{sech}}^2(r_b) J_{u} W_{b_{i}} \end{aligned}$$

(30)

$$\begin{aligned} V_{a_{i}}(k+1)=V_{a_{i}}(k)+\eta e(k) X Z_a {\mathrm{sech}}^2(r_a) J_{\hat{y}} W_{a_{i}} \end{aligned}$$

(31)

$$\begin{aligned} W_{b_{i}}(k+1)=W_{b_{i}}(k)+\eta e(k) X Z_b {\mathrm{sech}}^2(r_b) V_{b_i} J_u \end{aligned}$$

(32)

$$\begin{aligned} W_{a_{i}}(k+1)=W_{a_{i}}(k)+\eta e(k) X Z_a {\mathrm{sech}}^2(r_a) V_{a_i} J_{\hat{y}} \end{aligned}$$

(33)

1.3 ARX adaptation algorithms

$$\begin{aligned} X(k+1)=X(k)+\eta e(k) T \end{aligned}$$

(34)

$$\begin{aligned} Z_b(k+1)=Z_b(k)+\eta e(k) X r_b \end{aligned}$$

(35)

$$\begin{aligned} Z_a(k+1)=Z_a(k)+\eta e(k) X r_a \end{aligned}$$

(36)

$$\begin{aligned} Z_h(k+1)=Z_h(k)+ \eta e(k) X \end{aligned}$$

(37)

$$\begin{aligned} V_{b_{i}}(k+1)=V_{b_{i}}(k) + \eta e(k) X Z_b J_{u} W_{b_{i}} \end{aligned}$$

(38)

$$\begin{aligned} V_{a_{i}}(k+1)=V_{a_{i}}(k)+\eta e(k) X Z_a J_{\hat{y}} W_{a_{i}} \end{aligned}$$

(39)

$$\begin{aligned} W_{b_{i}}(k+1)=W_{b_{i}}(k)+\eta e(k) X Z_b V_{b_i} J_u \end{aligned}$$

(40)

$$\begin{aligned} W_{a_{i}}(k+1)=W_{a_{i}}(k)+\eta e(k) X Z_a V_{a_i} J_{\hat{y}} \end{aligned}$$

(41)

1.4 NARX adaptation algorithms

$$\begin{aligned} X(k+1)=X(k)+\eta e(k) \tanh (T) \end{aligned}$$

(42)

$$\begin{aligned} Z_b(k+1)=Z_b(k)+\eta e(k) X {\mathrm{sech}}^2(T) r_b \end{aligned}$$

(43)

$$\begin{aligned} Z_a(k+1)=Z_a(k)+\eta e(k) X {\mathrm{sech}}^2(T) r_a \end{aligned}$$

(44)

$$\begin{aligned} Z_h(k+1)=Z_h(k)+ \eta e(k) X {\mathrm{sech}}^2(T) \end{aligned}$$

(45)

$$\begin{aligned} V_{b_{i}}(k+1)=V_{b_{i}}(k) + \eta e(k) X {\mathrm{sech}}^2(T) Z_b J_{u} W_{b_{i}} \end{aligned}$$

(46)

$$\begin{aligned} V_{a_{i}}(k+1)=V_{a_{i}}(k)+\eta e(k) X {\mathrm{sech}}^2(T) Z_a J_{\hat{y}} W_{a_{i}} \end{aligned}$$

(47)

$$\begin{aligned} W_{b_{i}}(k+1)=W_{b_{i}}(k)+\eta e(k) X {\mathrm{sech}}^2(T) Z_b V_{b_i} J_u \end{aligned}$$

(48)

$$\begin{aligned} W_{a_{i}}(k+1)=W_{a_{i}}(k)+\eta e(k) X {\mathrm{sech}}^2(T) Z_a V_{a_i} J_{\hat{y}} \end{aligned}$$

(49)

with $\eta$, commonly referred to as learning rate, adapted according to the “search then convergence” algorithm presented in [8].

Appendix 2: Reduction methods applied to models FF2, ARX and NARX

Let us follow the proposed system identification procedure:

1.1 Model FF2

Step 1. Neural network training under particular assumptions Once the neural network model given by (4) is trained under Assumptions 1 and 2, we obtain:

$$\begin{aligned}&\hat{y}(k)=X^* T \nonumber\\&T=Z_b^* \varphi _2(r_b)+Z_a^* \varphi _2(r_a)+Z_h^*\nonumber \\&r_b=\sum _{i=1}^{nn}{V_{b_i}^*(J_{u}W_{b_{i}}^*)}\nonumber \\&r_a=\sum _{i=1}^{nn}{V_{a_i}^*(J_{\hat{y}}W_{a_i}^*)} \end{aligned}$$

(50)

Step 2. Model transformation Since the final values of the synaptic weights are: $W^*_{b_1}=W^*_{b_{j}},\,V^*_{b_1}=V^*_{b_{j}},\,W^*_{a_1}=W^*_{a_{j}}$ and $V^*_{a_1}=V^*_{a_{j}}$ with $j=2,3,\ldots ,nn$. Then, in (50), it is indeed possible to make the following algebraic operations:

$$\begin{aligned}&Z_B^*=X^*\times Z_b^*\\&Z_A^*=X^*\times Z_a^*\\&Z_H^*=X^*\times Z_h^*\\&W_{B_i}^*=V_{b_i}^*\times W_{b_i}^*\\&W_{A_i}^*=V_{a_i}^*\times W_{a_i}^*\\ \end{aligned}$$

Then, the 2nn-2-1 neuro-model given by (50) becomes:

$$\begin{aligned}&\hat{y}(k)=T\nonumber\\ &T=Z_B^*\varphi _2(r_b)+Z_A^*\varphi _2(r_a)+Z_H^* \nonumber \\ &r_b=\sum _{i=1}^{nn}{(J_{u}W_{B_{i}}^*)}\nonumber \\&r_a=\sum _{i=1}^{nn}{(J_{\hat{y}}W_{A_i}^*)}\end{aligned}$$

(51)

where $W^*_{B_1}=W^*_{B_{j}}$ and $W^*_{A_1}=W^*_{A_{j}}$ (with $j=2,3,\ldots ,nn$) due to Assumption 2.

A supplementary transformation is achieved in order to change the redefined 2nn-1 model (51) into a 2-1 representation by the following algebraic operations:

$$\begin{aligned}&\sum _{i=1}^{nn}{(J_{u}W_{B_{i}}^*)}=nn \times (J_{u}W_{B_1}^*)=J_{u}W_{B}^*\\&\sum _{i=1}^{nn}{(J_{\hat{y}}W_{A_i}^*)}= nn \times (J_{y_{n}}W_{A_1}^*)=J_{y_{n}}W_{A}^* \end{aligned}$$

The resulting model after the model transformation has the following mathematical form:

$$\begin{aligned} \hat{y}(k)=Z_{B}^*\varphi _2(J_{u}W_B^*)+ Z_{A}^*\varphi _2(J_{\hat{y}}W_A^*)+Z_H^* \end{aligned}$$

(52)

1.2 Model ARX

Step 1. Neural network training under particular assumptions Once the neural network model given by (5) is trained under Assumptions 1 and 2, we obtain

$$\begin{aligned}&\hat{y}(k)=X^* T\nonumber\\&T=Z_b^* (r_b)+Z_a^* (r_a)+Z_h^*\nonumber \\&r_b=\sum _{i=1}^{nn}{V_{b_i}^*(J_{u}W_{b_{i}}^*)}\nonumber \\&r_a=\sum _{i=1}^{nn}{V_{a_i}^*(J_{\hat{y}}W_{a_i}^*)} \end{aligned}$$

(53)

Step 2. Model transformation Since the final values of the synaptic weights are: $W^*_{b_1}=W^*_{b_{j}},\,V^*_{b_1}=V^*_{b_{j}},\,W^*_{a_1}=W^*_{a_{j}}$ and $V^*_{a_1}=V^*_{a_{j}}$ with $j=2,3,\ldots ,nn$. Then, in (53), it is indeed possible to make the following algebraic operations:

$$\begin{aligned}&Z_{H}^*=X^*\times Z_h^*\\&W_{B_i}^*=X^*\times Z_b^* \times V_{b_i}^*\times W_{b_i}^*\\&W_{A_i}^*=X^*\times Z_a^* \times V_{a_i}^*\times W_{a_i}^* \end{aligned}$$

Then, the 2nn-2-1 neuro-model given by (53) becomes:

$$\begin{aligned}&\hat{y}(k)=T \nonumber \\ &T=r_b+r_a+Z_H^* \nonumber \\ &r_b=\sum _{i=1}^{nn}{(J_{u}W_{B_{i}}^*)}\nonumber \\ &r_a=\sum _{i=1}^{nn}{(J_{\hat{y}}W_{A_i}^*)}\end{aligned}$$

(54)

where $W^*_{B_1}=W^*_{B_{j}}$ and $W^*_{A_1}=W^*_{A_{j}}$ (with $j=2,3,\ldots ,nn$) due to Assumption 2.

A supplementary transformation is achieved in order to change the redefined 2nn-1 model (54) into a 2-1 representation by the following algebraic operations:

$$\begin{aligned}&\sum _{i=1}^{nn}{(J_{u}W_{B_{i}}^*)}=nn \times (J_{u}W_{B_1}^*)=J_{u}W_B^*\\&\sum _{i=1}^{nn}{(J_{\hat{y}}W_{A_i}^*)}= nn \times (J_{\hat{y}}W_{A_1}^*)=J_{\hat{y}}W_A^* \end{aligned}$$

The resulting model after the model transformation has the following mathematical form:

$$\begin{aligned} \hat{y}(k)=(J_{u}W_B^*)+ (J_{\hat{y}}W_A^*)+Z_H^* \end{aligned}$$

(55)

1.3 Model NARX

Step 1. Neural network training under particular assumptions Once the neural network model given by (6) is trained under Assumptions 1 and 2, we obtain:

$$\begin{aligned}&\hat{y}(k)=X^* \varphi _{3}(T)\nonumber\\&T=Z_b^* (r_b)+Z_a^* (r_a)+Z_h^*\nonumber \\&r_b=\sum _{i=1}^{nn}{V_{b_i}^*(J_{u}W_{b_{i}}^*)}\nonumber \\&r_a=\sum _{i=1}^{nn}{V_{a_i}^*(J_{\hat{y}}W_{a_i}^*)} \end{aligned}$$

(56)

Step 2. Model transformation Since the final values of the synaptic weights are: $W^*_{b_1}=W^*_{b_{j}},\,V^*_{b_1}=V^*_{b_{j}},\,W^*_{a_1}=W^*_{a_{j}}$ and $V^*_{a_1}=V^*_{a_{j}}$ with $j=2,3,\ldots ,nn$.

Then, in (56), it is indeed possible to make the following algebraic operations:

$$\begin{aligned}&W_{B_i}^*=Z_b^* \times V_{b_i}^*\times W_{b_i}^*\\&W_{A_i}^*=Z_a^* \times V_{a_i}^*\times W_{a_i}^* \end{aligned}$$

Then, the 2nn-2-1 neuro-model given by (56) becomes:

$$\begin{aligned}&\hat{y}(k)=X^* \varphi _{3}(T) \nonumber\\ &T=r_b+r_a+Z_h^* \nonumber \\ &r_b=\sum _{i=1}^{nn}{(J_{u}W_{B_{i}}^*)}\nonumber \\ &r_a=\sum _{i=1}^{nn}{(J_{\hat{y}}W_{A_i}^*)}\end{aligned}$$

(57)

A supplementary transformation is achieved in order to change the redefined 2nn-1 model (57) into a 2-1 representation by the following algebraic operations:

$$\begin{aligned}&\sum _{i=1}^{nn}{(J_{u}W_{B_{i}}^*)}=nn \times (J_{u}W_{B_1}^*)=J_{u}W_B^*\\&\sum _{i=1}^{nn}{(J_{\hat{y}}W_{A_i}^*)}= nn \times (J_{y_{n}}W_{A_1}^*)=J_{\hat{y}}W_A^* \end{aligned}$$

The resulting model after the model transformation has the following mathematical form:

$$\begin{aligned} \hat{y}(k)=X^* \varphi _{3}((J_{u}W_B^*)+ (J_{\hat{y}}W_A^*)+Z_h^*) \end{aligned}$$

(58)

Appendix 2: Neuro-identification of a flexible robot arm

In order to show the flexibility of the proposed model families, a second system is identified. The data come from a flexible robot arm (see Fig. 9), and the arm is installed on an electrical motor. The applied persisting exciting input, corresponding to the reaction torque of the structure on the ground, is a periodic sine sweep. The output of the system is the acceleration of the flexible arm. This system identification case is issued from an example through DaISy (database for the identification of systems). http://homes.esat.kuleuven.be/~smc/daisy/daisydata.html.

Fig. 9

Flexible robot arm

The system is identified with the complex 2nn-2-1 neuro-model NARX (683 parameters before reduction) given by (59) with $nn=40,\,n_a=8$ and $n_b=7$. Notice that this neuro-model is derived from the proposed neural network architecture.

$$\begin{aligned}&\hat{y}(k)=X\tanh (T)\nonumber\\&T=Z_b(r_b)+Z_a(r_a)\nonumber \\&r_b=\sum _{i=1}^{nn}{V_{b_i}\left( J_{u}W_{b_{i}}\right) }\nonumber \\&r_a=\sum _{i=1}^{nn}{V_{a_i}\left( J_{\hat{y}}W_{a_i}\right) }\end{aligned}$$

(59)

Once the model (59) is trained under Assumptions 1 and 2, Theorem 1 (reduction approach) is applied in order to generate a reduced model of the form of (60).proposed approach has to be compared

$$\begin{aligned} \hat{y}(k)=X^*\tanh \left( J_{u}W_{B}^*+J_{\hat{y}}W_{A}^*\right) \end{aligned}$$

(60)

where the 16 parameters characterizing the system are as follows:

Proposed NARX model:

$$\begin{aligned}&W^*_B = [0.2883 \quad {-}0.3820 \quad 0.0655 \quad 0.1339 \quad {-}0.0821 \quad {-}0.2135 \quad 0.2335]\\&W^*_A = [{-}0.7387 \quad {-}0.0914 \quad 0.3221 \quad 0.3122 \quad {-}0.0812 \quad {-}0.3016 \quad {-}0.0196 \quad 0.3362]\\&X^*={-}1.4623\\ \end{aligned}$$

Naturally, we have to compare our approach with another black-box system identification method. For example, the same system is identified with the nonlinear system identification toolbox in MATLAB 2013. Here, the identification model used is the NLARX model, and the nonlinearity estimator is the sigmoid network with 5 units, $n_a= 6$ and $n_b=10$. The $619$ parameters of the MATLAB model are estimated by using the Levenberg–Marquardt learning algorithm. The comparison, realized in terms of the number of parameters $np$ and the validation results proposed in SYSID 2009 [39], is summarized in Table 3.

In order to compare the performances of both models, the frequency response function (FRF) that is mainly appreciated by the users in practical applications is computed from the measured data and the estimated output of both, the proposed neuro-model NARX and the NLARX model obtained by using the toolbox of MATLAB (see Fig. 10).

Fig. 10

Frequency response function (FRF)

Table 3 A comparative assessment with respect to another parametric method

1.1 Comments

Figure 10 exposes the interest of using neural networks, since the estimated models accurately represent the system behavior in a large frequency range.

Confronting the two black-box models (see Table 3), the deviation errors ($s_t$ and $e_{{\mathrm{RMS}}e}$) of the MATLAB model are only two times better, but the number of parameters is substantially larger ($619$ vs $16$). Conversely, in terms of $\mu _t$, the two models have almost the same accuracy, despite the use of a simple gradient training algorithm in the computation of the proposed NARX model. Remember that the main objective in this paper is to propose to the user a rather simple and efficient way to find a good balance between accuracy and complexity.

Appendix 3 :Neuro-identification of an acoustic duct

In order to show the flexibility of the proposed model families, a third system is identified. This experimental device is an acoustic waves guide made of Plexiglas, used to develop an active noise control (see Fig. 11). One end of the duct is almost anechoic and the other end is opened. The identification input signal is a pseudo-random binary sequence (PRBS) with a length $L = 2^{10} - 1$ and level $\pm$3 V sufficiently exciting applied to the control loudspeaker. The sampling period is TS = 500 $\upmu$s. In order to model, the first propagative modes of the waves guide (also called the secondary path) which lye in the frequency range $[0; 1{,}000\,\mathrm{Hz}]$. We shall use several data set of the same length, namely 1,024, measured by the output microphone. A prior measurement of the propagation delay confirms the analytical value $\tau \approx 7TS$.

Fig. 11

Schematic of semi-finite acoustic waves guide

This system is identified with the complex 2nn-2-1 neuro-model FF1 (1,418 parameters before reduction) given by (61) with $nn=40,\,n_a=15$ and $n_b=18$.

$$\begin{aligned}&\hat{y}(k)=X T\nonumber\\&T=Z_b r_b+Z_a r_a+Z_h\nonumber \\&r_b=\sum _{i=1}^{nn}{V_{b_i}\tanh \left( J_{u}W_{b_{i}}\right) }\nonumber \\&r_a=\sum _{i=1}^{nn}{V_{a_i}\tanh \left( J_{\hat{y}}W_{a_i}\right) } \end{aligned}$$

(61)

Once the model is trained under Assumptions 1 and 2, Theorem 1 (reduction approach) is applied in order to generate a reduced model of the form of (62).

$$\begin{aligned} \hat{y}(k)=V_{B}^*\tanh (J_{u}W_{B}^*)+ V_{A}^*\tanh \left( J_{\hat{y}}W_{A}^*\right) \end{aligned}$$

(62)

where the 35 parameters characterizing the system are as follows:

$$\begin{aligned} W^*_A &= \left[ {-}0.0828 \quad 0.0614 \quad{-}0.0718 \quad {-}0.1201 \quad {-}0.0043 \quad {-}0.0067 \quad {-}0.0530\quad {-}0.0474\right. \\&\left. \quad 0.0032 \quad {-}0.0173 \quad{-}0.0381 \quad {-}0.0135 \quad {-}0.0062 \quad {-}0.0151 \quad {-}0.0140\right] \\W^*_B&= \left[ {-}0.0325 \quad {-}0.0555 \quad 0.0256 \quad0.0393 \quad 0.0007 \quad 0.0274 \quad 0.0132 \quad 0.0202 \quad0.0138\right. \\&\left.\quad 0.0006 \quad 0.0059 \quad {-}0.0139 \quad0.0040 \quad {-}0.0188 \quad {-}0.0110 \quad {-}0.0120 \quad {-}0.0002 \quad{-}0.0058\right] \end{aligned}$$

$V^*_A=6.8643$ and $V^*_B=3.4414$

Naturally, the proposed approach has to be compared with another black-box system identification technique. Therefore, the same acoustic system is identified with the nonlinear system identification toolbox in MATLAB 2013. Here, the identification model used is the NLARX model, and the nonlinearity estimator is the sigmoid network with 5 units, $n_a= 14$ and $n_b=12$. The 1,519 parameters of the MATLAB model are estimated by using the Levenberg–Marquardt learning algorithm. The comparison, realized in terms of the number of parameters np and the validation results proposed in SYSID 2009 [39], is summarized in Table 4.

In order to compare the performances of both models in the frequency domain, the frequency response function (FRF) that is mainly appreciated by the users in practical applications is computed from the measured data and the estimated output of both, the proposed neuro-model NARX and the NLARX model obtained by using the toolbox of MATLAB (see Fig. 12).

Fig. 12

Frequency response function (FRF)

Table 4 A comparative assessment with respect to another parametric method

1.1 Comments

Figure 12 illustrates the relevant performance of both, the proposed neuro-model FF1 and the MATLAB model, with respect to the real system behavior. But, if the number of model parameters is taken into consideration, it seems remarkable that with only 35 parameters the proposed model provides a significant level of accuracy.

From Table 4, comparing the proposed model (FF1) and the model obtained by using the toolbox of MATLAB both models almost have the same accuracy, but once again, the number of parameters of the MATLAB model is substantially larger (1,519 vs 35). Then, we are proposing to the user a system identification approach with results comparable to the toolbox of MATLAB, but with a smaller number of parameters.