Constrained fuzzy evidential multivariate model identified by EM algorithm: a soft sensor to monitoring imprecise and uncertain process parameters

Abstract

This paper focuses on the problem of monitoring/estimating process parameters in the insufficient case when only imprecise and uncertain information can be obtained, possibly due to limited precision and reliability of sensors in industries. To solve this problem, a constrained fuzzy evidential multivariate model is proposed as a soft sensor to monitor imprecise and uncertain process parameters. The most challenging task involved in the modeling is how to identify structure parameters of the monitor model, especially under sets of constraints. To tackle this challenge, we represent the imprecise and uncertain information as fuzzy belief functions in the evidence theory framework, and then propose a restricted fuzzy evidential Expectation-Conditional Maximization algorithm (RFE2CM) for maximum likelihood estimation from fuzzy belief functions under linear inequality constraints. Also, the convergence property of the restricted fuzzy evidential EM algorithm is discussed. In order to validate the performance of the proposed model and algorithm, some numerical simulations are conducted as well as an experimental simulation on a real ball mill in a power plant. The numerical and experimental simulation results show that the proposed model and algorithm can not only be feasibly applicable to monitor the process parameters in insufficient informatics cases, but also have high prediction accuracy with small mean square errors.

Appendices

Appendix 1: Calculations of $\mathbf {z}^{( q)}={\mathtt {E}}_{\varvec{\psi } ^{(q)}} ( {\left. \mathbf {z} \right| {pl}})$,$\varvec{\alpha } _i^{( q)} ={\mathtt {E}}_{\varvec{\psi } ^{(q)}} ( {\left. {\mathbf {x}_i \mathbf {{x}'}_i } \right| {pl}_i })$ and $\beta _i^{( q)} ={\mathtt {E}}_{\varvec{\psi } ^{(q)}} ( {\left. {\mathbf {x}_i } \right| {pl}_i })$ in E-step of RFE2CM algorithm.

Let us assume that P is the distribution of a univariate normal random variable $X_{ij}$ with mean v, standard deviation $\sigma$ and p.d.f. g(x). A piece of (marginal) FBA $m_{ij}$ on $X_{ij}$, holding $r_{i}$ focal elements $F(m_{ij})$ = {$\tilde{A}_1$, $\tilde{A}_2$, ..., $\tilde{A}_{r_i }$}, is used to represent our partial and uncertain information about the realization $x_{ij}$. Let $\forall \tilde{A}=( {a,b,c})\in F( {m_{ij} })$be a triangular fuzzy number, with membership function:

$$\begin{aligned} \mu _{\tilde{A}} (x)=\left\{ {\begin{array}{l} \frac{x-a}{b-a}\qquad \mathrm{if}\quad a\le x\le b, \\ \frac{c-x}{c-b}\quad \quad \;\mathrm{if}\quad b\le x\le c, \\ 0\qquad \quad \,\, \mathrm{otherwise} \\ \end{array}} \right. \end{aligned}$$

(60)

Denoting by g(x) the p.d.f. of $X_{ij}$ with parameter $\varvec{\psi }$ = (v, $\sigma$)’, we have

$$\begin{aligned} L( {\varvec{\psi } ,{pl_{ij} }})= & {} {\mathtt {E}}\left[ {{pl_{ij} }( x)} \right] =\int {g( x){pl_{ij} }( x)\mathrm{d}x}\nonumber \\= & {} \int {g( x)\left( {\sum \limits _{\tilde{A}\in F( {m_{ij} })} {m( {\tilde{A}})\mu _{\tilde{A}} ( x)} }\right) \mathrm{d}x}\nonumber \\= & {} \sum \limits _{\tilde{A}\in F( {m_{ij} })} {m_{ij} ( {\tilde{A}})} \int {g( x)\mu _{\tilde{A}} ( x)\mathrm{d}x} \end{aligned}$$

(61)

which can be completed by

$$\begin{aligned} \int {\mu _{\tilde{A}} ( x)g( x){\text{ d }}x}= & {} \frac{1}{b-a}\int _a^b {xg( x){\text{ d }}x} \\&-\frac{a}{b-a}\left( {\Phi ( {b^*})-\Phi ( {a^*})}\right) \\&+\frac{c}{c-b}\left( {\Phi ( {c^*})-\Phi ( {b^*})}\right) \\&-\frac{1}{c-b}\int _b^c {xg( x){\text{ d }}x} . \end{aligned}$$

where $\Phi$(.) denotes the c.d.f. of the standard normal distribution, and $x_{ij}^{*}$ denotes ($x_{ij}$ – v)/ $\sigma$ for all $x_{ij}$. It is easy to show that

$$\begin{aligned}&\int _a^b {xg( x){\text{ d }}x} =\frac{\sigma }{\sqrt{2\pi } }\left[ {\exp \left( {-\frac{a^{*2}}{2}}\right) -\exp \left( {-\frac{b^{*2}}{2}}\right) } \right] \nonumber \\&\quad +v\left( {\Phi ( {b^*})-\Phi ( {a^*})}\right) \end{aligned}$$

(62)

Eq. (62) makes it possible to complete the Eq. (70).

Let us now compute the expectation of $X_{ij}$ given ${pl}_{ij}$for the conditional density of $X_{ij}$. We have

$$\begin{aligned} \beta _{ij}^{( q)}= & {} {\mathtt {E}}_{\varvec{\Psi } ^{( q)}} ( {\left. {X_{ij} } \right| {pl}_{ij} })=\int {xg( {\left. x \right| {pl_{ij} },\varvec{\psi } }){\text{ d }}x}\nonumber \\= & {} \frac{\int {xg( x){pl_{ij} }( x){\text{ d }}x} }{L( {\varvec{\psi } ,{pl_{ij} }})}\nonumber \\= & {} \frac{\sum \limits _{\tilde{A}\in F( {m_{ij} })} {m_{ij} ( {\tilde{A}})} \int {x\mu _{\tilde{A}} ( x)g( x){\text{ d }}x} }{L( {\varvec{\psi } ,{pl_{ij} }})} \end{aligned}$$

(63)

where the denominator is given by (61), and the numerator is

$$\begin{aligned} \int {x\mu _{\tilde{A}} ( x)g( x){\text{ d }}x}= & {} \frac{1}{b-a}\int _a^b {x^2g( x){\text{ d }}x}\nonumber \\&-\frac{a}{b-a}\int _a^b {xg( x){\text{ d }}x}\nonumber \\ \nonumber \\&+\frac{c}{c-b}\int _b^c {xg( x){\text{ d }}x}\nonumber \\&-\frac{1}{c-b}\int _b^c {x^2g( x){\text{ d }}x} \end{aligned}$$

(64)

which can be computed using Eq. (62)and

$$\begin{aligned}&\int _a^b {x^2g( x){\text{ d }}x} =\frac{\sigma ^2}{\sqrt{2\pi } }\left[ {a^*\exp \left( {-\frac{a^{*2}}{2}}\right) -b^*\exp \left( {-\frac{b^{*2}}{2}}\right) } \right] \nonumber \\&\quad +\frac{2\sigma v}{\sqrt{2\pi } }\left[ {a^*\exp \left( {-\frac{a^{*2}}{2}}\right) -b^*\exp \left( {-\frac{b^{*2}}{2}}\right) } \right] \nonumber \\&\quad +\left( {v^2+\sigma ^2}\right) \left( {\Phi ( {b^*})-\Phi ( {a^*})}\right) \end{aligned}$$

(65)

Therefore, we have

$$\begin{aligned}&\varvec{\beta } _i^{( q)} ={\mathtt {E}}_{\varvec{\psi } ^{(q)}} \left( {\left. {\mathbf {x}_i } \right| {pl}_i }\right) =\left( {\mathtt {E}}_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{i1} } \right| {pl}_{i1} }\right) ,\right. \nonumber \\&\left. \quad {\mathtt {E}}_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{i2} } \right| {pl}_{i2} }\right) ,\ldots ,{\mathtt {E}}_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{it} } \right| {pl}_{it} }\right) \right) ^\prime \nonumber \\&\quad =\left( {\beta _{i1}^{( q)} ,\beta _{i2}^{( q)} ,\ldots ,\beta _{it}^{( q)} }\right) ^\prime \end{aligned}$$

(66)

Because ${\varvec{\bar{\mathrm{x}}}}=( {{\varvec{\bar{\mathrm{x}'}}}_1 ,\varvec{{\bar{\mathrm{x}}'}}_2 ,\ldots ,{\varvec{\bar{\mathrm{x}'}}}_t })^\prime$ and $\mathbf {z}={{\varvec{\bar{\mathrm{U}}}}'D\bar{x}}$, we have

$$\begin{aligned} \mathbf {z}^{( q)}= & {} {\mathtt {E}}_{\varvec{\psi } ^{(q)}} ( {\left. \mathbf {z} \right| {pl}}) \nonumber \\= & {} {{\varvec{\bar{\mathrm{U}}}}^{\prime }{\mathbf {A}}}E_{\varvec{\psi } ^{(q)}} ( {\left. {{\varvec{\bar{\mathrm{x}}}}} \right| {pl}}) \nonumber \\= & {} {{\varvec{\bar{\mathrm{U}}}}^{\prime }{\mathbf {A}}}E_{\varvec{\psi } ^{(q)}} \left( {\left. {\left( {\varvec{{\bar{\mathrm{x}'}}}_1 ,{\varvec{\bar{\mathrm{x}'}}}_2 ,\ldots ,{\varvec{\bar{\mathrm{x}'}}}_t }\right) ^\prime } \right| {pl}}\right) \nonumber \\= & {} {\varvec{{\bar{\mathrm{U}}}}^{\prime }{\mathbf {D}}}\left( {\beta _{11}^{( q)} ,\beta _{21}^{( q)} ,\cdots ,\beta _{n1}^{( q)} ,\ldots ,\beta _{1t}^{( q)} ,\beta _{2t}^{( q)} ,\ldots ,\beta _{nt}^{( q)} }\right) ^\prime \nonumber \\ \end{aligned}$$

(67)

We finally compute

$$\begin{aligned} \alpha _{ij}^{( q)}= & {} {\mathtt {E}}\left( {\left. {X_{ij}^2 } \right| {pl}_{ij} }\right) =\int {x^2g\left( {\left. x \right| {pl}_{ij} ,\varvec{\psi } }\right) {\text{ d }}x}\nonumber \\= & {} \frac{\int {x^2g( x){pl}_{ij} ( x){\text{ d }}x} }{L\left( {\varvec{\psi } ,{pl}_{ij} }\right) }\nonumber \\ {}= & {} \frac{\sum \limits _{\tilde{A}\in F( {m_{ij} })} {m_{ij} ( {\tilde{A}})} \int {x^2\mu _{\tilde{A}} ( x)g( x){\text{ d }}x} }{L\left( {\varvec{\psi } ,{pl}_{ij} }\right) } \end{aligned}$$

(68)

The numerator in (68) is

$$\begin{aligned}&\int {x^2\mu _{\tilde{A}} ( x)g( x){\text{ d }}x} =\frac{1}{b-a}\int _a^b {x^3g( x){\text{ d }}x}\nonumber \\&\quad -\frac{a}{b-a}\int _a^b {x^2g( x){\text{ d }}x}\nonumber \\&\quad +\frac{c}{c-b}\int _b^c {x^2g( x){\text{ d }}x} -\frac{1}{c-b}\int _b^c {x^3g( x){\text{ d }}x} \end{aligned}$$

(69)

which can be computed using (65) and

$$\begin{aligned}&\int _a^b {x^3g( x){\text{ d }}x} \nonumber \\&\quad =\frac{\sigma ^3}{\sqrt{2\pi } }\left[ {\left( {2+a^*}\right) \exp \left( {-\frac{a^{\mathbf {*2}}}{2}}\right) -( {2+b^*})\exp \left( {-\frac{b^{\mathbf {*2}}}{2}}\right) } \right] \nonumber \\&\qquad +\,\frac{3\sigma ^2v}{\sqrt{2\pi } }\left[ {a^*\exp \left( {-\frac{a^{\mathbf {*2}}}{2}}\right) -b^*\exp \left( {-\frac{b^{\mathbf {*2}}}{2}}\right) } \right. \nonumber \\&\qquad \left. {+\,\sqrt{2\pi } \left( {\Phi ( {b^*})-\Phi ( {a^*})}\right) } \right] \nonumber \\&\qquad +\,\frac{3\sigma v^2}{\sqrt{2\pi } }\left[ {\exp \left( {-\frac{a^{\mathbf {*2}}}{2}}\right) -\exp \left( {-\frac{b^{\mathbf {*2}}}{2}}\right) } \right] \nonumber \\&\qquad +v^3\left( {\Phi ( {b^*})-\Phi ( {a^*})}\right) \end{aligned}$$

(70)

Then, we have

$$\begin{aligned}&\varvec{\alpha } _i^{( q)} ={\mathtt {E}}_{\varvec{\psi } ^{(q)}} \left( {\left. {\mathbf {x}_i \mathbf {{x}'}_i } \right| {pl}_i }\right) \\&\quad =E_{\varvec{\psi } ^{(q)}} \left( {\left. {\left( {{\begin{array}{llll} {X_{i1}^2 } &{} {X_{i1} X_{i2} } &{} \cdots &{} {X_{i1} X_{it} } \\ {X_{i2} X_{i1} } &{} {X_{i2}^2 } &{} \cdots &{} {X_{i2} X_{it} } \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ {X_{it} X_{i1} } &{} {X_{it} X_{i2} } &{} \cdots &{} {X_{it}^2 } \\ \end{array} }}\right) } \right| {pl}_i }\right) \\&\quad =\left( {{\begin{array}{llll} {E_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{i1}^2 } \right| {pl}_i }\right) } &{} {E_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{i1} X_{i2} } \right| {pl}_i }\right) } &{} \cdots &{} {E_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{i1} X_{it} } \right| {pl}_i }\right) } \\ {E_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{i2} X_{i1} } \right| {pl}_i }\right) } &{} {E_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{i2}^2 } \right| {pl}_i }\right) } &{} \cdots &{} {E_{\psi ^{(q)}} \left( {\left. {X_{i2} X_{it} } \right| {pl}_i }\right) } \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ {E_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{it} X_{i1} } \right| {pl}_i }\right) } &{} {E_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{it} X_{i2} } \right| {pl}_i }\right) } &{} \cdots &{} {E_{\varvec{\psi } ^{(q)}} \left( {\left. {X_{it}^2 } \right| {pl}_i }\right) } \\ \end{array} }}\right) =\left( {{\begin{array}{llll} {\alpha _{i1}^{( q)} } &{} {\beta _{i1}^{( q)} \beta _{i2}^{( q)} } &{} \cdots &{} {\beta _{i1}^{( q)} \beta _{it}^{( q)} } \\ {\beta _{i2}^{( q)} \beta _{i1}^{( q)} } &{} {\alpha _{i2}^{( q)} } &{} \cdots &{} {\beta _{i2}^{( q)} \beta _{it}^{( q)} } \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ {\beta _{it}^{( q)} \beta _{i1}^{( q)} } &{} {\beta _{it}^{( q)} \beta _{i2}^{( q)} } &{} \cdots &{} {\alpha _{it}^{( q)} } \\ \end{array} }}\right) . \end{aligned}$$

Appendix 2

Proof of Theorem 1

The Lagrange function corresponding to optimization problem (31) can be given by

$$\begin{aligned} l( \varvec{\theta } )=\left\| {\varvec{\theta } -\varvec{\gamma } ^{( q)}} \right\| ^2-{\varvec{\lambda } }'( {\mathbf {B}_0 \varvec{\theta } -\mathbf {a}}) \end{aligned}$$

(71)

where $\varvec{\lambda }$ is a non-negative l-dimensional vector.

Let $\varvec{\theta }$ be the solution to (31), we have

$$\begin{aligned} \frac{\partial l( \varvec{\theta } )}{\partial \varvec{\theta } }=2\left( {\varvec{\theta } -\varvec{\gamma } ^{( q)}}\right) -\mathbf {{B}'}_0 \varvec{\lambda } =0 \end{aligned}$$

(72)

$$\begin{aligned} \mathbf {B}_0 \varvec{\theta } -\mathbf {a}\ge \mathbf {0} \end{aligned}$$

(73)

$$\begin{aligned} {\varvec{\lambda } }'( {\mathbf {B}_0 \varvec{\theta } -\mathbf {a}})=\mathbf {0} \end{aligned}$$

(74)

If B $_{1}$ $\varvec{\theta }$ $>$ a $_{1}$ and B $_{2}$ $\varvec{\theta }$ = a $_{2}$, then (74) implies

$$\begin{aligned} {\varvec{\lambda } }'( {\mathbf {B}_0 \varvec{\theta } -\mathbf {a}})=( {{\varvec{\lambda } }'_1 ,{\varvec{\lambda } }'_2 })\left( {\begin{array}{l} \mathbf {B}_1 \varvec{\theta } -\mathbf {a}_1 \\ \mathbf {B}_2 \varvec{\theta } -\mathbf {a}_2 \\ \end{array}}\right) ={\varvec{\lambda } }'_1 ( {\mathbf {B}_1 \varvec{\theta } -\mathbf {a}_1 })=\mathbf {0} \end{aligned}$$

(75)

where $\varvec{\lambda }$ $_{1}$ and $\varvec{\lambda }$ $_{2}$ are, respectively, $l_{1}$- and $l_{2}$-dimensional vectors such that $l_{1}$+ $l_{2}=l$.

Therefore, we get $\varvec{\lambda }$ $_{1}$ = 0 and $\varvec{\lambda }$ $_{2}$ $\ge$ 0. From (72), we have

$$\begin{aligned} 2( {\varvec{\theta } -\varvec{\gamma } ^{( q)}})-\mathbf {{B}'}_2 \varvec{\lambda } _2 =\mathbf {0} \end{aligned}$$

(76)

Multiplying B $_{2}$ on both sides of (76), we have

$$\begin{aligned} 2\mathbf {B}_2 ( {\varvec{\theta } -\varvec{\gamma } ^{( q)}})-\mathbf {B}_2 \mathbf {{B}'}_2 \varvec{\lambda } _2 =\mathbf {0} \end{aligned}$$

(77)

From (77), we can obtain the solution of $\varvec{\lambda }$ $_{2}$ as following:

$$\begin{aligned} \varvec{\lambda } _2= & {} 2( {\mathbf {B}_2 \mathbf {{B}'}_2 })^{-1}\mathbf {B}_2 ( {\varvec{\theta } -\varvec{\gamma } ^{( q)}})\nonumber \\= & {} 2( {\mathbf {B}_2 \mathbf {{B}'}_2 })^{-1}\left( {\mathbf {B}_2 \varvec{\theta } -\mathbf {B}_2 \varvec{\gamma } ^{( q)}}\right) \nonumber \\= & {} 2\left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\left( {\mathbf {a}_2 -\mathbf {B}_2 \varvec{\gamma } ^{( q)}}\right) \end{aligned}$$

(78)

By inserting (78) into (77), we can obtain the final solution to (31):

$$\begin{aligned} \varvec{\theta } =\left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) \varvec{\gamma } ^{( q)}+\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2\nonumber \\ \end{aligned}$$

(79)

In addition, by Kuhn–Tucker Theorem, we have B $_{1}$ $\varvec{\theta }$ - a $_{1} \quad >$0 and $\varvec{\lambda }$ $_{2} \ge$ 0, i.e.,

$$\begin{aligned} \left\{ {\begin{array}{l} \mathbf {B}_1 \left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) \varvec{\gamma } ^{( q)}\\ \quad +\mathbf {B}_1 \mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 -\mathbf {a}_1 >0 \\ \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\left( {\mathbf {a}_2 -\mathbf {B}_2 \varvec{\gamma } ^{( q)}}\right) \ge 0 \\ \end{array}} \right. . \quad \end{aligned}$$

$\square$

Proof of Proposition 1

According to Theorem 1, the MLEs of parameters with linear inequalities is

$$\begin{aligned} \varvec{\hat{{\theta } }}\mathbf {=}\left( {\mathbf {I}_k -\mathbf {{B}'}_2 ( {\mathbf {B}_2 \mathbf {{B}'}_2 })^{-1}\mathbf {B}_2 }\right) \varvec{\gamma } ^{( q)}+\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 \end{aligned}$$

We have

$$\begin{aligned}&E\left( {\varvec{\hat{{\theta } }}}\right) \mathbf {=}E\left( {\left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) \varvec{\gamma } ^{( q)}+\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 }\right) \\&\quad =\left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) E\left( {\varvec{\gamma } ^{( q)}}\right) +\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 \\&\quad =\left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) E\left( {\left( {\Lambda ^{( q)}}\right) ^{-1/2}\mathbf {z}^{( q)}}\right) \\&\qquad +\,\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 \\&\quad =\left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) \left( {\Lambda ^{( q)}}\right) ^{-1/2}E_{\varvec{\psi } ^{(q)}} \left( {\left. \mathbf {z} \right| {pl}}\right) \\&\qquad +\,\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 \\&\quad =\left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) \left( {\Lambda ^{( q)}}\right) ^{-1/2}{\varvec{{\bar{\mathrm{U}}}}^{\prime }{\mathbf {A}}}^{( q)}E_{\varvec{\psi } ^{(q)}} \left( {\left. {\varvec{{\bar{\mathrm{x}}}}} \right| {pl}}\right) \\&\qquad +\,\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 \\&\quad =\left( {\mathbf {I}_k -\mathbf {{B}'}_2 ( {\mathbf {B}_2 \mathbf {{B}'}_2 })^{-1}\mathbf {B}_2 }\right) \left( {\Lambda ^{( q)}}\right) ^{-1/2}{{\varvec{\bar{\mathrm{U}}}}^{\prime }{\mathbf {A}}}^{( q)}E_{\varvec{\psi } ^{(q)}} \left( {\left. {{\varvec{\bar{\mathrm{U}}}{\mathbf {b}}}} \right| {pl}}\right) \\&\qquad +\,\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 \\&\quad =\left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) \left( {\Lambda ^{( q)}}\right) ^{-1/2}{{\varvec{\bar{\mathrm{U}}}}^{\prime }{\mathbf {A}}}^{( q)}{\varvec{\bar{\mathrm{U}}}}E_{\varvec{\psi } ^{(q)}} \left( {\left. \mathbf {b} \right| {pl}}\right) \\&\quad +\,\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 \end{aligned}$$

Then

$$\begin{aligned}&E\left( {{\varvec{\hat{\mathrm{b}}}}}\right) \mathbf {=}\left( {\varvec{\Lambda }^{( q)}}\right) ^{-1/2}E\left( {\varvec{\hat{{\theta } }}}\right) \nonumber \\&\quad =\left( {\varvec{\Lambda }^{( q)}}\right) ^{-1/2}\left( {\mathbf {I}_k -\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {B}_2 }\right) \nonumber \\&\qquad \times \left( {\Lambda ^{( q)}}\right) ^{-1/2}{{\varvec{\bar{\mathrm{U}}}}'D}^{( q)}{\varvec{\bar{\mathrm{U}}}}E_{\varvec{\psi } ^{(q)}} \left( {\left. \mathbf {b} \right| {pl}}\right) \nonumber \\&\qquad +\left( {\varvec{\Lambda }^{( q)}}\right) ^{-1/2}\mathbf {{B}'}_2 \left( {\mathbf {B}_2 \mathbf {{B}'}_2 }\right) ^{-1}\mathbf {a}_2 \nonumber \\&\quad =\mathbf {K}_1 E_{\varvec{\psi } ^{(q)}} \left( {\left. \mathbf {b} \right| {pl}}\right) +{\varvec{K}}_2 \ne \mathbf {b}\nonumber \\ \end{aligned}$$

(80)

where $\mathbf {K}_1 =( {\varvec{\Lambda }^{( q)}})^{-1/2}( {\mathbf {I}_k -\mathbf {{B}'}_2 ( {\mathbf {B}_2 \mathbf {{B}'}_2 })^{-1}\mathbf {B}_2 })( {\Lambda ^{( q)}})^{-1/2}{{\varvec{\bar{\mathrm{U}}}}'{\mathbf {D}}}^{( q)}{\varvec{\bar{\mathrm{U}}}}$, and$\varvec{K}_2 =( {\varvec{\Lambda }^{( q)}})^{-1/2}\mathbf {{B}'}_2 ( {\mathbf {B}_2 \mathbf {{B}'}_2 })^{-1}\mathbf {a}_2$. $\square$