A unified analytical framework for distributed variable step size LMS algorithms in sensor networks

Abstract

Internet of Things (IoT) is helping to create a smart world by connecting sensors in a seamless fashion. With the forthcoming fifth generation (5G) wireless communication systems, IoT is becoming increasingly important since 5G will be an important enabler for the IoT. Sensor networks for IoT are increasingly used in diverse areas, e.g., in situational and location awareness, leading to proliferation of sensors at the edge of physical world. There exist several variable step-size strategies in literature to improve the performance of diffusion-based Least Mean Square (LMS) algorithm for estimation in wireless sensor networks. However, a major drawback is the complexity in the theoretical analysis of the resultant algorithms. Researchers use several assumptions to find closed-form analytical solutions. This work presents a unified analytical framework for distributed variable step-size LMS algorithms. This analysis is then extended to the case of diffusion based wireless sensor networks for estimating a compressible system and steady state analysis is carried out. The approach is applied to several variable step-size strategies for compressible systems. Theoretical and simulation results are presented and compared with the existing algorithms to show the superiority of proposed work.

Appendix A

Here, we present the detailed mean-square analysis. Applying the expectation operator to the weighting matrix of (14) gives

$$\begin{aligned} {\mathbb {E}}\left[ {{\hat{\varvec{\Sigma }}}} \right]= & {} {\mathbf{G}}^T {\varvec{\Sigma }} {\mathbf{G}} - {\mathbf{G}}^T {\varvec{\Sigma }} {\mathbb {E}}\left[ {\mathbf{Y}}(i) {\mathbf{U}}(i) \right] \nonumber \\&-\,{\mathbb {E}}\left[ {\mathbf{U}}^T(i) {\mathbf{Y}}^T(i) \right] {\varvec{\Sigma }} {\mathbf{G}}\nonumber \\&+\,{\mathbb {E}}\left[ {\mathbf{U}}^T(i) {\mathbf{Y}}^T(i) {\varvec{\Sigma }} {\mathbf{Y}}(i) {\mathbf{U}}(i) \right] \nonumber \\= & {} {\mathbf{G}}^T {\varvec{\Sigma }} {\mathbf{G}} -\,{\mathbf{G}}^T {\varvec{\Sigma }} {\mathbf{G}}{\mathbb {E}}\left[ {{\mathbf{D}}(i) } \right] {\mathbb {E}}\left[ {{\mathbf{U}}^T(i) {\mathbf{U}}(i) } \right] \nonumber \\&-\,{\mathbb {E}}\left[ {{\mathbf{U}}^T(i) {\mathbf{U}}(i) } \right] {\mathbb {E}}\left[ {{\mathbf{D}}(i) } \right] {\mathbf{G}}^T {\varvec{\Sigma }} {\mathbf{G}} \nonumber \\&+\,{\mathbb {E}}\left[ {{\mathbf{U}}^T(i) {\mathbf{Y}}^T(i) {\varvec{\Sigma }} {\mathbf{Y}}(i) {\mathbf{U}}(i) } \right] , \end{aligned}$$

(24)

For ease of notation, we denote ${\mathbb {E}}\left[ {{\hat{\varvec{\Sigma }}}} \right] = {\varvec{\Sigma }} '$ for the remaining analysis.

Next, using the Gaussian transformed variables as gives in Sect. 3.2, (14) and (24) are rewritten, respectively, as

$$\begin{aligned} {\mathbb {E}}\left[ {\left\| {{{\bar{\mathbf{w}}}}\left( i+1\right) } \right\| _{\bar{\varvec{\Sigma }} }^2 } \right]= & {} {\mathbb {E}}\left[ {\left\| {{{\bar{\mathbf{w}}}}(i) } \right\| _{\bar{\varvec{\Sigma }} '}^2 } \right] \nonumber \\&+\, {\mathbb {E}}\left[ {{\mathbf{v}}^T(i) {{\bar{\mathbf{Y}}}}^T(i) \bar{\varvec{\Sigma }} {{\bar{\mathbf{Y}}}}(i) {\mathbf{v}}(i) } \right] , \end{aligned}$$

(25)

and

$$\begin{aligned} \bar{\varvec{\Sigma }} '= & {} {{\bar{\mathbf{G}}}}^T \bar{\varvec{\Sigma }} {{\bar{\mathbf{G}}}} - {{\bar{\mathbf{G}}}}^T \bar{\varvec{\Sigma }} {{\bar{\mathbf{G}}}}{\mathbb {E}}\left[ {{\mathbf{D}}(i) } \right] {\mathbb {E}}\left[ {{{\bar{\mathbf{U}}}}^T(i) {{\bar{\mathbf{U}}}}(i) } \right] \nonumber \\&-\, {\mathbb {E}}\left[ {{{\bar{\mathbf{U}}}}^T(i) {{\bar{\mathbf{U}}}}(i) } \right] {\mathbb {E}}\left[ {{\mathbf{D}}(i) } \right] {{\bar{\mathbf{G}}}}^T \bar{\varvec{\Sigma }} {{\bar{\mathbf{G}}}} \nonumber \\&+\, {\mathbb {E}}\left[ {{{\bar{\mathbf{U}}}}^T(i) {{\bar{\mathbf{Y}}}}(i) \bar{\varvec{\Sigma }} {{\bar{\mathbf{Y}}}}(i) {{\bar{\mathbf{U}}}}(i) } \right] \nonumber \\= & {} {{\bar{\mathbf{G}}}}^T \bar{\varvec{\Sigma }} {{\bar{\mathbf{G}}}} - {{\bar{\mathbf{G}}}}^T \bar{\varvec{\Sigma }} {{\bar{\mathbf{G}}}}{\mathbb {E}}\left[ {{\mathbf{D}}(i) } \right] {\varvec{\Lambda }}\nonumber \\&-\, {\varvec{\Lambda }} {\mathbb {E}}\left[ {{\mathbf{D}}(i) } \right] {{\bar{\mathbf{G}}}}^T \bar{\varvec{\Sigma }} {{\bar{\mathbf{G}}}} \nonumber \\&+\, {\mathbb {E}}\left[ {{{\bar{\mathbf{U}}}}^T(i) {{\bar{\mathbf{Y}}}}(i) \bar{\varvec{\Sigma }} {{\bar{\mathbf{Y}}}}(i) {{\bar{\mathbf{U}}}}(i) } \right] , \end{aligned}$$

(26)

where ${{\bar{\mathbf{Y}}}}(i) = {{\bar{\mathbf{G}} D}}(i) {{{\bar{\mathbf{U}}}}^T(i) }$ and ${\mathbb {E}}\left[ {{{\bar{\mathbf{U}}}}^T(i) {{\bar{\mathbf{U}}}}(i) } \right] = {\varvec{\Lambda }}$.

The two terms that need to be solved are ${\mathbb {E}}\Big [ {\mathbf{v}}^T(i) {{\bar{\mathbf{Y}}}}^T(i) \bar{\varvec{\Sigma }}{{\bar{\mathbf{Y}}}}(i) {\mathbf{v}}(i) \Big ]$ and ${\mathbb {E}}\left[ {{{\bar{\mathbf{U}}}}^T(i) {{\bar{\mathbf{Y}}}}(i) \bar{\varvec{\Sigma }} {{\bar{\mathbf{Y}}}}(i) {{\bar{\mathbf{U}}}}(i) } \right]$. Using the $\text{ bvec }\{.\}$ operator and the block Kronecker product, denoted by $\odot$ [34], the two terms are simplified as

$$\begin{aligned} {\mathbb {E}}\left[ {{\mathbf{v}}^T(i) {{\bar{\mathbf{Y}}}}^T(i) \bar{\varvec{\Sigma }} {{\bar{\mathbf{Y}}}}(i) {\mathbf{v}}(i) } \right] = {\mathbf{b}}^T(i) \bar{\varvec{\sigma }}, \end{aligned}$$

(27)

and

$$\begin{aligned}&\text{ bvec } \left\{ {{\mathbb {E}}\left[ {{{\bar{\mathbf{U}}}}^T(i) {{\bar{\mathbf{Y}}}}^T(i) \bar{\varvec{\Sigma }} {{\bar{\mathbf{Y}}}}(i) {{\bar{\mathbf{U}}}}(i) } \right] } \right\} \nonumber \\&\quad = \left( {{\mathbb {E}}\left[ {{\mathbf{D}}(i) \odot {\mathbf{D}}(i) } \right] } \right) \mathbf{A}\left( {{\mathbf{G}}^T \odot {\mathbf{G}}^T } \right) \bar{\varvec{\sigma }}, \end{aligned}$$

(28)

where $\bar{\varvec{\sigma }} = \text{ bvec } \left\{ {\bar{\varvec{\Sigma }} } \right\} ,{\mathbf{b}}(i) = \text{ bvec } \left\{ {{\mathbf{R}}_{\mathbf{v}} {\mathbb {E}}\left[ {{\mathbf{D}}^2(i) } \right] {\varvec{\Lambda }} } \right\} ,{\mathbf{R}}_{\mathbf{v}} = {\varvec{\Lambda }} _{\mathbf{v}} \odot \mathrm{{\mathbf{I}}}_M,\varvec{\Lambda }_{\mathbf{v}}$ is a diagonal noise variance matrix for the network and ${\mathbf{A}} = \text{ diag } \left\{ {{\mathbf{A}}_1 ,{\mathbf{A}}_2 ,\ldots ,{\mathbf{A}}_N } \right\}$ [10], with each matrix ${\mathbf{A}}_k$ defined as

$$\begin{aligned} {\mathbf{A}}_k= & {} \text{ diag } \left\{ {\varvec{\Lambda }} _1 \otimes {\varvec{\Lambda }} _k ,\ldots ,{\varvec{\lambda }} _k {\varvec{\lambda }} _k^T \right. \nonumber \\&\left. +\, 2{\varvec{\Lambda }} _k \otimes {\varvec{\Lambda }} _k ,\ldots ,{\varvec{\Lambda }} _N \otimes {\varvec{\Lambda }} _k \right\} , \end{aligned}$$

(29)

where ${\varvec{\Lambda }}_k$ is the diagonal eigenvalue matrix and $\lambda _k$ is the corresponding eigenvalue vector for node k. Applying the $\text{ bvec }\{.\}$ operator to (26) and simplifying gives

$$\begin{aligned} \text{ bvec } \left\{ {\bar{\varvec{\Sigma }} '} \right\} = \bar{\varvec{\sigma }} ' = {\mathbf{F}}(i) \bar{\varvec{\sigma }}, \end{aligned}$$

(30)

where ${\mathbf{F}}(i)$ is given by (18). Thus, (14) is rewritten as

$$\begin{aligned} {\mathbb {E}}\left[ {\left\| {{{\bar{\mathbf{w}}}}\left( i+1\right) } \right\| _{\bar{\varvec{\sigma }} }^2 } \right] = {\mathbb {E}}\left[ {\left\| {{{\bar{\mathbf{w}}}}(i) } \right\| _{{\mathbf{F}}(i) \bar{\varvec{\sigma }} }^2 } \right] + {\mathbf{b}}^T(i) \bar{\varvec{\sigma }}, \end{aligned}$$

(31)

which characterizes the transient behavior of the network. Although not explicitly visible from (31), (18) clearly shows the effect of the VSS algorithm on the performance of the algorithm through the presence of the diagonal step-size matrix ${\mathbf{D}} (i)$.

Now, using (31) and (18), the analysis iterates as

$$\begin{aligned} {\mathbb {E}}\left[ {\left\| {{{\bar{\mathbf{w}}}}\left( 0\right) } \right\| _{\bar{\varvec{\sigma }} }^2 } \right]= & {} \left\| {{{\bar{\mathbf{w}}}}^{(o)} } \right\| _{\bar{\varvec{\sigma }} }^2,\\ {\mathbf{F}}(0)= & {} \left[ {\mathbf{I}}_{M^2 N^2 } - \left( {{\mathbf{I}}_{MN} \odot \varvec{\Lambda } {\mathbb {E}}\left[ {\mathbf{D}} (0) \right] } \right) \right. \nonumber \\&\left. - \,\left( {\varvec{\Lambda } {\mathbb {E}}\left[ {\mathbf{D}} (0) \right] \odot {\mathbf{I}}_{MN} } \right) \right. \nonumber \\&\left. +\, \left( {{\mathbb {E}}\left[ {{\mathbf{D}}(0) \odot {\mathbf{D}}(0)} \right] } \right) \mathbf{A} \right] \\&.\left( {{\mathbf{G}}^T \odot {\mathbf{G}}^T } \right) , \end{aligned}$$

where ${{\mathbb {E}}\left[ {\mathbf{D}} (0) \right] = \text{ diag } \left\{ {\mu _{1}(0) {{\mathbf{I}}}_M ,\ldots ,\mu _{N}(0) \mathrm{{\mathbf{I}}}_M } \right\} }$ as these are the initial step-size values. The first iterative update is given by

$$\begin{aligned} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(1) \right\| ^2_{\bar{\varvec{\sigma }}} \right]= & {} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(0) \right\| ^2_{\mathbf{F}(0) \bar{\varvec{\sigma }}} \right] + {\mathbf{b}}^T(0) \bar{\varvec{\sigma }} \\= & {} \left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{\mathbf{F}(0) \bar{\varvec{\sigma }}} + {\mathbf{b}}^T(0) \bar{\varvec{\sigma }} \\ {\mathbf{F}}(1)= & {} \left[ {\mathbf{I}}_{M^2 N^2 } - \left( {{\mathbf{I}}_{MN} \odot \varvec{\Lambda } {\mathbb {E}}\left[ {\mathbf{D}} (1) \right] } \right) \right. \nonumber \\&\left. -\,\left( {\varvec{\Lambda } {\mathbb {E}}\left[ {\mathbf{D}} (1) \right] \odot {\mathbf{I}}_{MN} } \right) \right. \nonumber \\&\left. +\,\left( {{\mathbb {E}}\left[ {{\mathbf{D}}\left( 1\right) \odot {\mathbf{D}}(1)} \right] } \right) \mathbf{A} \right] \\&.\left( {{\mathbf{G}}^T \odot {\mathbf{G}}^T } \right) , \end{aligned}$$

where ${\mathbf{b}}(0) = \text{ bvec } \left\{ {{\mathbf{R}}_{\mathbf{v}} {\mathbb {E}}\left[ {{\mathbf{D}}^2(0) } \right] {\varvec{\Lambda }} } \right\} ,{\mathbb {E}}\left[ {\mathbf{D}} (1) \right]$ is the first step-size update. The matrix ${\mathbf{F}}(i)$ is updated with (18) using the ith update for the step-size matrix ${\mathbb {E}}\left[ {\mathbf{D}} (i) \right]$, which is updated using the VSS approach that is being applied to the algorithm. The second iterative update is given by

$$\begin{aligned} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(2) \right\| ^2_{\bar{\varvec{\sigma }}} \right]= & {} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(1) \right\| ^2_{\mathbf{F}(1) \bar{\varvec{\sigma }}} \right] + {\mathbf{b}}^T(1) \bar{\varvec{\sigma }} \\= & {} \left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{\mathbf{F}(0)\mathbf{F}(1) \bar{\varvec{\sigma }}} \nonumber \\&+\,{\mathbf{b}}^T(0) \mathbf{F}(1) \bar{\varvec{\sigma }} \nonumber \\&+\,{\mathbf{b}}^T(1) \bar{\varvec{\sigma }}. \end{aligned}$$

Continuing, the third iterative update is given by

$$\begin{aligned} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(3) \right\| ^2_{\bar{\varvec{\sigma }}} \right]= & {} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(2) \right\| ^2_{\mathbf{F}(2) \bar{\varvec{\sigma }}} \right] + {\mathbf{b}}^T\left( 2 \right) \bar{\varvec{\sigma }} \\= & {} \left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{\mathbf{F}(0)\mathbf{F}(1)\mathbf{F}(2) \bar{\varvec{\sigma }}} + {\mathbf{b}}^T\left( 2 \right) \bar{\varvec{\sigma }} \nonumber \\&+\,{\mathbf{b}}^T(0) \mathbf{F}(1)\mathbf{F}(2) \bar{\varvec{\sigma }} + {\mathbf{b}}^T(1) \mathbf{F}(2) \bar{\varvec{\sigma }} \\= & {} \left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{{{\mathcal {A}}}(2)\mathbf{F}(2) \bar{\varvec{\sigma }}} + {\mathbf{b}}^T\left( 2 \right) \bar{\varvec{\sigma }}\nonumber \\&+\, \left[ {\sum \limits _{k = 0}^1 { \left\{ {\mathbf{b}}^T\left( k \right) \prod \limits _{m = k+1}^{2} {{\mathbf{F}}(m)} \right\} } } \right] {\varvec{\sigma }}, \end{aligned}$$

where the weighting matrix ${{\mathcal {A}}}(2) = \mathbf{F}(0)\mathbf{F}(1)$. Similarly, the fourth iterative update is given by

$$\begin{aligned} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(4) \right\| ^2_{\bar{\varvec{\sigma }}} \right]= & {} \left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{{{\mathcal {A}}}(3) \mathbf{F}(3) \bar{\varvec{\sigma }}} + {\mathbf{b}}^T\left( 3 \right) \bar{\varvec{\sigma }} \nonumber \\&+\,\left[ {\sum \limits _{k = 0}^2 { \left\{ {\mathbf{b}}^T\left( k \right) \prod \limits _{m = k+1}^{3} {{\mathbf{F}}(m)} \right\} } } \right] {\varvec{\sigma }}, \end{aligned}$$

where the weighting matrix ${{\mathcal {A}}}(3) = {{\mathcal {A}}}(2) \mathbf{F}(2)$. Now, from the third and fourth iterative updates, we generalize the recursion for the ith update as

$$\begin{aligned} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(i) \right\| ^2_{\bar{\varvec{\sigma }}} \right]= & {} \left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{{{\mathcal {A}}}(i-1)\mathbf{F}(i-1) \bar{\varvec{\sigma }}} + {\mathbf{b}}^T\left( i-1 \right) \bar{\varvec{\sigma }} \nonumber \\&+\,\left[ {\sum \limits _{k = 0}^{i-2} { \left\{ {\mathbf{b}}^T\left( k \right) \prod \limits _{m = k+1}^{i-1} {{\mathbf{F}}(m)} \right\} } } \right] {\varvec{\sigma }}. \end{aligned}$$

(32)

where ${{\mathcal {A}}}(i-1) = {{\mathcal {A}}}(i-2) \mathbf{F}(i-2)$. The recursion for the $(i+1)$th update is given by

$$\begin{aligned}&{\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(i+1) \right\| ^2_{\bar{\varvec{\sigma }}} \right] = \left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{{{\mathcal {A}}}(i)\mathbf{F}(i) \bar{\varvec{\sigma }}} + {\mathbf{b}}^T(i) \bar{\varvec{\sigma }}\nonumber \\&+\,\left[ {\sum \limits _{k = 0}^{i-1} { \left\{ {\mathbf{b}}^T\left( k \right) \prod \limits _{m = k+1}^{i} {{\mathbf{F}}(m)} \right\} } } \right] {\varvec{\sigma }}, \end{aligned}$$

(33)

where ${{\mathcal {A}}}(i) = {{\mathcal {A}}}(i-1) \mathbf{F}(i-1)$. Subtracting (32) from (33) and simplifying gives the overall recursive update equation

$$\begin{aligned} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(i+1) \right\| ^2_{\bar{\varvec{\sigma }}} \right]= & {} {\mathbb {E}} \left[ \left\| {\bar{\mathbf{w}}}(i) \right\| ^2_{\bar{\varvec{\sigma }}} \right] \nonumber \\&+\,\left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{{{\mathcal {A}}}(i)\mathbf{F}(i) \bar{\varvec{\sigma }}} \nonumber \\&-\,\left\| {\bar{\mathbf{w}}}^{(o)} \right\| ^2_{{{\mathcal {A}}}(i-1)\mathbf{F}(i-1) \bar{\varvec{\sigma }}} \nonumber \\&+\,{\mathbf{b}}^T(i) \bar{\varvec{\sigma }} - {\mathbf{b}}^T\left( i-1 \right) \bar{\varvec{\sigma }} \nonumber \\&+\,\left[ {\sum \limits _{k = 0}^{i-1} { \left\{ {\mathbf{b}}^T\left( k \right) \prod \limits _{m = k+1}^{i} {{\mathbf{F}}(m)} \right\} } } \right] \bar{\varvec{\sigma }} \nonumber \\&-\,\left[ {\sum \limits _{k = 0}^{i-2} { \left\{ {\mathbf{b}}^T\left( k \right) \prod \limits _{m = k+1}^{i-1} {{\mathbf{F}}(m)} \right\} } } \right] \bar{\varvec{\sigma }},\nonumber \\ \end{aligned}$$

(34)

Simplifying (34) and rearranging the terms gives the final recursive update equation (17), where

$$\begin{aligned} {{\mathcal {B}}}(i)= & {} \sum \limits _{k = 0}^{i-2} { \left\{ {\mathbf{b}}^T\left( k \right) \prod \limits _{m = k+1}^{i-1} {{\mathbf{F}}(m)} \right\} } \nonumber \\&+\,\mathbf{b}^T (i-1) \mathbf{I}_{M^2 N^2}. \end{aligned}$$

(35)

The final set of iterative equations for the mean-square learning curve are given by (17), (18), (19) and (20).