Improvement of energy-efficient resources for cognitive internet of things using learning automata

Abstract

The increasing demand for data collection from various Internet of Things (IoT) devices and limited energy of nodes in these networks led to the complex network conditions. Currently, the role of energy consumption for a large number of interconnected nodes in IoT is one of the research subjects. In many IoT applications, IoT devices are deployed in environments that are difficult to physically access. Therefore, an energy-efficient mechanism is important. This article proposes a new method for energy-efficient improvement based on machine learning from the point of view of cognitive networks, which is estimated using learning automata of network parameters. Then, transmission power of the network nodes to improve energy consumption is adjusted in a self-organized, self-aware and dynamic manner, and the behavior of the network nodes is adapted according to the current conditions of the network. One of the strengths of this method compared to the existing methods is that network conditions are estimated through parameters of Delay (D), Channel Status (S) and data rate, and this mechanism makes all decisions based on the network conditions. The results of the experiments show that examining energy-efficient from the perspective of cognitive network not only leads to the improvement of Quality of Service (QoS) parameters such as operational power and end to end delay in the network, but also an increase in the life of the network compared to other energy-efficient methods.

Appendices

Appendix

Proofs of Convergence for CALA algorithm: We can write the CALA algorithm in the form.

• 1. $Z\left(k+1\right)=Z\left(k\right)+\lambda G\left(Z\left(k\right),\xi \left(k\right)\right)$

  where: $Z(k) = {\left[\mu \left(K\right) \sigma \left(k\right)\right]}^{T}$ is the state at k, $\xi \left(k\right)={\left[x\left(k\right) {\beta }_{x\left(k\right) }{\beta }_{\mu \left(k\right)}\right]}^{T}$ is random and the probability of its taking different values depends on Z(k), G symbolically represents the updating of two components of Z(k) in formulate updating $\mu \left(k\right)\ and\ \sigma \left(k\right).$

  The algorithm updates $\mu \left(k\right) , \sigma \left(k\right)$ parameters of the action probability distribution at k.

• 2. $\left(k+1\right)= \mu \left(k\right)+ \lambda {F}_{1}\left(\mu \left(K\right),\sigma \left(K\right),X\left(K\right),{\beta }_{X\left(K\right)},{\beta }_{\mu \left(K\right)}\right)$, $\sigma \left(K+1\right)= \sigma \left(K\right)+ \lambda {F}_{2}\left(\mu \left(K\right),\sigma \left(K\right),X\left(K\right),{\beta }_{X\left(K\right)}, {\beta }_{\mu \left(K\right)}\right)-\lambda K\left[\sigma \left(k\right)-{\sigma }_{l}\right]$

  where: ${F}_{1}\left(\mu ,\sigma ,X,{\beta }_{X},{\beta }_{\mu }\right)=\left(\frac{{\beta }_{X}-{\beta }_{\mu }}{\phi \left(\sigma \right)}\right)(\frac{X-\mu }{\phi \left(\sigma \right)})$, ${F}_{2}\left(\mu ,\sigma ,X,{\beta }_{X},{\beta }_{\mu }\right)=(\frac{{\beta }_{X}-{\beta }_{\mu }}{\phi \left(\sigma \right)}) [{\left(\frac{X-\mu }{\phi \left(\sigma \right)}\right)}^{2}-1]$.

  x(k) is action selected at k and ${\beta }_{X\left(K\right)}$, ${\beta }_{\mu \left(K\right)}$ are the reinforcements for the two actions x(k) and $\mu (k)$ respectively, ${F}_{1}\ and\ {F}_{2}$ are functions with five variables.

  f(x) Denote the reward probability function.

• 3. $f\left(x\right)=E\left[{\beta }_{x\left(k\right)}\left|x\left(k\right)=x\right)\right]$

  where ${\beta }_{x(k)}$ is reinforcement and $x\left(k\right)$ is action selected at k.

• 4. $\mathrm{J}\left(\mu ,\sigma \right)=\int f\left(x\right) d N\left(\mu ,\sigma \right)=\int f\left(x\right) d N\left(\mu ,\sigma \right)\ dx$

  where N ($\mu,\ \sigma$) = $\frac{1}{\sigma \sqrt{2\pi }} {e}^{-\frac{{\left(x-\mu \right)}^{2}}{2{\sigma }^{2}}}$ is the normal density. (In general, d N (a, b) denote integration with respect to the normal distribution with mean a and variance ${b}^{2}$). With a simple coordinate transformation, we can rewrite formula 4 as:

• 5. $\mathrm{J}\left(\mu ,\sigma \right)=\int f\left(\sigma x+\mu \right)\ d\ N\left(\mathrm{0,1}\right)$, Assuming that $\mu =0$, $\sigma =1$

  The g is a function of X, have two Components, ${g}_{1}, {g}_{2}$

  We can equivalently denote is as a function of two variables: $\mu , \sigma$, we can calculate ${g}_{1}, {g}_{2}$ as follows:

• 6. ${g}_{1}\left(\mu ,\sigma \right)$ = E[${F}_{1}(\mu (K),\sigma (K),X(K),\beta \_X(K) ,\beta \_\mu (K)|\mu (k)=\mu , \sigma (k)= \sigma )$] = $\int \left\{E\left[\frac{{\beta }_{X}-{\beta }_{\mu }}{\phi \left(\sigma \right)}\right]|x\left(k\right)=x, \mu \left(k\right)=\mu , \sigma \left(k\right)=\sigma \right\}\left(\frac{X-\mu }{\phi \left(\sigma \right)}\right)$ d N($\mu ,\phi \left(\sigma \right)$) = $\int [(\frac{f\left(x\right)-f\left(\mu \right)}{\phi \left(\sigma \right)}]$)($\frac{X-\mu }{\phi \left(\sigma \right)}$) d N($\mu ,\phi (\sigma$) = $\int \frac{f\left(x\right)}{\phi \left(\sigma \right)}$ ($\frac{X-\mu }{\phi \left(\sigma \right)}$) d N($\mu ,\phi (\sigma$)

  The last step above follows because $f(\mu )$ is independent of x and hence comes out of the integral and the rest of integral in the term is zero, with respect $\mu$, we now get.

• 7. ${g}_{1}\left(\mu ,\sigma \right)=\frac{\partial J}{\partial \mu }\left(\mu ,\phi \left(\sigma \right)\right)$

  where $\frac{\partial J}{\partial \mu}$ is partial derivative of the ratio J function and $\mu$ and $\phi \left(\sigma \right) ={\sigma }_{l}\hspace{0.33em}for\hspace{0.33em}\sigma \le {\sigma }_{l},\ \sigma \hspace{0.33em}for\hspace{0.33em}\sigma \succ {\sigma }_{l}\succ 0$

  In a similar fashion, we can show that

• 8. ${g}_{2}\left(\mu ,\sigma \right)= E\left[{F}_{2}\left(\mu \left(K\right),\sigma \left(K\right),X\left(K\right),{\beta }_{X\left(K\right)}, {\beta }_{\mu \left(K\right)}\right)-K\left(\sigma \left(k\right)-{\sigma }_{l}\right)\left|\mu \left(k\right)= \mu , \sigma \left(k\right)= \sigma \right.\right]$ = $\frac{\partial J}{\partial \sigma }$ ($\mu ,\phi \left(\sigma \right)$) – K[$\sigma -{\sigma }_{l}$]

  where $\frac{\partial J}{\partial \sigma}$ is partial derivative of the ratio J function and $\sigma ,$ K is constant. Having calculated the g function, the approximating ODE would be $\dot{Z}=g\left(Z\right).$ Since the state here has two components, namely $\mu\ and\ \sigma$ using formulas 7 and 8 the approximating ODE for CALA algorithm is:

• 9. $\frac{d\mu }{dt} = \frac{\partial J}{\partial \mu }$ ($\mu ,\phi \left(\sigma \right)$)

• 10. $\frac{d\sigma }{dt} = \frac{\partial J}{\partial \sigma }$ ($\mu ,\phi \left(\sigma \right)$) – K [$\sigma -{\sigma }_{l}$]

  This completes calculation of the approximating ODE for CALA algorithm to prove that this approximating ODE, all we need to do is to verify assumptions A1 to A4.

Assumptions

We make following assumptions on the system.

• A1. $\left\{({X}_{k}^{b} , {\xi }_{k}^{b} , k\ge 1)\right\}$ is a Markov process.

• A2. For any appropriate Borel set B. 

  $\mathrm{Prob}\left[{\xi }_{k}^{b} \in B \left|{X}_{k}^{b} , {\xi }_{k-1}^{ b} \right.\right]=Prob\left[{\xi }_{k}^{b} \in B \left|{X}_{k}^{b}\right.\right].$

  That is, conditioned on ${X}_{k}^{b} , {\xi }_{k}^{b}$ is independent of${\xi }_{k-1}^{ b}$.

• A3. Define g: ${R}^{N} \to {R}^{N}$ by g(x) = E [G (${X}_{k},{\xi }_{k}$)$\left|{X}_{k}=x\right.$]

  We assume that g(.) is independent of k and that is globally Lipschutz.

• A4. Define ${\theta }_{k}^{b}$ = G (${X}_{k}^{b}$,${\xi }_{k}^{b}$) – g (${X}_{k}^{b}$)

  We assume that E ${\Vert {\theta }_{k}^{b}\Vert }^{2}<M < \infty$, for some M and $\forall k.$

  From the structure of the Algorithm it is immediately obvious that assumptions A1 and A2 are satisfied. To satisfy A3 and A4, we make the following additional assumptions on the unknown reward function, f(.).

• B1. F(.) is a bounded and is continuously differentiable, Let L denote the bound of F(.).

• B2. The derivative of f, namely, ${f}^{\prime}\left(.\right)$, is globally Lipschutz, that is, $\left|{f}^{\prime}\left(x\right)- {f}^{\prime}\left(y\right)\right| \le {K}_{0}\Vert x-y\Vert$

• B3. Define zero mean random variables ${\tau }_{x}= {\beta }_{x}-f\left(x\right)$. We assume that Var (${\tau }_{x}$)$\le {\sigma }_{M}^{2}$ for some ${\sigma }_{M}<\infty$ and that E [${\tau }_{x}, {\tau }_{y}\left|x,y\right.$] = 0 for x $\ne y.$

  Under the assumptions B1—B3, we verify A3 – A4. under the integral, we get

• 11. $\frac{\partial J}{\partial \mu }\left(\mu , \sigma \right) = \int {f}^{\prime}\left(\sigma x+\mu \right)x\ d\ N\ \left(\mathrm{0,1}\right)$

• 12. $\frac{\partial J}{\partial \sigma }\left(\mu , \sigma \right)= \int {f}^{\prime}\left(\sigma x+\mu \right)x\ d\ N\ \left(\mathrm{0,1}\right)$  

  Now we have:

• 13. $\left|{g}_{1}\left({\mu }_{1},{\sigma }_{1}\right)- {g}_{1}\left({\mu }_{2},{\sigma }_{2}\right)\right| = \left|\frac{\partial J}{\partial \mu }\left({\mu }_{1},{\sigma }_{1}\right)-\frac{\partial J}{\partial \mu }\left({\mu }_{2},{\sigma }_{2}\right)\right| \le \int \left|{f}^{\mathrm{^{\prime}}}\left({\left(\sigma \right)}_{1}x+{\mu }_{1}\right)- {f}^{\mathrm{^{\prime}}}\left({\sigma }_{2}x + {\mu }_{2} \right)\right|$ d N (0,1) $\le {K}_{0} \int \left|{\sigma }_{1}x + {\mu }_{1}- {\sigma }_{2}x- {\mu }_{2} \right| d N \left(\mathrm{0,1}\right) \le {K}_{0}\left|{\mu }_{1}- {\mu }_{2}\right| \int d N \left(\mathrm{0,1}\right)+ { K}_{0}\left|{\sigma }_{1}- {\sigma }_{2}\right| \int \left|x\right| d N \left(\mathrm{0,1}\right) {K}_{1} \Vert \left({\mu }_{1},{\sigma }_{1}\right)-\left({\mu }_{2},{\sigma }_{2}\right)\Vert$

  where ${K}_{1}$ depend on ${K}_{0}$ and first absolute moment of standard normal distribution. Hence can show that

• 14. $\left|{g}_{2}\left({\mu }_{1},{\sigma }_{1}\right)- {g}_{2}\left({\mu }_{2},{\sigma }_{2}\right)\right| \le {{K}^{\mathrm{^{\prime}}}}_{2} \Vert \left({\mu }_{1},{\sigma }_{1}\right)- \left({\mu }_{2},{\sigma }_{2}\right)\Vert$ + K $\left|{\sigma }_{1}- {\sigma }_{2}\right| \le {K}_{2 }\Vert \left({\mu }_{1},{\sigma }_{1}\right)-\left({\mu }_{2},{\sigma }_{2}\right)\Vert$

  The proves that g is globally Lipschitz and thus verify A3, we note here that if instead of assumption B2. We had assumed that ${f}^{\prime}$ in Lipschitz on compact sets then the above proves that g is Lipschitz on compact sets.

• 15. E [${G}_{1}{\left(X,\xi \right)}^{2}\left|X\right.$] = E [${\left(\frac{{\beta }_{X}-{\beta }_{\mu }}{\phi \left(\sigma \right)}.\frac{X-\mu }{\phi \left(\sigma \right)}\right)}^{2}\left|\mu ,\sigma \right.$] = E [E [${\left(\frac{{\beta }_{X}-{\beta }_{\mu }}{\phi \left(\sigma \right)}\right)}^{2} {\left(\frac{X-\mu }{\phi \left(\sigma \right)}\right)}^{2}\left|x,\mu ,\sigma \right.$]$\left|\mu ,\sigma \right.$] = E [${\left(\frac{X-\mu }{\phi \left(\sigma \right)}\right)}^{2}$ E[${\left(\frac{{\beta }_{X}-{\beta }_{\mu }}{\phi \left(\sigma \right)}\right)}^{2} \left|x,\mu ,\sigma \right.$] $\left|\mu ,\sigma \right.$] $\le$ E [${\left(\frac{X-\mu }{\phi \left(\sigma \right)}\right)}^{2} \frac{1}{{\phi }^{2}\left(\sigma \right)}$ [${\left(f\left(x\right)-f\left(\mu \right)\right)}^{2}$ + 2 ${\left({\sigma }_{M}\right)}^{2}$] $\left|\mu ,\sigma \right.$] = $\int {\left(\frac{X-\mu }{\phi \left(\sigma \right)}\right)}^{2}\frac{1}{{\phi }^{2}\left(\sigma \right)}$ [${\left(f\left(x\right)-f\left(\mu \right)\right)}^{2}$ + 2 ${\left({\sigma }_{M}\right)}^{2}$] d N($\mu ,\phi \left(\sigma \right)$) $\le \frac{4{L}^{2}+2{{\sigma }^{2}}_{M}}{{\phi }^{2}\left(\sigma \right)} \int {y}^{2} d N\left(\mathrm{0,1}\right)$

  This shows that E [${G}_{1}^{2}$ (X,$\xi$)$\left|X\right.$] $\le {K}_{4}$ for some constant ${K}_{4}$ which is independent of X, now we have, by definition of J (.,.),

• 16. $\left|\frac{\partial J}{\partial \mu } \left(\mu ,\sigma \right)\right|$ = $\left|\frac{\partial }{\partial \mu } \int f\left(x\right) d N\left(\mu ,\phi \left(\sigma \right)\right)\right|$= $\left|\int f\left(x\right)\frac{x- \mu }{{\phi }^{2}\left(\sigma \right)} d N\left(\mu ,\phi \left(\sigma \right)\right)\right| \le$ L $\int \left|x\right| d N\left(\mathrm{0,1}\right)$.

  Which shows ${g}_{1}$ is also bounded. assumption A4 is satisfied, since all assumptions are satisfied, we can conclude that is the approximating ODE for the CALA algorithm. The main features of the algorithm that result in the specific from of this ODE are as follows.

• 17. J ($\mu ,\sigma$) = $\oint f\left(x\right)d\ N\left(\mu ,\sigma \right)$ = $\oint \frac{1}{\sigma \sqrt{2\pi }} {e}^{-\frac{{\left(x-\mu \right)}^{2}}{2{\sigma }^{2}}}$

  where $N\left(\mu ,\sigma \right)$ denotes the gaussian density Function with mean $\mu$ and standard deviation $\sigma$. Let

• 18. ${F}_{1} \left(\mu , \sigma ,x,\beta ,{\beta }^{\prime}\right)= \frac{(\beta - {\beta }^{\prime})}{\phi (\sigma )}\cdot \frac{(x - \mu )}{\phi (\sigma )}$

• 19. ${F}_{2}\left(\mu , \sigma ,x,\beta ,{\beta }^{\prime}\right) = \frac{(\beta - {\beta }^{\prime})}{\phi (\sigma )}\left[{(\frac{(x - \mu )}{\phi (\sigma )})}^{2}-1\right]$

  Where that these functions capture the prominent terms in the learning algorithm by formula 24 paper.

  We have

• 20. E [${F}_{1}\left(\mu \left(K\right),\sigma \left(K\right),x\left(k\right),{\beta }_{x\left(k\right)},{\beta }_{\mu \left(k\right)}\right) \left|\mu \left(k\right)\right.= \mu , \sigma \left(k\right)= \sigma$] = $\frac{\partial J}{\partial \mu }\left(\mu ,\phi \left(\sigma \right)\right)$

• 21. E [${F}_{2}\left(\mu \left(K\right),\sigma \left(K\right),x\left(k\right),{\beta }_{x\left(k\right)},{\beta }_{\mu \left(k\right)}\right) \left|\mu \left(k\right)= \mu , \sigma \left(k\right)= \sigma \right.$] = $\frac{\partial J}{\partial \sigma }\left(\mu ,\phi \left(\sigma \right)\right)$

  The above equations show that expectation of the changes $\mu \left(K\right)\ and\ \sigma \left(K\right)$ conditioned on their current values, is proportional to the gradient of the average reinforcement, J. the ODE associated with the CALA algorithm is given by

• 22. $\frac{d\mu }{dt} = \frac{\partial J}{\partial \mu }$ ($\mu ,\phi \left(\sigma \right)$), $\frac{d\sigma }{dt} = \frac{\partial J}{\partial \sigma }$ ($\mu ,\phi \left(\sigma \right)$) – K [$\sigma -{\sigma }_{l}$]

• 23. Given any $\Delta\ >0\ and\ any\ Compact\ Set\ \widetilde{K} \in R$, There exists a ${\sigma }^{*} >0$ such that ${SUP}_{\mu \in \widetilde{K}}\left|\frac{\partial J}{\partial \mu } \left(\mu ,\sigma \right)- {f}^{\prime}(\mu )\right| < \Delta$, for $0 <\sigma < {\sigma }^{*}$, This result shows that the Convergence of $\frac{\partial J}{\partial \mu }$ to ${f}^{\prime}\left(\mu \right)= \frac{df(u)}{du}$ as $\sigma \to$ 0 is uniform over all $\mu \epsilon \widetilde{K}$ for any Compact Set $\widetilde{K}$.

• 24. $\frac{\partial J}{\partial \sigma }(\mu ,\phi \left(\sigma \right)-\mathrm{K}\left(\sigma - {\sigma }_{L}\right)=0$.

  Putting all this together, we can conclude that by choosing a small value of ${\sigma }_{l} >0,$ a small step size $\lambda >0 ,$ and sufficiently high K$>0$, we can ensure that the iterates $\mu \left(K\right)$ of the CALA algorithm will be close to maximum of the function f with high probability after a long [39].