Distributed localization for IoT with multi-agent reinforcement learning

Abstract

Localization has become one of the important techniques for Internet of Things (IoT). However, most existing localization methods need a central controller and operate on an off-line manner, which cannot satisfy the requirements of real-time IoT applications. In order to address this issue, a novel distributed localization scheme based on multi-agent reinforcement learning (MARL) is proposed. The localization problem is first reformulated as a stochastic game for maximizing the sum of the negative localization error. Each non-anchor node is then modeled as an intelligent agent, where its action space corresponds to possible locations. After that, we invoke a MARL framework on the basis of conventional Q-learning framework to learn the optimal policy, and to maximize the long-term expected reward. The novel strategy is also proposed to reduce the localization error. Extensive simulations demonstrate that the proposed localization method is superior to game theoretic-based distributed localization algorithm and virtual force-based distributed localization algorithm in terms of both localization accuracy and convergence speed, and is suitable for on-line localization scenarios.

Appendix A: Proof of Theorem 1

We proof the Theorem 1 based on theory in [42]. The Q-learning algorithm is iterative process of the stochastic game. Therefore, we consider to use stochastic approximation theory to prove the convergence of the Q-learning. The result of approximation theory given in [42] can be concluded in Lemma 1.

Lemma 1

The random iterative process $\varDelta ^{t+1}(x)$, which is given by

$$\begin{aligned} {{\varDelta ^{t+1}}({x})} = \left( 1 - {\alpha ^{t}}({x})\right) {{\varDelta ^{t}}({x})} + {\beta ^{t}}({x}){\varPhi ^{t}}({x}), \end{aligned}$$

(A.1)

converges to zero with the probability 1 only if the following conditions are met.

• 1. The state space is finite;

• 2. $\sum _{t=0}^{+\infty }{\alpha ^t} = {\infty }$, $\sum _{t=0}^{+\infty }{({\alpha ^t})}^2 {<} {\infty }$, $\sum _{t=0}^{+\infty }{\beta ^t} = {\infty }$, $\sum _{t=0}^{+\infty }{({\beta ^t})}^2 {<} {\infty }$, and $E\{{\beta }^{t}({x})|{\varGamma }^{t}\}$ $\le$ $E\{{\alpha }^{t}({x})|{\varGamma }^{t}\}$ with the probability 1;

• 3. $\parallel {E\{{\varPhi }^{t}({x})|{\varGamma }^{t}\}}\parallel _\mathrm{w} \le$ $\delta \parallel \varDelta ^{t}{\parallel }_\mathrm{w}$, where ${\delta }\in (0,1)$;

• 4. Var$\{{\varPhi }^{t}({x})|{\varGamma }^{t}\}$ $\le$ Z$(1+\parallel {\varDelta ^{t}}\parallel _\mathrm{w})^{2}$, where Z is a constant and Z > 0.

Here, $\varGamma ^{t}$ = {${\varDelta ^{t}},{\varDelta ^{t-1}},\ldots ,{\varPhi ^{t-1}},\ldots ,{\alpha ^{t-1}},\ldots ,{\beta ^{t-1}}$} denotes the past situation before time slot t. Symbol $\parallel \cdot \parallel _\mathrm{w}$ denotes the weighted maximum norm.

Now, we prove the Theorem 1 as follows.

The updating rule of Q-value is shown in (27), we now let both side of (27) subtract $Q_{i}^{*}({s _{i}},{a_i})$, we can derive

$$\begin{aligned} {{\varDelta _{i}^{t+1}}\left( {s_i},{a_i}\right) } = \left( 1 - {\alpha ^{t}}\right) {{\varDelta _{i}^{t}}\left( {s_i},{a_i}\right) } + {\alpha ^{t}}{\gamma }{\varPhi ^{t}}\left( {s_i},{a_i}\right) , \end{aligned}$$

(A.2)

where

$$\begin{aligned} {{\varDelta _{i}^{t+1}}\left( {s_i},{a_i}\right) }&= Q_{i}^{t}\left( {s_{i}},{a_i}\right) - {Q_{i}^{*}\left( {s_{i}},{a_i}\right) }, \end{aligned}$$

(A.3)

$$\begin{aligned} {{\varPhi _{i}^{t+1}}\left( {s_i},{a_i}\right) }&= {r_i^t} + {\gamma }\underset{{{a_i^{'}}}\in {{\mathcal {A}_i}}}{max}{Q_i^t}\left( {s_i^{'}},{a_i^{'}}\right) - {Q_{i}^{*}\left( {s_{i}},{a_i}\right) }. \end{aligned}$$

(A.4)

Observing the above analysis, we know that the Q-learning algorithm is the specific case of Lemma 1 with $\beta ^{t}$ = $\alpha ^{t}$.

Then, we prove the Q-learning algorithm satisfy the condition (3) and (4) in Lemma 1. We first introduce the contraction mapping.

Definition 4

For a set $\mathcal {Y}$, the mapping H: $\mathcal {Y} \rightarrow \mathcal {Y}$ is a contraction mapping, if there is a constant $\gamma$ and ${\gamma }\in (0,1)$, for any ${y_1}, {y_2}$ have

$$\begin{aligned} {\parallel \mathbf{H}{y_1}-\mathbf{H}{y_2}\parallel \le {\gamma }\parallel {y_1}-{y_2}\parallel }. \end{aligned}$$

(A.5)

Proposition 3

There is a contraction mapping H for function q and the q is the optimal Q-function in (A.7). We have

$$\begin{aligned} {{\parallel \mathbf{H}{q_1}({s_i},{a_i})-\mathbf{H}{q_2}({s_i},{a_i})\parallel _\infty } \le {\gamma }{\parallel {q_1}({s_i},{a_i})-{q_2}({s_i},{a_i})\parallel }_\infty }. \end{aligned}$$

(A.6)

Proof

The optimal Q-function in (26) can be reformulated to

$$\begin{aligned} {Q_{i}^{*}}\left( {s_{i}},{a_{i}}\right) =&{\sum _{{s_{i}^{'}}}}{{T}\left( {s_{i}},{a_{i}},{s_{i}^{'}}\right) }\nonumber \\&\times {\left[ {R_{i}}{\left( {s_{i}},{a_{i}},{s_{i}^{'}}\right) }+{\gamma }{\max \limits _{a_{i}^{'}}}{Q_{i}^{*}}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) }\right] }. \end{aligned}$$

(A.7)

Then, we have

$$\begin{aligned} \mathbf{H}{q}{\left( s_i,a_i\right) } =&{\sum _{{s_{i}^{'}}}}{{T}\left( {s_{i}},{a_{i}},{s_{i}^{'}}\right) }\nonumber \\&\times {\left[ {R_{i}}{\left( {s_{i}},{a_{i}},{s_{i}^{'}}\right) }+{\gamma }{\max \limits _{a_{i}^{'}}}{q}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) }\right] }. \end{aligned}$$

(A.8)

In order to derive the in Eq. (A.6), we make the following calculations.

$$\begin{aligned}&{\parallel \mathbf{H}{q_1\left( {s_{i},{a_{i}}}\right) } - \mathbf{H}{q_2\left( {s_{i},{a_{i}}}\right) }\parallel }_\infty \nonumber \\&\quad = {\max \limits _{s_{i},{a_i}}}{\gamma }{\bigg |}{\sum _{{s_{i}^{'}}}}{{T}\left( {s_{i}},{a_{i}},{s_{i}^{'}}\right) }\left[ {\max \limits _{a_{i}^{'}}}{q_1}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) } - {\max \limits _{a_{i}^{'}}}{q_2}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) }\right] {\bigg |}\nonumber \\&\quad \le {\max \limits _{s_{i},{a_i}}}{\gamma }{\sum _{{s_{i}^{'}}}}{{T}\left( {s_{i}},{a_{i}},{s_{i}^{'}}\right) }{\bigg |}{\max \limits _{a_{i}^{'}}}{q_1}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) } - {\max \limits _{a_{i}^{'}}}{q_2}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) }{\bigg |}\nonumber \\&\quad \le {\max \limits _{s_{i},{a_i}}}{\gamma }{\sum _{{s_{i}^{'}}}}{{T}({s_{i}},{a},{s_{i}^{'}})}{\max \limits _{a_{i}^{'}}}{\bigg |}{q_1}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) } - {q_2}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) }{\bigg |}\nonumber \\&\quad = {\max \limits _{s_{i},{a_i}}}{\gamma }{\sum _{{s_{i}^{'}}}}{{T}\left( {s_{i}},{a},{s_{i}^{'}}\right) }{\parallel }{q_1}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) } - {q_2}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) }{\parallel }_\infty \nonumber \\&\quad = {\gamma }{\parallel }{q_1}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) } - {q_2}{\left( {s_{i}^{'}},{a_{i}^{'}}\right) }{\parallel }_\infty . \end{aligned}$$

(A.9)

We can derive the first equation in (A.9) from the definition of q, the second and the third inequations follow the properties of absolute value inequalities. The definition of infinity norm is used in the fourth equation. After the maximum calculation, we can derive the last equation. $\square$

According to (A.4) and (A.8), we have

$$\begin{aligned} {E}\left\{ {{\varPhi _{i}^{t+1}}\left( {s_i},{a_i}\right) }\right\}&= {\sum _{{s_{i}^{'}}}}{{T}\left( {s_{i}},{a_{i}},{s_{i}^{'}}\right) }\nonumber \\&\quad \times \,\left[ {r_i^t} + {\gamma }\underset{{{a_i^{'}}}\in {{\mathcal {A}_i}}}{max}{Q_i^t}\left( {s_i^{'}},{a_i^{'}}\right) - {Q_{i}^{*}\left( {s_{i}},{a_i}\right) }\right] \nonumber \\&= \mathbf{H}{Q_{i}^{t}}\left( {s_{i},{a_{i}}}\right) - {Q_{i}^{*}}\left( {s_{i},{a_{i}}}\right) \nonumber \\&= \mathbf{H}{Q_{i}^{t}}\left( {s_{i},{a_{i}}}\right) - \mathbf{H}{Q_{i}^{*}}\left( {s_{i},{a_{i}}}\right) . \end{aligned}$$

(A.10)

Note that the optimal Q-value ${Q_{i}^{*}}({s_{i},{a_{i}}})$ is a constant so that we have ${Q_{i}^{*}}({s_{i},{a_{i}}})$ = H${Q_{i}^{*}}({s_{i},{a_{i}}})$. Hence, based on (A.3), Proposition 3, and (A.10), we have

$$\begin{aligned} \parallel {E}\{{{\varPhi _{i}^{t+1}}\left( {s_i},{a_i}\right) }\}\parallel _\infty&\le {\gamma }{\parallel {{Q_i^t}\left( {s_i},{a_i}\right) } - {Q_{i}^{*}\left( {s_{i}},{a_i}\right) }\parallel }_\infty \nonumber \\&={\gamma }{\parallel }{\varDelta _{i}^{t}\left( {s_{i},{a_{i}}}\right) {\parallel }_\infty }. \end{aligned}$$

(A.11)

According to (A.11), the condition (3) of Lemma 1 is proved. Finally, we consider the condition (4) of Lemma 1.

$$\begin{aligned}&\mathrm{Var}\left\{ {{\varPhi _{i}^{t+1}}\left( {s_i},{a_i}\right) }\right\} \nonumber \\&\quad = {E}\left\{ {r_i^t} + {\gamma }\underset{{{a_i^{'}}}\in {{\mathcal {A}_i}}}{max}{Q_i^t}\left( {s_i^{'}},{a_i^{'}}\right) - {Q_{i}^{*}\left( {s_{i}},{a_i}\right) }\right. \nonumber \\&\qquad \left. -\, \mathbf{H}{Q_{i}^{t}}\left( {s_{i},{a_{i}}}\right) + {Q_{i}^{*}}\left( {s_{i},{a_{i}}}\right) \right\} \nonumber \\&\quad = {E}\left\{ {r_i^t} + {\gamma }\underset{{{a_i^{'}}}\in {{\mathcal {A}_i}}}{max}{Q_i^t}\left( {s_i^{'}},{a_i^{'}}\right) - \mathbf{H}{Q_{i}^{t}}\left( {s_{i},{a_{i}}}\right) \right\} \nonumber \\&\quad = \mathrm{Var}\left\{ {r_i^t} +{\gamma }\underset{{{a_i^{'}}}\in {{\mathcal {A}_i}}}{max}{Q_i^t}\left( {s_i^{'}},{a_i^{'}}\right) \right\} . \end{aligned}$$

(A.12)

As shown in (11), the strategy region for each agent is finite and thus the state and action space of each agent is finite. And the reward of each non-anchor node $v_i$ is only related to its neighbors and the number of the nodes in $N_i$ is finite. Therefore, the value of reward function $r_i^t$ for $v_i$ is bounded at any time slot t. Hence, we have

$$\begin{aligned} \mathrm{Var}\left\{ {r_i^t} + {\gamma }\underset{{{a_i^{'}}}\in {{\mathcal {A}_i}}}{max}{Q_i^t}\left( {s_i^{'}},{a_i^{'}}\right) \right\} \le {Z}{\left( 1+\parallel {\varDelta _{i}^{t}}{\left( {s_{i},a_{i}}\right) }\parallel _\mathrm{w}^{2}\right) }, \end{aligned}$$

(A.13)

where Z is a constant.

Therefore, ${\parallel }{\varDelta _m^{t}}({s_i},{a_i}){\parallel }$ converges to zero with the probability 1, that is $Q_{m}^{t}({s_{i},a_{i}})$ converges to $Q_{m}^{*}({s_{i},a_{i}})$ with the probability 1.