Learning algorithms for joint resource block and power allocation in underlay D2D networks

Abstract

We investigate the problem of resource block (RB) and power allocation jointly and in a distributed manner using game theoretic learning solutions, in an underlay device-to-device network where device pairs communicate directly with each other by reusing the spectrum allocated to the cellular users. We formulate the joint RB and power allocation as multi-agent learning problems with discrete strategy sets; and suggest partially distributed and fully distributed learning algorithms to determine the RB and power level to be used by each device pair. The partially distributed algorithms, viz., Fictitious Play and its variant Fading Memory Joint Strategy Fictitious Play with Inertia, achieve Nash Equilibrium (NE) of the sum-rate maximization game in a static wireless environment. The completely distributed and uncoupled Stochastic Learning Algorithm converges to pure strategy NE of the interference mitigation game in a time-varying radio environment. We provide proofs for the existence of NE and convergence of the learning algorithms to the NE. Performance of the proposed schemes are evaluated in log-normal, Rayleigh and Nakagami fading environments and compared with an existing hybrid scheme and a centralized scheme. The simulation results show that the partially distributed schemes give the same performance as the centralized scheme, and the fully distributed scheme gives similar performance as the hybrid scheme but with much reduced signaling and computation overhead.

Appendices

Appendix A: Proof of Theorem 1

We choose the potential function as

$$\begin{aligned} \phi (a_{i},{\varvec{a}}_{-i})= & {} \!-\!\!\sum _{i\in {{\mathscr {M}}}}p_{i}~\left( \sum _{j\in {\{{\mathscr {M}}\setminus \{i\}}\}}p_{j}{\bar{h}}_{ji}^{s_{i}}f(s_{j},s_{i})\!+\!p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \nonumber \\ \end{aligned}$$

(27)

$$\begin{aligned} \phi (a_{i},{\varvec{a}}_{-i})= & {} -U(a_{i},{\varvec{a}}_{-i}) \end{aligned}$$

(28)

where $U(a_{i},{\varvec{a}}_{-i})$ is given by (8) where $a_{i}$ is one of the L possible transmit configurations. Let $\pi (s_{i})$ be the set of all D2D pairs other than i that choose the same RB $s_{i}$.Hence, we have,

$$\begin{aligned} f(s_{i},s_{j})= {\left\{ \begin{array}{ll} 1,\forall j \in \pi (s_i)\\ 0,\forall j \notin \pi (s_i) \end{array}\right. } \end{aligned}$$

For any D2D pair i playing action $a_{i}$ at iteration t, there are three possible choices for $\bar{a_{_{i}}}$ played in iteration $t+1$.

Case-(i) D2D pair i changes channel from $s_{i}$ in iteration t to $\bar{s_{i}}$ in iteration $t+1$ but transmit power is constant at $p_{i}$, i.e., $a_{i}=[s_{i},p_{i}]$ and $\bar{a_{i}}=[\bar{s_{i}},p_{i}]$.

$$\begin{aligned} u_{i}(a_{i},{\varvec{a}}_{-i})= & {} D-p_{i} \left( \sum _{j\in {{\mathscr {M}}\setminus \{i\}}}p_{j}{\bar{h}}_{ji}^{s_{i}}f(s_{i},s_{j}) +p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \\ u_{i}(\bar{a_{i}},{\varvec{a}}_{-i})= & {} D-p_{i}\left( \sum _{j\in {{\mathscr {M}}\setminus \{i\}}}p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}f (\bar{s_{i}},s_{j})+p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}}\right) \end{aligned}$$

$$\begin{aligned}&u_{i}(\bar{a_{i}},{\varvec{a}}_{-i})-u_{i}(a_{i},{\varvec{a}}_{-i})\nonumber \\&\quad = -p_{i}\left( \sum _{j\in {{\mathscr {M}}\setminus \{i\}}}p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}f(\bar{s_{i}},s_{j})+p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}}\right) \nonumber \\&\quad +p_{i}\left( \sum _{j\in {{\mathscr {M}}\setminus \{i\}}}p_{j}{\bar{h}}_{ji}^{s_{i}}f(s_{i},s_{j})+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \end{aligned}$$

(29)

$$\begin{aligned}&u_{i}\left( \bar{a_{i}},{\varvec{a}}_{-i})-u_{i}(a_{i},{\varvec{a}}_{-i})=-p_{i}(\sum _{j\in {\pi (\bar{s_{i}})}}p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}+p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}}\right) \nonumber \\&\quad + p_{i}\left( \sum _{j\in \pi (s_{i})}p_{j}{\bar{h}}_{ji}^{s_{i}}+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \end{aligned}$$

(30)

Change in potential function when D2D pair i switches from RB $s_{i}$ to $\bar{s_{i}}= -$ (change in utility of i $+$ change in utility of D2D pairs in $s_i$ when i switches from $s_{i}$ to $\bar{s_{i}} +$ change in utility of D2D pairs in $\bar{s_i}$ when i switches from $s_{i}$ to $\bar{s_{i}} +$ change in utility of D2D pairs $k\in {{\mathscr {M}}}\setminus \{{i}~\cup ~\pi (s_{i})~\cup ~\pi (\bar{s_i})\}$) when i switches from $s_{i}$ to $\bar{s_{i}}$).

The fourth term in the above summation is zero since D2D pairs k are not affected by the strategy change of D2D pair i. Hence, we have

$$\begin{aligned}&\phi (\bar{a_{i}},{\varvec{a}}_{-i})-\phi (a_{i},{\varvec{a}}_{-i})\nonumber \\&\quad =\!p_{i}\left( \sum _{j\in {\pi (\bar{s_{i}})}}\!-\!p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}-p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}}\!+\!\! \sum _{j\in \pi (s_{i})}p_{j}{\bar{h}}_{ji}^{s_{i}}+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \nonumber \\&\quad -\left( \sum _{j\in \pi (s_{i})}p_ip_{j}{\bar{h}}_{ij}^{\bar{s_{i}}}f(\bar{s_i},s_j)-\!\!\!\sum _{j\in \pi (s_{i})}p_ip_{j}{\bar{h}}_{ij}^{s_{i}}f(s_i,s_j)\right) \nonumber \\&\quad -\left( \sum _{j\in \pi (\bar{s_{i}})}p_ip_{j}{\bar{h}}_{ij}^{\bar{s_{i}}}f(\bar{s_i},s_j)-\!\!\!\sum _{j\in \pi (\bar{s_{i}})}p_ip_{j}{\bar{h}}_{ij}^{s_{i}}f(s_i,s_j)\right) \nonumber \\ \end{aligned}$$

(31)

which reduces to

$$\begin{aligned}&\phi (\bar{a_{i}},{\varvec{a}}_{-i})-\phi (a_{i},{\varvec{a}}_{-i})\nonumber \\&\quad =2p_{i}\left( \sum _{j\in {\pi (\bar{s_{i}})}}-p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}+ \sum _{j\in \pi (s_{i})}p_{j}{\bar{h}}_{ji}^{s_{i}}\right) \nonumber \\&\qquad \quad -p_{i}p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}}+p_{i}p_{o}{\bar{h}}_{oi}^{s_{i}} \end{aligned}$$

(32)

since ${\bar{h}}_{ji}^{\bar{s_{i}}}={\bar{h}}_{ji}^{s_{i}}={\bar{h}}_{ij}^{s_{i}}={\bar{h}}_{ij}^{\bar{s_{i}}}$.

Case-(ii) D2D pair i changes transmit power from $p_{i}$ in iteration t to $\bar{p_{i}}$ in iteration $t+1$ but chooses the same RB $s_i$, i.e.,

$a_{i}=[s_{i},p_{i}]$ and $\bar{a_{i}}=[s_{i},\bar{p_{i}}]$.

$$\begin{aligned} u_{i}(a_{i},{\varvec{a}}_{-i})=D-p_{i}\left( \sum _{j\in {{\mathscr {M}}\setminus \{i\}}}p_{j}{\bar{h}}_{ji}^{s_{i}}f(s_{i},s_{j})+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \\ u_{i}(\bar{a_{i}},{\varvec{a}}_{-i})=D-\bar{p_{i}}\left( \sum _{j\in {{\mathscr {M}}\setminus \{i\}}}p_{j}{\bar{h}}_{ji}^{s_{i}}f(s_{i},s_{j})+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \end{aligned}$$

$$\begin{aligned}&u_{i}(\bar{a_{i}},{\varvec{a}}_{-i})-u_{i}(a_{i},{\varvec{a}}_{-i})\!=\!-\bar{p_{i}}\left( \sum _{j\in {\pi (s_{i})}}p_{j}{\bar{h}}_{ji}^{s_{i}}\!+\!p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \nonumber \\&\quad +p_{i}\left( \sum _{j\in \pi (s_{i})}p_{j}{\bar{h}}_{ji}^{s_{i}}+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \end{aligned}$$

(33)

Change in potential function when D2D pair i switches from transmit power $p_i$ to $\bar{p_i}$ on RB $s_i= -$ (change in utility of i $+$ change in utility of D2D pairs in $s_i$ when i switches from $p_{i}$ to $\bar{p_{i}} +$ change in utility of D2D pairs $k\in {{\mathscr {M}}}\setminus \{{i}~\cup ~\pi (s_{i}\}$ when i switches from $p_{i}$ to $\bar{p_{i}}$).

The third term in the above summation is zero since D2D pairs k are not affected by the strategy change of D2D pair i. Hence, we have

$$\begin{aligned} \begin{aligned}&\phi (\bar{a_{i}},{\varvec{a}}_{-i})-\phi (a_{i},{\varvec{a}}_{-i})\\&\quad = -\bar{p_{i}}\left( \sum _{j\in {\pi (s_{i})}}p_{j}{\bar{h}}_{ji}^{s_{i}}+p_{o}{\bar{h}}_{oi}^{s_{i}})+ p_{i}(\sum _{j\in \pi (s_{i})}p_{j}{\bar{h}}_{ji}^{s_{i}}+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \\&\qquad \quad -\left( \sum _{j\in \pi (s_{i})}\bar{p_i}p_{j}{\bar{h}}_{ij}^{s_{i}}f(s_i,s_j)-\sum _{j\in \pi (s_{i})}p_ip_{j}{\bar{h}}_{ij}^{s_{i}}f(s_i,s_j)\right) \end{aligned} \end{aligned}$$

(34)

Using the same reasoning as in Case-1 we have

$$\begin{aligned} \begin{aligned}&\phi (\bar{a_{i}},{\varvec{a}}_{-i})-\phi (a_{i},{\varvec{a}}_{-i})\\&\quad =-2\bar{p_{i}}\left( \sum _{j\in {\pi (s_{i})}}p_{j}{\bar{h}}_{ji}^{s_{i}}\right) +2p_{i}\left( \sum _{j\in \pi (s_{i})}p_{j}{\bar{h}}_{ji}^{s_{i}}\right) \\&\qquad \qquad -\bar{p_{i}}p_{o}{\bar{h}}_{oi}^{s_{i}}+p_{i}p_{o}{\bar{h}}_{oi}^{s_{i}} \end{aligned} \end{aligned}$$

(35)

Case-(iii) D2D pair i changes RB from $s_{i}$ in iteration t to $\bar{s_{i}}$ in iteration $t+1$ and transmit power from $p_{i}$ to $\bar{p_{i}}$, i.e., $a_{i}=[s_{i},p_{i}]$ and $\bar{a_{i}}=[\bar{s_{i}},\bar{p_{i}}]$.

$$\begin{aligned}&u_{i}(a_{i},{\varvec{a}}_{-i})=D-p_{i}(\sum _{j\in {{\mathscr {M}}\setminus \{i\}}}p_{j}{\bar{h}}_{ji}^{s_{i}}f(s_{i},s_{j})+p_{o}{\bar{h}}_{oi}^{s_{i}})\\&\quad u_{i}(\bar{a_{i}},{\varvec{a}}_{-i})=D-\bar{p_{i}}(\sum _{j\in {{\mathscr {M}}\setminus \{i\}}}p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}f(\bar{s_{i}},s_{j})+p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}}) \end{aligned}$$

$$\begin{aligned} \begin{aligned}&u_{i}(\bar{a_{i}},{\varvec{a}}_{-i})-u_{i}(a_{i},{\varvec{a}}_{-i})\\&\qquad =-\bar{p_{i}}\left( \sum _{j\in {\pi (\bar{s_{i}})}}p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}+p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}}\right) \\&\qquad \quad +p_{i}\left( \sum _{j\in \pi (s_{i})}p_{j}{\bar{h}}_{ji}^{s_{i}}+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \end{aligned} \end{aligned}$$

(36)

Change in potential function when D2D pair i switches from RB $s_{i}$ to $\bar{s_{i}}= -$ (change in utility of i $+$ change in utility of D2D pairs in $s_i$ when i switches RB from $s_{i}$ to $\bar{s_{i}}$ and transmit power from $p_i$ to $\bar{p_i} +$ change in utility of D2D pairs in $\bar{s_i}$ when i switches from $s_{i}$ to $\bar{s_{i}}$ and transmit power from $p_i$ to $\bar{p_i} +$ change in utility of D2D pairs $k\in {{\mathscr {M}}}\setminus \{{i}~\cup ~\pi (s_{i})~\cup ~\pi (\bar{s_i})\}$) when i switches from $s_{i}$ to $\bar{s_{i}}$).

The fourth term in the above summation is zero since D2D pairs k are not affected by the strategy change of D2D pair i. Hence, we have

$$\begin{aligned} \begin{aligned}&\phi (\bar{a_{i}},{\varvec{a}}_{-i})-\phi (a_{i},{\varvec{a}}_{-i})\\&\quad \!=\!-\bar{p_{i}}\left( \sum _{j\in {\pi (\bar{s_{i}})}}p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}+p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}}\right) \\&+p_{i}\left( \sum _{j\in \pi (s_{i})}p_{j}{\bar{h}}_{ji}^{s_{i}}+p_{o}{\bar{h}}_{oi}^{s_{i}}\right) \\&\quad \!-\!\left( \sum _{j\in \pi (s_{i})}\bar{p_i}p_{j}{\bar{h}}_{ij}^{\bar{s_{i}}}f(\bar{s_i},s_j)-\sum _{j\in \pi (s_{i})}p_ip_{j}{\bar{h}}_{ij}^{s_{i}}f(s_i,s_j)\right) \\&\quad \!-\!\left( \sum _{j\in \pi (\bar{s_{i}})}\bar{p_i}p_{j}{\bar{h}}_{ij}^{\bar{s_{i}}}f(\bar{s_i},s_j)-\sum _{j\in \pi (\bar{s_{i}})}p_ip_{j}{\bar{h}}_{ij}^{s_{i}}f(s_i,s_j)\right) \end{aligned} \end{aligned}$$

(37)

which reduces to

$$\begin{aligned} \begin{aligned}&\phi (\bar{a_{i}},{\varvec{a}}_{-i})-\phi (a_{i},{\varvec{a}}_{-i})\\&\quad =2p_{i}\sum _{j\in {\pi (s_{i})}}p_{j}{\bar{h}}_{ji}^{s_{i}}- 2\bar{p_{i}}\sum _{j\in \pi (\bar{s_{i}})}p_{j}{\bar{h}}_{ji}^{\bar{s_{i}}}\\&\qquad \quad +p_{i}p_{o}{\bar{h}}_{oi}^{s_{i}}-\bar{p_{i}}p_{o}{\bar{h}}_{oi}^{\bar{s_{i}}} \end{aligned} \end{aligned}$$

(38)

From (32), (35) and (38), we have

$$\begin{aligned}&\phi (\bar{a_{i}},{\varvec{a}}_{-i})-\phi (a_{i},{\varvec{a}}_{-i})>0 \nonumber \\&\qquad \qquad \qquad \Longleftrightarrow u_{i}(\bar{a_{i}},{\varvec{a}}_{-i})-u_{i}(a_{i},{\varvec{a}}_{-i})>0 \end{aligned}$$

(39)

Hence, we proved that the interference mitigation game is an ordinal potential game with potential function given by (27). $\square$

Appendix B: Proof of Theorem 2

As in [18] we can re-write the ODE in (22) as

$$\begin{aligned} \dfrac{dv_{ik}}{dt}=H_{ik}(\mathbf{v }), \forall i \in M, 1\le k \le L \end{aligned}$$

(40)

Using (18), (19) and (23), (40) can be written as

$$\begin{aligned} \begin{aligned}&\dfrac{dv_{ik_{1}}}{dt}=\dfrac{1}{D}\Big (v_{ik_{1}}(1-v_{ik_{1}})\mathbf{E }[r_{i}|(k_{1},\mathbf{v }_{-i})]\\&\quad +\sum _{k_{2}=1,k_{2} \ne k_{1}}^{L} v_{ik_{2}}(-v_{ik_{1}})\mathbf{E }[r_{i}|(k_{2},\mathbf{v }_{-i})] \Big )\\&\quad = \dfrac{v_{ik_{1}}}{D} \sum _{k_{2}=1}^{L}v_{ik_{2}}[q_{ik_{1}}(\mathbf{v })-q_{ik_{2}}(\mathbf{v })] \end{aligned} \end{aligned}$$

(41)

We have

$$\begin{aligned} \dfrac{ \partial Q(\mathbf{v })}{ \partial v_{ik_{1}}}=Q(k_{1},\mathbf{v }_{-i}) \end{aligned}$$

(42)

since

$$\begin{aligned} Q(\mathbf{v })= \sum _{k_{1}=1}^{L} v_{ik_{1}}Q(k_{1},\mathbf{v }_{-i}) \end{aligned}$$

(43)

Applying (41), (42) and (24), we have

$$\begin{aligned} \dfrac{dQ(\mathbf{v })}{dt}&= \sum _{i,k_{1}}\dfrac{ \partial Q(\mathbf{v })}{ \partial v_{ik_{1}}} \dfrac{dv_{ik_{1}}}{dt}\nonumber \\&= \sum _{i,k_{1}}Q(k_{1},\mathbf{v }_{-i})\dfrac{v_{ik_{1}}}{D} \sum _{k_{2}} v_{ik_{2}}[q_{ik_{1}}(\mathbf{v })-q_{ik_{2}}(\mathbf{v })]\nonumber \\&=\dfrac{1}{2D} \sum _{i,k_{1},k_{2}} \mathbf{Q }(i,k_{1},k_{2})\ge 0 \end{aligned}$$

(44)

where

$$\begin{aligned} \mathbf{Q }(i,k_{1},k_{2})= & {} \dfrac{1}{2D}\sum _{i,k_{1},k_{2}}v_{ik_{1}}v_{ik_{2}}\Big [(Q(k_{1},\mathbf{v }_{-i})\\&-Q(k_{2},\mathbf{v }_{-i})) (q_{ik_{1}}(\mathbf{v })-q_{ik_{2}}(\mathbf{v }))\Big ] \end{aligned}$$

From (44), (41) and (40), we have

$$\begin{aligned} \begin{aligned}&\dfrac{dH(\mathbf{v })}{dt}=0\Rightarrow v_{ik_{1}}v_{ik_{2}}\Big [(Q(k_{1},\mathbf{v }_{-i})-Q(k_{2},\mathbf{v }_{-i}))\\&\quad (q_{ik_{1}}(\mathbf{v })-q_{ik_{2}}(\mathbf{v }))\Big ]=0\\&\quad \forall i, k_1, k_2. \end{aligned} \end{aligned}$$

(45)

Using (24) we have

$$\begin{aligned}&\dfrac{dH(\mathbf{v })}{dt}=0\Rightarrow H_{ik_{1}}(\mathbf{v })=0~\forall ~i,k_1\\&\quad \Rightarrow \mathbf{v }{\textit{ is a stationary point of }}\,(22). \end{aligned}$$

${\mathbf{v }(t)}$ converges to a stationary point of the ODE of (22) . Thus from Proposition 2, Theorem 2 follows. $\square$