Online Learning and Optimization for Computation Offloading in D2D Edge Computing and Networks

Abstract

This paper introduces a framework of device-to-device edge computing and networks (D2D-ECN), a new paradigm for computation offloading and data processing with a group of resource-rich devices towards collaborative optimization between communication and computation. However, the computation process of task intensive applications would be interrupted when capacity-limited battery energy run out. In order to tackle this issue, the D2D-ECN with energy harvesting technology is applied to provide a green computation network and guarantee service continuity. Specifically, we design a reinforcement learning framework in a point-to-point offloading system to overcome challenges of the dynamic nature and uncertainty of renewable energy, channel state and task generation rates. Furthermore, to cope with high-dimensionality and continuous-valued action of the offloading system with multiple cooperating devices, we propose an online approach based on Lyapunov optimization for computation offloading and resource management without priori energy and network information. Numerical results demonstrate that our proposed scheme can reduce system operation cost with low task execution time in D2D-ECN.

Appendices

Appendix: Convergence and optimality analysis of algorithm 1

Given the Lagrange multiplier η, Algorithm 1 can converge to an optimal state-action value (e.g, Q(t), to approximate Q^∗) after finite number of iterations as it satisfies following conditions [30]: i) the state transition $p(\left . {s^{\prime }} \right |s,a)$ in Eq. 10 is stationary under any observed state s ∈ S with the corresponding action a ∈ A; ii) the Lagrange function ρ in Eq. 13 is bounded, that is the unconstrained cost satisfies $\left | \rho (s,a)\right | < \rho _{max}$ for each possible state-action pair (s,a), and ρ_max is a constant; (iii) each state s and Lagrange function could be infinitely explored at the each episode. As a consequence, the Q^η converges to the optimal Q^∗η with probability 1. Moreover, it is worth mentioning that the executive time conducted by interactions between agent and external network environment have considerable impact on the feasibility and efficiency of Q-learning algorithm.

On the other hand, we impose the following additional conditions on the learning rate α(t) and β(t) to guarantee convergence of Theorem 1 [26],

$$ \left\{ \begin{array}{l} \sum\nolimits_{t = 0}^{\infty} {\alpha (t)} = \infty ,\sum\nolimits_{t = 0}^{\infty} {\beta (t)} = \infty \\ \sum\nolimits_{t = 0}^{\infty} {{\alpha^{2}}(t)} \le \infty ,\sum\nolimits_{t = 0}^{\infty} {{\beta^{2}}(t)} \le \infty \\ \underset{t \to \infty }{\lim } \frac{{\beta (t)}}{{\alpha (t)}} \to 0 \end{array} \right. $$

(28)

Consequently, Eq. 19 can ensure to obtain optimal policy in finite iterations. ■

Appendix B: Proof for theorem 1

Based on the inequality $\max {[x,0]^{2}} \le {x^{2}}$, the battery energy state transition can be reformulated as

$$\begin{array}{@{}rcl@{}} {[b(t + 1)]^{2}} &\le& {[b(t)]^{2}} + {[e(t)]^{2}} + {[g(t)]^{2}}\\ &&- 2b(t)[g(t) - e(t)] \end{array} $$

(29)

Then, substituting the Lyapunov function (23) and the above inequality (29) into the Lyapunov drift function (24), we obtain

$$\begin{array}{@{}rcl@{}} E\left\{ {L(t + 1) - L(t)} \right\} &\le& \frac{1}{2}E\left\{ {\left({\left. {{{[e(t)]}^{2}} + {{[g(t)]}^{2}}} \right|b(t)} \right)} \right.\\ &&-\left({\left. {2b(t)[g(t) - e(t)]} \right|b(t)} \right) \end{array} $$

(30)

Taking the renewable energy arrival rate and energy utilization to hold for the inequalities $\left | {e(t)} \right | \le {e_{\max }}$ and $\left | {g(t)} \right | \le {g_{\max }}$, we define the following equation

$$ \vartheta = \frac{1}{2}E\left\{ {\left({\left. {{{[{e_{\max }}]}^{2}} + {{[{g_{\max }}]}^{2}}} \right|b(t)} \right)} \right. $$

(31)

Therefore, adding the $\pi E\left \{ {c(t)} \right \}$ into each side of the inequality (30), which can be rewritten as the Eq. 26. ■