Performance and optimization of dynamic-TDD intercell coordination with elastic traffic

Abstract

We study the intercell coordination problem between two interfering cells combined with dynamic TDD serving elastic data traffic. We model the system at the so-called flow level, where the number of flows varies dynamically and each flow has a random service requirement. Due to the interference between the stations the system is modeled as a set of four interacting processor sharing queues. Our objective is to consider, by using several approaches, the gains from dynamic policies that utilize instantaneous state information to minimize the total mean number of flows. Assuming that the capacity is shared dynamically according to the notion of balanced fairness yields a general policy that is also Pareto optimal, and we are able to explicitly analyze its performance. For exponential service times, we also apply the theory of Markov decision processes and the policy iteration algorithm to minimize the number of flows in the system. Finally, we define priority policies for certain special cases and show that they are even stochastically optimal. Our numerical results show that the gains from the dynamic policies compared with a statically optimized policy can be significant. Surprisingly, the gains can be achieved largely by our novel robust heuristic policy that only relies on instantaneous information about the number of flows.

Appendices

Appendix 1: Proof of Theorem 1

Theorem 1

For $\lambda ^\mathrm {d}_1/\mu ^\mathrm {dd}_1 > \lambda ^\mathrm {d}_2/\mu ^\mathrm {dd}_2$, there exists a stable static policy if and only if

$$\begin{aligned} \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} + \frac{1}{\mu ^\mathrm {du}_1}\left( \lambda ^\mathrm {d}_1 - \mu ^\mathrm {dd}_1 \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} \right) < 1. \nonumber \\ \end{aligned}$$

(26)

If $\lambda ^\mathrm {d}_1/\mu ^\mathrm {dd}_1 \le \lambda ^\mathrm {d}_2/\mu ^\mathrm {dd}_2$, there exists a stable static policy if and only if

$$\begin{aligned} \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\lambda ^\mathrm {d}_1}{\mu ^\mathrm {dd}_1} + \frac{1}{\mu ^\mathrm {ud}_2}\left( \lambda ^\mathrm {d}_2 - \mu ^\mathrm {dd}_2 \frac{\lambda ^\mathrm {d}_1}{\mu ^\mathrm {dd}_1} \right) < 1. \nonumber \\ \end{aligned}$$

(27)

Proof

Consider first the case with $\lambda ^\mathrm {d}_1/\mu ^\mathrm {dd}_1 > \lambda ^\mathrm {d}_2/\mu ^\mathrm {dd}_2$.

$1^\circ$ Assume that condition (26) is satisfied, i.e.,

$$\begin{aligned} \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} + \frac{1}{\mu ^\mathrm {du}_1}\left( \lambda ^\mathrm {d}_1 - \mu ^\mathrm {dd}_1 \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} \right) < 1. \end{aligned}$$

Consider the static policy defined by

$$\begin{aligned}&p^{\mathrm {uu}} = \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\epsilon }{3}, \quad p^{\mathrm {ud}} = 0, \\&p^{\mathrm {dd}} = \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} + \frac{\epsilon }{3}, \quad p^{\mathrm {du}} = \frac{1}{\mu ^\mathrm {du}_1}\left( \lambda ^\mathrm {d}_1 - \mu ^\mathrm {dd}_1 \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} \right) + \frac{\epsilon }{3}, \end{aligned}$$

where

$$\begin{aligned} \epsilon= & {} 1 - \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} \\&- \,\frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} - \frac{1}{\mu ^\mathrm {du}_1}\left( \lambda ^\mathrm {d}_1 - \mu ^\mathrm {dd}_1 \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2}\right) > 0. \end{aligned}$$

It is easy to see that all conditions (9) are satisfied so that the policy is well-defined and stable.

$2^\circ$ Assume now that condition (26) is not satisfied, i.e.,

$$\begin{aligned} \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} + \frac{1}{\mu ^\mathrm {du}_1}\left( \lambda ^\mathrm {d}_1 - \mu ^\mathrm {dd}_1 \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2}\right) \ge 1.\nonumber \\ \end{aligned}$$

(28)

Consider first the static policies for which

$$\begin{aligned} p^{\mathrm {dd}} \le \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2}. \end{aligned}$$

(29)

Now

$$\begin{aligned}&p^{\mathrm {uu}} + p^{\mathrm {ud}} + p^{\mathrm {du}} + p^{\mathrm {dd}} \\&\quad \mathop {>}\limits ^{(9)} \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{1}{\mu ^\mathrm {du}_1}(\lambda ^\mathrm {d}_1-p^\mathrm {dd} \mu ^\mathrm {dd}_1) \\&\qquad + \frac{1}{\mu ^\mathrm {ud}_2}(\lambda ^\mathrm {d}_2-p^\mathrm {dd} \mu ^\mathrm {dd}_2) + p^\mathrm {dd} \\&\quad = \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\lambda ^\mathrm {d}_1}{\mu ^\mathrm {du}_1} + \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {ud}_2} - \\&\qquad p^\mathrm {dd} \left( \frac{\mu ^\mathrm {dd}_1}{\mu ^\mathrm {du}_1} + \frac{\mu ^\mathrm {dd}_2}{\mu ^\mathrm {ud}_2} -1 \right) \\&\quad \mathop {\ge }\limits ^{(2),(30)} \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} + \frac{1}{\mu ^\mathrm {du}_1}\left( \lambda ^\mathrm {d}_1 - \mu ^\mathrm {dd}_1 \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2}\right) \\&\quad \mathop {\ge }\limits ^{(28)} 1, \end{aligned}$$

which contradicts with the requirement that the vector $p$ constitues a probability distribution. Thus, there is no stable static policy satisfying (29).

Consider then the static policies for which

$$\begin{aligned} p^{\mathrm {dd}} > \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2}. \end{aligned}$$

(30)

Now

$$\begin{aligned}&p^{\mathrm {uu}} + p^{\mathrm {du}} + p^{\mathrm {dd}} \\&\quad \mathop {>}\limits ^{(9)} \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{1}{\mu ^\mathrm {du}_1}(\lambda ^\mathrm {d}_1-p^\mathrm {dd} \mu ^\mathrm {dd}_1) + p^\mathrm {dd} \\&\quad = \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\lambda ^\mathrm {d}_1}{\mu ^\mathrm {du}_1} + p^\mathrm {dd} \left( 1 - \frac{\mu ^\mathrm {dd}_1}{\mu ^\mathrm {du}_1}\right) \\&\quad \mathop {>}\limits ^{(2),(30)} \max \left\{ \frac{\lambda ^\mathrm {u}_1}{\mu ^\mathrm {uu}_1},\frac{\lambda ^\mathrm {u}_2}{\mu ^\mathrm {uu}_2} \right\} + \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2} + \frac{1}{\mu ^\mathrm {du}_1}\left( \lambda ^\mathrm {d}_1 - \mu ^\mathrm {dd}_1 \frac{\lambda ^\mathrm {d}_2}{\mu ^\mathrm {dd}_2}\right) \\&\quad \mathop {\ge }\limits ^{(28)} 1, \end{aligned}$$

which again contradicts with the requirement that $p$ is a probability distribution. Thus, there is neither any stable static policy that satisfies (30).

Finally, the proof in the case with $\lambda ^\mathrm {d}_1/\mu ^\mathrm {dd}_1 \le \lambda ^\mathrm {d}_2/\mu ^\mathrm {dd}_2$ can be done in a similar manner as above. $\square$

Appendix 2: Proof of Proposition 1

Proposition 1

The BF coordination policy is Pareto optimal.

Proof

Fix state x, and consider the corresponding BF service rate vector $\phi (x)$. Since, by (11),

$$\begin{aligned} \phi ^\delta _i(x) = \frac{{\varPhi }(x - e^\delta _i)}{{\varPhi }(x)}, \end{aligned}$$

we observe from (17) that BF results in service rates $\phi ^\delta _i(x)$ where at least one of the constraints (15) is satisfied as an equality.

(i) Assume now that the first one of the constraints (15) is satisfied as an equality so that

$$\begin{aligned} a_1 \phi ^\mathrm {d}_1 + b_1 \phi ^\mathrm {d}_2 + c_1 \phi ^\mathrm {u}_1 = 1. \end{aligned}$$

(31)

Let then $\tilde{\phi }(x) \in {\mathcal {C}}$ denote such a service rate vector for which

$$\begin{aligned}&\tilde{\phi }^\mathrm {d}_1(x) \ge \phi ^\mathrm {d}_1(x), \quad \tilde{\phi }^\mathrm {d}_2(x) \ge \phi ^\mathrm {d}_2(x), \\&\tilde{\phi }^\mathrm {u}_1(x) \ge \phi ^\mathrm {u}_1(x), \quad \tilde{\phi }^\mathrm {u}_2(x) \ge \phi ^\mathrm {u}_2(x). \end{aligned}$$

However, since, by (15),

$$\begin{aligned} a_1 \tilde{\phi }^\mathrm {d}_1(x) + b_1 \tilde{\phi }^\mathrm {d}_2(x) + c_1 \tilde{\phi }^\mathrm {u}_1(x) \le 1, \end{aligned}$$

we deduce that

$$\begin{aligned} \tilde{\phi }^\mathrm {d}_1(x) = \phi ^\mathrm {d}_1(x), \; \tilde{\phi }^\mathrm {d}_2(x) = \phi ^\mathrm {d}_2(x), \; \tilde{\phi }^\mathrm {u}_1(x) = \phi ^\mathrm {u}_1(x). \end{aligned}$$

In addition, by (7), we have

$$\begin{aligned} \tilde{\phi }^\mathrm {u}_2(x) = \tilde{\phi }^\mathrm {u}_1(x) \frac{\mu ^\mathrm {uu}_2}{\mu ^\mathrm {uu}_1} = \phi ^\mathrm {u}_1(x) \frac{\mu ^\mathrm {uu}_2}{\mu ^\mathrm {uu}_1} = \phi ^\mathrm {u}_2(x). \end{aligned}$$

Thus, we necessarily have $\tilde{\phi }(x) = \phi (x)$, which implies that, under assumption (31), there is no feasible service rate vector that is strictly better than $\phi (x)$ (having at least as high service rate components as $\phi (x)$ with at least one of them being strictly higher than the corresponding component of $\phi (x)$).

(ii) The remaining three constraints of (15) can be handled in a symmetric way. $\square$

Appendix 3: Proof of Theorem 2

Theorem 2

The BF balance function ${\varPhi }(x)$ can be calculated recursively as follows. For any state x,

$$\begin{aligned}&{\varPhi }(x) \nonumber \\&\quad ={x^\mathrm {d}_1 + x^\mathrm {d}_2 + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2} \left( \frac{1}{\mu ^\mathrm {uu}_1} \right) ^{x^\mathrm {u}_1} \left( \frac{1}{\mu ^\mathrm {uu}_2} \right) ^{x^\mathrm {u}_2} {\varPhi }(x^\mathrm {d}_1, x^\mathrm {d}_2), \end{aligned}$$

(32)

where ${\varPhi }(0,0) = 1$, ${\varPhi }(u,v) = 0$ if $u < 0$ or $v < 0$, and, for any state $(u,v,0,0) \ne (0,0,0,0)$,

$$\begin{aligned}&{\varPhi }(u,v) \nonumber \\&\quad =\max \left\{ \frac{1}{\mu ^\mathrm {du}_1} {\varPhi }(u - 1, v) + \frac{\mu ^\mathrm {du}_1 - \mu ^\mathrm {dd}_1}{\mu ^\mathrm {du}_1 \mu ^\mathrm {dd}_2} {\varPhi }(u, v - 1), \right. \nonumber \\&\qquad \left. \frac{\mu ^\mathrm {ud}_2 - \mu ^\mathrm {dd}_2}{\mu ^\mathrm {ud}_2 \mu ^\mathrm {dd}_1} {\varPhi }(u - 1, v) + \frac{1}{\mu ^\mathrm {ud}_2} {\varPhi }(u, v - 1) \right\} . \end{aligned}$$

(33)

Proof

$1^\circ$ Consider first the case $x^\mathrm {d}_1 = x^\mathrm {d}_2 = 0$. Equation (32) is trivially true for the state $x = (0,0,0,0)$,

$$\begin{aligned} {\varPhi }(0,0,0,0) = 1 = {\varPhi }(0,0). \end{aligned}$$

Let $m \ge 0$, and assume that (32) is true for any state $x = (0, 0, x^\mathrm {u}_1, x^\mathrm {u}_2)$ such that $x^\mathrm {u}_1 + x^\mathrm {u}_2 \le m$. Consider now state $x = (0, 0, x^\mathrm {u}_1, x^\mathrm {u}_2)$ such that $x^\mathrm {u}_1 + x^\mathrm {u}_2 = m + 1$. By (17) and the induction assumption (IA), we have

$$\begin{aligned}&{\varPhi }(0, 0, x^\mathrm {u}_1, x^\mathrm {u}_2) \\&\quad \mathop {=}\limits ^{(17)} \max \left\{ c_1 {\varPhi }(0, 0, x^\mathrm {u}_1 - 1, x^\mathrm {u}_2), c_2 {\varPhi }(0, 0, x^\mathrm {u}_1, x^\mathrm {u}_2 - 1) \right\} \\&\quad \mathop {=}\limits ^{\mathrm {IA}} \max \left\{ c_1 \, c_1^{x^\mathrm {u}_1 - 1} c_2^{x^\mathrm {u}_2} {\varPhi }(0, 0), c_2 \, c_1^{x^\mathrm {u}_1} c_2^{x^\mathrm {u}_2 - 1} {\varPhi }(0, 0) \right\} \\&\quad = {x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2} c_1^{x^\mathrm {u}_1} c_2^{x^\mathrm {u}_2} {\varPhi }(0, 0). \end{aligned}$$

Thus, (32) is also true for any state $x = (0, 0, x^\mathrm {u}_1, x^\mathrm {u}_2)$ such that $x^\mathrm {u}_1 + x^\mathrm {u}_2 = m + 1$.

$2^\circ$ Consider then the general case where $x^\mathrm {d}_1, x^\mathrm {d}_2 \ge 0$. Let $n \ge 0$. Assume that we have, for any state $x = (u, v, x^\mathrm {u}_1, x^\mathrm {u}_2)$ such that $u + v \le n$,

$$\begin{aligned} {\varPhi }(u, v, x^\mathrm {u}_1, x^\mathrm {u}_2) = {u + v + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2} c_1^{x^\mathrm {u}_1} c_2^{x^\mathrm {u}_2} {\varPhi }(u, v, 0, 0),\nonumber \\ \end{aligned}$$

(34)

Consider now state $x = (u, v, x^\mathrm {u}_1, x^\mathrm {u}_2)$ such that $u + v = n + 1$. Note that (34) is trivially true if $x^\mathrm {u}_1 = x^\mathrm {u}_2 = 0$. In addition, by (17), we have

$$\begin{aligned}&{\varPhi }(u, v, 0, 0) \\&\quad \mathop {=}\limits ^{(17)}\max \{ a_1 {\varPhi }(u - 1, v, 0, 0) + b_1 {\varPhi }(u, v - 1, 0, 0), \\&\qquad b_2 {\varPhi }(u - 1, v, 0, 0) + a_2 {\varPhi }(u, v - 1, 0, 0) \}. \end{aligned}$$

Thus, we may choose ${\varPhi }(u, v) = {\varPhi }(u, v, 0, 0)$.

Assume then that (34) is true for any state $x = (u, v, x^\mathrm {u}_1, x^\mathrm {u}_2)$ such that $u + v = n + 1$ and $x^\mathrm {u}_1 + x^\mathrm {u}_2 \le m$, where $m \ge 0$. By (17), we have

$$\begin{aligned}&{\varPhi }(u, v, x^\mathrm {u}_1, x^\mathrm {u}_2) \mathop {=}\limits ^{(17)} \max \left\{ a_1 {\varPhi }(u - 1, v, x^\mathrm {u}_1, x^\mathrm {u}_2) \right. \\&\quad + b_1 {\varPhi }(u, v - 1, x^\mathrm {u}_1, x^\mathrm {u}_2) + c_1 {\varPhi }(u, v, x^\mathrm {u}_1 - 1, x^\mathrm {u}_2), \\&\quad a_1 {\varPhi }(u - 1, v, x^\mathrm {u}_1, x^\mathrm {u}_2) + b_1 {\varPhi }(u, v - 1, x^\mathrm {u}_1, x^\mathrm {u}_2) \\&\quad +c_2 {\varPhi }(u, v, x^\mathrm {u}_1, x^\mathrm {u}_2 - 1), b_2 {\varPhi }(u - 1, v, x^\mathrm {u}_1, x^\mathrm {u}_2) \\&\quad +a_2 {\varPhi }(u, v - 1, x^\mathrm {u}_1, x^\mathrm {u}_2) + c_1 {\varPhi }(u, v, x^\mathrm {u}_1 - 1, x^\mathrm {u}_2), \\&\quad b_2 {\varPhi }(u - 1, v, x^\mathrm {u}_1, x^\mathrm {u}_2) + a_2 {\varPhi }(u, v - 1, x^\mathrm {u}_1, x^\mathrm {u}_2) \\&\quad +\left. c_2 {\varPhi }(u, v, x^\mathrm {u}_1, x^\mathrm {u}_2 - 1) \right\} . \end{aligned}$$

Then by the induction assumptions (IA) we obtain

$$\begin{aligned}&{\varPhi }(u, v, x^\mathrm {u}_1, x^\mathrm {u}_2) \mathop {=}\limits ^{\mathrm {IA}} {n + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2} c_1^{x^\mathrm {u}_1} c_2^{x^\mathrm {u}_2} \\&\quad \times \max \left\{ a_1 {\varPhi }(u - 1, v, 0, 0) + b_1 {\varPhi }(u, v - 1, 0, 0), \right. \\&\quad \left. b_2 {\varPhi }(u - 1, v, 0, 0) + a_2 {\varPhi }(u, v - 1, 0, 0) \right\} \\&\quad +{n + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2 - 1} c_1^{x^\mathrm {u}_1} c_2^{x^\mathrm {u}_2} {\varPhi }(u, v, 0, 0) \end{aligned}$$

Finally, again by (17) we get

$$\begin{aligned} {\varPhi }(u, v, x^\mathrm {u}_1, x^\mathrm {u}_2)&\mathop {=}\limits ^{(17)} {n + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2} c_1^{x^\mathrm {u}_1} c_2^{x^\mathrm {u}_2} {\varPhi }(u, v, 0, 0) \\&\quad + {n + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2 - 1} c_1^{x^\mathrm {u}_1} c_2^{x^\mathrm {u}_2} {\varPhi }(u, v, 0, 0) \\&= {n + 1 + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2} c_1^{x^\mathrm {u}_1} c_2^{x^\mathrm {u}_2} {\varPhi }(u, v, 0, 0). \end{aligned}$$

Thus, (34) is also true for any state $x = (u, v, x^\mathrm {u}_1, x^\mathrm {u}_2)$ such that $u + v = n + 1$ and $x^\mathrm {u}_1 + x^\mathrm {u}_2 = m + 1$. It follows that (34) is true for any state $x = (u, v, x^\mathrm {u}_1, x^\mathrm {u}_2)$ such that $u + v = n + 1$. $\square$

Appendix 4: Proof of Proposition 2

Proposition 2

Consider the BF balance function ${\varPhi }(u,v)$ defined in (33). Let $f(0) = 0$. For any $n \in \{1,2,\ldots \}$ there is $f(n) \in (f(n-1),f(n-1)+1]$ such that

$$\begin{aligned}&{\varPhi }(u,n-u) \nonumber \\&= \left\{ \begin{array}{ll} \displaystyle \frac{1}{\mu ^\mathrm {du}_1} {\varPhi }(u - 1, n - u) \\ \displaystyle \qquad + \frac{\mu ^\mathrm {du}_1 - \mu ^\mathrm {dd}_1}{\mu ^\mathrm {du}_1 \mu ^\mathrm {dd}_2} {\varPhi }(u, n - u - 1), &{} u \ge f(n); \\ \displaystyle \frac{\mu ^\mathrm {ud}_2 - \mu ^\mathrm {dd}_2}{\mu ^\mathrm {ud}_2 \mu ^\mathrm {dd}_1} {\varPhi }(u - 1, n - u) \\ \displaystyle \qquad + \frac{1}{\mu ^\mathrm {ud}_2} {\varPhi }(u, n - u - 1), &{} \hbox {otherwise}. \end{array} \right. \end{aligned}$$

(35)

Proof

$1^\circ$ Consider first the case $n = 1$. According to our low downlink interference assumption (3), we have

$$\begin{aligned}&a_1 = \frac{1}{\mu ^\mathrm {du}_1} \mathop {\ge }\limits ^{(3)} \frac{\mu ^\mathrm {ud}_2 - \mu ^\mathrm {dd}_2}{\mu ^\mathrm {ud}_2 \mu ^\mathrm {dd}_1} = b_2, \\&a_2 = \frac{1}{\mu ^\mathrm {ud}_2} \mathop {\ge }\limits ^{(3)} \frac{\mu ^\mathrm {du}_1 - \mu ^\mathrm {dd}_1}{\mu ^\mathrm {du}_1 \mu ^\mathrm {dd}_2} = b_1. \end{aligned}$$

By further taking into account Theorem 2, we get

$$\begin{aligned} {\varPhi }(1,0)= & {} \max \{ a_1 {\varPhi }(0, 0), b_2 {\varPhi }(0, 0) \} \\= & {} \max \{ a_1, b_2 \} = a_1 = a_1 {\varPhi }(0, 0), \\ {\varPhi }(0,1)= & {} \max \{ b_1 {\varPhi }(0, 0), a_2 {\varPhi }(0, 0) \} \\= & {} \max \{ b_1, a_2 \} = a_2 = a_2 {\varPhi }(0, 0), \end{aligned}$$

which implies that (35) is satisfied for $n = 1$ with choice $f(n) = 1/2$.

$2^\circ$ Let then $n \in \{1,2,\ldots \}$, and assume that there is $f(n) \in [f(n-1),f(n-1)+1]$ satisfying (35).

Let $u \in \{0,1,\ldots ,n+1\}$ such that $u \ge f(n) + 1$, and let $v = n + 1 - u$. Now, by (33)

$$\begin{aligned} {\varPhi }(u,v) \mathop {=}\limits ^{(33)} \max \{&a_1 {\varPhi }(u - 1, v) + b_1 {\varPhi }(u, v - 1), \\&b_2 {\varPhi }(u - 1, v) + a_2 {\varPhi }(u, v - 1) \}. \end{aligned}$$

Using the induction assumption (IA) gives

$$\begin{aligned} {\varPhi }(u,v) \mathop {=}\limits ^{\mathrm {IA}} \max \{&a_1 (a_1 {\varPhi }(u - 2, v) + b_1 {\varPhi }(u - 1, v - 1)) \\&+ b_1 (a_1 {\varPhi }(u - 1, v - 1) + b_1 {\varPhi }(u, v - 2)), \\&b_2 (a_1 {\varPhi }(u - 2, v) + b_1 {\varPhi }(u - 1, v - 1)) \\&+ a_2 (a_1 {\varPhi }(u - 1, v - 1) + b_1 {\varPhi }(u, v - 2)) \}. \end{aligned}$$

Finally, rearranging and using the induction assumption again yields

$$\begin{aligned} {\varPhi }(u,v)&= \max \{ a_1 (a_1 {\varPhi }(u - 2, v) + b_1 {\varPhi }(u - 1, v - 1)) \\&\quad + b_1 (a_1 {\varPhi }(u - 1, v - 1) + b_1 {\varPhi }(u, v - 2)), \\&\quad a_1 (b_2 {\varPhi }(u - 2, v) + a_2 {\varPhi }(u - 1, v - 1)) \\&\quad + b_1 (b_2 {\varPhi }(u - 1, v - 1) + a_2 {\varPhi }(u, v - 2)) \} \\&\mathop {=}\limits ^{\mathrm {IA}} a_1 {\varPhi }(u - 1, v) + b_1 {\varPhi }(u, v - 1). \end{aligned}$$

Thus, (35) is satisfied with choice $f(n + 1) \le f(n) + 1$.

Let then $u \in \{0,1,\ldots ,n+1\}$ such that $u \le f(n)$, and let $v = n + 1 - u$. Again, by (33) we get

$$\begin{aligned} {\varPhi }(u,v) \mathop {=}\limits ^{(33)} \max \{&a_1 {\varPhi }(u - 1, v) + b_1 {\varPhi }(u, v - 1), \\&b_2 {\varPhi }(u - 1, v) + a_2 {\varPhi }(u, v - 1) \}, \end{aligned}$$

and the induction assumption (IA) yields,

$$\begin{aligned} {\varPhi }(u,v) \mathop {=}\limits ^{\mathrm {IA}} \max \{&a_1 (b_2 {\varPhi }(u - 2, v) + a_2 {\varPhi }(u - 1, v - 1)) \\&+ b_1 (b_2 {\varPhi }(u - 1, v - 1) + a_2 {\varPhi }(u, v - 2)), \\&b_2 (b_2 {\varPhi }(u - 2, v) + a_2 {\varPhi }(u - 1, v - 1)) \\&+ a_2 (b_2 {\varPhi }(u - 1, v - 1) + a_2 {\varPhi }(u, v - 2)) \}. \end{aligned}$$

By rearranging and using the induction assumption we finally obtain

$$\begin{aligned} {\varPhi }(u,v)&= \max \{ b_2 (a_1 {\varPhi }(u - 2, v) + b_1 {\varPhi }(u - 1, v - 1)) \\&\quad + a_2 (a_1 {\varPhi }(u - 1, v - 1) + b_1 {\varPhi }(u, v - 2)), \\&\quad b_2 (b_2 {\varPhi }(u - 2, v) + a_2 {\varPhi }(u - 1, v - 1)) \\&\quad + a_2 (b_2 {\varPhi }(u - 1, v - 1) + a_2 {\varPhi }(u, v - 2)) \} \\&\mathop {=}\limits ^{\mathrm {IA}} b_2 {\varPhi }(u - 1, v) + a_2 {\varPhi }(u, v - 1). \end{aligned}$$

Thus, (35) is satisfied with choice $f(n+1) > f(n)$.

The proof is completed by observing that there is a unique integer u belonging to the half-open interval $(f(n),f(n)+1]$. If

$$\begin{aligned}&a_1 {\varPhi }(u - 1, n + 1 - u) + b_1 {\varPhi }(u, n - u) \\&\quad \ge b_2 {\varPhi }(u - 1, n + 1 - u) + a_2 {\varPhi }(u, n - u), \end{aligned}$$

then we can choose any $f(n + 1) \in (f(n), u]$. Otherwise, we choose any $f(n + 1) \in (u, f(n) + 1]$. $\square$

Appendix 5: Proof of Theorem 3

Theorem 3

In the symmetric case (4), the BF balance function ${\varPhi }(x)$ for any state x is given by

$$\begin{aligned} {\varPhi }(x)&= {\varPhi }(x^\mathrm {d}_1, x^\mathrm {d}_2, x^\mathrm {u}_1 + x^\mathrm {u}_2) \nonumber \\&= {x^\mathrm {d}_1 + x^\mathrm {d}_2 + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2} \left( \frac{1}{\mu ^\mathrm {uu}} \right) ^{x^\mathrm {u}_1 + x^\mathrm {u}_2} {\varPhi }(x^\mathrm {d}_1, x^\mathrm {d}_2), \nonumber \\ \end{aligned}$$

(36)

where ${\varPhi }(0,0) = 1$, and, for any state $(u,v,0,0) \ne (0, 0, 0, 0)$ such that $u \ge v$,

$$\begin{aligned}&{\varPhi }(u,v) = {\varPhi }(v,u) \nonumber \\&\quad = \sum _{i = 0}^v {u - 1 + v - i \atopwithdelims ()v - i} \frac{u - v + i}{u} \nonumber \\&\qquad \times \left( \frac{1}{\mu ^\mathrm {d}} \right) ^u \left( \frac{1}{\mu ^\mathrm {dd}} - \frac{1}{\mu ^\mathrm {d}} \right) ^{v - i} \left( \frac{1}{\mu ^\mathrm {dd}} \right) ^i. \end{aligned}$$

(37)

Proof

Consider the symmetric case (4). By Theorem 2 and Corollary 1, we have, for any state x,

$$\begin{aligned} {\varPhi }(x) = {x^\mathrm {d}_1 + x^\mathrm {d}_2 + x^\mathrm {u}_1 + x^\mathrm {u}_2 \atopwithdelims ()x^\mathrm {u}_1 + x^\mathrm {u}_2} \left( \frac{1}{\mu ^\mathrm {uu}} \right) ^{x^\mathrm {u}_1 + x^\mathrm {u}_2} {\varPhi }(x^\mathrm {d}_1, x^\mathrm {d}_2), \nonumber \\ \end{aligned}$$

(38)

where ${\varPhi }(0,0) = 1$, and, for any state $(u,v,0,0) \ne (0,0,0,0)$ such that $u \ge v$,

$$\begin{aligned} {\varPhi }(u,v) = {\varPhi }(v,u) = a \, {\varPhi }(u - 1, v) + b \, {\varPhi }(u, v - 1), \end{aligned}$$

(39)

where we have used notation

$$\begin{aligned} a = \frac{1}{\mu ^\mathrm {d}}, \quad b = \frac{1}{\mu ^\mathrm {dd}} - \frac{1}{\mu ^\mathrm {d}}. \end{aligned}$$

Thus, it remains to prove that ${\varPhi }(u,v)$ defined in (37) satisfies recursion (39).

Consider first a state $x = (u, v, 0, 0)$ such that $u > v$. Now, by (37), we have

$$\begin{aligned}&a \, {\varPhi }(u - 1, v) + b \, {\varPhi }(u, v - 1) \\&\quad \mathop {=}\limits ^{(37)} a \, \sum _{i = 0}^v {u + v - i - 2 \atopwithdelims ()v - i} \frac{u - v + i - 1}{u - 1} \, a^{u - 1} b^{v - i} (a + b)^i \\&\qquad + b \, \sum _{i = 0}^{v - 1} {u + v - i - 2 \atopwithdelims ()v - i - 1} \frac{u - v + i + 1}{u} \, a^u b^{v - i - 1} (a + b)^i \\&\quad = \sum _{i = 0}^{v - 1} \left[ {u + v - i - 2 \atopwithdelims ()v - i - 1} \left( \frac{u - v + i - 1}{v - i} + \frac{u - v + i + 1}{u} \right) \right] \\&\qquad \times a^u b^{v - i} (a + b)^i + a^u (a + b)^v \\&\quad = \sum _{i = 0}^{v - 1} {u + v - i - 1 \atopwithdelims ()v - i} \frac{u - v + i}{u} \, a^u b^{v - i} (a + b)^i + a^u (a + b)^v \\&\quad \mathop {=}\limits ^{(37)} {\varPhi }(u, v). \end{aligned}$$

Thus, recursion (39) is satisfied in this case.

Consider now the remaining case, where $u = v > 0$. By (37), we have

$$\begin{aligned}&a \, {\varPhi }(u - 1, u) + b \, {\varPhi }(u, u - 1) \\&\quad \mathop {=}\limits ^{(37)} (a + b) \, \sum _{i = 0}^{u - 1} {2 u - i - 2 \atopwithdelims ()u - i - 1} \frac{i + 1}{u} \, a^u b^{u - i - 1} (a + b)^i \\&\quad \; = \sum _{i = 1}^{u} {2 u - i - 1 \atopwithdelims ()u - i} \frac{i}{u} \, a^u b^{u - i} (a + b)^i \\&\quad \mathop {=}\limits ^{(37)} {\varPhi }(u, u). \end{aligned}$$

Thus, recursion (39) is satisfied in this case as well. $\square$

Appendix 6: Proof of Theorem 4

For the optimality results below, we need the following (easily-proved) lemma.

Lemma 1

If $a_1 \ge a_2 \ge 0$ and $b_1 \ge b_2 \ge 0$, then $a_1 b_1 + a_2 b_2 \ge a_1 b_2 + a_2 b_1$.

Theorem 4

The PU policy is stochastically optimal if

$$\begin{aligned} \lambda ^\mathrm {u}_1 > 0, \quad \lambda ^\mathrm {u}_2 = 0 \quad \hbox {and}\quad \mu ^\mathrm {uu} \ge 2 \mu ^\mathrm {dd} > \mu ^\mathrm {d}. \end{aligned}$$

(40)

Proof

Assume condition (40). The result is proved by induction. More precisely, we will show that the following equations are valid for any $k \in \{0,1,\ldots$}:

$$\begin{aligned}&\mu _k(x,\mathrm {uu}) \le \min \{\mu _k(x,\mathrm {dd}), \mu _k(x,\mathrm {du}), \mu _k(x,\mathrm {ud})\}, \nonumber \\&\quad \hbox {if } x^\mathrm {u}_1 > 0, x^\mathrm {d}_1 \ge 0, \hbox { and } x^\mathrm {d}_2 \ge 0; \end{aligned}$$

(41)

$$\begin{aligned}&\mu _k(x,\mathrm {dd}) \le \min \{\mu _k(x,\mathrm {du}), \mu _k(x,\mathrm {ud})\}, \nonumber \\&\quad \hbox {if } x^\mathrm {u}_1 \ge 0, x^\mathrm {d}_1 > 0, \hbox { and } x^\mathrm {d}_2 > 0; \end{aligned}$$

(42)

$$\begin{aligned}&\mu _k(x,\mathrm {dd}) \le \min \{\mu _k(x,\mathrm {uu}), \mu _k(x,\mathrm {du}), \mu _k(x,\mathrm {ud})\}, \nonumber \\&\quad \hbox {if } x^\mathrm {u}_1 = 0, x^\mathrm {d}_1 > 0, \hbox { and } x^\mathrm {d}_2 > 0; \end{aligned}$$

(43)

$$\begin{aligned}&\mu _k(x,\mathrm {du}) \le \min \{\mu _k(x,\mathrm {uu}), \mu _k(x,\mathrm {dd}), \mu _k(x,\mathrm {ud})\}, \nonumber \\&\quad \hbox {if } x^\mathrm {u}_1 = 0, x^\mathrm {d}_1 > 0, \hbox { and } x^\mathrm {d}_2 = 0; \end{aligned}$$

(44)

$$\begin{aligned}&\mu _k(x,\mathrm {ud}) \le \min \{\mu _k(x,\mathrm {uu}), \mu _k(x,\mathrm {dd}), \mu _k(x,\mathrm {du})\}, \nonumber \\&\quad \hbox {if } x^\mathrm {u}_1 = 0, x^\mathrm {d}_1 = 0, \hbox { and } x^\mathrm {d}_2 > 0, \end{aligned}$$

(45)

where $\mu _k(x,\cdot )$ are given by (24). Note that the optimality of the PU policy follows already from Eqs. (41), (43), (44), and (45). The remaining Eq. (42) is needed for the proof of (43), as can be seen from the proof below.

$1^\circ$ In this part we prove that Eqs. (41)–(45) are valid for $k = 0$.

(a) Assume first that $x^\mathrm {u}_1 > 0$, $x^\mathrm {d}_1 \ge 0$, and $x^\mathrm {d}_2 \ge 0$. Now, by Lemma 1 (that we refer to by “L1”), we have

$$\begin{aligned} \mu _0(x,\mathrm {uu})&\mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {uu} V_0(x - e^\mathrm {u}_1) + (2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_0(x) \\&\mathop {\le }\limits ^{\hbox {L1}} 2 \mu ^\mathrm {dd} V_0(x - e^\mathrm {u}_1) + (\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_0(x) \\&\mathop {=}\limits ^{(21)} 2 \mu ^\mathrm {dd} 1_{\{|x| - 1 > s\}} +\, (\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_0(x) \\&{\le } \mu ^\mathrm {dd} 1_{\{|(x - e^\mathrm {d}_1)^+| > s\}} + \mu ^\mathrm {dd} 1_{\{|(x - e^\mathrm {d}_2)^+| > s\}} \\&\quad + \,(\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_0(x) \\&\mathop {=}\limits ^{(21)} \mu ^\mathrm {dd} V_0((x - e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_0((x - e^\mathrm {d}_2)^+) \\&\quad + \,(\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_0(x) \\&\mathop {=}\limits ^{\hbox {def.}} \mu _0(x,\mathrm {dd}). \end{aligned}$$

Note that the use of Lemma 1 is possible due to Proposition 3 and our assumption that $\mu ^\mathrm {uu} \ge 2 \mu ^\mathrm {dd}$.

Similarly,

By symmetry, we also have $\mu _0(x,\mathrm {uu}) \le \mu _0(x,\mathrm {ud})$. Thus, Eq. (41) is valid for $k = 0$.

(b) Consider then any state x such that $x^\mathrm {u}_1 \ge 0$, $x^\mathrm {d}_1 > 0$, and $x^\mathrm {d}_2 > 0$. Now

$$\begin{aligned} \mu _0(x,\mathrm {dd})&\mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {dd} V_0(x - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_0(x - e^\mathrm {d}_2) \\&\quad +\, (\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_0(x) \\&\mathop {\le }\limits ^{\hbox {L1}} (\mu ^\mathrm {d}/2) V_0(x - e^\mathrm {d}_1) + (\mu ^\mathrm {d}/2) V_0(x - e^\mathrm {d}_2) \\&\quad + \,(\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_0(x) \\&\mathop {=}\limits ^{(21)} \mu ^\mathrm {d} 1_{\{|x| - 1 > s\}} +\, (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_0(x) \\&\mathop {=}\limits ^{(21)} \mu ^\mathrm {d} V_0(x - e^\mathrm {d}_1) + (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_0(x) \\&\mathop {=}\limits ^{\hbox {def.}} \mu _0(x,\mathrm {du}). \end{aligned}$$

Above, the use of Lemma 1 is justified by Proposition 3 and our low downlink interference assumption $\mu ^\mathrm {dd} \ge \mu ^\mathrm {d}/2$. By symmetry, we also have $\mu _1(x,\mathrm {dd}) \le \mu _1(x,\mathrm {ud})$. Thus, Eq. (42) is valid for $k = 0$.

(c) Consider then any state x such that $x^\mathrm {u}_1 = 0$, $x^\mathrm {d}_1 > 0$, and $x^\mathrm {d}_2 > 0$. Due to (b), it remains to prove that $\mu _0(x,\mathrm {dd}) \le \mu _0(x,\mathrm {uu})$ in this case. However, it follows easily from Proposition 3 (that we refer to by “P3”) as shown below:

$$\begin{aligned} \mu _0(x,\mathrm {dd})&\mathop {=}\limits ^{\text{ def. }} \mu ^\mathrm {dd} V_0(x - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_0(x - e^\mathrm {d}_2) \\&\quad + (\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_0(x) \\&\mathop {\le }\limits ^{\text{ P3 }} (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_0(x) \\&\mathop {=}\limits ^{\text{ def. }} \mu _0(x,\mathrm {uu}). \end{aligned}$$

Thus, Eq. (43) is valid for $k = 0$.

(d) Next we consider any state x such that $x^\mathrm {u}_1 = 0$, $x^\mathrm {d}_1 > 0$, and $x^\mathrm {d}_2 = 0$. Note first that

$$\begin{aligned} \mu _0(x,\mathrm {du})&\mathop {=}\limits ^{\text{ def. }} \mu ^\mathrm {d} V_0(x - e^\mathrm {d}_1) + (2 \mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_0(x) \\ {}&\mathop {\le }\limits ^{\text{ P3 }} (2 \mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_0(x) \\ {}&\mathop {=}\limits ^{\text{ def. }} \min \{\mu _0(x,\mathrm {uu}), \mu _0(x,\mathrm {ud})\}. \end{aligned}$$

In addition,

$$\begin{aligned} \mu _0(x,\mathrm {du})&\mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {d} V_0(x - e^\mathrm {d}_1) + (2 \mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_0(x) \\&\mathop {\le }\limits ^{\hbox {L1}} \mu ^\mathrm {dd} V_0(x - e^\mathrm {d}_1) + (2 \mu ^\mathrm {uu} + \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_0(x) \\&\mathop {=}\limits ^{\hbox {def.}} \mu _0(x,\mathrm {dd}). \end{aligned}$$

The use of Lemma 1 is justified here by Proposition 3 and the fact that $\mu ^\mathrm {d} > \mu ^\mathrm {dd}$. It follows from the results above that Eq. (44) is valid for $k = 0$.

(e) By symmetry, Eq. (45), which is related to states x such that $x^\mathrm {u}_1 = 0$, $x^\mathrm {d}_1 = 0$, and $x^\mathrm {d}_2 > 0$, is also valid for $k = 0$.

$2^\circ$ Assume now that that Eqs. (41)-(45) are valid for $k \ge 0$. We will show below that they are also valid for $k + 1$.

(a) Consider any state x such that $x^\mathrm {u}_1 > 0$, $x^\mathrm {d}_1 \ge 0$, and $x^\mathrm {d}_2 \ge 0$.

We will first show that $\mu _{k+1}(x,\mathrm {uu}) \le \mu _{k+1}(x,\mathrm {dd})$. Now

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {uu} V_{k+1}(x - e^\mathrm {u}_1) + (2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_{k+1}(x) \\&\quad \le \mu ^\mathrm {uu}[\lambda _k(x - e^\mathrm {u}_1) + \mu _k(x - e^\mathrm {u}_1,\mathrm {dd})] \\&\qquad + 2 \mu ^\mathrm {dd}[\lambda _k(x) + \mu _k(x,\mathrm {dd})] + \mu ^\mathrm {d} V_{k+1}(x) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {uu}[\lambda ^\mathrm {u}_1 V_k(x) + \lambda ^\mathrm {d}_1 V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_1) \\&\qquad +\, \lambda ^\mathrm {d}_2 V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_2)]+2 \mu ^\mathrm {dd}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1) \\&\qquad +\, \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1) + \lambda ^\mathrm {d}_2 V_k(x + e^\mathrm {d}_2)] \\&\qquad \mu ^\mathrm {uu}[\mu ^\mathrm {dd} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_2)^+) \\&\qquad +\, (\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_k(x - e^\mathrm {u}_1)] + \\&\qquad 2 \mu ^\mathrm {dd}[\mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_2)^+) \\&\qquad + \,(\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_k(x)] + \mu ^\mathrm {d} V_{k+1}(x) \end{aligned}$$

By rearranging the terms we obtain

$$\begin{aligned} \mu&_{k+1}(x,\mathrm {uu}) \\&= \lambda ^\mathrm {u}_1[\mu ^\mathrm {uu} V_k(x) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1)] \\&\quad + \,\lambda ^\mathrm {d}_1[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_1) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_1)] \\&\quad +\,\lambda ^\mathrm {d}_2[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_2) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_2)] \\&\quad +\, \mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + 2 \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1)^+)] \\&\quad + \,\mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_2)^+) + 2 \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_2)^+)] \\&\quad +\, \mu ^\mathrm {uu}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + 2 \mu ^\mathrm {dd} V_k(x)] \\&\quad +\,\mu ^\mathrm {d}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + 2 \mu ^\mathrm {dd} V_k(x)] + \mu ^\mathrm {d} V_{k+1}(x). \end{aligned}$$

By the induction assumption (that we refer to by “IA”) related to (41), we have

$$\begin{aligned}&\mu ^\mathrm {uu} V_k(x) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1) \\&\quad = \mu _k(x + e^\mathrm {u}_1,\mathrm {uu}) - \mu ^\mathrm {d} V_k(x + e^\mathrm {u}_1) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu _k(x + e^\mathrm {u}_1,\mathrm {dd}) - \mu ^\mathrm {d} V_k(x + e^\mathrm {u}_1) \\&\quad = \mu ^\mathrm {dd} V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_2)^+) \\&\qquad + \,\mu ^\mathrm {uu} V_k(x + e^\mathrm {u}_1). \end{aligned}$$

By using a similar argument, we also observe that

$$\begin{aligned}&\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_1) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_1) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu ^\mathrm {dd} V_k(x) + \mu ^\mathrm {dd} V_k((x + e^\mathrm {d}_1 - e^\mathrm {d}_2)^+) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {d}_1), \\&\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_2) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_2) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1 + e^\mathrm {d}_2)^+) + \mu ^\mathrm {dd} V_k(x) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {d}_2), \\&\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + 2 \mu ^\mathrm {dd} V_k(x) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_2)^+) + \mu ^\mathrm {uu} V_k(x). \end{aligned}$$

Thus, by the induction assumption we get

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \lambda ^\mathrm {u}_1[\mu ^\mathrm {dd} V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_2)^+) \\&\qquad +\, \mu ^\mathrm {uu} V_k(x + e^\mathrm {u}_1)] + \lambda ^\mathrm {d}_1[\mu ^\mathrm {dd} V_k(x) \\&\qquad + \mu ^\mathrm {dd} V_k((x + e^\mathrm {d}_1 - e^\mathrm {d}_2)^+) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {d}_1)]\\&\qquad +\,\lambda ^\mathrm {d}_2[\mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1 + e^\mathrm {d}_2)^+) + \mu ^\mathrm {dd} V_k(x) \\&\qquad +\, \mu ^\mathrm {uu} V_k(x + e^\mathrm {d}_2)] + \mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+)\\ {}&\qquad +\, \mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_2)^+) + 2 \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1)^+)] \\&\qquad + 2 \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_2)^+)]+ \mu ^\mathrm {uu}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1)\\&\qquad + 2 \mu ^\mathrm {dd} V_k(x)] + \mu ^\mathrm {d}[\mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1)^+) \\&\qquad +\, \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_2)^+)+ \mu ^\mathrm {uu} V_k(x)] + \mu ^\mathrm {d} V_{k+1}(x). \end{aligned}$$

Rearranging the terms gives

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad =\mu ^\mathrm {dd}[\lambda ^\mathrm {u}_1 V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_1)^+)) + \lambda ^\mathrm {d}_1 V_k(x) \\&\qquad +\, \lambda ^\mathrm {d}_2 V_k((x - e^\mathrm {d}_1 + e^\mathrm {d}_2)^+)] \\&\qquad +\,\mu ^\mathrm {dd}[\lambda ^\mathrm {u}_1 V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_2)^+)) + \lambda ^\mathrm {d}_1 V_k((x + e^\mathrm {d}_1 - e^\mathrm {d}_2)^+) \\&\qquad + \,\lambda ^\mathrm {d}_2 V_k(x)] +\mu ^\mathrm {uu}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1) + \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1) \\&\qquad + \lambda ^\mathrm {d}_2 V_k(x + e^\mathrm {d}_2)] + \mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) \\&\qquad + \,(2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_k((x - e^\mathrm {d}_1)^+)] \\&\qquad +\,\mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_2)^+) \\&\qquad + \,(2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_k((x - e^\mathrm {d}_2)^+)] \\&\qquad +\, \mu ^\mathrm {uu}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + (2 \mu ^\mathrm {dd} \\&\qquad +\, \mu ^\mathrm {d}) V_k(x)] + \mu ^\mathrm {d} V_{k+1}(x). \end{aligned}$$

By Proposition 3 we obtain

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {\le }\limits ^{\text{ P3 }} \mu ^\mathrm {dd}[\lambda ^\mathrm {u}_1 V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_1)^+)) + \lambda ^\mathrm {d}_1 V_k((x - e^\mathrm {d}_1)^+ + e^\mathrm {d}_1) \\&\qquad + \,\lambda ^\mathrm {d}_2 V_k((x - e^\mathrm {d}_1 + e^\mathrm {d}_2)^+)] \\&\qquad +\,\mu ^\mathrm {dd}[\lambda ^\mathrm {u}_1 V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_2)^+)) + \lambda ^\mathrm {d}_1 V_k((x + e^\mathrm {d}_1 - e^\mathrm {d}_2)^+) \\&\qquad +\, \lambda ^\mathrm {d}_2 V_k((x - e^\mathrm {d}_2)^+ + e^\mathrm {d}_2)] \\&\qquad +\,\mu ^\mathrm {uu}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1) + \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1) + \lambda ^\mathrm {d}_2 V_k(x + e^\mathrm {d}_2)] \\&\qquad +\,\mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) \\&\qquad +\, (2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_k((x - e^\mathrm {d}_1)^+)] \\&\qquad +\,\mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_2)^+) \\&\qquad +\, (2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_k((x - e^\mathrm {d}_2)^+)] \\&\qquad + \,\mu ^\mathrm {uu}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + (2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_k(x)] + \mu ^\mathrm {d} V_{k+1}(x) \end{aligned}$$

From this it follows by definition and using the induction assumption that

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {dd}[\lambda _k((x - e^\mathrm {d}_1)^+) + \mu _k((x - e^\mathrm {d}_1)^+,\mathrm {uu})] \\&\qquad +\mu ^\mathrm {dd}[\lambda _k((x - e^\mathrm {d}_2)^+) + \mu _k((x - e^\mathrm {d}_2)^+,\mathrm {uu})] \\&\qquad +\mu ^\mathrm {uu}[\lambda _k(x) + \mu _k(x,\mathrm {uu})] + \mu ^\mathrm {d} V_{k+1}(x) \\&\quad \mathop {=}\limits ^{\hbox {IA}} \mu ^\mathrm {dd} V_{k+1}((x - e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_{k+1}((x - e^\mathrm {d}_2)^+) \\&\qquad + (\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_{k+1}(x) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu _{k+1}(x,\mathrm {dd}). \end{aligned}$$

Next we will show that $\mu _{k+1}(x,\mathrm {uu}) \le \mu _{k+1}(x,\mathrm {du})$. Similarly as above, we have

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {uu} V_{k+1}(x - e^\mathrm {u}_1) + (2 \mu ^\mathrm {dd} + \mu ^\mathrm {d}) V_{k+1}(x) \\&\quad \le \mu ^\mathrm {uu}[\lambda _k(x - e^\mathrm {u}_1) + \mu _k(x - e^\mathrm {u}_1,\mathrm {du})] \\&\qquad + \mu ^\mathrm {d}[\lambda _k(x) + \mu _k(x,\mathrm {du})] + 2 \mu ^\mathrm {dd} V_{k+1}(x). \end{aligned}$$

By definition we have then

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {uu}[\lambda ^\mathrm {u}_1 V_k(x) + \lambda ^\mathrm {d}_1 V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_1) + \lambda ^\mathrm {d}_2 V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_2)] \\&\qquad +\,\mu ^\mathrm {d}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1) + \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1) + \lambda ^\mathrm {d}_2 V_k(x + e^\mathrm {d}_2)] \\&\qquad +\,\mu ^\mathrm {uu}[\mu ^\mathrm {d} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_k(x - e^\mathrm {u}_1)] \\&\qquad +\,\mu ^\mathrm {d}[\mu ^\mathrm {d} V_k((x - e^\mathrm {d}_1)^+) + (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_k(x)] \\&\qquad + \,2 \mu ^\mathrm {dd} V_{k+1}(x). \end{aligned}$$

Rearranging the terms finally yields

$$\begin{aligned} \mu _{k+1}&(x,\mathrm {uu}) \\&\quad =\lambda ^\mathrm {u}_1[\mu ^\mathrm {uu} V_k(x) + \mu ^\mathrm {d} V_k(x + e^\mathrm {u}_1)] \\&\qquad \,+ \lambda ^\mathrm {d}_1[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_1) + \mu ^\mathrm {d} V_k(x + e^\mathrm {d}_1)] \\&\qquad \,+\lambda ^\mathrm {d}_2[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x + e^\mathrm {d}_2)] \\&\qquad \,+\mu ^\mathrm {d}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + \mu ^\mathrm {d} V_k((x - e^\mathrm {d}_1)^+)] \\&\qquad \,+\mu ^\mathrm {uu}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + \mu ^\mathrm {d} V_k(x)] \\&\qquad \,+\,2 \mu ^\mathrm {dd}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + \mu ^\mathrm {d} V_k(x)] + 2 \mu ^\mathrm {dd} V_{k+1}(x). \end{aligned}$$

Again, by the induction assumption related to (41), we have

$$\begin{aligned}&\mu ^\mathrm {uu} V_k(x) + \mu ^\mathrm {d} V_k(x + e^\mathrm {u}_1) \\&\quad = \mu _k(x + e^\mathrm {u}_1,\mathrm {uu}) - 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1) \\&\mathop {\le }\limits ^{\hbox {IA}} \mu _k(x + e^\mathrm {u}_1,\mathrm {du}) - 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1) \\&\quad = \mu ^\mathrm {d} V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {u}_1). \end{aligned}$$

By using a similar argument, we also observe that

$$\begin{aligned}&\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_1) + \mu ^\mathrm {d} V_k(x + e^\mathrm {d}_1) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu ^\mathrm {d} V_k(x) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {d}_1), \\&\qquad \mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1 + e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x + e^\mathrm {d}_2) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu ^\mathrm {d} V_k((x - e^\mathrm {d}_1 + e^\mathrm {d}_2)^+) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {d}_2), \\&\qquad \mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + \mu ^\mathrm {d} V_k(x) \\&\quad \mathop {\le }\limits ^{\hbox {IA}}\mu ^\mathrm {d} V_k((x - e^\mathrm {d}_1)^+) + \mu ^\mathrm {uu} V_k(x). \end{aligned}$$

Thus, by the induction assumption we have

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \lambda ^\mathrm {u}_1[\mu ^\mathrm {d} V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {u}_1)] \\&\qquad \,+\lambda ^\mathrm {d}_1[\mu ^\mathrm {d} V_k(x) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {d}_1)] \\&\qquad \,+\lambda ^\mathrm {d}_2[\mu ^\mathrm {d} V_k((x - e^\mathrm {d}_1 + e^\mathrm {d}_2)^+) + \mu ^\mathrm {uu} V_k(x + e^\mathrm {d}_2)] \\&\qquad \,+\mu ^\mathrm {d}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) + \mu ^\mathrm {d} V_k((x - e^\mathrm {d}_1)^+)] \\&\qquad \, + \mu ^\mathrm {uu}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + \mu ^\mathrm {d} V_k(x)] \\&\qquad \,+ 2 \mu ^\mathrm {dd}[\mu ^\mathrm {d} V_k((x - e^\mathrm {d}_1)^+) + \mu ^\mathrm {uu} V_k(x)] + 2 \mu ^\mathrm {dd} V_{k+1}(x). \end{aligned}$$

Rearranging the terms gives

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad =\mu ^\mathrm {d}[\lambda ^\mathrm {u}_1 V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_1)^+)) + \lambda ^\mathrm {d}_1 V_k(x) \\&\qquad + \lambda ^\mathrm {d}_2 V_k((x - e^\mathrm {d}_1 + e^\mathrm {d}_2)^+)] \\&\qquad +\mu ^\mathrm {uu}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1) + \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1) + \lambda ^\mathrm {d}_2 V_k(x + e^\mathrm {d}_2)] \\&\qquad +\mu ^\mathrm {d}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) \\&\qquad + (\mu ^\mathrm {d} + 2\mu ^\mathrm {dd}) V_k((x - e^\mathrm {d}_1)^+)] \\&\qquad +\mu ^\mathrm {uu}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + (\mu ^\mathrm {d} \\&\qquad + 2\mu ^\mathrm {dd}) V_k(x)] + 2 \mu ^\mathrm {dd} V_{k+1}(x). \end{aligned}$$

By Proposition 3 we obtain

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {\le }\limits ^{\text{ P3 }} \mu ^\mathrm {d}[\lambda ^\mathrm {u}_1 V_k((x + e^\mathrm {u}_1 - e^\mathrm {d}_1)^+)) + \lambda ^\mathrm {d}_1 V_k((x - e^\mathrm {d}_1)^+ + e^\mathrm {d}_1) \\&\qquad + \,\lambda ^\mathrm {d}_2 V_k((x - e^\mathrm {d}_1 + e^\mathrm {d}_2)^+)]\\ {}&\qquad +\,\mu ^\mathrm {uu}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1) + \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1) + \lambda ^\mathrm {d}_2 V_k(x + e^\mathrm {d}_2)] \\&\qquad +\,\mu ^\mathrm {d}[\mu ^\mathrm {uu} V_k((x - e^\mathrm {u}_1 - e^\mathrm {d}_1)^+) \\&\qquad +\, (\mu ^\mathrm {d} + 2\mu ^\mathrm {dd}) V_k((x - e^\mathrm {d}_1)^+)] \\&\qquad +\,\mu ^\mathrm {uu}[\mu ^\mathrm {uu} V_k(x - e^\mathrm {u}_1) + (\mu ^\mathrm {d} + 2\mu ^\mathrm {dd}) V_k(x)] \\&\qquad +\, 2 \mu ^\mathrm {dd} V_{k+1}(x). \end{aligned}$$

Finally, it follows by definition and using the induction assumption that

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {uu}) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {d}[\lambda _k((x - e^\mathrm {d}_1)^+) + \mu _k((x - e^\mathrm {d}_1)^+,\mathrm {uu})] \\&\qquad + \mu ^\mathrm {uu}[\lambda _k(x) + \mu _k(x,\mathrm {uu})] + 2 \mu ^\mathrm {dd} V_{k+1}(x) \\&\quad \mathop {=}\limits ^{\hbox {IA}} \mu ^\mathrm {d} V_{k+1}((x - e^\mathrm {d}_1)^+) + (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_{k+1}(x) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu _{k+1}(x,\mathrm {du}). \end{aligned}$$

By symmetry, we also have $\mu _{k+1}(x,\mathrm {uu}) \le \mu _{k+1}(x,\mathrm {ud})$. Thus, (41) is valid for $k + 1$.

(b) Consider then any state x such that $x^\mathrm {u}_1 \ge 0$, $x^\mathrm {d}_1 > 0$, and $x^\mathrm {d}_2 > 0$. Now we have

$$\begin{aligned} \mu _{k+1}(x,\mathrm {dd})&\mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {dd} V_{k+1}(x - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_{k+1}(x - e^\mathrm {d}_2) \\&\quad + \, (\mu ^\mathrm {uu} + \mu ^\mathrm {d}) V_{k+1}(x) \\&\le \mu ^\mathrm {dd}[\lambda _k(x - e^\mathrm {d}_1) + \mu _k(x - e^\mathrm {d}_1,\mathrm {du})] \\&\quad + \, \mu ^\mathrm {dd}[\lambda _k(x - e^\mathrm {d}_2) + \mu _k(x - e^\mathrm {d}_2,\mathrm {du})] \\&\quad + \, \mu ^\mathrm {d}[\lambda _k(x) + \mu _k(x,\mathrm {du})] + \mu ^\mathrm {uu} V_{k+1}(x). \end{aligned}$$

By definition this equals

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {dd}) \\&\quad \mathop {=}\limits ^{\hbox {def.}}\mu ^\mathrm {dd}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1 - e^\mathrm {d}_1) + \lambda ^\mathrm {d}_1 V_k(x) \\&\qquad + \, \lambda ^\mathrm {d}_2 V_k(x - e^\mathrm {d}_1 + e^\mathrm {d}_2)] \\&\qquad + \, \mu ^\mathrm {dd}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1 - e^\mathrm {d}_2) \\&\qquad + \, \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1 - e^\mathrm {d}_2) + \lambda ^\mathrm {d}_2 V_k(x)] \\&\qquad + \, \mu ^\mathrm {d}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1) + \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1) + \lambda ^\mathrm {d}_2 V_k(x + e^\mathrm {d}_2)] \\&\qquad + \, \mu ^\mathrm {dd}[\mu ^\mathrm {d} V_k((x - 2 e^\mathrm {d}_1)^+) + (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_k(x - e^\mathrm {d}_1)] \\&\qquad + \, \mu ^\mathrm {dd}[\mu ^\mathrm {d} V_k((x - e^\mathrm {d}_1 - e^\mathrm {d}_2)^+) \\&\qquad + \, (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_k(x - e^\mathrm {d}_2)] \\&\qquad + \, \mu ^\mathrm {d}[\mu ^\mathrm {d} V_k(x - e^\mathrm {d}_1) + (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_k(x)] \\&\qquad + \, \mu ^\mathrm {uu} V_{k+1}(x). \end{aligned}$$

Rearranging the terms finally yields

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {dd}) \\&\quad = \lambda ^\mathrm {u}_1[\mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1 - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1- e^\mathrm {d}_2) \\&\qquad + \, \mu ^\mathrm {d} V_k(x + e^\mathrm {u}_1)] \\&\qquad + \, \lambda ^\mathrm {d}_1[\mu ^\mathrm {dd} V_k(x) + \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_1 - e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x + e^\mathrm {d}_1)] \\&\qquad + \, \lambda ^\mathrm {d}_2[\mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_1 + e^\mathrm {d}_2) + \mu ^\mathrm {dd} V_k(x) + \mu ^\mathrm {d} V_k(x + e^\mathrm {d}_2)] \\&\qquad + \, \mu ^\mathrm {d}[\mu ^\mathrm {dd} V_k((x - 2 e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1 - e^\mathrm {d}_2)^+) \\&\qquad + \, \mu ^\mathrm {d} V_k(x - e^\mathrm {d}_1)] \\&\qquad +\, 2 \mu ^\mathrm {dd}[\mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x)] \\&\qquad +\, \mu ^\mathrm {uu}[\mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x)] \\&\qquad + \, \mu ^\mathrm {uu} V_{k+1}(x). \end{aligned}$$

By the induction assumption related to (42), we have

$$\begin{aligned}&\mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1 - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1 - e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x + e^\mathrm {u}_1) \\&\quad = \mu _k(x + e^\mathrm {u}_1,\mathrm {dd}) - \mu ^\mathrm {uu} V_k(x + e^\mathrm {u}_1) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu _k(x + e^\mathrm {u}_1,\mathrm {du}) - \mu ^\mathrm {uu} V_k(x + e^\mathrm {u}_1) \\&\quad = \mu ^\mathrm {d} V_k(x + e^\mathrm {u}_1 - e^\mathrm {d}_1) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1). \end{aligned}$$

Note that even if $x^\mathrm {u}_1 = 0$ for the original state x, the induction assumption related to (43) is not sufficient to justify the result above. This is the reason why we need to include Eq. (42) in our induction hypothesis.

On the other hand, by the induction assumption related to (43), we have

$$\begin{aligned}&\mu ^\mathrm {dd} V_k(x) + \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_1 - e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x + e^\mathrm {d}_1)\\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu ^\mathrm {d} V_k(x) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_1), \\&\qquad \mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_1 + e^\mathrm {d}_2) + \mu ^\mathrm {dd} V_k(x) + \mu ^\mathrm {d} V_k(x + e^\mathrm {d}_2) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu ^\mathrm {d} V_k(x - e^\mathrm {d}_1 + e^\mathrm {d}_2) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_2), \\&\qquad \mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \mu ^\mathrm {d} V_k(x - e^\mathrm {d}_1) + 2 \mu ^\mathrm {dd} V_k(x). \end{aligned}$$

Thus, the induction assumption gives

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {dd}) \\&\quad \mathop {\le }\limits ^{\hbox {IA}} \lambda ^\mathrm {u}_1[\mu ^\mathrm {d} V_k(x + e^\mathrm {u}_1 - e^\mathrm {d}_1) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {u}_1)] \\&\qquad +\,\lambda ^\mathrm {d}_1[\mu ^\mathrm {d} V_k(x) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_1)] \\&\qquad +\,\lambda ^\mathrm {d}_2[\mu ^\mathrm {d} V_k(x - e^\mathrm {d}_1 + e^\mathrm {d}_2) + 2 \mu ^\mathrm {dd} V_k(x + e^\mathrm {d}_2)] \\&\qquad +\,\mu ^\mathrm {d}[\mu ^\mathrm {dd} V_k((x - 2 e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1 - e^\mathrm {d}_2)^+) \\&\qquad +\, \mu ^\mathrm {d} V_k(x - e^\mathrm {d}_1)] \\&\qquad +\,2 \mu ^\mathrm {dd}[\mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_2) + \mu ^\mathrm {d} V_k(x)] \\&\qquad +\,\mu ^\mathrm {uu}[\mu ^\mathrm {d} V_k(x - e^\mathrm {d}_1) + 2 \mu ^\mathrm {dd} V_k(x)] + \mu ^\mathrm {uu} V_{k+1}(x). \end{aligned}$$

Rearranging the terms yields

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {dd}) \\&\quad =\mu ^\mathrm {d}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1 - e^\mathrm {d}_1)) + \lambda ^\mathrm {d}_1 V_k(x) + \lambda ^\mathrm {d}_2 V_k(x - e^\mathrm {d}_1 + e^\mathrm {d}_2)] \\&\qquad +\,2 \mu ^\mathrm {dd}[\lambda ^\mathrm {u}_1 V_k(x + e^\mathrm {u}_1) + \lambda ^\mathrm {d}_1 V_k(x + e^\mathrm {d}_1) + \lambda ^\mathrm {d}_2 V_k(x + e^\mathrm {d}_2)] \\&\qquad +\,\mu ^\mathrm {d}[\mu ^\mathrm {dd} V_k((x - 2 e^\mathrm {d}_1)^+) + \mu ^\mathrm {dd} V_k((x - e^\mathrm {d}_1 - e^\mathrm {d}_2)^+) \\&\qquad +\, (\mu ^\mathrm {d} + \mu ^\mathrm {uu}) V_k(x - e^\mathrm {d}_1)] \\&\qquad +\,2 \mu ^\mathrm {dd}[\mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_1) + \mu ^\mathrm {dd} V_k(x - e^\mathrm {d}_2) \\&\qquad + \,(\mu ^\mathrm {d} + \mu ^\mathrm {uu}) V_k(x)] \\&\qquad + \,\mu ^\mathrm {uu} V_{k+1}(x). \end{aligned}$$

From this it follows by definition and using the induction assumption that

$$\begin{aligned}&\mu _{k+1}(x,\mathrm {dd}) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu ^\mathrm {d}[\lambda _k(x - e^\mathrm {d}_1) + \mu _k(x - e^\mathrm {d}_1,^\mathrm {uu})] \\&\qquad + 2 \mu ^\mathrm {dd}[\lambda _k(x) + \mu _k(x,\mathrm {uu})] + \mu ^\mathrm {uu} V_{k+1}(x) \\&\quad \mathop {=}\limits ^{\hbox {IA}} \mu ^\mathrm {d} V_{k+1}((x - e^\mathrm {d}_1)^+) + (\mu ^\mathrm {uu} + 2 \mu ^\mathrm {dd}) V_{k+1}(x) \\&\quad \mathop {=}\limits ^{\hbox {def.}} \mu _{k+1}(x,\mathrm {du}). \end{aligned}$$

By symmetry, we also have $\mu _{k+1}(x,\mathrm {dd}) \le \mu _{k+1}(x,\mathrm {ud})$. Thus, Eq. (42) is valid for $k + 1$.

(c) Consider then any state x such that $x^\mathrm {u}_1 = 0$, $x^\mathrm {d}_1 > 0$, and $x^\mathrm {d}_2 > 0$. Due to (b), it remains to prove that $\mu _{k+1}(x,\mathrm {dd}) \le \mu _{k+1}(x,\mathrm {uu})$ in this case. However, it can easily be proved by using a similar argument as in $1^\circ$ (for $k = 0$). Thus, we conclude that Eq. (43) is valid for $k + 1$.

(d) Also Eqs. (44) and (45) can easily be proved to be valid for $k + 1$ by using a similar argument as in $1^\circ$ (for $k = 0$), which completes the proof. $\square$

Appendix 7: Proof of Theorem 5

To save space, we only give a sketch of the proof as it is technically very similar to the proof of Theorem 4.

Theorem 5

The PD policy is stochastically optimal if

$$\begin{aligned} \lambda ^\mathrm {d}_1 > 0, \quad \lambda ^\mathrm {d}_2 = 0 \quad \hbox {and}\quad \mu ^\mathrm {d} \ge 2 \mu ^\mathrm {uu}. \end{aligned}$$

(46)

Proof

Assume condition (46). Since $\lambda ^\mathrm {d}_2 = 0$, it is obvious that the ‘ud’ mode should not be used. In addition, due to the fact that $\mu ^\mathrm {d} \ge \mu ^\mathrm {dd}$, it is also clear that there is no need to use the ‘dd’ mode. Thus, it is sufficient to consider the ‘du’ and ‘uu’ modes.

The result is proved by induction following similar lines as in the proof of Theorem 4. More precisely, we are able to show that the following equations are valid for any $k \in \{0,1,\ldots$}:

$$\begin{aligned} \mu _k(x,\mathrm {du}) \le \mu _k(x,\mathrm {uu}),&\hbox {if } x^\mathrm {d}_1 > 0; \end{aligned}$$

(47)

$$\begin{aligned} \mu _k(x,\mathrm {uu}) \le \mu _k(x,\mathrm {du}),&\hbox {if } x^\mathrm {d}_1 = 0, \end{aligned}$$

(48)

where $\mu _k(x,\cdot )$ are defined in (24). The optimality then follows from (47) and (48). $\square$