Optimal assignment of servers to tasks when collaboration is inefficient

Abstract

Consider a Markovian system of two stations in tandem with finite intermediate buffer and two servers. The servers are heterogeneous, flexible, and more efficient when they work on their own than when they collaborate. We determine how the servers should be assigned dynamically to the stations with the goal of maximizing the system throughput. We show that the optimal policy depends on whether or not one server is dominant (i.e., faster at both stations) and on the magnitude of the efficiency loss of collaborating servers. In particular, if one server is dominant then he must divide his time between the two stations, and we identify the threshold policy the dominant server should use; otherwise each server should focus on the station where he is the faster server. In all cases, servers only collaborate to avoid idleness when the first station is blocked or the second station is starved, and we determine when collaboration is preferable to idleness as a function of the efficiency loss of collaborating servers.

Appendix

1.1 Proof of Theorem 3.4

It follows from our assumptions on service rates that $\mu _{11}>0$ and $\mu _{22}>0$. The set of possible actions is given by $A=\{a_{\sigma _1\sigma _2}: \sigma _i \in \{0,1,2\}, \forall i=1,2\}$, where for all $i\in \{1,2\}, \sigma _i=0$ when server $i$ is idle and $\sigma _i=j\in \{1,2\}$ when server $i$ is assigned to station $j$.

The set $A_s$ of allowable actions in state $s$ is given as

$$\begin{aligned} A_s = \left\{ \begin{array}{ll} \{a_{11}\}&\text{ for}\, s =0, \\ \{a_{11},a_{12},a_{21},a_{22}\}&\text{ for}\, s \in \{1,\ldots ,B+1\}, \\ \{a_{22}\}&\text{ for}\, s=B+2, \end{array} \right. \end{aligned}$$

where we use sample path arguments similar to those of Lemma 2.1 and Corollary 2.2 of Kırkızlar et al. [16] to eliminate actions that allow servers to idle (exploiting the fact that when servers collaborate, their combined service rate at each station is faster than the rate of the faster server at that station). Since the number of possible states and actions are both finite, the existence of an optimal Markovian stationary deterministic policy follows from Theorem 9.1.8 of Puterman [18], which provides sufficient conditions under which such a policy exists.

Under our assumptions on the service rates, the policy described in Theorem 3.4 corresponds to an irreducible Markov chain, and, hence, we have a communicating Markov decision process. Therefore, we use the policy iteration algorithm for communicating models (see pp. 479–480 of Puterman [18]) to prove the optimality of the policy.

Let $p(s^{\prime } | s, d(s))$ be the probability of going to state $s^{\prime }\in S$ in one step when the action prescribed by decision rule $d$ is taken in state $s$ and $P_d$ be the corresponding $(B+3) \times (B+3)$-dimensional probability transition matrix. Similarly, $r(s,d(s))$ denotes the immediate reward obtained when the action prescribed by decision rule $d$ is taken in state $s$ and $r_d$ denotes the corresponding $(B+3)$-dimensional reward vector.

As the initial policy of the policy iteration algorithm, we choose

$$\begin{aligned} d_0 (s) = \left\{ \begin{array}{ll} a_{11}&\quad \text{ for}\, s = 0, \\ a_{12}&\quad \text{ for}\, 1\le s \le B+1, \\ a_{22}&\quad \text{ for}\, s=B+2, \end{array} \right. \end{aligned}$$

corresponding to the policy described in Theorem 3.4. Then

$$\begin{aligned} r(s,d_0(s)) = \left\{ \begin{array}{ll} 0&\quad \text{ for}\, s = 0, \\ \mu _{22}&\quad \text{ for}\, 1\le s \le B+1, \\ \alpha (\mu _{12}+\mu _{22})&\quad \text{ for}\, s=B+2, \end{array} \right. \end{aligned}$$

and

$$\begin{aligned} p(s^{\prime } | s,d_0(s))= \left\{ \begin{array}{ll} \frac{\alpha (\mu _{11}+\mu _{21})}{q}&\quad \text{ for}\, s =0, s^{\prime } =1, \\ 1 - \frac{\alpha (\mu _{11}+\mu _{21})}{q}&\quad \text{ for}\, s = s^{\prime } = 0, \\ \frac{\mu _{22}}{q}&\quad \text{ for}\, 1\le s \le B+1, s^{\prime } =s-1, \\ 1-\frac{\mu _{11}+\mu _{22}}{q}&\quad \text{ for}\, 1\le s \le B+1, s^{\prime }=s, \\ \frac{\mu _{11}}{q}&\quad \text{ for}\, 1\le s \le B+1, s^{\prime } = s+1, \\ \frac{\alpha (\mu _{12}+\mu _{22})}{q}&\quad \text{ for}\, s =B+2, s^{\prime } =B+1, \\ 1 - \frac{\alpha (\mu _{12}+\mu _{22})}{q}&\quad \text{ for}\, s = s^{\prime } =B+2, \end{array} \right. \end{aligned}$$

where $q$ is the uniformization constant. Since the policy $(d_0)^{\infty }$ (corresponding to the decision rule $d_0$) yields an irreducible Markov chain, we find a scalar $g_0$ and a vector $h_0$ solving

$$\begin{aligned} r_{d_0} - g_0 e + (P_{d_0} - I) h_0 = 0, \end{aligned}$$

(10)

subject to $h_{0}(0) = 0$, where $e$ is a column vector of ones and $I$ is the identity matrix. Then

$$\begin{aligned} g_0&\!=\!&\frac{\alpha (\mu _{11}\!+\!\mu _{21})(\mu _{12}\!+\!\mu _{22}) \sum _{j=0}^{B\!+\!1}\mu _{11}^j\mu _{22}^{B\!+\!1\!-\!vj}}{\mu _{11}^{B\!+\!1}(\mu _{11}\!+\! \mu _{21})\!+\!\mu _{22}^{B+1}(\mu _{12}\!+\!\mu _{22})\!+\!\alpha (\mu _{11}\!+\!\mu _{21}) (\mu _{12}\!+\!\mu _{22})\sum _{j=0}^{B}\mu _{11}^j\mu _{22}^{B-j}}, \end{aligned}$$

$h_0(0)=0$, and

$$\begin{aligned} h_0(s)&= \frac{q g_0}{\alpha (\mu _{11}+\mu _{21})\mu _{11}^{s-1}}\left[(\alpha (\mu _{11}+\mu _{21})-\mu _{11}+\mu _{22})\sum _{j=0}^{s-2}(j+1)\mu _{11}^j\mu _{22}^{s-2-j}+s \mu _{11}^{s-1}\right] \\&- \frac{q \mu _{22}}{\mu _{11}^{s-1}}\sum _{j=0}^{s-2}(j+1)\mu _{11}^j\mu _{22}^{s-2-j} \end{aligned}$$

for $1\le s \le B+2$ constitute a solution to Eq. (10).

For the next step of the policy iteration algorithm, we choose

$$\begin{aligned} d_1 (s) \in \arg \max _{a \in A_s} \left\{ r (s,a) + \sum _{j \in S} p (j| s, a) h_0 (j) \right\} ,\quad \forall s \in S, \end{aligned}$$

setting $d_1 (s) = d_0 (s)$ if possible. We now show that $d_1(s)=d_0(s)$ for all $s \in S$. For all $s \in S \setminus \{ 0, B + 2 \}$ and $a \in A_s \setminus \{ d_0 (s) \}$, we will compute the differences

$$\begin{aligned} \epsilon (s,a)&= r (s, d_0 (s)) + \sum _{j \in S} p (j| s, d_0 (s)) h_0 (j) -\left( r (s, a) + \sum _{j \in S} p (j| s, a) h_0 (j) \right) \end{aligned}$$

and show that the differences are non-negative. For $s = 0$ and $s=B+2$, there is nothing to prove because there is only one possible action in these states, namely $d_0 (0) = a_{11}$ and $d_0 (B+2) = a_{22}$.

For $s\in \{1,\ldots , B+1\}$, we have that $d_0 (s) = a_{12}$. We will specify $\epsilon (s,a)$ for actions $a_{11}, a_{21}$, and $a_{22}$. With some algebra we obtain

$$\begin{aligned} \epsilon (s,a_{11})&= \frac{\alpha (\mu _{11}+\mu _{21})\mu _{11}^{B+1-s} \sum _{j=0}^{s-1}\mu _{11}^j\mu _{22}^{s-1-j}\Upsilon _1(\alpha )}{\Upsilon }, \end{aligned}$$

where

$$\begin{aligned} \Upsilon _1(\alpha )&= -\alpha (\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})+2\mu _{11}\mu _{22}+ \mu _{11}\mu _{12}+\mu _{21}\mu _{22} \end{aligned}$$

(11)

and

$$\begin{aligned} \Upsilon \!=\! \mu _{11}^{B+1}(\mu _{11}\!+\!\mu _{21})\!+\!\mu _{22}^{B+1}(\mu _{12}\!+\!\mu _{22})\!+\! \alpha (\mu _{11}\!+\!\mu _{21})(\mu _{12}\!+\!\mu _{22}) \sum _{j=0}^{B} \mu _{11}^j\mu _{22}^{B-j}>0. \end{aligned}$$

Note that $\Upsilon _1(\alpha )\ge 0$ for $\alpha \le 1$ as long as $\mu _{11}\mu _{22}\ge \mu _{12}\mu _{21}$, which follows from our assumptions on the service rates. Hence, $\epsilon (s,a_{11})\ge 0$ with equality only when $\mu _{11}\mu _{22}=\mu _{12}\mu _{21}$ and $\alpha =1$.

Similarly,

$$\begin{aligned} \epsilon (s,a_{21})&= \frac{\mu _{22}^B\Delta _1(\alpha )+\sum _{j=0}^{B-s}\mu _{11}^{j+1}\mu _{22}^{B-j-1}\Delta _1(\alpha )+\sum _{j=B-s+1}^{B}\mu _{11}^j\mu _{22}^{B-j}\Delta _2(\alpha )}{\Upsilon }, \end{aligned}$$

where $\Delta _1(\alpha )$ and $\Delta _2(\alpha )$ are defined in the proof of Lemma 4.8. In this case, $\Delta _1(\alpha )$ is non-decreasing in $\alpha$. Since

$$\begin{aligned} \Delta _1\left(\frac{\mu _{11}}{\mu _{11}+\mu _{21}}\right)&= \mu _{22}(\mu _{12}+\mu _{22})(\mu _{11}-\mu _{21})\ge 0, \end{aligned}$$

we can conclude that $\Delta _1(\alpha )\ge 0$ for all $\max \{\frac{\mu _{11}}{\mu _{11}+\mu _{21}},\frac{\mu _{22}}{\mu _{12}+\mu _{22}}\} \le \alpha \le 1$. Moreover, $\Delta _2(\alpha )$ is non-decreasing in $\alpha$. Since

$$\begin{aligned} \Delta _2\left(\frac{\mu _{22}}{\mu _{12}+\mu _{22}}\right)&= \mu _{11}(\mu _{11}+\mu _{21})(\mu _{22}-\mu _{12})\ge 0, \end{aligned}$$

we can conclude that $\Delta _2(\alpha )\ge 0$ for all $\max \{\frac{\mu _{11}}{\mu _{11}+\mu _{21}},\frac{\mu _{22}}{\mu _{12}+\mu _{22}}\} \le \alpha \le 1$. Hence, $\epsilon (s,a_{21})\ge 0$ with equality only when $\mu _{11}=\mu _{21}$ and $\mu _{12}=\mu _{22}$.

Finally,

$$\begin{aligned} \epsilon (s,a_{22})&= \frac{\alpha (\mu _{12}+\mu _{22})\mu _{22}^{s-1}\sum _{j=0}^{B-s+1} \mu _{11}^j\mu _{22}^{B-s+1-j}\Upsilon _1(\alpha )}{\Upsilon }, \end{aligned}$$

which is non-negative since $\Upsilon _1(\alpha )\ge 0$ with equality only when $\mu _{11}\mu _{22}=\mu _{12}\mu _{21}$ and $\alpha =1$. This proves that $d_1(s)=d_0(s)$ for all $s\in S$. Thus, the policy described in Theorem 3.4 is optimal. The proof of the uniqueness of the optimal policy is similar to the uniqueness proof in Theorem 3.1 of Andradóttir and Ayhan [6] (the lower bound on $\alpha$ needs to be strict to ensure that the policies in Theorems 3.2 and 3.3 are not optimal and that idling actions are not optimal in states $1,\ldots ,B+1$).

1.2 Proof of Theorem 4.4

The proof of Theorem 4.4 is similar to the proof of Theorem 3.4. It follows from our assumptions on service rates that $\mu _{11}>0$, $\mu _{12}>0$, and either $\mu _{21}>0$ or $\mu _{22}>0$.

The set $A_s$ of allowable actions in state $s$ is the same as the one described in the proof of Theorem 3.4, where we again use the results of Kırkızlar et al. [16] to eliminate idling actions for the specified range of $\alpha$. Since the number of possible states and actions are both finite, the existence of an optimal Markovian stationary deterministic policy follows from Theorem 9.1.8 of Puterman [18].

Under our assumptions on the service rates, the policy described in Theorem 4.4 corresponds to a unichain Markov chain, and, hence, we have a weakly communicating Markov decision process. Therefore, we use the policy iteration algorithm for communicating models (see pp. 479–480 of Puterman [18]) to prove the optimality of the policy described in Theorem 4.4. This time as the initial policy of the policy iteration algorithm, we choose

$$\begin{aligned} d^{\prime }_0 (s) = \left\{ \begin{array}{ll} a_{11}&\text{ for}\; s = 0, \\ a_{12}&\text{ for}\; 1\le s \le s^*-1, \\ a_{21}&\text{ for}\; s^* \le s \le B+1, \\ a_{22}&\text{ for}\; s=B+2, \end{array} \right. \end{aligned}$$

corresponding to the policy described in Theorem 4.4. Then

$$\begin{aligned} r(s,d^{\prime }_0(s)) = \left\{ \begin{array}{ll} 0&\text{ for}\, s = 0, \\ \mu _{22}&\text{ for}\; 1\le s \le s^*-1, \\ \mu _{12}&\text{ for}\; s^*\le s \le B+1, \\ \alpha (\mu _{12}+\mu _{22})&\text{ for}\; s=B+2, \end{array} \right. \end{aligned}$$

and

$$\begin{aligned} p(s^{\prime } | s, d^{\prime }_0(s))= \left\{ \begin{array}{ll} \frac{\alpha (\mu _{11}+\mu _{21})}{q}&\text{ for}\; s =0, s^{\prime } =1, \\ 1 - \frac{\alpha (\mu _{11}+\mu _{21})}{q}&\text{ for}\; s = s^{\prime } = 0, \\ \frac{\mu _{22}}{q}&\text{ for}\; 1\le s \le s^*-1, s^{\prime } =s-1, \\ 1-\frac{\mu _{11}+\mu _{22}}{q}&\text{ for}\; 1\le s \le s^*-1, s^{\prime }=s, \\ \frac{\mu _{11}}{q}&\text{ for}\; 1\le s \le s^*-1, s^{\prime } = s+1, \\ \frac{\mu _{12}}{q}&\text{ for}\; s^*\le s \le B+1, s^{\prime } =s-1, \\ 1-\frac{\mu _{21}+\mu _{12}}{q}&\text{ for}\; s^*\le s \le B+1, s^{\prime }=s, \\ \frac{\mu _{21}}{q}&\text{ for}\; s^*\le s \le B+1, s^{\prime } = s+1, \\ \frac{\alpha (\mu _{12}+\mu _{22})}{q}&\text{ for}\; s =B+2, s^{\prime } =B+1, \\ 1 - \frac{\alpha (\mu _{12}+\mu _{22})}{q}&\text{ for}\; s = s^{\prime } =B+2, \end{array} \right. \end{aligned}$$

where $q$ is the uniformization constant. Since the policy $(d^{\prime }_0)^{\infty }$ (corresponding to the decision rule $d^{\prime }_0$) is irreducible, we find a scalar $g^{\prime }_0$ and a vector $h^{\prime }_0$ solving

$$\begin{aligned} r_{d^{\prime }_0} - g^{\prime }_0 e + (P_{d^{\prime }_0} - I) h^{\prime }_0 = 0, \end{aligned}$$

(12)

subject to $h^{\prime }_{0}(0) = 0$, where $e$ is again a column vector of ones and $I$ is the identity matrix. Then

$$\begin{aligned} g^{\prime }_0&= \frac{\Theta _1}{\Theta _2}, \end{aligned}$$

where

$$\begin{aligned} \Theta _1\!&= \! \alpha (\mu _{11}\!+\!\mu _{21})\left(\frac{\sum _{j=0}^{s^*-1} \mu _{11}^j\mu _{22}^{s^*-1-j}}{\mu _{22}^{s^*-1}}+\frac{\mu _{11}^ {s^*-1}\mu _{21}\sum _{j=0}^{B+1-s^*}\mu _{21}^j\mu _{12}^{B+1-s^*-j}}{\mu _{12}^{B+2-s^*}\mu _{22}^{s^*-1}}\right),\\ \Theta _2&= 1+\frac{\alpha (\mu _{11}+\mu _{21})\sum _{j=0}^{s^*-2} \mu _{11}^j\mu _{22}^{s^*-2-j}}{\mu _{22}^{s^*-1}}+\frac{\alpha (\mu _{11}+\mu _{21})\mu _{11}^{s^*-1}}{\mu _{12}^{B+2-s^*} \mu _{22}^{s^*-1}}\\&\left(\sum _{j=0}^{B+1-s^*}\mu _{21}^j\mu _{12} ^{B+1-s^*-j}+\frac{\mu _{21}^{B+2-s^*}}{\alpha (\mu _{12}+\mu _{22})}\right), \end{aligned}$$

$h^{\prime }_0(0)=0$,

$$\begin{aligned} h^{\prime }_0(s)&= \frac{q g^{\prime }_0}{\alpha (\mu _{11}+\mu _{21})\mu _{11}^{s-1}}\left[(\alpha (\mu _{11}+\mu _{21})-\mu _{11}+\mu _{22})\sum _{j=0}^{s-2}(j+1)\mu _{11}^j\mu _{22}^{s-2-j}+s \mu _{11}^{s-1}\right] \\&- \frac{q \mu _{22}}{\mu _{11}^{s-1}}\sum _{j=0}^{s-2}(j+1)\mu _{11}^j\mu _{22}^{s-2-j} \end{aligned}$$

for $1\le s \le s^*$, and

$$\begin{aligned} h^{\prime }_0(s)&= h^{\prime }_0(s^*)+\frac{q \mu _{12}}{\mu _{21}^{s-s^*}\mu _{11}^{s^*-1}}\sum _{j=0}^{s-s^*-1} \mu _{21}^j\mu _{12}^{s-s^*-1-j}\Big [\frac{g^{\prime }_0}{\alpha (\mu _{11}+ \mu _{21})}\\&\left(\sum _{k=0}^{s^*-2}\mu _{11}^k\mu _{22}^{s^*-2-k}(\alpha (\mu _{11}+\mu _{21})-\mu _{11}+\mu _{22})+ \mu _{11}^{s^*-1}\right)\\&\quad -\mu _{22}\sum _{i=0}^{s^*-2}\mu _{11}^i \mu _{22}^{s^*-2-i}\Big ]+\frac{q}{\mu _{21}^{s-s^*}}(g^{\prime }_0-\mu _{12}) \sum _{j=0}^{s-s^*-1}(j+1)\mu _{21}^j\mu _{12}^{s-s^*-j-1} \end{aligned}$$

for $s^*+1 \le s \le B+2$, constitute a solution to Eq. (12).

For the next step of the policy iteration algorithm, we choose

$$\begin{aligned} d^{\prime }_1 (s) \in \arg \max _{a \in A_s} \left\{ r (s,a) + \sum _{j \in S} p (j| s, a) h^{\prime }_0 (j) \right\} ,\quad \forall s \in S, \end{aligned}$$

setting $d^{\prime }_1 (s) = d^{\prime }_0 (s)$ if possible. We now show that $d^{\prime }_1(s)=d^{\prime }_0(s)$ for all $s \in S$. In particular, for all $s \in S \setminus \{ 0, B + 2 \}$ and $a \in A_s \setminus \{ d^{\prime }_0 (s) \}$, we will compute the differences

$$\begin{aligned} \epsilon ^{\prime }(s,a)&= r (s, d^{\prime }_0 (s)) + \sum _{j \in S} p (j| s, d^{\prime }_0 (s)) h^{\prime }_0 (j) -\Big ( r (s, a) + \sum _{j \in S} p (j| s, a) h^{\prime }_0 (j) \Big ) \end{aligned}$$

and show that the differences are non-negative. Note that for $s = 0$ and $s=B+2$, there is nothing to prove because there is only one possible action in these states, namely $d^{\prime }_0 (0) = a_{11}$ and $d^{\prime }_0 (B+2) = a_{22}$.

For $s\in \{1,\ldots , s^*-1\}$, we have that $d^{\prime }_0 (s) = a_{12}$. We will specify $\epsilon ^{\prime }(s,a)$ for actions $a_{11}, a_{21}$, and $a_{22}$. Without loss of generality, assume that $s^*>1$ because otherwise, this set of states is empty and there is nothing to prove. With some algebra, we obtain

$$\begin{aligned} \epsilon ^{\prime }(s,a_{11})&= \frac{\alpha (\mu _{11}+\mu _{21})\mu _{11}^{s^*-s-1} \sum _{j=0}^{s-1}\mu _{11}^j\mu _{22}^{s-1-j}\Big [\mu _{21}^{B+2-s^*} \Upsilon _1(\alpha )+\sum _{j=0}^{B+1-s^*}\mu _{21}^j\mu _{12}^{B+1-s^*-j} \Delta _1(\alpha )\Big ]}{\Upsilon ^{\prime }}, \end{aligned}$$

where

$$\begin{aligned} \Upsilon ^{\prime }&= \alpha (\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})\left[\mu _{12}^{B+2-s^*} \sum _{j=0}^{s^*-2}\mu _{11}^j\mu _{22}^{s^*-2-j}\right.\nonumber \\&\left.+\mu _{11}^{s^*-1} \sum _{j=0}^{B+1-s^*}\mu _{12}^j\mu _{21}^{B+1-s^*-j}\right]\\&\quad + (\mu _{12}+\mu _{22})\mu _{22}^{s^*-1}\mu _{12}^{B+2-s^*}+(\mu _{11}+ \mu _{21})\mu _{21}^{B+2-s^*}\mu _{11}^{s^*-1}\\&> 0 \end{aligned}$$

and $\Upsilon _1(\alpha )$ is defined in the proof of Theorem 3.4. As mentioned in the proof of Theorem 3.4, if $\alpha \le 1$ and $\mu _{11}\mu _{22}\ge \mu _{12} \mu _{21}$, then $\Upsilon _1(\alpha )\ge 0$. On the other hand, if $\mu _{12}\mu _{21} > \mu _{11}\mu _{22}$, then $\mu _{12}>\mu _{22}$ and $\Upsilon _1(\alpha )\le 0$ if and only if $\alpha \ge \frac{2\mu _{11}\mu _{22}+\mu _{11}\mu _{12}+\mu _{21}\mu _{22}}{(\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})}\ge \frac{\mu _{11}\mu _{12}-\mu _{21}\mu _{22}}{(\mu _{11}+\mu _{21})(\mu _{12}-\mu _{22})}$. Then we know from part (ii) of Lemma 4.8 that $f_4(i,\alpha )< 0$ for all $i\in \{2,\ldots ,B+2\}$. But then $s^*=1$, which is a contradiction. Similarly, if $\mu _{11}\mu _{22}\ge \mu _{12}\mu _{21}$, we know from the proof of part (i) of Lemma 4.8 that $\Delta _1(\alpha )\ge 0$. On the other hand, if $\mu _{12}\mu _{21}> \mu _{11}\mu _{22}$, then $\Delta _1(\alpha ) \le 0$ if and only if $\alpha \ge \frac{\mu _{11}\mu _{12}-\mu _{21}\mu _{22}}{(\mu _{11}+\mu _{21})(\mu _{12}-\mu _{22})}$. Then we know from part (ii) of Lemma 4.8 that $f_4(i,\alpha )< 0$ for all $i\in \{2,\ldots ,B+2\}$ . But then $s^*=1$, which is a contradiction. Thus, $\epsilon ^{\prime }(s,a_{11})\ge 0$ with an equality only if $\mu _{11}\mu _{22}=\mu _{12} \mu _{21}$.

Similarly,

$$\begin{aligned} \epsilon ^{\prime }(s,a_{21})&= \frac{f_4(s^*,\alpha )}{\Upsilon ^{\prime }}+\\&\frac{(\mu _{11}\mu _{12}\!-\!\mu _{21}\mu _{22})\mu _{22}^{s-1} \sum _{j=0}^{s^*\!-\!s\!-\!2}\mu _{11}^j\mu _{22}^{s^*-s-2-j}\Big [\mu _{21} ^{B+2-s^*}\Upsilon _1(\alpha )\!+\!\sum _{j=0}^{B+1-s^*}\mu _{21}^j \mu _{12}^{B+1-s^*-j}\Delta _1(\alpha )\Big ]}{\Upsilon ^{\prime }}. \end{aligned}$$

One can immediately conclude that $\epsilon ^{\prime }(s,a_{21})\ge 0$ (with an equality only if $\mu _{11}\mu _{22}=\mu _{12}\mu _{21}$ and $f_4(s^*,\alpha )=0$) since $\Upsilon ^{\prime }>0, f_4(s^*,\alpha )\ge 0, \mu _{11}\ge \mu _{21}, \mu _{12}\ge \mu _{22}, \Upsilon _1(\alpha )\ge 0$, and $\Delta _1(\alpha )\ge 0$ as discussed above.

On the other hand,

$$\begin{aligned}&\epsilon ^{\prime }(s,a_{22})\\&=\alpha (\mu _{12}+\mu _{22})\Big [\frac{\mu _{22}^{s-1} (\mu _{12}^{B+2-s^*}\sum _{j=0}^{s^*-s-1}\mu _{11}^j \mu _{22}^{s^*-s-1-j}\Upsilon _1(\alpha )+\mu _{11}^{s^*-s} \sum _{j=0}^{B+1-s^*}\mu _{21}^j\mu _{12}^{B+1-s^*-j} \Upsilon _2(\alpha ))}{\Upsilon ^{\prime }}\\&\quad +\frac{\mu _{11}^{s^*-s}\sum _{j=0}^{s-2}\mu _{11}^j\mu _{22} ^{s-2-j}\sum _{k=0}^{B+1-s^*}\mu _{21}^k\mu _{12}^{B+1-s^*-k} (-\Delta _2(\alpha ))}{\Upsilon ^{\prime }}\Big ], \end{aligned}$$

where

$$\begin{aligned} \Upsilon _2(\alpha )&= -\alpha (\mu _{11}+\mu _{21})(\mu _{12}+ \mu _{22})+2\mu _{12}\mu _{21}+\mu _{11}\mu _{12}+\mu _{21}\mu _{22}. \end{aligned}$$

Note that $\Upsilon _1(\alpha )\ge 0$ as mentioned above. If $\mu _{12}\mu _{21} \ge \mu _{11}\mu _{22}$ and $\alpha \le 1$, then $\Upsilon _2(\alpha )\ge 0$. On the other hand, if $\mu _{11}\mu _{22} > \mu _{12} \mu _{21}$, then $\Upsilon _2(\alpha )\le 0$ if and only if $\alpha \ge \frac{2\mu _{12}\mu _{21}+\mu _{11}\mu _{12}+\mu _{21}\mu _{22}}{(\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})}\ge \frac{\mu _{11}\mu _{12}-\mu _{21}\mu _{22}}{(\mu _{11}-\mu _{21})(\mu _{12}+\mu _{22})}$. Then we know from part (i) of Lemma 4.8 that $f_4(i,\alpha )> 0$ for all $i\in \{2,\ldots ,B+2\}$. But then $s^*=B+2$, which implies that $\sum _{j=0}^{B+1-s^*}\mu _{21}^j\mu _{12}^{B+1-s^*-j}\Upsilon _2(\alpha )=0$. Finally, we know from the proof of Lemma 4.8 that if $\mu _{12} \mu _{21} \ge \mu _{11}\mu _{22}$, then $\Delta _2(\alpha )\le 0$. If $\mu _{11}\mu _{22}> \mu _{12} \mu _{21}$, then $\Delta _2(\alpha )\ge 0$ if and only if $\alpha \ge \frac{\mu _{11}\mu _{12}-\mu _{21}\mu _{22}}{(\mu _{11}-\mu _{21})(\mu _{12}+\mu _{22})}$ and part (i) of Lemma 4.8 implies that $f_4(i,\alpha )> 0$ for all $i\in \{2,\ldots ,B+2\}$. But then $s^*=B+2$, and, hence, $\sum _{j=0}^{B+1-s^*}\mu _{21}^j\mu _{12}^{B+1-s^*-j}(-\Delta _2(\alpha ))=0$. Thus, $\epsilon ^{\prime }(s,a_{22})\ge 0$ with an equality only if $\mu _{11}\mu _{22}=\mu _{12}\mu _{21}$.

Next we consider $s\in \{s^*,\ldots , B+1\}$, where we have that $d^{\prime }_0 (s) = a_{21}$. We will specify $\epsilon ^{\prime }(s,a)$ for actions $a_{11}, a_{12}$, and $a_{22}$. Without loss of generality, assume that $s^*<B+2$ because otherwise, this set of states is empty and there is nothing to prove. With some algebra we obtain

$$\begin{aligned} \epsilon ^{\prime }(s,a_{11})&= \alpha (\mu _{11}+\mu _{21})\Big [\frac{\sum _{j=0}^{s^*-2} \mu _{11}^j\mu _{22}^{s^*-2-j}\Delta _1(\alpha )(\mu _{12}^{s-s^*+1} \sum _{k=0}^{B-s}\mu _{21}^k\mu _{12}^{B-s-k}+\mu _{12}^{B+1-s^*})}{\Upsilon ^{\prime }}\\&+ \frac{\mu _{12}\mu _{21}^{B+1-s^*}\sum _{j=0}^{s^*-2} \mu _{11}^j\mu _{22}^{s^*-2-j}\Upsilon _1(\alpha )+\mu _{11}^{s^*-1} \mu _{21}^{B+1-s}\sum _{i=0}^{s-s^*}\mu _{21}^i\mu _{12}^{s-s^*-i} \Upsilon _2(\alpha )}{\Upsilon ^{\prime }}\Big ]. \end{aligned}$$

It follows from the proof of part (i) of Lemma 4.8 that if $\mu _{11}\mu _{22}\ge \mu _{12}\mu _{21}$, then $\Delta _1(\alpha )\ge 0$. On the other hand, if $\mu _{12}\mu _{21} > \mu _{11}\mu _{22}$, as discussed above $\sum _{j=0}^{s^*-2}\mu _{11}^j\mu _{22}^{s^*-2-j}\Delta _1(\alpha )=0$ because $s^*=1$. Similarly, if $\mu _{11}\mu _{22}\ge \mu _{12}\mu _{21}$, then $\Upsilon _1(\alpha )\ge 0$. If $\mu _{12}\mu _{21} > \mu _{11}\mu _{22}$ then $\sum _{j=0}^{s^*-2}\mu _{11}^j\mu _{22}^{s^*-2-j}\Upsilon _1(\alpha )=0$ because $s^*=1$. Finally, if $\mu _{12}\mu _{21} \ge \mu _{11}\mu _{22}$, then $\Upsilon _2(\alpha )\ge 0$. On the other hand, if $\mu _{12}\mu _{21} < \mu _{11}\mu _{22}$, then $s^*=B+2$, which is a contradiction. Thus, $\epsilon ^{\prime }(s,a_{11})\ge 0$ with an equality only if $\mu _{11}\mu _{22}=\mu _{12}\mu _{21}$.

Next we have

$$\begin{aligned} \epsilon ^{\prime }(s,a_{12})&= \frac{-f_4(s^*+1,\alpha )}{\Upsilon ^{\prime }}+\\&\frac{(\mu _{11}\mu _{12}-\mu _{21}\mu _{22})\mu _{21}^{B+1-s} \sum _{j=0}^{s-s^*-1}\mu _{21}^j\mu _{12}^{s-s^*-1-j}\big [\mu _{22}^ {s^*-1}\Upsilon _2(\alpha )+\sum _{j=0}^{s^*-2}\mu _{11}^j\mu _{22}^ {s^*-2-j}(-\Delta _2(\alpha ))\big ]}{\Upsilon ^{\prime }}. \end{aligned}$$

From the definition of $s^*, f_4(s^*+1,\alpha )\le 0$ and $\mu _{11}\mu _{12}-\mu _{21}\mu _{22}\ge 0$ from our assumptions on the service rates. If $\mu _{12}\mu _{21} \ge \mu _{11}\mu _{22}$ then $\Upsilon _2(\alpha )\ge 0$. On the other hand, as discussed above, if $\mu _{12}\mu _{21} < \mu _{11}\mu _{22}$, then $s^*=B+2$, which is a contradiction. Moreover, we know from the proof of Lemma 4.8 that if $\mu _{12}\mu _{21} \ge \mu _{11}\mu _{22}$, then $\Delta _2(\alpha )\le 0$. However, if $\mu _{11}\mu _{22} > \mu _{12}\mu _{21}$, then as discussed above, $s^*=B+2$, which is a contradiction. Thus, $\epsilon ^{\prime }(s,a_{12})\ge 0$ with an equality only if $\mu _{11}\mu _{22}=\mu _{21}\mu _{12}$ and $f_4(s^*+1,\alpha )= 0$.

Finally,

$$\begin{aligned} \epsilon ^{\prime }(s,a_{22})&= \frac{\alpha (\mu _{12}+\mu _{22})\sum _{j=0}^{B+1-s} \mu _{21}^j\mu _{12}^{B+1-s-j}\mu _{12}^{s-s^*}\Big [\sum _{j=0} ^{s^*-2}\mu _{11}^j\mu _{22}^{s^*-2-j}(-\Delta _2(\alpha ))+\mu _{22} ^{s^*-1}\Upsilon _2(\alpha )\Big ]}{\Upsilon ^{\prime }}. \end{aligned}$$

Using the arguments in the previous paragraph, one can immediately conclude that $\epsilon ^{\prime }(s,a_{22})\ge 0$ with an equality only if $\mu _{11}\mu _{22}=\mu _{12}\mu _{21}$. This proves that $d^{\prime }_1(s)=d^{\prime }_0(s)$ for all $s\in S$. By Theorem 9.5.1 of Puterman [18] (which says that in a (weakly) communicating model, policy iteration terminates with an optimal policy) this proves that the policy described in Theorem 4.4 is optimal. The proof of the uniqueness of the optimal policy is similar to the uniqueness proof of Theorem 3.4 (the lower bound on $\alpha$ needs to be strict to ensure that the policies in Theorems 4.2 and 4.3 are not optimal and that idling actions are not optimal in states $1,\ldots ,B+1$).

1.3 Proof of Proposition 4.1

We first show that when $\mu _{11}\mu _{22}>\mu _{12} \mu _{21}, s^*$ can take at most two adjacent integer values for each $0\le \alpha \le 1$. Note that our conditions on the service rates (i.e., $\mu _{11}\mu _{22}>\mu _{12}\mu _{21}, \mu _{11}\ge \mu _{21}$, and $\mu _{12}\ge \mu _{22}$) imply that $\mu _{11},\mu _{12},\mu _{22}>0$ and $\mu _{11}>\mu _{21}$. If $\mu _{21}=0, \mu _{12}=\mu _{22}$, and $\alpha \le \frac{\mu _{12}}{\mu _{12}+\mu _{22}}$, then Remark 4.2 implies that $s^*=B+2$. On the other hand, if $\alpha \le \frac{\mu _{12}}{\mu _{12}+\mu _{22}}$ and either $\mu _{21}>0$ or $\mu _{12}>\mu _{22}$, it follows from expressions (2), (3), and (4) of Işık, Andradóttir, and Ayhan [14] that $f_1(i)$ is strictly decreasing in $i\in S\setminus \{0\}$, implying that $s^*$ is either uniquely defined or can be chosen from two adjacent integers (depending on whether $f_1(i)=0$ for some $i\in S\setminus \{0\}$). Moreover, when $\frac{\mu _{12}}{\mu _{12}+\mu _{22}} \le \alpha \le \frac{\mu _{11}}{\mu _{11}+\mu _{21}}$ and $\beta _2(\alpha )\ge 0$, (2) is strictly positive in the proof of Lemma 4.6 (because the second summation is strictly positive since $\mu _{11}>\mu _{21}$) and when $\beta _2(\alpha )<0$, (3) is strictly negative (note that $\mu _{21}=0$ implies that $\beta _2(\alpha )\ge 0$). Thus, either $s^*=B+2$ or $f_2(i,\alpha )$ is strictly decreasing in $i\in S\setminus \{0\}$, and, hence, $s^*$ can take at most two adjacent integer values. Finally, we know from part (i) of Lemma 4.8 that $f_4(i,\alpha )>0$ when $\frac{\mu _{11}\mu _{12}-\mu _{21}\mu _{22}}{(\mu _{11}-\mu _{21})(\mu _{12}+\mu _{22})} \le \alpha \le 1$, and, hence, $s^*=B+2$. Otherwise, $\beta _2(\alpha )<0$, implying that the inequalities (8) and (9) (in the proof of part (v) of Lemma 4.8) are strict, and, thus, $f_4(i, \alpha )$ is strictly decreasing in $i\in S\setminus \{0\}$ and $s^*$ can take at most two adjacent integer values. In order to see this, note that (8) and (9) are strictly negative as long as $\mu _{21}> 0$, but if $\mu _{21}= 0$, then part (i) of Lemma 4.8 will apply.

If $\alpha \le \frac{\mu _{12}}{\mu _{12}+\mu _{22}}$, then $s^*$ is a constant function of $\alpha$ since $f_1(i)$ does not depend on $\alpha$ (as long as $s^*$ is chosen consistently when $f_1(i)=0$ for some $i$). Moreover, for all $i\in \{1,\ldots ,B+2\}$,

$$\begin{aligned} \frac{\text{ d}f_2(i,\alpha )}{\text{ d} \alpha }&= \mu _{21}^{B+2-i}\sum _{j=0}^{i-2}\mu _{22}^j\mu _{11}^ {i-1-j}(\mu _{12}+\mu _{22})(\mu _{11}-\mu _{21})\\&+\mu _{22}^{i-1}\sum _{j=0}^{B-i+2}\mu _{21}^j\mu _{12}^{B-i+2-j} (\mu _{12}+\mu _{22})(\mu _{11}-\mu _{21})\\&> 0. \end{aligned}$$

Thus, $f_2(i,\alpha )$ is strictly increasing in $\alpha$, which implies that $s^*$ is non-decreasing in $\alpha$ when $\mu _{11}\mu _{22}> \mu _{12}\mu _{21}$ and $\frac{\mu _{12}}{\mu _{12}+\mu _{22}}\le \alpha \le \frac{\mu _{11}}{\mu _{11}+\mu _{21}}$. Furthermore, with some algebra, we have

$$\begin{aligned} f_2\left(i,\frac{\mu _{12}}{\mu _{12}+\mu _{22}}\right)=f_1(i) \end{aligned}$$

for all $i\in \{1,\ldots ,B+2\}$. Thus, when $\alpha =\frac{\mu _{12}}{\mu _{12}+\mu _{22}}, f_1(\cdot )$ and $f_2(\cdot ,\cdot )$ yield the same optimal switch points.

Note that from the proof of Lemma 4.8, one can immediately see that

$$\begin{aligned} \frac{\text{ d}f_4(i,\alpha )}{\text{ d} \alpha }&= (\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})\left[\sum _{j=0}^{B+2-i}\mu _{21}^j\mu _{12}^{B+2-i-j} \mu _{22}^{i-2} (\mu _{22}-\mu _{12})\right.\\&+\left. \sum _{j=0}^{i-2}\mu _{22}^j \mu _{11}^{i-2-j} \mu _{21}^{B+2-i}(\mu _{11}-\mu _{21})\right] \end{aligned}$$

for all $i\in \{2,\ldots ,B+2\}$. Thus, for a fixed $i\in \{2,\ldots ,B+2\}, f_4(i,\alpha )$ could be either strictly increasing or non-increasing in $\alpha$. Let $0 \le \alpha \le 1$ and $i\le s^*$, so that $f_4(i,\alpha )\ge 0$. If $f_4(i,\alpha )$ is strictly increasing, then $i\le s^*$ for all $\alpha ^{\prime } \in [\alpha ,1]$. On the other hand, if $f_4(i,\alpha )$ is non-increasing in $\alpha$, one can conclude that $f_4(i,\alpha ^{\prime })> 0$, and, hence, $i\le s^*$, for all $\alpha \le \alpha ^{\prime } \le \frac{\mu _{11}\mu _{12}-\mu _{21}\mu _{22}}{(\mu _{11}-\mu _{21})(\mu _{12}+\mu _{22})}$, because otherwise part (i) of Lemma 4.8 would contradict the assumption that $f_4(i,\alpha )$ is non-increasing in $\alpha$. Finally, when $\alpha \in [\frac{\mu _{11}\mu _{12}-\mu _{21}\mu _{22}}{(\mu _{11}-\mu _{21})(\mu _{12}+\mu _{22})},1]$, part (i) of Lemma 4.8 implies that $s^*=B+2\ge i$. Since $\alpha$ and $i\le s^*$ are arbitrary, we have shown that $s^*$ is non-decreasing in $\alpha$ for $\frac{\mu _{11}}{\mu _{11}+\mu _{21}} \le \alpha \le 1$. Moreover, with some algebra, we have

$$\begin{aligned} f_2\left(i,\frac{\mu _{11}}{\mu _{11}+\mu _{21}}\right)=\frac{\mu _{11}}{\mu _{11}+\mu _{21}}f_4\left(i,\frac{\mu _{11}}{\mu _{11}+\mu _{21}}\right) \end{aligned}$$

for all $i\ge 0$, which implies that when $\alpha =\frac{\mu _{11}}{\mu _{11}+\mu _{21}}, f_2(\cdot ,\cdot )$ and $f_4(\cdot ,\cdot )$ yield the same optimal switch points. In all cases, the continuity of the functions $f_1(\cdot ), f_2(\cdot ,\cdot )$, and $f_4(\cdot ,\cdot )$ in $\alpha$ and the fact that $f_1(\cdot ), f_2(\cdot ,\cdot )$, and $f_4(\cdot ,\cdot )$ are strictly decreasing in $i$ whenever $s^*<B+2$ imply that jumps in $s^*$ are of size one. Thus, the proof is complete. $\square$