Skip to main content
Log in

Profit maximization in flexible serial queueing networks

  • Published:
Queueing Systems Aims and scope Submit manuscript

Abstract

We analyze the tradeoff between efficiency and service quality in tandem systems with flexible servers and finite buffers. We reward efficiency by assuming that a revenue is earned each time a job is completed, and penalize poor service quality by incorporating positive holding costs. We study the dynamic assignment of servers to tasks with the objective of maximizing the long-run average profit. For systems of arbitrary size, structured service rates, and linear or nonlinear holding costs, we determine the server assignment policy that maximizes the profit. For systems with two stations, two servers with arbitrary service rates, and linear holding costs, we show that the optimal server assignment policy is of threshold type and determine the value of this threshold as a function of the revenue and holding cost. The threshold can be interpreted as the best possible buffer size, and hence our results prove the equivalence of addressing service quality via a holding cost and via limiting the buffer size. Furthermore, we identify the optimal buffer size when each buffer space comes at a cost. We provide numerical results that suggest that the optimal policy also has a threshold structure for nonlinear holding costs. Finally, for larger systems with arbitrary service rates, we propose effective server assignment heuristics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Ahn, H.-S., Duenyas, I., Lewis, M.E.: The optimal control of a two-stage tandem queueing system with flexible servers. Probab. Eng. Inf. Sci. 16, 453–469 (2002)

    Article  Google Scholar 

  2. Ahn, H.-S., Duenyas, I., Zhang, R.: Optimal scheduling of a 2-stage tandem queue with parallel servers. Adv. Appl. Probab. 31, 1095–1117 (1999)

    Article  Google Scholar 

  3. Ahn, H.-S., Duenyas, I., Zhang, R.: Optimal control of a flexible server. Adv. Appl. Probab. 36, 139–170 (2004)

    Article  Google Scholar 

  4. Ahn, H.-S., Righter, R.: Dynamic load balancing with flexible workers. Adv. Appl. Probab. 38, 621–642 (2006)

    Article  Google Scholar 

  5. Andradóttir, S., Ayhan, H.: Throughput maximization for tandem lines with two stations and flexible servers. Oper. Res. 53, 516–531 (2005)

    Article  Google Scholar 

  6. Andradóttir, S., Ayhan, H., Down, D.G.: Server assignment policies for maximizing the steady-state throughput of finite queueing systems. Manag. Sci. 47, 1421–1439 (2001)

    Article  Google Scholar 

  7. Andradóttir, S., Ayhan, H., Down, D.G.: Dynamic server allocation for queueing networks with flexible servers. Oper. Res. 51, 952–968 (2003)

    Article  Google Scholar 

  8. Andradóttir, S., Ayhan, H., Down, D.G.: Dynamic assignment of dedicated and flexible servers in tandem lines. Probab. Eng. Inf. Sci. 21, 497–538 (2007)

    Article  Google Scholar 

  9. Andradóttir, S., Ayhan, H., Kırkızlar, E.: Flexible servers in tandem lines with setup costs. Queueing Syst. 70, 165–186 (2012)

    Article  Google Scholar 

  10. Bartholdi III, J.J., Bunimovich, L.A., Eisenstein, D.D.: Dynamics of two- and three-worker “bucket brigade” production lines. Oper. Res. 47, 488–491 (1999)

    Article  Google Scholar 

  11. Bartholdi III, J.J., Eisenstein, D.D.: A production line that balances itself. Oper. Res. 44, 21–34 (1996)

    Article  Google Scholar 

  12. Bartholdi III, J.J., Eisenstein, D.D., Foley, R.D.: Performance of bucket brigades when work is stochastic. Oper. Res. 49, 710–719 (2001)

    Article  Google Scholar 

  13. Bell, S.L., Williams, R.J.: Dynamic scheduling of a system with two parallel servers in heavy traffic with complete resource pooling: asymptotic optimality of a threshold policy. Ann. Appl. Probab. 11, 608–649 (2001)

    Article  Google Scholar 

  14. Bell, S.L., Williams, R.J.: Dynamic scheduling of a parallel server system in heavy traffic with complete resource pooling: asymptotic optimality of a threshold policy. Electron. J. Probab. 10, 1044–1115 (2005)

    Article  Google Scholar 

  15. Bhulai, J., Koole, G.: A queueing model for call blending in call centers. IEEE Trans. Autom. Control 48, 1421–1426 (2003)

    Article  Google Scholar 

  16. Billingsley, P.: Probability and Measure. Wiley, New York (1995)

    Google Scholar 

  17. Farrar, T.M.: Optimal use of an extra server in a two station tandem queueing network. IEEE Trans. Autom. Control 38, 1296–1299 (1993)

    Article  Google Scholar 

  18. Gans, N., Zhou, Y.-P.: A call routing problem with service level constraints. Oper. Res. 51, 255–271 (2003)

    Article  Google Scholar 

  19. Gel, E.S., Hopp, W.J., Van Oyen, M.P.: Factors affecting opportunity of worksharing as a dynamic line balancing mechanism. IIE Trans. 34, 847–863 (2002)

    Google Scholar 

  20. Harrison, J.M., López, M.J.: Heavy traffic resource pooling in parallel-server systems. Queueing Syst. 33, 339–368 (1999)

    Article  Google Scholar 

  21. Hopp, W.J., Tekin, E., Van Oyen, M.P.: Benefits of skill chaining in serial production lines with cross-trained workers. Manag. Sci. 50, 83–98 (2004)

    Article  Google Scholar 

  22. Iravani, S.M.R., Posner, M.J.M., Buzacott, J.M.: A two-stage tandem queue attended by a moving server with holding and switching costs. Queueing Syst. 26, 203–228 (1997)

    Article  Google Scholar 

  23. Kaufman, D.L., Ahn, H.-S., Lewis, M.E.: On the introduction of an agile, temporary workforce into a tandem queueing system. Queueing Syst. 51, 135–171 (2005)

    Article  Google Scholar 

  24. Kırkızlar, E., Andradóttir, S., Ayhan, H.: Robustness of efficient server assignment policies to service time distributions in finite-buffered lines. Nav. Res. Logist. 57, 563–582 (2010)

    Article  Google Scholar 

  25. Kırkızlar, E., Andradóttir, S., Ayhan, H.: Flexible servers in understaffed tandem lines. Prod. Oper. Manag. 21, 761–777 (2012)

    Article  Google Scholar 

  26. Kulkarni, V.G.: Modeling and Analysis of Stochastic Systems. Chapman & Hall, London (1995)

    Google Scholar 

  27. Lim, Y.F., Yang, K.K.: Maximizing throughput of bucket brigades on discrete work stations. Prod. Oper. Manag. 18, 48–59 (2009)

    Article  Google Scholar 

  28. Lippman, S.A.: Applying a new device in the optimization of exponential queueing system. Oper. Res. 23, 687–710 (1975)

    Article  Google Scholar 

  29. Mandelbaum, A., Stolyar, A.L.: Scheduling flexible servers with convex delay costs: heavy-traffic optimality of the generalized \(c \mu \)-rule. Oper. Res. 52, 836–855 (2004)

    Article  Google Scholar 

  30. Mayorga, M.E., Taaffe, K.M., Arumugam, R.: Allocating flexible servers in serial systems with switching costs. Ann. Oper. Res. 172, 231–242 (2009)

    Article  Google Scholar 

  31. McClain, J.O., Thomas, L.J., Sox, C.: On-the-fly line balancing with very little WIP. Int. J. Prod. Econ. 27, 283–289 (1992)

    Article  Google Scholar 

  32. Pandelis, D.G.: Optimal use of excess capacity in two interconnected queues. Math. Methods Oper. Res. 65, 179–192 (2007)

    Article  Google Scholar 

  33. Puterman, M.L.: Markov Decision Processes. Wiley, New York (1994)

    Book  Google Scholar 

  34. Rosberg, Z., Varaiya, P.P., Walrand, J.C.: Optimal control of service in tandem queues. IEEE Trans. Autom. Control 27, 600–609 (1982)

    Article  Google Scholar 

  35. Saghafian, S., Van Oyen, M.P., Kolfal, B.: The “W” network and the dynamic control of unreliable flexible servers. IIE Trans. 43, 893–907 (2011)

    Article  Google Scholar 

  36. Schiefermayr, K., Weichbold, J.: A complete solution for the optimal stochastic scheduling of a two-stage tandem queue with two flexible servers. J. Appl. Probab. 42, 778–796 (2005)

    Article  Google Scholar 

  37. Sennott, L.I., Van Oyen, M.P., Iravani, S.M.R.: Optimal dynamic assignment of a flexible server on an open production line with specialists. Eur. J. Oper. Res. 170, 541–566 (2006)

    Article  Google Scholar 

  38. Tassiulas, L., Bhattacharya, P.P.: Allocation of interdependent resources for maximal throughput. Stoch. Models 16, 27–48 (2000)

    Article  Google Scholar 

  39. Tassiulas, L., Ephrimedes, A.: Stability properties of constrained queueing systems and scheduling properties for maximum throughput in multihop radio networks. IEEE Trans. Autom. Control 37, 1936–1948 (1992)

    Article  Google Scholar 

  40. Van Oyen, M.P., Gel, E.G.S., Hopp, W.J.: Performance opportunity for workforce agility in collaborative and noncollabortive work systems. IIE Trans. 33, 761–777 (2001)

    Article  Google Scholar 

  41. Wolff, R.W.: Stochastic Modeling and the Theory of Queues. Prentice Hall, Englewood Cliffs (1989)

    Google Scholar 

  42. Wu, C.-H., Down, D.G., Lewis, M.E.: Heuristics for allocation of reconfigurable resources in a serial line with reliability considerations. IIE Trans. 40, 595–611 (2008)

    Article  Google Scholar 

  43. Wu, C.-H., Lewis, M.E., Veatch, M.: Dynamic allocation of reconfigurable resources in a two-stage tandem queueing system with reliability considerations. IEEE Trans. Autom. Control 51, 309–314 (2006)

    Article  Google Scholar 

  44. Zavadlav, E., McClain, J.O., Thomas, L.J.: Self-buffering, self-balancing, self-flushing production lines. Manag. Sci. 42, 1151–1164 (1996)

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the National Science Foundation under Grant CMMI–0856600. The third author was also supported by the National Science Foundation under grant CMMI–0969747. The authors would like to thank the Associate Editor and two anonymous referees whose comments led to significant improvements to the paper.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Eser Kırkızlar.

Appendix

Appendix

Details of the problem formulation (5):

The discrete time Markov chain \(\{ X_\pi ' (k) \}\) has state space \({\mathcal {S}}\) and transition probabilities \(p_d (x, x') = q_d (x, x') / q\) if \(x' \not = x\) and \(p_d (x, x) = 1 - \sum _{x' \in {\mathcal {S}}, x' \not = x} q_d (x, x') / q\) for all \(x \in {\mathcal {S}}\). We will generate a Poisson process \(\{ K (t) \}\) with rate \(q\) and the next state of the continuous time Markov chain \(\{ X_\pi (t) \}\) is generated (independently from \(\{ K (t) \}\)) using the transition probabilities of the discrete time Markov chain \(\{ X_\pi ' (k) \}\) at the times of the events of \(\{ K (t) \}\).

For all \(x, x' \in {\mathcal {S}}\) and \(\pi _d \in \Pi \), let

$$\begin{aligned} R_d '\left( x, x'\right) = \left\{ \begin{array}{ll} 1 - \frac{\sum _{j=2}^N h_j x_j}{q} &{} \quad \text{ if }\,x' \in {\mathcal {D}}_x,\\ -\, \frac{\sum _{j=2}^N h_j x_j}{q} &{} \quad \text{ otherwise }, \end{array} \right. \end{aligned}$$

where \({\mathcal {D}} = \{ x \in {\mathcal {S}} : x_{N-1} >0 \}\), \({\mathcal {D}}_x = \{ (x_1, \ldots , x_{N-2}, x_{N-1}-1) \}\) for all \(x \in {\mathcal {D}}\), and \({\mathcal {D}}_x= \emptyset \) for all \(x \not \in {\mathcal {D}}\) (note that the set \({\mathcal {D}}_x\) has at most one state \(x'\)). As \(\sum _{j=2}^N h_j x_j\) is the holding cost per unit time in state \(x\) and \(\frac{1}{q}\) is the expected time between transitions, an expected holding cost of \(\frac{\sum _{j=2}^N h_j x_j}{q}\) is incurred for each transition into state \(x\). Moreover, a revenue of \(r=1\) is gained if there is a departure from the system. Hence, \(R_d'(x,x')\) equals the expected immediate reward associated with each transition from state \(x\) to state \(x'\). Consequently, for all \(\pi _d \in \Pi \),

$$\begin{aligned} P_{\pi ,{\mathcal {B}}}' =\lim _{t \rightarrow \infty } \mathrm{I}\!\mathrm{E}\left\{ \frac{K (t)}{t} \times \frac{1}{K (t)} \sum _{k=1}^{K (t)} R_d' \left( X_\pi ' (k-1), X_\pi ' (k)\right) \right\} . \end{aligned}$$
(8)

It is clear that \(K (t) / t \rightarrow q\) almost surely as \(t \rightarrow \infty \) by the elementary renewal theorem. Moreover, it is clear that for all \(\pi _d \in \Pi \), the limit \(\lim _{K \rightarrow \infty } \sum _{k=1}^K R_d' \left( X_\pi ' (k-1), X_\pi ' (k)\right) / K\) exists almost surely by the strong law of large numbers for Markov chains (see, for example, [41], p. 164; and Sect. 3.8 of [26]). Uniform integrability shows that for all \(\pi _d \in \Pi \), we have

$$\begin{aligned} P_{\pi ,{\mathcal {B}}}' = q \lim _{K \rightarrow \infty } \mathrm{I}\!\mathrm{E}\left\{ \frac{1}{K} \sum _{k=1}^K R_d' \left( X_\pi ' (k-1), X_\pi ' (k)\right) \right\} \end{aligned}$$

(see for example the corollary to Theorem 25.12 in [16]) since

$$\begin{aligned} \left| \frac{1}{K} \sum _{k=1}^K R_d' \left( X_\pi ' (k-1), X_\pi ' (k)\right) \right| \le 1 +\frac{ N}{q} \left( \max _{2 \le k\le N} | B_k|+2\right) \left( \max _{2 \le k\le N} | h_k|\right) < \infty \end{aligned}$$

for all \(K \ge 1\) and \(\sup _{t \ge 0} \mathrm{I}\!\mathrm{E}\{ [K (t) / t]^2 \} < \infty \) (because \(K (t)\) is a Poisson random variable with mean \(q t\)). Hence, the optimization problem (4) has the same solution as

$$\begin{aligned} \max _{\pi _d \in \Pi } \lim _{K \rightarrow \infty } \mathrm{I}\!\mathrm{E}\left\{ \frac{1}{K} \sum _{k=1}^K R_d' \left( X_\pi ' (k-1), X_\pi ' (k)\right) \right\} . \end{aligned}$$

Using the strong law of large numbers for Markov chains, we obtain for all \(\pi _d \in \Pi \) that

$$\begin{aligned} \quad \lim _{K \rightarrow \infty } \frac{1}{K} \sum _{k=1}^K R_d' \left( X_\pi ' (k-1), X_\pi ' (k)\right) = \lim _{K \rightarrow \infty } \frac{1}{K} \sum _{k=1}^K R_d'' \left( X_\pi ' (k-1)\right) \quad \quad \mathrm{a.s.}, \end{aligned}$$

where

$$\begin{aligned} R_d'' (x) = \sum _{x' \in {\mathcal {D}}_x} \frac{q_d \left( x, x'\right) }{q} - \frac{\sum _{j=2}^N h_jx_j}{q} \end{aligned}$$

for all \(x \in {\mathcal {S}}\). Uniform integrability now gives that

$$\begin{aligned} \lim _{K \rightarrow \infty } \mathrm{I}\!\mathrm{E}\left\{ \frac{1}{K} \sum _{k=1}^K R_d' \left( X_\pi ' (k-1), X_\pi ' (k)\right) \right\} = \lim _{K \rightarrow \infty } \mathrm{I}\!\mathrm{E}\left\{ \frac{1}{K} \sum _{k=1}^K R_d'' \left( X_\pi ' (k-1)\right) \right\} . \end{aligned}$$

Hence, the optimization problem (4) has the same solution as

$$\begin{aligned} \max _{\pi _d \in \Pi } \lim _{K \rightarrow \infty } \mathrm{I}\!\mathrm{E}\left\{ \frac{1}{K} \sum _{k=1}^K R_d'' \left( X_\pi ' (k-1)\right) \right\} . \end{aligned}$$

As \(R_d(x)=qR_d''(x)\) for all \(x \in {\mathcal {S}}\) and \(\pi _d \in \Pi \), it follows that the optimization problem (4) has the same solution as the Markov decision problem (5).

Proof of Lemma 3.1

If all the servers are idle in a state \(x\), then \(x\) is an absorbing state and the long-run average profit is at most zero. Since \(\sum _{j=2}^N\frac{h_j}{\Sigma _j}<r\), the expedite policy \(\pi ^e\) is profitable, and hence at least one of the servers should not be idle in state \(x\). Let \(\pi \) be any policy that idles one server when the other server is assigned to a station \(j\) in state \(x\). Now compare \(\pi \) with the policy \(\pi '\) that assigns both servers to station \(j\) in state \(x\) and agrees with \(\pi \) otherwise. The next state visited, \(x'\), will be the same under both policies, and the transition time from state \(x\) to state \(x'\) is never longer for \(\pi '\) compared to \(\pi \). Hence, the number of departures is never smaller under \(\pi '\). Moreover, the holding cost per item produced is never bigger under \(\pi '\), because both policies go through the same sequence of states and the time between transitions is never longer under \(\pi '\). Hence, \(\frac{D_{\pi ,{\mathcal {B}}} (t)}{t} \ge \frac{D_{\pi ',{\mathcal {B}}} (t)}{t}\) and \(C_{\pi ,{\mathcal {B}}}(t) \le C_{\pi ',{\mathcal {B}}}(t)\) for all \(t \ge 0\). Equation (6) shows that \(P_{\pi ,{\mathcal {B}}}' \ge P_{\pi ',{\mathcal {B}}}'\), and consequently there exists an optimal policy that does not idle any servers in state \(x\). \(\square \)

Proof of Lemma 3.2

Let \(i,j \in \mathrm{I}\!\mathrm{N}^+\) and \(i<j\). We have \(f_1(i,j) \ge 0\) trivially because of our assumption that \(\mu _{11}\mu _{22} \ge \mu _{12}\mu _{21}\). Hence, we only need to show that \(f_2(i,j)>0\). Some algebra shows that when \(\mu _{11} \ne \mu _{22}\),

$$\begin{aligned} f_2(i,j)&= \frac{\mu _{22}^{j+i-4}}{\mu _{11}^{i-1}}(\mu _{22}-\mu _{11})^2 \left[ \mu _{12}\mu _{22}^2\left( \sum _{k=1}^{j-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}+\sum _{k=1}^{j-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k}\right) \right. \\&+\,\mu _{12}\mu _{22}(\mu _{12}+\mu _{21})\sum _{k=1}^{j-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\\&\quad +\,(\mu _{11}\mu _{22}-\mu _{12}\mu _{21})\mu _{22}\sum _{k=i}^{j-1}\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\\&\quad \left. +\,(\mu _{11}+\mu _{12})(\mu _{11}+\mu _{21})\left( \mu _{12}\sum _{k=i}^{j-1}\sum _{l=1}^{i-1}(k-l)(\frac{\mu _{11}}{\mu _{22}})^{k+l-2}\right. \right. \nonumber \\&\quad \left. \left. +\,\mu _{22}\sum _{k=i+1}^{j}\sum _{l=1}^{i}(k-l)\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k+l-3} \right) \right] . \end{aligned}$$

Hence, it is clear that \(f_2(i,j)>0\) because \(\mu _{11} \ne \mu _{22}\) and \(\mu _{11}\mu _{22} \ge \mu _{12}\mu _{21}\). Finally, when \(\mu _{11}=\mu _{22}\), it is clear that \(f_2(i,j)>0\) because \(i<j\) and \(\mu _{22}>0\). Hence the proof is complete. \(\square \)

Proof of Lemma 3.3

Note that when \(\mu _{11}\mu _{22}=\mu _{12}\mu _{21}\), we have \(\frac{f_1(i,i+1)}{f_2(i,i+1)}=0\) for all \(i \in \mathrm{I}\!\mathrm{N}^+\) and the result follows trivially. Hence, in the rest of the proof we assume that \(\mu _{11}\mu _{22}>\mu _{12}\mu _{21}\).

When \(\mu _{11} \ne \mu _{22}\), some algebra shows that

$$\begin{aligned}&\frac{f_1(i,i+1)}{f_2(i,i+1)} >\frac{f_1(i+1,i+2)}{f_2(i+ 1,i+2)} \\&\quad \Leftrightarrow \mu _{11}\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{i-1}(\mu _{11}+\mu _{12})(\mu _{22}-\mu _{11}) \\&\qquad \times \left( \mu _{12}\mu _{21}\left( \mu _{22}^i-\mu _{11}^i\right) +(\mu _{12}+\mu _{21})\left( \mu _{22}^{i+1} -\mu _{11}^{i+1}\right) + \left( \mu _{22}^{i+2}-\mu _{11}^{i+2}\right) \right) > 0. \end{aligned}$$

The last inequality is correct because \(\mu _{11}\ne \mu _{22}\), hence the lemma holds. When \(\mu _{11}=\mu _{22}\), we can show that the lemma holds because

$$\begin{aligned}&\frac{f_1(i,i+1)}{f_2(i,i+1)} > \frac{f_1(i+1,i+2)}{f_2(i+1,i+2)} \\&\quad \Leftrightarrow (\mu _{12} +\mu _{22})\left[ \left( \mu _{22}^2 -\mu _{12}\mu _{21}\right) +(i+1)\left( \mu _{12}\mu _{21} +(\mu _{12}+\mu _{21})\mu _{22}+\mu _{22}^{2}\right) \right] > 0. \end{aligned}$$

Hence the proof is complete. \(\square \)

Proof of Lemma 3.4

First note that when \(\mu _{11}\mu _{22}=\mu _{12}\mu _{21}\), we have \(\frac{f_1(i,j)}{f_2(i,j)}=0\) for all \(i,j \in \mathrm{I}\!\mathrm{N}^+\) and \(i<j\) and the result follows trivially. Hence, in the rest of the proof we assume that \(\mu _{11}\mu _{22}>\mu _{12}\mu _{21}\).

It is sufficient to show that for all \(i,j \in \mathrm{I}\!\mathrm{N}^+\) with \(i<j-1\)

$$\begin{aligned} \frac{f_1(i,j)}{f_2(i,j)} > \frac{f_1(i+1,j)}{f_2(i+1,j)}. \end{aligned}$$
(9)

First assume that \(\mu _{11} \ne \mu _{22}\). Then, some algebra shows that inequality (9) holds if and only if

$$\begin{aligned}&\mu _{22}^{j-i-1}(\mu _{22}-\mu _{11})\left( \mu _{22}^{j+1}-\mu _{11}^{j+1}+\left( \mu _{22}^{j} -\mu _{11}^{j}\right) (\mu _{12}+\mu _{21}) +\left( \mu _{22}^{j-1}- \mu _{11}^{j-1}\right) \mu _{12}\mu _{21}\right) \nonumber \\&\qquad \times \left( \mu _{12}\sum _{k=i+1}^{j-1}(k-i)\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1} +\mu _{22} \sum _{k=i+2}^{j}(k-i-1)\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1} \right) > 0. \end{aligned}$$
(10)

When \(\mu _{11}=\mu _{22}\), we observe that inequality (9) holds if and only if

$$\begin{aligned}&(\mu _{12}+\mu _{22})\frac{(j-i)(j-i-1)}{2}\nonumber \\&\quad \times \left[ \left( \mu _{22}^2-\mu _{12}\mu _{21}\right) + j\left( \mu _{12}\mu _{21}+(\mu _{12}+\mu _{21})\mu _{22}+\mu _{22}^{2}\right) \right] > 0. \end{aligned}$$
(11)

Inequalities (10) and (11) hold because \(j>i\) and \(\mu _{22}>0\). Hence the proof is complete. \(\square \)

Proof of Lemma 3.5

First note that when \(\mu _{11}\mu _{22}=\mu _{12}\mu _{21}\), we have \(\frac{f_1(i,i+1)}{f_2(i,i+1)}=0\) for all \(i \in \mathrm{I}\!\mathrm{N}^+\) and the result follows trivially. Hence, in the rest of the proof we assume that \(\mu _{11}\mu _{22}>\mu _{12}\mu _{21}\).

When \(\mu _{11} \ne \mu _{22}\), using the \(f_2\) expression provided in the proof of Lemma 3.2 we obtain

$$\begin{aligned} \frac{f_1(i,i+1)}{f_2(i,i+1)}&= \left( (\mu _{12}+\mu _{22})(\mu _{11}\mu _{22}-\mu _{12}\mu _{21})\right) \Big / \\&\quad \left[ \mu _{12}\mu _{22}\left( \sum _{k=1}^{i}k\left( \frac{\mu _{22}}{\mu _{11}}\right) ^{i-k}+\sum _{k=1}^{i}k\left( \frac{\mu _{22}}{\mu _{11}}\right) ^{i-k-1}\right) \right. \\&\quad \left. +\,\mu _{12}(\mu _{12}+\mu _{21})\sum _{k=1}^{i}k\left( \frac{\mu _{22}}{\mu _{11}}\right) ^{i-k} +(\mu _{11}\mu _{22}-\mu _{12}\mu _{21})\right. \\&\quad \left. +\,(\mu _{11}+\mu _{12})(\mu _{11}+\mu _{21})\left( \frac{\mu _{12}}{\mu _{22}}\sum _{l=1}^{i-1}(i-l)\left( \frac{\mu _{22}}{\mu _{11}}\right) ^{i-l+1}\right. \right. \nonumber \\&\quad \left. \left. +\,\sum _{l=1}^{i}(i+1-l)\left( \frac{\mu _{22}}{\mu _{11}}\right) ^{i-l+1}\right) \right] . \end{aligned}$$

Note that the numerator of this expression is positive, and that all the terms in the denominator are nonnegative and the last two terms are positive. Moreover, \(\sum _{k=1}^{i}k(\frac{\mu _{22}}{\mu _{11}})^{i-k} \ge i\), because the last term in the summation equals \(i\). Hence, it is clear that the denominator approaches infinity as \(i \rightarrow \infty \), and it follows that \(\frac{f_1(i,i+1)}{f_2(i,i+1)} \rightarrow 0\) as \(i \rightarrow \infty \). Finally, when \(\mu _{11}=\mu _{22}\), we observe that

$$\begin{aligned}&\frac{f_1(i,i+1)}{f_2(i,i+1)}\\&\quad =\frac{\mu _{22}(\mu _{12}+\mu _{22})\left( \mu _{22}^2-\mu _{12}\mu _{21}\right) }{\left( \mu _{12}i+\mu _{22}(i+1)\right) \left( \mu _{22}^2-\mu _{12}\mu _{21}\right) +\frac{i(i+1)}{2}(\mu _{12}+\mu _{22}) \left( \mu _{12}\mu _{21}+(\mu _{12}+\mu _{21})\mu _{22}+\mu _{22}^{2}\right) }. \end{aligned}$$

It is clear that the numerator and denominator of this expression are positive and that the denominator approaches infinity as \(i \rightarrow \infty \). Hence the proof is complete. \(\square \)

Proof of Theorem 3.2

Lemma 3.1 shows that it suffices to consider the policies that are non-idling. Then, in each state \(i \in {\mathcal {S}}\), it is sufficient to consider the following set of allowable actions:

$$\begin{aligned} A_i = \left\{ \begin{array}{l@{\quad }l} \{ a_{11} \}&{} \text{ for }\,i = 0, \\ \{a_{11},a_{12},a_{21},a_{22}\} &{} \text{ for }\,i \in \{1,\ldots ,B+1\}, \\ \{ a_{22} \}&{} \text{ for }\,i = B+2. \\ \end{array} \right. \end{aligned}$$

The policy described in the theorem corresponds to a Markov chain with a single recurrent class and possibly some transient states. Thus, we can use the policy iteration algorithm for weakly communicating models as described in Sect. 9.5.1 of [33].

Let \(P_d\) be the probability transition matrix corresponding to the policy \(\pi _d\), and \(r_d(i)\) denote the reward in state \(i\) when policy \(\pi _d\) is employed. It follows from Lemmas 3.2, 3.3, and 3.5 that \(s_1\) is well defined. Let us define \(\frac{f_1(0,1)}{f_2(0,1)}=\infty \), so that \(\frac{f_1(s_1,s_1+1)}{f_2(s_1,s_1+1)} < h \le \frac{f_1(s_1-1,s_1)}{f_2(s_1-1,s_1)}\). We start the policy iteration algorithm by considering the policy \(\pi _0=\pi _{d_0}\), where

$$\begin{aligned} d_0(i) = \left\{ \begin{array}{l@{\quad }l} a_{11} &{} \text{ for }\,i=0,\\ a_{12} &{} \text{ for }\,1 \le i \le s_B^*-1, \\ a_{22} &{} \text{ for }\,s_B^* \le i \le B+2. \end{array} \right. \end{aligned}$$

Then we obtain

$$\begin{aligned} r_{d_0}(i) = \left\{ \begin{array}{l@{\quad }l} 0 &{} \text{ for }\,i=0,\\ \mu _{22}-ih &{} \text{ for }\,1 \le i \le s_B^*-1, \\ \mu _{12}+\mu _{22}-ih&{} \text{ for }\,s_B^* \le i \le B+2, \end{array} \right. \end{aligned}$$

and

$$\begin{aligned} P_{d_{0}}(i,i') = \left\{ \begin{array}{ll} \frac{\mu _{11}+\mu _{21}}{q} &{} \quad \text{ for }\,i = 0\,\mathrm{and}\,i'=1,\\ \frac{\mu _{12}+\mu _{22}}{q} &{} \quad \text{ for }\,i = i' = 0, \\ \frac{\mu _{11}}{q} &{} \quad \text{ for }\,i \in \{1,\ldots ,s_B^*-1\}\,\mathrm{and}\,i'=i+1, \\ \frac{\mu _{22}}{q} &{} \quad \text{ for }\,i \in \{1,\ldots ,s_B^*-1\}\,\mathrm{and}\,i'=i-1, \\ \frac{\mu _{12}+\mu _{21}}{q} &{} \quad \text{ for }\,i=i' \in \{1,\ldots ,s_B^*-1\}, \\ \frac{\mu _{12}+\mu _{22}}{q} &{} \quad \text{ for }\,i \in \{s_B^*,\ldots ,B+2\}\,\mathrm{and}\,i'=i-1, \\ \frac{\mu _{11}+\mu _{21}}{q} &{} \quad \text{ for }\,i = i' \in \{s_B^*,\ldots ,B+2\}. \\ \end{array} \right. \end{aligned}$$

For all \(i,i' \in {\mathcal {S}}\) and \(a \in A_i\), we use \(r(i,a)\) to denote the immediate reward in state \(i\) when action \(a\) is taken and \(p(i' | i,a)\) to denote the one-step probability of going from state \(i\) to state \(i'\) when action \(a\) is chosen in state \(i\). Under our assumptions on the service rates (\(\mu _{11}\mu _{22} \ge \mu _{12}\mu _{21}\), \(\sum _{k=1}^M\mu _{kj}>0\) for \(j \in \{1,\ldots ,N\}\), and \(\sum _{j=1}^N\mu _{kj}>0\) for \(k \in \{1,\ldots ,M\}\)), it is clear that \(\mu _{11}>0\) and \(\mu _{22}>0\). Hence, the discrete time Markov chain \(\{ X_{\pi _0}' (t) \}\) found by uniformization of \(\{ X_{\pi _0} (t) \}\) is unichain, and we can solve the following set of equations to find a scalar \(g_0\) and a vector \(h_0\), letting \(h_0(0)=0\),

$$\begin{aligned} r_{d_{0}}-g_0e+(P_{d_{0}}-I)h_0=0, \end{aligned}$$
(12)

where \(e\) is the column vector of ones and \(I\) is the identity matrix. We can show that

$$\begin{aligned} g_0 = \left\{ \begin{array}{l@{\quad }l} (\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})\left( \left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}\right) \right. \\ \quad \left. -\frac{h}{(\mu _{22}-\mu _{11})(\mu _{12}+\mu _{22})} \left[ (\mu _{12}+\mu _{22}) \left( \mu _{22}^{s_B^*}-s_B^*\mu _{11}^{s_B^*-1}\mu _{22}\right. \right. \right. \\ \quad \times \left. \left. \left. +\,(s_B^*-1)\mu _{11}^{s_B^*}\right) +s_B^*\mu _{11}^{s_B^*-1}(\mu _{22}-\mu _{11})^2 \right] \right) \\ \quad \Big / \left( \mu _{22}^{s_B^*+1}-\mu _{11}^{s_B^*+1}+(\mu _{22}^{s_B^*}-\mu _{11}^{s_B^*})(\mu _{12}+\mu _{21})\right. \\ \quad \left. +\,(\mu _{22}^{s_B^*-1}-\mu _{11}^{s_B^*-1})\mu _{12}\mu _{21}\right) &{} \mathrm{if }\ \mu _{11} \ne \mu _{22}; \\ \quad \times (\mu _{11}+\mu _{21})(\mu _{11}+\mu _{22}) s_B^* \left( \mu _{22}- \frac{h}{2(\mu _{12}+\mu _{22})}\right. \\ \quad \times \left[ (s_B^*-1)(\mu _{12}+\mu _{22})+2 \mu _{22} \right] \Big )\\ \quad \Big / \left( \mu _{22}(\mu _{11}+\mu _{21})+(s_B^*-1)\right. \\ \quad \times \left. (\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})+\mu _{22}(\mu _{12}+\mu _{22})\right) &{}\mathrm{if }\ \mu _{11} = \mu _{22}; \\ \end{array} \right. \nonumber \\ \end{aligned}$$
(13)

and

$$\begin{aligned} h_0(i) = \left\{ \begin{array}{ll} g_0q\frac{(\mu _{11}+\mu _{21})\sum _{j=0}^{i-2}(j+1)\mu _{11}^j\mu _{22}^{i-2-j}+\sum _{j=0}^{i-1}\mu _{11}^j\mu _{22}^{i-1-j}}{\mu _{11}^{i-1}(\mu _{11}+\mu _{21})} -q\mu _{22}\frac{\sum _{j=0}^{i-2}(j+1)\mu _{11}^j\mu _{22}^{i-2-j}}{\mu _{11}^{i-1}}\\ \quad +\,hq\frac{\sum _{j=0}^{i-2}\mu _{11}^j\mu _{22}^{i-2-j}\sum _{k=0}^{j+1}k}{\mu _{11}^{i-1}} \text{ for }\,1 \le i \le s_B^*; \\ h_0(s_B^*)+q(i-s_B^*)-qg_0\frac{(i-s_B^*)}{\mu _{12}+\mu _{22}}-hq\frac{\sum _{k=s_B^*+1}^{i}k}{\mu _{12}+\mu _{22}} \text{ for }\,s_B^*+1 \le i \le B+1; \\ \end{array} \right. \end{aligned}$$

is a solution to (12), with the convention that summation over an empty set is equal to zero.

For the remainder of the proof, we assume that \(\mu _{11} \ne \mu _{22}\) since the proof for the case with \(\mu _{11}=\mu _{22}\) is similar. When \(\mu _{11}\mu _{22}=\mu _{12}\mu _{21}\), the service rates are structured (with \(\mu _1=\mu _{11}\), \(\mu _2=\mu _{21}\), \(\gamma _1=1\), and \(\gamma _2=\frac{\mu _{12}}{\mu _{11}}\)). Consequently, Theorem 3.1 shows that the expedite policy maximizes the profit. Moreover, when \(\mu _{11}\mu _{22}=\mu _{12}\mu _{21}\), we observe that \(\frac{f_1(1,2)}{f_2(1,2)}=0\), and hence \(s_B^*=s_1=1\) in Theorem 3.2, corresponding to the expedite policy. Hence, in the rest of the proof, we only consider the case where \(\mu _{11}\mu _{22}>\mu _{12}\mu _{21}\).

Next, we compute \(d_1(i)\), where

$$\begin{aligned} d_1(i) \in \arg \max _{a \in A_i} \left\{ r(i,a) + \sum _{i' \in {\mathcal {S}}} p(i'|i,a)h_0(i')\right\} ,\quad \forall i \in {\mathcal {S}}, \end{aligned}$$

and set \(d_1(i) = d_{0}(i)\) whenever possible. If one can show \(d_1(i)=d_{0}(i)\) for all \(i \in {\mathcal {S}}\), then the policy \(\pi _0\) is optimal according to Theorem 9.5.1 of [33]. Consequently, for all \(i \in {\mathcal {S}}\) and \(a \in A_i\), we want to show that the following inequality holds:

$$\begin{aligned} \Delta (i,a)&= \left( r(i,d_{0}(i)) + \sum _{i' \in {\mathcal {S}}} p(i'|i,d_{0}(i))h_0(i')\right) \nonumber \\&\quad -\, r(i,a) - \sum _{i' \in {\mathcal {S}}} p(i'|i,a)h_0(i') \ge 0. \end{aligned}$$
(14)

It is clear that the action \(d_0(i)\) is optimal for \(i \in \{0,B+2\}\) because the action space consists of only one action in these states. Hence, it is sufficient to check if the inequality (14) is satisfied in states \(1 \le i \le B+1\). For \(i \in \mathrm{I}\!\mathrm{N}^+\), let us define \(f_j(i)\) for \(j \in \{3,4,5,6\}\) as follows:

$$\begin{aligned} f_3(i)&= \mu _{11}^{s_B^*-i-1}(\mu _{11}+\mu _{21})\left( \mu _{22}^i-\mu _{11}^i\right) (\mu _{22}-\mu _{11})(\mu _{11}\mu _{22}-\mu _{12}\mu _{21}), \\ f_4(i)&= \frac{\mu _{22}^{i-1}}{\mu _{11}^{i}}(\mu _{22}-\mu _{11})\left[ \left( \mu _{11}\sum _{k=1}^{i-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}+ \mu _{21}\sum _{k=1}^{i}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right) \right. \\&\left. \times \left( \mu _{12}\mu _{21}\left( \mu _{22}^{s_B^*-1}-\mu _{11}^{s_B^*-1}\right) +(\mu _{12}+\mu _{21})\left( \mu _{22}^{s_B^*} -\mu _{11}^{s_B^*}\right) +\mu _{22}^{s_B^*+1} -\mu _{11}^{s_B^*+1}\right) \right. \\&\left. -\,\mu _{22}^{s_B^*-i-1}\left( \mu _{12}\sum _{k=1}^{s_B^*-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}+ \mu _{22}\sum _{k=1}^{s_B^*}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right) \right. \nonumber \\&\times \left. (\mu _{11}+\mu _{21})(\mu _{21}+\mu _{22})\left( \mu _{22}^i-\mu _{11}^i\right) \right] , \\ f_5(i)&= (\mu _{22}-\mu _{11})(\mu _{11}\mu _{22}-\mu _{12}\mu _{21}) \\&\times \left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}+\mu _{12}\mu _{22}^{i-1}\left( \mu _{22}^{s_B^*-i}-\mu _{11}^{s_B^*-i}\right) +\mu _{11}^{s_B^*-i-1}\mu _{21}\left( \mu _{22}^{i}-\mu _{11}^{i}\right) \right) , \\ f_6(i)&= \frac{\mu _{22}^{i-2}}{\mu _{11}^{i}}(\mu _{22}-\mu _{11})\left[ \mu _{22}^{s_B^*-i}\left( \mu _{12}\sum _{k=1}^{s_B^*-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}+ \mu _{22}\sum _{k=1}^{s_B^*}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right) \right. \\&\left. \times \left( \mu _{11}^i(\mu _{11}+\mu _{21})(\mu _{21}+\mu _{22}-\mu _{11}-\mu _{12})\right. \right. \\&\left. \left. +\,\mu _{22}^{i-1}(\mu _{21}+\mu _{22})(\mu _{11}\mu _{12}-\mu _{21}\mu _{22})\right) \right. \\&\left. -\left( \mu _{11}\mu _{12}\sum _{k=1}^{i-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1} -\mu _{21}\mu _{22}\sum _{k=1}^{i}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}+i\frac{\mu _{11}^i}{\mu _{22}^{i-2}}\right) \right. \\&\left. \times \left( \mu _{12}\mu _{21}\left( \mu _{22}^{s_B^*-1}-\mu _{11}^{s_B^*-1}\right) +(\mu _{12}+\mu _{21})\left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}\right) \right. \right. \\&\left. \left. +\,\mu _{22}^{s_B^*+1}-\mu _{11}^{s_B^*+1}\right) \right] . \end{aligned}$$

Moreover, let

$$\begin{aligned} \Psi&= (\mu _{22}-\mu _{11})\left( \mu _{22}^{s_B^*+1}-\mu _{11}^{s_B^*+1}+(\mu _{12}+\mu _{21})\left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}\right) \right. \\&\left. \quad +\,\mu _{12}\mu _{21} \left( \mu _{22}^{s_B^*-1}-\mu _{11}^{s_B^*-1}\right) \right) . \end{aligned}$$

First, let us consider the states \(i \in \{1,\ldots , s_B^*-1\}\). Recall that \(d_{0}(i)=a_{12}\) for \(i \in \{1,\ldots , s_B^*-1\}\). With some algebra we have

$$\begin{aligned} \Delta (i,a_{11})=\frac{f_3(i)-hf_4(i)}{\Psi }. \end{aligned}$$

The denominator of this expression is always positive. We observe that \(f_3(i) > 0\) for \(i \in \{1,\ldots ,s_B^*-1\}\). If \(f_4(i) \le 0\), then \(\Delta (i,a_{11}) > 0\). Next, we prove that \(\Delta (i,a_{11}) > 0\) when \(f_4(i) >0\), by showing that \(h < \frac{f_3(i)}{f_4(i)}\) (note that we do not have to show this when \(f_4(i) \le 0\)). Extensive algebra, together with Lemma 3.2, shows that for \(i \in \{1,\ldots ,s_B^*-1\}\), we have

$$\begin{aligned}&\frac{f_1(i,s_B^*)}{f_2(i,s_B^*)} < \frac{f_3(i)}{f_4(i)} \Leftrightarrow (\mu _{22}-\mu _{11}) \nonumber \\&\quad \times \left( \mu _{22}^{s_B^*+1}-\mu _{11}^{s_B^*+1}+(\mu _{12}+\mu _{21})\left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}\right) +\mu _{12}\mu _{21} \left( \mu _{22}^{s_B^*-1}-\mu _{11}^{s_B^*-1}\right) \right) \nonumber \\&\quad \times \left[ (\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})\mu _{22}^{s_B^*-1}\sum _{k=i}^{s_B^*-1}\sum _{j=1}^{i-1}(k-j)\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k+j-1} \right. \nonumber \\&\left. \quad +\left( s_B^*-i\right) (\mu _{11}+\mu _{21})\mu _{11}^{s_B^*-1}\mu _{22}\sum _{k=i}^{s_B^*-1}\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k}\right. \nonumber \\&\left. \quad +\,(\mu _{12}\!+\!\mu _{22})\mu _{22}^{s_B^*-1}\left( \mu _{11}\sum _{k=i}^{s_B^*-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1} \!+\!\mu _{21}\sum _{k=i}^{s_B^*-1}(k\!-\!i)\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right) \right] \!>\! 0 .\nonumber \\ \end{aligned}$$
(15)

We observe that inequality (15) holds trivially because \(\mu _{11} \ne \mu _{22}\). Our assumption that \(h \le \frac{f_1(s_1-1,s_1)}{f_2(s_1-1,s_1)}\), together with Lemmas 3.3 and 3.4, implies that \(h \le \frac{f_1(i,s_B^*)}{f_2(i,s_B^*)}\) for \(i \in \{1,\ldots ,s_B^*-1\}\). Consequently, the inequality (15) shows that \(h < \frac{f_3(i)}{f_4(i)}\), and hence \(\Delta (i,a_{11}) > 0\) for \(i \in \{1,\ldots ,s_B^*-1\}\).

Similarly, we can show that

$$\begin{aligned} \Delta (i,a_{21}) =\frac{f_5(i)-hf_6(i)}{ \Psi }. \end{aligned}$$

The denominator of this expression is always positive. We see that \(f_5(i) > 0\) for \(i \in \{1,\ldots ,s_B^*-1\}\). If \(f_6(i) \le 0\), then \(\Delta (i,a_{21}) \ge 0\). Next, we prove that \(\Delta (i,a_{21}) > 0\) when \(f_6(i) >0\), by showing that \(h < \frac{f_5(i)}{f_6(i)}\) (note that we do not have to show this when \(f_6(i) \le 0\)). Extensive algebra, together with Lemma 3.2, shows that for \(i \in \{1,\ldots ,s_B^*-1\}\), we have

$$\begin{aligned}&\frac{f_1(i,s_B^*)}{f_2(i,s_B^*)} < \frac{f_5(i)}{f_6(i)} \Leftrightarrow \frac{(\mu _{22}-\mu _{11})}{\mu _{11}} \nonumber \\&\quad \times \left( \mu _{22}^{s_B^*+1}-\mu _{11}^{s_B^*+1}+(\mu _{12}+\mu _{21})\left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}\right) +\mu _{12}\mu _{21} \left( \mu _{22}^{s_B^*-1}-\mu _{11}^{s_B^*-1}\right) \right) \nonumber \\&\quad \times \left[ (\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})\mu _{22}^{s_B^*-1}\sum _{k=i}^{s_B^*-1}\sum _{j=1}^{i-1}(k-j)\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k+j-1}\right. \nonumber \\&\left. \quad +\left( s_B^*-i\right) (\mu _{11}+\mu _{21})\mu _{11}^{s_B^*-1}\mu _{22} \left( \sum _{k=0}^{i-1}\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k}\right) ^2\right. \nonumber \\&\left. \quad +\,(\mu _{12}\!+\!\mu _{22})\mu _{22}^{s_B^*-1}\left( \mu _{11}\sum _{k=i}^{s_B^*-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1} \!+\!\mu _{21}\sum _{k=i}^{s_B^*-1}(k\!-\!i)\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right) \!\right] \!>\! 0 .\nonumber \\ \end{aligned}$$
(16)

We observe that the inequality (16) holds trivially because \(\mu _{11} \ne \mu _{22}\). Our assumption that \(h \le \frac{f_1(s_1-1,s_1)}{f_2(s_1-1,s_1)}\), together with Lemmas 3.3 and 3.4, implies that \(h \le \frac{f_1(i,s_B^*)}{f_2(i,s_B^*)}\) for \(i \in \{1,\ldots ,s_B^*-1\}\). Consequently, the inequality (16) shows that \(h < \frac{f_5(i)}{f_6(i)}\), and hence \(\Delta (i,a_{21}) > 0\) for \(i \in \{1,\ldots ,s_B^*-1\}\).

Finally, some algebra shows that

$$\begin{aligned} \Delta (i,a_{22}) =\frac{f_1(i,s_B^*)-hf_2(i,s_B^*)}{ \Psi }. \end{aligned}$$

The denominator of this expression is always positive, and our assumption that \(h \le \frac{f_1(s_1-1,s_1)}{f_2(s_1-1,s_1)}\), together with Lemmas 3.2, 3.3, and 3.4, shows that \(\Delta (i,a_{22}) \ge 0\) for \(i \in \{1,\ldots ,s_B^*-1\}\) (note that \(\Delta (i,a_{22}) > 0\) when \(h < \frac{f_1(s_B^*-1,s_B^*)}{f_2(s_B^*-1,s_B^*)}\)).

For \(i \in \mathrm{I}\!\mathrm{N}^+\) and \(i \ge s\), let us define \(\bar{f}_j(i)\) for \(j \in \{1,2,3\}\) as follows:

$$\begin{aligned} \bar{f}_1(i)&= \frac{(\mu _{22}\!-\!\mu _{11})}{(\mu _{12}\!+\!\mu _{22})}\left[ \left( i(\mu _{12}\!+\!\mu _{22})\!+\!(i+1)(\mu _{11}\!+\!\mu _{21})\right) \!\times \! \left( \mu _{12}\mu _{21}\left( \mu _{22}^{s_B^*-1}\!-\!\mu _{11}^{s_B^*-1}\right) \right. \right. \\&\left. \left. +\,(\mu _{12}\!+\!\mu _{21})\left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}\right) \,{+}\,\mu _{22}^{s_B^*+1}-\mu _{11}^{s_B^*+1}\right) \!-\!\mu _{22}^{s_B^*-2}(\mu _{22}\!-\!\mu _{11})(\mu _{11}\!+\!\mu _{21})\right. \\&\left. \times \left( \mu _{11}\!+\!\mu _{12}\!+\!\mu _{21}\!+\!\mu _{22}\right) \left( \mu _{12}\sum _{k=1}^{s_B^*-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\!+\! \mu _{22}\sum _{k=1}^{s_B^*}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right) \right] , \\ \bar{f}_2(i)&= \frac{\mu _{22}^{s_B^*-2}}{\mu _{11}^{s_B^*-1}}(\mu _{22}\!-\!\mu _{11})\left[ \left( i\mu _{12}\!+\!(i+1)\mu _{11}\right) \right. \\&\left. \times \left( \mu _{12}\mu _{21}\left( \mu _{22}^{s_B^*-1}\!-\!\mu _{11}^{s_B^*-1}\right) \!+\!(\mu _{12}\!+\!\mu _{21})\left( \mu _{22}^{s_B^*}\!-\!\mu _{11}^{s_B^*}\right) \!+\!\mu _{22}^{s_B^*+1}-\mu _{11}^{s_B^*+1}\right) \right. \\&\left. -\mu _{22}^{s_B^*-2}(\mu _{22}\!-\!\mu _{11})(\mu _{11}\!+\!\mu _{12})(\mu _{11}\!+\!\mu _{21})\right. \\&\left. \quad \left( \mu _{12}\sum _{k=1}^{s_B^*-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\!+\! \mu _{22}\sum _{k=1}^{s_B^*}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right) \right] , \\ \bar{f}_3(i)&= \frac{(\mu _{22}-\mu _{11})}{(\mu _{12}\!+\!\mu _{22})}\left[ \left( i\mu _{22}\!+\!(i+1)\mu _{21}\right) \right. \\&\left. \times \left( \mu _{12}\mu _{21}\left( \mu _{22}^{s_B^*-1}-\mu _{11}^{s_B^*-1}\right) +(\mu _{12}+\mu _{21})\left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}\right) +\mu _{22}^{s_B^*+1}-\mu _{11}^{s_B^*+1}\right) \right. \\&\left. -\mu _{22}^{s_B^*-2}(\mu _{22}-\mu _{11})(\mu _{11}+\mu _{21})(\mu _{21}+\mu _{22})\right. \\&\quad \left. \left( \mu _{12}\sum _{k=1}^{s_B^*-1}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}+ \mu _{22}\sum _{k=1}^{s_B^*}k\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right) \right] . \end{aligned}$$

Now, consider the states \(i \in \{ s_B^*,\ldots ,B+1 \}\), whenever this set is nonempty. Recall that \(d_{0}(i)=a_{22}\) for \(i \in \{ s_B^*,\ldots ,B+1 \}\). With some algebra we have

$$\begin{aligned} \Delta (i,a_{11})\!=\!\frac{(\mu _{11}\!+\!\mu _{21}) (\mu _{11}\mu _{22}\!-\!\mu _{12}\mu _{21})(\mu _{22}\!-\!\mu _{11})\left( \mu _{22}^{s_B^*-1}-\mu _{11}^{s_B^*-1}\right) \!+\!h\bar{f}_1(i)}{ \Psi }. \end{aligned}$$

The denominator and the first term in the numerator are nonnegative. Moreover, \(\bar{f}_1(i) \ge 0\) is nondecreasing in \(i\), and hence we only need to show that \(\Delta (s_B^*,a_{11}) \ge 0\). Some algebra shows that

$$\begin{aligned} \bar{f}_1(s_B^*)&= \mu _{22}^{s_B^*-2}\frac{(\mu _{22}\!-\!\mu _{11})^2}{(\mu _{12}\!+\!\mu _{22})} \left[ (\mu _{12}\!+\!\mu _{22})\mu _{22} \left( \left( s_B^*\!+\!1\right) (\mu _{11}\!+\!\mu _{21})\!+\!s_B^*(\mu _{12}\!+\!\mu _{22})\right) \right. \\&\left. +\,(\mu _{11}\!+\!\mu _{21}) \left( (\mu _{12}\!+\!\mu _{22})^2\sum _{k=1}^{s_B^*-1}\left( s_B^*-k\right) \left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1} \right. \right. \\&\left. \left. +\,\mu _{12}(\mu _{11}\!+\!\mu _{21})\sum _{k=1}^{s_B^*-1}\left( s_B^*\!+\!1-k\right) \left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1}\right. \right. \\&\quad \left. \left. +\,\mu _{22}(\mu _{11}+\mu _{21})\sum _{k=1}^{s_B^*}\left( s_B^*+1-k\right) \left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1} \right) \right] . \end{aligned}$$

It is clear that \(\bar{f}_1(s_B^*) >0\) because \(\mu _{11} \ne \mu _{22}\). Hence we can conclude that \(\Delta (s_B^*,a_{11}) > 0\) and consequently \(\Delta (i,a_{11}) > 0\) for \(i \in \{s_B^*,\ldots ,B+1\}\).

Similarly, we can show that

$$\begin{aligned} \Delta (i,a_{12}) =\frac{-f_1(s_B^*,s_B^*+1)+h\bar{f}_2(i)}{ \Psi }. \end{aligned}$$

The denominator of this expression is always positive. We observe that \(\bar{f}_2(s_B^*)=f_2(s_B^*,s_B^*+1)\). Moreover, \(\bar{f}_2(i)\) is positive (by Lemma 3.2) and increasing in \(i\). Hence, it is sufficient to show that \(-f_1(s_B^*,s_B^*+1)+h\bar{f}_2(s_B^*)=-f_1(s_B^*,s_B^*+1)+hf_2(s_B^*,s_B^*+1) \ge 0\). This follows from Lemma 3.3 since \(h>\frac{f_1(s_1,s_1+1)}{f_2(s_1,s_1+1)}\), and thus \(\Delta (i,a_{12}) > 0\) for \(i \in \{s_B^*,\ldots ,B+1\}\).

Finally, some algebra shows that

$$\begin{aligned} \Delta (i,a_{21})\!=\!\frac{(\mu _{11}\mu _{22}\!-\!\mu _{12}\mu _{21})(\mu _{22}\!-\!\mu _{11})\left( \left( \mu _{22}^{s_B^*}-\mu _{11}^{s_B^*}\right) \!+\!\mu _{21}\left( \mu _{22}^{s_B^*-1}\!-\!\mu _{11}^{s_B^*-1}\right) \right) \!+\!h\bar{f}_3(i)}{ \Psi }. \end{aligned}$$

The denominator and the first term in the numerator are nonnegative. Moreover, \(\bar{f}_3(i)\ge 0\) is increasing in \(i\), and hence we only need to show that \(\Delta (s_B^*,a_{21}) \ge 0\). Some algebra shows that

$$\begin{aligned} \bar{f}_3(s_B^*)&= \frac{(\mu _{22}-\mu _{11})^2}{(\mu _{12}+\mu _{22})}\left[ \mu _{21}\left( \mu _{11}^{s_B^*-1}(\mu _{11}+\mu _{21})+\mu _{22}^{s_B^*-1}(\mu _{12}+\mu _{22})\right) \right. \\&\quad \left. +\,s_B^*\mu _{22}^{s_B^*-1}(\mu _{12}+\mu _{22})(\mu _{21}+\mu _{22})+\mu _{22}^{s_B^*-2}(\mu _{11}+\mu _{21})(\mu _{12}+\mu _{22})\right. \\&\quad \left. \left( \mu _{21}\sum _{k=1}^{s_B^*-1}(s_B^*+1-k)\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{k-1} +\mu _{22}\sum _{k=1}^{s_B^*-1}(s_B^*-k)(\frac{\mu _{11}}{\mu _{22}})^{k-1} \right) \right] . \end{aligned}$$

It is clear that \(\bar{f}_3(s_B^*) > 0\) because \(\mu _{11}\ne \mu _{22}\). Hence, we can conclude that \(\Delta (s_B^*,a_{21}) > 0\) and consequently \(\Delta (i,a_{21}) > 0\) for \(i \in \{s_B^*,\ldots ,B+1\}\).

Note that any action that takes the process to one of the recurrent states can be used in states \(\{s_B^*+1,\ldots ,B+2\}\) because they are transient. For the reasons explained in the proof of Lemma 3.1, it is strictly suboptimal to idle both servers in a state. Similarly, when \(\mu _{ij}>0\) for \(i,j \in \{1,2\}\), idling one server in a recurrent state is strictly suboptimal compared to assigning the server to the station where the other server is working in that state, because in the problem formulation (5) the reward term is smaller in the idling policy and every recurrent state has a positive steady-state probability. The uniqueness of the optimal actions in states \(\{0,\ldots ,s_B^*\}\) when \(\mu _{11}\mu _{22} > \mu _{12}\mu _{21}>0\) and \(h\ne \frac{f_1(s_B^*-1,s_B^*)}{f_2(s_B^*-1,s_B^*)}\) now follows from the proof of Theorem 3.1 of [5] and the discussion in Sect. 9.5.2 of [33]. \(\square \)

Proof of Lemma 3.6

Note that if \(s_1 \le B+2\), then we have \(P'(B+1) = P'(B)\) (see Theorem 3.2 and Remark 3.3). Moreover, \(P'(B) \ge P'(B-1)\) (because the optimal policy for \(B-1\) is feasible for \(B\)). Hence, the inequality \(P'(B+1) - P'(B) \le P'(B) - P'(B-1)\) is satisfied trivially when \(s_1 \le B + 2\).

In the remainder of the proof, we prove the lemma for \(s_1 > B+2\) (so that \(s_{B-1}^* = B+1\), \(s_B^* = B+2\), and \(s_{B+1}^* = B+3\)). The expression for \(P' (B)\) is provided in equation (13). Note that we have \(P'(B) \ge 0\) for all \(B \ge 0\) because of our assumption that \(\sum _{j=2}^N\frac{h_j}{\Sigma _j} < r\).

First, consider the case with \(\mu _{11} \ne \mu _{22}\). It follows from Eq. (13) that \(P'(B) \ge 0\) if and only if \(\alpha _1 - h \alpha _2 \ge 0\), where

$$\begin{aligned} \alpha _1&= \left( \mu _{22}^{B+2}-\mu _{11}^{B+2}\right) (\mu _{22}-\mu _{11}) (\mu _{12} + \mu _{22}), \\ \alpha _2&= (\mu _{12}+\mu _{22}) \left( \mu _{22}^{B+2}-(B+2)\mu _{11}^{B+1}\mu _{22}+(B+1)\mu _{11}^{B+2}\right) \\&\quad +\,(B+2)\mu _{11}^{B+1}(\mu _{22}-\mu _{11})^2. \end{aligned}$$

Note that

$$\begin{aligned}&\left( \mu _{22}^{B+2}-(B+2)\mu _{11}^{B+1}\mu _{22}+(B+1)\mu _{11}^{B+2}\right) \\&\quad = \mu _{22}^{B+1}(\mu _{22}-\mu _{11})\sum _{k=0}^{B}\left( \left( \frac{\mu _{11}}{\mu _{22}}\right) ^k-\left( \frac{\mu _{11}}{\mu _{22}}\right) ^{B+1} \right) \ge 0. \end{aligned}$$

This implies that \(\alpha _2 \ge 0\), and hence we have that \(P'(B) \ge 0\) if and only if \(h \le \frac{\alpha _1}{\alpha _2}\). Some algebra shows that the inequality \(P'(B+1)- P'(B) \le P'(B) - P'(B-1)\) holds if and only if \(\beta _1- h \beta _2 \ge 0\), where

$$\begin{aligned} \beta _1&= (\mu _{11}-\mu _{22})^2(\mu _{11}\mu _{22}-\mu _{12}\mu _{21})\\&\quad \left( \mu _{11}^{B+3}(\mu _{11}+\mu _{12})(\mu _{11}+\mu _{21}) + \mu _{22}^{B+3}(\mu _{11}+\mu _{22})(\mu _{21}+\mu _{22})\right) , \\ \beta _2&= \mu _{11}^{B+2}\mu _{22}(\mu _{11}+\mu _{12})(\mu _{11}+\mu _{21})\\&\quad \left( \mu _{11}\mu _{21}(\mu _{11}+2\mu _{12}) +\mu _{11}\mu _{22}(\mu _{12}+\mu _{21})+\mu _{22}^2(2\mu _{11}+\mu _{12}) \right) \\&-\, \mu _{11}\mu _{22}^{B+2}(\mu _{12}+\mu _{22})(\mu _{21}+\mu _{22})\\&\quad \left( \mu _{12}\mu _{22}(2\mu _{21}+\mu _{22}) +\mu _{11}\mu _{22}(\mu _{12}+\mu _{21})+\mu _{11}^2(\mu _{21}+2\mu _{22}) \right) \\&-\, (B+2) \mu _{11}\mu _{22}(\mu _{11}-\mu _{22})(\mu _{11}+\mu _{12})(\mu _{21}+\mu _{22}) \times \\&\qquad \quad \left( \mu _{11}^{B+1}(\mu _{11}+\mu _{12})(\mu _{11}+\mu _{21}) + \mu _{22}^{B+1}(\mu _{12}+\mu _{22})(\mu _{21}+\mu _{22})\right) . \end{aligned}$$

If \(\beta _2 \le 0\), the result follows trivially. Otherwise, if \(\beta _2 > 0\), some algebra shows that

$$\begin{aligned}&\frac{\beta _1}{\beta _2} \ge \frac{\alpha _1}{\alpha _2} \Leftrightarrow (\mu _{22}-\mu _{11})\nonumber \\&\quad \left( \mu _{22}^{B+3}-\mu _{11}^{B+3}+\left( \mu _{22}^{B+2}-\mu _{11}^{B+2}\right) (\mu _{12}+\mu _{21}) +\left( \mu _{22}^{B+1}-\mu _{11}^{B+1}\right) \mu _{12}\mu _{21}\right) \nonumber \\&\quad \times \left[ \mu _{22}\left( (\mu _{12}+\mu _{21})\gamma (B+2) + \mu _{12}\mu _{21} \gamma (B+1) + \gamma (B+3) \right) \right. \nonumber \\&\quad + \left. \left( \mu _{11}\mu _{22} - \mu _{12}\mu _{21}\right) (\mu _{22}-\mu _{11})\left( \mu _{22}^{B+1}-\mu _{11}^{B+1}\right) \right] \ge 0, \end{aligned}$$
(17)

where \(\gamma (t) = \sum _{k=0}^{B+1} (\mu _{11}^{t-k} -\mu _{22}^{t-k})(\mu _{11}^{k} -\mu _{22}^{k})\) for \(t \ge 0 \) . The assumption that \(\mu _{11}\mu _{22} \ge \mu _{12}\mu _{21}\) clearly implies that inequality (17) holds, and hence \(\frac{\beta _1}{\beta _2} \ge \frac{\alpha _1}{\alpha _2}\). The fact that \(h \le \frac{\alpha _1}{\alpha _2}\) now implies that \(\beta _1- h \beta _2 \ge 0\), and hence \(P'(B+1)- P'(B) \le P'(B) - P'(B-1)\).

Finally, consider the case with \(\mu _{11} = \mu _{22}\). It follows from Eq. (13) that \(P'(B) \ge 0\) if and only if

$$\begin{aligned} h \le \frac{2\mu _{22}(\mu _{12}+\mu _{22})}{(B+1)(\mu _{12}+\mu _{22})+2\mu _{22}}. \end{aligned}$$

Some algebra shows that

$$\begin{aligned} P'(B+1)- P'(B)&\le P'(B) - P'(B-1) \Leftrightarrow 2(\mu _{12}+\mu _{22})(\mu _{21}+\mu _{22})\\&- \,h(\mu _{21}-\mu _{12}) \ge 0. \end{aligned}$$

If \(\mu _{12} \ge \mu _{21}\), the result follows trivially. Otherwise, we have

$$\begin{aligned}&\frac{2(\mu _{12}+\mu _{22})(\mu _{21}+\mu _{22})}{\mu _{21}-\mu _{12}} \ge \frac{2\mu _{22}(\mu _{12}+\mu _{22})}{(B+1)(\mu _{12}+\mu _{22})+2\mu _{22}}\\&\quad \Leftrightarrow (\mu _{22}^2-\mu _{12}\mu _{21})+ (B+2)(\mu _{12}+\mu _{22})(\mu _{21}+\mu _{22}) \ge 0. \end{aligned}$$

The last inequality holds because the assumptions that \(\mu _{11} = \mu _{22}\) and \(\mu _{11}\mu _{22} \ge \mu _{12}\mu _{21}\) together imply that \(\mu _{22}^2 \ge \mu _{12} \mu _{21}\). Hence, the fact that \(P'(B) \ge 0\) now implies that \(P'(B+1)- P'(B) \le P'(B) - P'(B-1)\). The proof is complete. \(\square \)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kırkızlar, E., Andradóttir, S. & Ayhan, H. Profit maximization in flexible serial queueing networks. Queueing Syst 77, 427–464 (2014). https://doi.org/10.1007/s11134-014-9393-y

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11134-014-9393-y

Keywords

Mathematics Subject Classification

Navigation