Skip to main content
SpringerLink
Log in

Optimal control of a multiclass queueing system when customers can change types

  • Published:
Queueing Systems Aims and scope Submit manuscript

Abstract

It is well known that the \(c\mu \)-rule is optimal for serving multiple types of customers to minimize the expected total waiting cost. What happens when less valuable customers (those with lower \(c\mu \)) can change to valuable ones? In this paper, we study this problem by considering two types of customers. The first type of customers is less valuable, but it may change to the second type (i.e., more valuable customers) after a random amount of time. The resulting problem is a continuous-time Markov decision process with countable state space and unbounded transition rates, which is known to be technically challenging. We first prove the existence of optimal non-idling stationary policies. Based on the smoothed rate truncation, we derive conditions under which a modified \(c\mu \)-rule remains optimal. For other cases, we develop a simple heuristic policy for serving customers. Our numerical study shows that the heuristic policy performs close to the optimal, with the worst case within 2.47 % of the optimal solution and 95 % of the examples within 1 % of the optimal solution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Akan, M., Alagoz, O., Ata, B., Erenay, F., Said, A.: A broader view of designing the liver allocation system. Oper. Res. 60(4), 626–633 (2012)

    Article  Google Scholar 

  2. Anderson, W.J.: Continous-Time Markov Chains. Springer, Berlin (1991)

    Book  Google Scholar 

  3. Bassamboo, A., Randhawa, R.S.: On the accuracy of fluid models for capacity sizing in queueing systems with impatient customers. Oper. Res. 58, 1398–1413 (2010)

    Article  Google Scholar 

  4. Bertsekas, D.P.: Dynamic Programming and Optimal Control, II edn. Athena Scientific, Belmont, Massechusetts (2001)

    Google Scholar 

  5. Bhulai, S., Brooms, A.C., Spieksma, F.M.: On structural properties of the value function for an unbounded jump Markov process with an application to a processor sharing retrial queue. Queueing Syst. 76(4), 425–446 (2014)

    Article  Google Scholar 

  6. Blok, H.: Markov decision processes with unbounded transition rates: structural properties of the relative value function. Master thesis, Utrecht University (2011)

  7. Blok, H., Spieksman, F.M.: Continuity and ergodicity properties of a parametrised collection of countable Markov processes. Techinical report, Leiden University (2013)

  8. Bremaud, P.: Point Processes and Queues: Martingale Dynamics. Springer Series in Statistics, New York (1981)

    Book  Google Scholar 

  9. Buyukkoc, C., Varaiya, P., Walrand, J.: The \( c\mu \)-rule revisited. Adv. Appl. Probab. 17(1), 237–238 (1985)

    Article  Google Scholar 

  10. Choi, B.D., Kim, B.: Non-ergodicity criteria for denumerable continuous time Markov processes. Oper. Res. Lett. 32, 574–580 (2004)

    Article  Google Scholar 

  11. Cox, D.R., Smith, W.L.: Queues. Methuen, London (1961)

    Google Scholar 

  12. Down, D.G., Lewis, M.E.: The N-network model with upgrades. Probab. Eng. Inf. Sci. 24(2), 171–200 (2010)

    Article  Google Scholar 

  13. Down, D.G., Koole, G.M., Lewis, M.E.: Dynamic control of a single server system with abandonments. Queueing Syst. 69, 63–90 (2011)

    Article  Google Scholar 

  14. Foss, S.G.: Queues with customers of several types. In: Borovkov, A.A. (ed.) Advances in Probability Theory: Limit Theorems and Related Problems, pp. 348–377. Optimization Software (1984)

  15. Glazebrook, K.D.: Scheduling tasks with exponential service times on parallel processors. J. Appl. Probab. 685–689 (1979)

  16. Glazebrook, K.D.: Scheduling stochastic jobs on a single machine subject to breakdowns. Nav. Res. Logist. Q. 31(2), 251–264 (1984)

    Article  Google Scholar 

  17. Guo, X.P.: Contrained optimization for average-cost continuous-time Markov decision processes. IEEE Trans. Autom. Control 52, 1139–1143 (2007)

    Article  Google Scholar 

  18. Guo, X.P., Hernandez-Lerma, O.: Continous-Time Markov Decision Processes: Theory and Applications. Springer, London (2009)

    Book  Google Scholar 

  19. Guo, X.P., Piunovskiy, A.: Discounted continuous-time Markov decision processes with constraints: unbounded transition and loss rates. Math. Oper. Res. 36, 105–132 (2011)

    Article  Google Scholar 

  20. Guo, X.P., Ye, L.: New discount and average optimality conditions for continuous-time Markov decision processes. Adv. Appl. Probab. 42, 953–985 (2010)

    Article  Google Scholar 

  21. Guo, X.P., Zhang, W.: Convergence of controlled models and finite-state approximation for discounted continuous-time Markov decision processes with constraints. Eur. J. Oper. Res. 238, 486–496 (2014)

    Article  Google Scholar 

  22. He, Q.-M., Xie, J.G., Zhao, X.B.: Priority queue with customer upgrades. Nav. Res. Logist. 59, 362–375 (2012)

    Article  Google Scholar 

  23. Kitaev, M.Y., Rykov, V.V.: A service system with a branching flow of secondary customers. Autom. Remote Control 41(9), 52–61 (1980)

    Google Scholar 

  24. Koole, G.M.: Monotonicity in Markov reward and decision chains: theory and applications. Found. Trends Stoch. Syst. 1, 1–76 (2006)

    Article  Google Scholar 

  25. Muller, A., Stoyan, D.: Comparison Methods for Stochastic Models and Risks. Wiley, Baffins Lane, Chichester (2002)

    Google Scholar 

  26. Neuts, M.: Matrix-Geometric Solutions in Stochastic Models: An Algorithmic Approach. The Johns Hopkins University Press, Baltimore (1981)

    Google Scholar 

  27. Oksendal, B., Sulem, A.: Applied Stochastic Control of Jump Diffusions. Springer, Berlin (2005)

    Google Scholar 

  28. Pang, G., Yao, D.D.: Heavy-traffic limits for a many-server queueing network with switchover. Adv. Appl. Probab. 45(3), 645–672 (2013)

    Article  Google Scholar 

  29. Piunovskiy, A.: Controlled jump Markov processes with local transitions and their fluid approximation. WSEAS Trans. Syst. Control 4(8), 399–412 (2009)

    Google Scholar 

  30. Piunovskiy, A., Zhang, Y.: The transformation method for continuous-time Markov decision processes. J. Optim. Theory Appl. 154(2), 691–712 (2012)

    Article  Google Scholar 

  31. Prieto-Rumeau, T., Hernandez-Lerma, O.: Discounted continuous-time controlled Markov chains: convergence of control models. J. Appl. Probab. 49, 1072–1090 (2012)

    Article  Google Scholar 

  32. Prieto-Rumeau, T., Lorenzo, J.M.: Approximating ergodic average reward continuous-time controlled Markov chains. IEEE Trans. Autom. Control 55(1), 201–207 (2010)

    Article  Google Scholar 

  33. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994)

    Book  Google Scholar 

  34. Ross, S.: Stochastic Processes, 2nd edn. Wiley, New York (1996)

    Google Scholar 

  35. Sennott, L.I.: Average-cost optimal stationary policies in infinite state Markov decision processes with unbounded costs. Oper. Res. 37(4), 626–633 (1989)

    Article  Google Scholar 

  36. Sennott, L.I.: Stochastic Dynamic Programming and the Control of Queueing Systems. Wiley, New York (1999)

    Google Scholar 

  37. Serfozo, R.F.: An equivalence between continuous and discrete time Markov decision processes. Oper. Res. 27(3), 616–620 (1979)

    Article  Google Scholar 

  38. Shaked, M., Shanthikumar, J.G.: Stochastic Orders. Springer, New York (2007)

    Book  Google Scholar 

  39. Smith, W.E.: Various optimizers for single-stage production. Nav. Res. Logist. Q. 3, 59–66 (1956)

    Article  Google Scholar 

  40. Stidham, S., Weber, R.: A survey of Markov decision models for control of networks of queues. Queueing Syst. 13, 291–314 (1993)

    Article  Google Scholar 

  41. Ungureanu, V., Melamed, B., Katehakis, M., Bradford, P.G.: Deferred assignment scheduling in cluster-based servers. Clust. Comput. 9(1), 57–65 (2006)

    Article  Google Scholar 

  42. Ungureanu, V., Melamed, B., Katehakis, M.: Effective load balancing for cluster-based servers employing job preemption. Perform. Eval. 65(8), 606–622 (2008)

    Article  Google Scholar 

  43. Van Mieghem, J.A.: Dynamic scheduling with convex delay costs: the generalized \(c\mu \)-rule. Ann. Appl. Probab. 5(3), 809–833 (1995)

    Article  Google Scholar 

  44. Weber, R.R.: Scheduling jobs with stochastic processing requirements on parallel machines to minimize makespan or flowtime. J. Appl. Probab. 167–182 (1982)

  45. Weber, R.R., Varaiya, P., Walrand, J.: Scheduling jobs with stochastically ordered processing times on parallel machines to minimize expected flowtime. J. Appl. Probab. 841–847 (1986)

  46. Xie, J.G., He, Q.-M., Zhao, X.B.: Stability of a priority queueing system with customer transfers. Oper. Res. Lett. 36, 705–709 (2008)

    Article  Google Scholar 

  47. Xie, J.G., He, Q.-M., Zhao, X.B.: On the stationary distribution of queue lengths in a multi-class priority queueing system with customer transfers. Queueing Syst. 62(3), 255–277 (2009)

    Article  Google Scholar 

  48. Zhang, Y.: Average optimality for continuous-time Markov decision process under weak continuity conditions. J. Appl. Probab. 51, 954–970 (2014)

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the NSFC under Grants 71201154, 71401159, and 71571176, and the Fundamental Research Funds for the Central Universities under Grants WK2040160009 and WK2040160011. The authors are grateful to Prof. Xiuli Chao and Prof. Xin Chen for their helpful discussions, and to the Editor-in-Chief Prof. Sergey Foss, an area editor and two referees for their thoughtful comments.

Author information

Authors and Affiliations

  1. School of Management, University of Science and Technology of China, Hefei, Anhui, China

    Ping Cao & Jingui Xie

Authors
  1. Ping Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingui Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingui Xie.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, P., Xie, J. Optimal control of a multiclass queueing system when customers can change types. Queueing Syst 82, 285–313 (2016). https://doi.org/10.1007/s11134-015-9466-6

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11134-015-9466-6

Keywords

Mathematics Subject Classification

Search

Navigation