Skip to main content
Log in

Continuous-Review Tracking Policies for Dynamic Control of Stochastic Networks

  • Published:
Queueing Systems Aims and scope Submit manuscript

Abstract

This paper is concerned with dynamic control of stochastic processing networks. Specifically, it follows the so called “heavy traffic approach,” where a Brownian approximating model is formulated, an associated Brownian optimal control problem is solved, the solution of which is then used to define an implementable policy for the original system. A major challenge is the step of policy translation from the Brownian to the discrete network. This paper addresses this problem by defining a general and easily implementable family of continuous-review tracking policies. Each such policy has the following structure: at each point in time t, the controller observes the current vector of queue lengths q and chooses (i) a target position z(q) of where the system should be at some point in the near future, say at time t+l, and (ii) an allocation vector v(q) that describes how to split the server's processing capacity amongst job classes in order to steer the state from q to z(q). Implementation of such policies involves the enforcement of small safety stocks. In the context of the “heavy traffic” approach, the solution of the approximating Brownian control problem is used in selecting the target state z(q). The proposed tracking policy is shown to be asymptotically optimal in the heavy traffic limiting regime, where the Brownian model approximation becomes valid, for multiclass queueing networks that admit orthant Brownian optimal controls; this is a form of pathwise, or greedy, optimality. Several extensions are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. F. Avram, D. Bertsimas and M. Ricard, Fluid models of sequencing problems in open queueing networks; an optimal control approach, in: Stochastic Networks, eds. F. Kelly and R. Williams, Proceedings of the IMA, Vol. 71 (Springer, New York, 1995) pp. 199–234.

    Google Scholar 

  2. S. Axsäter, Continuous review policies for multi-level inventory systems with stochastic demand, in: Logistics of Production and Inventory, eds. S.C. Graves, A.H.G.R. Kan and P.H. Zipkin, Handbooks in Operations Research andManagement Science, Vol. 4 (North-Holland, Amsterdam, 1993) pp. 175– 197.

    Google Scholar 

  3. N. Bäuerle, Asymptotic optimality of tracking policies in stochastic networks, Ann. Appl. Probab. 10(4) (2000) 1065–1083.

    Google Scholar 

  4. S.L. Bell and R.J. Williams, Dynamic scheduling of a system with two parallel servers in heavy traffic with resource pooling: Asymptotic optimality of a threshold policy, Ann. Appl. Probab. 11 (2001) 608–649.

    Google Scholar 

  5. V. Böhm, On the continuity of the optimal policy set for linear programs, SIAM J. Appl. Math. 28(2) (1975) 303–306.

    Google Scholar 

  6. M. Bramson, Stability of two families of queueing networks and a discussion of fluid limits, Queueing Systems 28 (1998) 7–31.

    Google Scholar 

  7. M. Bramson, State space collapse with applications to heavy-traffic limits for multiclass queueing networks, Queueing Systems 30 (1998) 89–148.

    Google Scholar 

  8. M. Bramson and J. Dai, Heavy traffic limits for some queueing networks, Preprint (1999).

  9. H. Chen, Fluid approximations and stability of multiclass queueing networks: Work-conserving policies, Ann. Appl. Probab. 5 (1995) 637–655.

    Google Scholar 

  10. H. Chen and D. Yao, Dynamic scheduling of a multiclass fluid network, Oper. Res. 41(6) (1993) 1104–1115.

    Google Scholar 

  11. M. Chen, C. Pandit and S. Meyn, In search of sensitivity in network optimization (2002) submitted for publication.

  12. D. Clarke,Advances in Model Predictive Control (Oxford Univ. Press, Oxford, 1994).

    Google Scholar 

  13. J.G. Dai, On positive Harris recurrence of multiclass queueing networks: A unified approach via fluid limit models, Ann. Appl. Probab. 5 (1995) 49–77.

    Google Scholar 

  14. J. Dai and G. Weiss, A fluid heuristic for minimizing makespan in job-shops, Oper. Res. (2002) to appear.

  15. A. Dembo and T. Zajik, Large deviations: From empirical mean and measure to partial sum processes, Stochastic Process. Appl. 57 (1995) 191–224.

    Google Scholar 

  16. A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Applications of Mathematics (Springer, New York, 1998).

    Google Scholar 

  17. P. Dupuis and K. Ramanan, A multiclass feedback queueing network with a regular Skorokhod problem, Preprint (1999).

  18. J.M. Harrison, Brownian Motion and Stochastic Flow Systems (Wiley, New York, 1985).

    Google Scholar 

  19. J.M. Harrison, Brownian models of queueing networks with heterogeneous customer populations, in: Stochastic Differential Systems, Stochastic Control Theory and Applications, eds. W. Fleming and P.L. Lions, Proceedings of the IMA, Vol. 10 (Springer, New York, 1988) pp. 147–186.

    Google Scholar 

  20. J.M. Harrison, The BIGSTEP approach to flow management in stochastic processing networks, in: Stochastic Networks: Theory and Applications, eds. F. Kelly, S. Zachary and I. Ziedins (Oxford Univ. Press, Oxford, 1996) pp. 57–90.

    Google Scholar 

  21. J.M. Harrison, Heavy traffic analysis of a system with parallel servers: Asymptotic optimality of discrete-review policies, Ann. Appl. Probab. 8 (1998) 822–848.

    Google Scholar 

  22. J.M. Harrison, A broader view of Brownian networks (2000) submitted.

  23. J.M. Harrison, Stochastic networks and activity analysis, in:Analytic Methods in Applied Probability, ed. Y. Suhov, In memory of Fridrih Karpelevich (Amer. Math. Soc., Providence, RI, 2002).

  24. J.M. Harrison and J.A. Van Mieghem, Dynamic control of brownian networks: State space collapse and equivalent workload formulations, Ann. Appl. Probab. 7 (1996) 747–771.

    Google Scholar 

  25. J.M. Harrison and L.M. Wein, Scheduling network of queues: Heavy traffic analysis of a simple open network, Queueing Systems 5 (1989) 265–280.

    Google Scholar 

  26. J.M. Harrison and L.M. Wein, Scheduling networks of queues: Heavy traffic analysis of a two-station closed network, Oper. Res. 38(6) (1990) 1052–1064.

    Google Scholar 

  27. F.P. Kelly and C.N. Laws, Dynamic routing in open queueing models: Brownian models, cut constraints and resource pooling, Queueing Systems 13 (1993) 47–86.

    Google Scholar 

  28. S. Kumar, Two-server closed networks in heavy traffic: Diffusion limits and asymptotic optimality, Ann. Appl. Probab. 10(3) (2000) 930–961.

    Google Scholar 

  29. S. Kumar and M. Muthuraman, A numerical method for solving singular Brownian control problems, in: Proc. IEEE Conf. on Decision and Control, 2000, to appear.

  30. P.R. Kumar and T.I. Seidman, Dynamic instabilities and stabilization methods in distributed real-time scheduling of manufacturing systems, IEEE Trans. Automat. Control 35(3) (1990) 289–298.

    Google Scholar 

  31. H.J. Kushner and P. Dupuis, Numerical Methods for Stochastic Control Problems in Continuous Time (Springer, New York, 1992).

    Google Scholar 

  32. H.J. Kushner and L.F. Martins, Heavy traffic analysis of a controlled multiclass queueing network via weak convergence methods, SIAM J. Control Optim. 34(5) (1996) 1781–1797.

    Google Scholar 

  33. C. Maglaras, Dynamic scheduling in multiclass queueing networks: Stability under discrete-review policies, Queueing Systems 31(3) (1999) 171–206.

    Google Scholar 

  34. C. Maglaras, Discrete-review policies for scheduling stochastic networks: Trajectory tracking and fluid-scale asymptotic optimality, Ann. Appl. Probab. 10(3) (2000) 897–929.

    Google Scholar 

  35. L.F. Martins, S.E. Shreve and H.M. Soner, Heavy traffic convergence of a controlled multiclass queueing network, SIAM J. Control Optim. 34(6) (1996) 2133–2171.

    Google Scholar 

  36. S.P. Meyn, Stability and optimization of queueing networks and their fluid models, in: Mathematics of Stochastic Manufacturing Systems, eds. G.G. Yin and Q. Zhang, Lectures in Applied Mathematics, Vol. 33 (Amer. Math. Soc., Providence, RI, 1997) pp. 175–200.

  37. S.P. Meyn, Sequencing and routing in multiclass queueing networks Part I: Feedback regulation, SIAM J. Control Optim. 40(3) (2001) 741–776.

    Google Scholar 

  38. S.P. Meyn, Sequencing and routing in multiclass queueing networks Part II: Workload relaxations (2001) submitted for publication.

  39. C.H. Papadimitriou and J.N. Tsitsiklis, The complexity of optimal queueing network control, Math. Oper. Res. 24(2) (1999) 293–305.

    Google Scholar 

  40. A.N. Rybko and A.L. Stolyar, Ergodicity of stochastic processes describing the operations of open queueing networks, Problems Inform. Transmission 28 (1992) 199–220.

    Google Scholar 

  41. S. Sethi and G. Sorger, A theory of rolling horizon decision making, Ann. Oper. Res. 29 (1991) 387–416.

    Google Scholar 

  42. A. Shwartz and A. Weiss, Large Deviations for Performance Analysis: Queues, Communications and Computing (Chapman & Hall, London, 1995).

    Google Scholar 

  43. Y.C. Teh, Dynamic scheduling for queueing networks derived from discrete-review policies, in:Analysis of Communication Networks: Call Centres, Traffic and Performance, eds. D.R. McDonald and S.R.E. Turner, Fields Institute Communications, Vol. 28 (Amer. Math. Soc., Providence, RI, 2000).

  44. R.J. Williams, On the approximation of queueing networks in heavy traffic, in: Stochastic Networks: Theory and Applications, eds. F. Kelly, S. Zachary and I. Ziedins (Oxford Univ. Press, Oxford,1996).

    Google Scholar 

  45. R.J. Williams, An invariance principle for semimartingale reflecting Brownian motion, Queueing Systems 30 (1998) 5–25.

    Google Scholar 

  46. R.J. Williams, Diffusion approximations for open multiclass queueing networks: Sufficient conditions involving for state space collapse, Queueing Systems 30 (1998) 27–88.

    Google Scholar 

  47. P. Yang, Least controls for a class of constrained linear stochastic systems, Math. Oper. Res. 18 (1993) 275–291.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Maglaras, C. Continuous-Review Tracking Policies for Dynamic Control of Stochastic Networks. Queueing Systems 43, 43–80 (2003). https://doi.org/10.1023/A:1021800414253

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1021800414253

Navigation