Abstract
This paper is concerned with dynamic control of stochastic processing networks. Specifically, it follows the so called “heavy traffic approach,” where a Brownian approximating model is formulated, an associated Brownian optimal control problem is solved, the solution of which is then used to define an implementable policy for the original system. A major challenge is the step of policy translation from the Brownian to the discrete network. This paper addresses this problem by defining a general and easily implementable family of continuous-review tracking policies. Each such policy has the following structure: at each point in time t, the controller observes the current vector of queue lengths q and chooses (i) a target position z(q) of where the system should be at some point in the near future, say at time t+l, and (ii) an allocation vector v(q) that describes how to split the server's processing capacity amongst job classes in order to steer the state from q to z(q). Implementation of such policies involves the enforcement of small safety stocks. In the context of the “heavy traffic” approach, the solution of the approximating Brownian control problem is used in selecting the target state z(q). The proposed tracking policy is shown to be asymptotically optimal in the heavy traffic limiting regime, where the Brownian model approximation becomes valid, for multiclass queueing networks that admit orthant Brownian optimal controls; this is a form of pathwise, or greedy, optimality. Several extensions are discussed.
Similar content being viewed by others
References
F. Avram, D. Bertsimas and M. Ricard, Fluid models of sequencing problems in open queueing networks; an optimal control approach, in: Stochastic Networks, eds. F. Kelly and R. Williams, Proceedings of the IMA, Vol. 71 (Springer, New York, 1995) pp. 199–234.
S. Axsäter, Continuous review policies for multi-level inventory systems with stochastic demand, in: Logistics of Production and Inventory, eds. S.C. Graves, A.H.G.R. Kan and P.H. Zipkin, Handbooks in Operations Research andManagement Science, Vol. 4 (North-Holland, Amsterdam, 1993) pp. 175– 197.
N. Bäuerle, Asymptotic optimality of tracking policies in stochastic networks, Ann. Appl. Probab. 10(4) (2000) 1065–1083.
S.L. Bell and R.J. Williams, Dynamic scheduling of a system with two parallel servers in heavy traffic with resource pooling: Asymptotic optimality of a threshold policy, Ann. Appl. Probab. 11 (2001) 608–649.
V. Böhm, On the continuity of the optimal policy set for linear programs, SIAM J. Appl. Math. 28(2) (1975) 303–306.
M. Bramson, Stability of two families of queueing networks and a discussion of fluid limits, Queueing Systems 28 (1998) 7–31.
M. Bramson, State space collapse with applications to heavy-traffic limits for multiclass queueing networks, Queueing Systems 30 (1998) 89–148.
M. Bramson and J. Dai, Heavy traffic limits for some queueing networks, Preprint (1999).
H. Chen, Fluid approximations and stability of multiclass queueing networks: Work-conserving policies, Ann. Appl. Probab. 5 (1995) 637–655.
H. Chen and D. Yao, Dynamic scheduling of a multiclass fluid network, Oper. Res. 41(6) (1993) 1104–1115.
M. Chen, C. Pandit and S. Meyn, In search of sensitivity in network optimization (2002) submitted for publication.
D. Clarke,Advances in Model Predictive Control (Oxford Univ. Press, Oxford, 1994).
J.G. Dai, On positive Harris recurrence of multiclass queueing networks: A unified approach via fluid limit models, Ann. Appl. Probab. 5 (1995) 49–77.
J. Dai and G. Weiss, A fluid heuristic for minimizing makespan in job-shops, Oper. Res. (2002) to appear.
A. Dembo and T. Zajik, Large deviations: From empirical mean and measure to partial sum processes, Stochastic Process. Appl. 57 (1995) 191–224.
A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Applications of Mathematics (Springer, New York, 1998).
P. Dupuis and K. Ramanan, A multiclass feedback queueing network with a regular Skorokhod problem, Preprint (1999).
J.M. Harrison, Brownian Motion and Stochastic Flow Systems (Wiley, New York, 1985).
J.M. Harrison, Brownian models of queueing networks with heterogeneous customer populations, in: Stochastic Differential Systems, Stochastic Control Theory and Applications, eds. W. Fleming and P.L. Lions, Proceedings of the IMA, Vol. 10 (Springer, New York, 1988) pp. 147–186.
J.M. Harrison, The BIGSTEP approach to flow management in stochastic processing networks, in: Stochastic Networks: Theory and Applications, eds. F. Kelly, S. Zachary and I. Ziedins (Oxford Univ. Press, Oxford, 1996) pp. 57–90.
J.M. Harrison, Heavy traffic analysis of a system with parallel servers: Asymptotic optimality of discrete-review policies, Ann. Appl. Probab. 8 (1998) 822–848.
J.M. Harrison, A broader view of Brownian networks (2000) submitted.
J.M. Harrison, Stochastic networks and activity analysis, in:Analytic Methods in Applied Probability, ed. Y. Suhov, In memory of Fridrih Karpelevich (Amer. Math. Soc., Providence, RI, 2002).
J.M. Harrison and J.A. Van Mieghem, Dynamic control of brownian networks: State space collapse and equivalent workload formulations, Ann. Appl. Probab. 7 (1996) 747–771.
J.M. Harrison and L.M. Wein, Scheduling network of queues: Heavy traffic analysis of a simple open network, Queueing Systems 5 (1989) 265–280.
J.M. Harrison and L.M. Wein, Scheduling networks of queues: Heavy traffic analysis of a two-station closed network, Oper. Res. 38(6) (1990) 1052–1064.
F.P. Kelly and C.N. Laws, Dynamic routing in open queueing models: Brownian models, cut constraints and resource pooling, Queueing Systems 13 (1993) 47–86.
S. Kumar, Two-server closed networks in heavy traffic: Diffusion limits and asymptotic optimality, Ann. Appl. Probab. 10(3) (2000) 930–961.
S. Kumar and M. Muthuraman, A numerical method for solving singular Brownian control problems, in: Proc. IEEE Conf. on Decision and Control, 2000, to appear.
P.R. Kumar and T.I. Seidman, Dynamic instabilities and stabilization methods in distributed real-time scheduling of manufacturing systems, IEEE Trans. Automat. Control 35(3) (1990) 289–298.
H.J. Kushner and P. Dupuis, Numerical Methods for Stochastic Control Problems in Continuous Time (Springer, New York, 1992).
H.J. Kushner and L.F. Martins, Heavy traffic analysis of a controlled multiclass queueing network via weak convergence methods, SIAM J. Control Optim. 34(5) (1996) 1781–1797.
C. Maglaras, Dynamic scheduling in multiclass queueing networks: Stability under discrete-review policies, Queueing Systems 31(3) (1999) 171–206.
C. Maglaras, Discrete-review policies for scheduling stochastic networks: Trajectory tracking and fluid-scale asymptotic optimality, Ann. Appl. Probab. 10(3) (2000) 897–929.
L.F. Martins, S.E. Shreve and H.M. Soner, Heavy traffic convergence of a controlled multiclass queueing network, SIAM J. Control Optim. 34(6) (1996) 2133–2171.
S.P. Meyn, Stability and optimization of queueing networks and their fluid models, in: Mathematics of Stochastic Manufacturing Systems, eds. G.G. Yin and Q. Zhang, Lectures in Applied Mathematics, Vol. 33 (Amer. Math. Soc., Providence, RI, 1997) pp. 175–200.
S.P. Meyn, Sequencing and routing in multiclass queueing networks Part I: Feedback regulation, SIAM J. Control Optim. 40(3) (2001) 741–776.
S.P. Meyn, Sequencing and routing in multiclass queueing networks Part II: Workload relaxations (2001) submitted for publication.
C.H. Papadimitriou and J.N. Tsitsiklis, The complexity of optimal queueing network control, Math. Oper. Res. 24(2) (1999) 293–305.
A.N. Rybko and A.L. Stolyar, Ergodicity of stochastic processes describing the operations of open queueing networks, Problems Inform. Transmission 28 (1992) 199–220.
S. Sethi and G. Sorger, A theory of rolling horizon decision making, Ann. Oper. Res. 29 (1991) 387–416.
A. Shwartz and A. Weiss, Large Deviations for Performance Analysis: Queues, Communications and Computing (Chapman & Hall, London, 1995).
Y.C. Teh, Dynamic scheduling for queueing networks derived from discrete-review policies, in:Analysis of Communication Networks: Call Centres, Traffic and Performance, eds. D.R. McDonald and S.R.E. Turner, Fields Institute Communications, Vol. 28 (Amer. Math. Soc., Providence, RI, 2000).
R.J. Williams, On the approximation of queueing networks in heavy traffic, in: Stochastic Networks: Theory and Applications, eds. F. Kelly, S. Zachary and I. Ziedins (Oxford Univ. Press, Oxford,1996).
R.J. Williams, An invariance principle for semimartingale reflecting Brownian motion, Queueing Systems 30 (1998) 5–25.
R.J. Williams, Diffusion approximations for open multiclass queueing networks: Sufficient conditions involving for state space collapse, Queueing Systems 30 (1998) 27–88.
P. Yang, Least controls for a class of constrained linear stochastic systems, Math. Oper. Res. 18 (1993) 275–291.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Maglaras, C. Continuous-Review Tracking Policies for Dynamic Control of Stochastic Networks. Queueing Systems 43, 43–80 (2003). https://doi.org/10.1023/A:1021800414253
Issue Date:
DOI: https://doi.org/10.1023/A:1021800414253