Abstract
We deal with a very useful numerical method for both controlled and uncontrolled queuing and multiplexing type systems. The basic idea starts with a heavy traffic approximation, but it is shown that the results are very good even when working far from the heavy traffic regime. The underlying numerical method is a version of what is known as the Markov chain approximation method. It is a powerful methodology for controlled and uncontrolled stochastic systems, which can be approximated by diffusion or reflected diffusion type systems, and has been used with success on many other problems in stochastic control. We give a complete development of the relevant details, with an emphasis on multiplexing and particular queueing systems. The approximating process is a controlled or uncontrolled Markov chain which retains certain essential features of the original problem. This problem is generally substantially simpler than the original physical problem, and there are associated convergence theorems. The non-classical associated ergodic cost problem is derived, and put into a form such that reliable and good numerical algorithms, based on multigrid type ideas, can be used. Data for both controlled and uncontrolled problems shows the value of the method.
Similar content being viewed by others
References
M. Akian, Résolution numéricque d'équations d'Hamilton-Jacobi-Bellman au moyen d'algorithmes multigrilles et d'iterations sur les politiques, in:8th Conf. on Analysis and Optimization of Systems, Antibes, France (INRIA, 1988).
M. Akian, Méthodes multigrilles en controle stochastique, PhD thesis, University of Paris (1990).
D. Anick, D. Mitra and M.M. Sondhi, Stochastic theory of a data handling system with multiple sources, Bell Syst. Techn. J. 61 (1982) 1971–1894.
D.P. Bertsekas,Dynamic Programming: Deterministic and Stochastic Models (Prentice-Hall, Englewood Cliffs, NJ, 1987).
D.P. Bertsekas and D.A. Castañon, Adaptive aggregation methods for infinite horizon dynamic programming, IEEE Trans. Auto. Contr. AC-34 (1989) 589–598.
K.-L. Chung,Markov Chains with Stationary Transition Probabilities (Springer, Berlin, 1960).
J. Dai, Steady state analysis of reflected Brownian motions: characterization, numerical methods and queueing applications, PhD thesis, Operations Research Dept., Stanford University (1990).
A.I. Elwalid and D. Mitra, Fluid models for the analysis and design of statistical multiplexing with loss priorities on multiple classes of bursty traffic, in:Proc. IEEE INFOCOM'92 (IEEE Press, New York, 1992) pp. 415–425.
J.M. Harrison and V. Nguyen, The qnet method for two moment analysis of open queueing networks, Queueing Syst. 61 (1990) 1–32.
D.L. Iglehart and W. Whitt, Multiple channel queues in heavy traffic, Adv. Appl. Prob. 2 (1970) 150–177.
C. Knessl and J.A. Morrison, Heavy traffic analysis of a data handling system with multiple sources, SIAM J. Appl. Math. 51 (1991) 187–213.
H.J. Kushner,Probability Methods for Approximations in Stochastic Control and for Elliptic Equations (Academic Press, New York, 1977).
H.J. Kushner, Robustness and approximation of escape times and large deviations estimates for systems with small noise effects, SIAM J. Appl. Math. 44 (1984) 160–182.
H.J. Kushner, Numerical methods for stoachastic control problems in continuous time, SIAM J. Contr. Optim. 28 (1990) 999–1048.
H.J. Kushner and P. Dupuis,Numerical Methods for Stochastic Control Problems in Continuous Time (Springer, New York and Berlin, 1992).
H.J. Kushner and L.F. Martins, Heavy traffic analysis of a data transmission system with many independent sources, Technical report, Brown University, Lefschetz Center for Dynamical Systems, LCDS#91-12 (1991), SIAM J. Appl. Math., to appear.
H.J. Kushner and L.F. Martins, Numerical methods for stochastic singular control problems, SIAM J. Contr. Optim. 29 (1991) 1443–1475.
H.J. Kushner and L.F. Martins, Numerical methods for controlled and uncontrolled multiplexing and queueing systems, Technical report, Brown University, Lefschetz Center for Dynamical Systems (1993).
H.J. Kushner and K.M. Ramachandran, Optimal and approximately optimal control policies for queues in heavy traffic, SIAM J. Contr. Optim. 27 (1989) 1293–1318.
H.J. Kushner and J. Yang, Numerical methods for controlled routing in large trunk line systems, Technical report, Brown University, Lefschetz Center for Dynamical Systems (1992), to appear in ORSA J. Comp.
J.P. Lehoczky and S.E. Shreve, Absolutely continuous and singular stochastic control, Stochastics 17 (1986) 91–110.
L.F. Martins and H.J. Kushner, Routing and singular control for queueing networks in heavy traffic, SIAM J. Contr. Optim. 28 (1990) 1209–1233.
D. Mitra, Stochastic theory of a fluid model of producers and consumers coupled by buffer, Adv. Appl. Prob. 20 (1988) 646–676.
M.L. Puterman, Markov decision processes, in:Stochastic models, Vol. 2, eds. D.P. Heyman and M.J. Sobel (North-Holland, Amsterdam, 1991) chapter 8.
M.R. Reiman, Asymptotically optimal trunk reservation for large trunk groups, in:Proc. 28th Conf. on Decision and Control, New York (1989).
M.R. Reiman, Optimal trunk reservations for a critically loaded line, in:Teletraffic and Datatraffic in a Period of Change, eds. A. Jensen and V.B. Jensen (North-Holland Elsevier, 1991).
M. Taksar, Average optimal singular control and a related stopping problem, Math. Oper. Res. 10 (1985) 63–81.
L.M. Wein, Optimal control of a two station Brownian network, Math. Oper. Res. 15 (1990) 215–242.
D.J. White, Dynamic programming, Markov chains and the method of successive approximations, J. Math. Anal. Appl. 6 (1963) 373–376.
Author information
Authors and Affiliations
Additional information
Supported by ARO contract DAAL-03-92-G-0115, AFOSR contract F49620-92-J-0081, and DARPA contract AFOSR-91-0375.
Formerly at Brown University. Supported by DARPA contract AFOSR-91-0375.
Rights and permissions
About this article
Cite this article
Kushner, H.J., Martins, L.F. Numerical methods for controlled and uncontrolled multiplexing and queueing systems. Queueing Syst 16, 241–285 (1994). https://doi.org/10.1007/BF01158957
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF01158957