Abstract
In this paper we study the continuous-time Markov decision processes with a denumerable state space, a Borel action space, and unbounded transition and cost rates. The optimality criterion to be considered is the finite-horizon expected total cost criterion. Under the suitable conditions, we propose a finite approximation for the approximate computations of an optimal policy and the value function, and obtain the corresponding error estimations. Furthermore, our main results are illustrated with a controlled birth and death system.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bäuerle N, Rieder U (2011) Markov decision processes with applications to finance. Springer, Berlin
Gihman II, Skohorod AV (1979) Controlled stochastic processes. Springer, Berlin
Guo XP, Hernández-Lerma O (2009) Continuous-time Markov decision processes: theory and applications. Springer, Berlin
Guo XP, Huang XX, Huang YH (2015) Finite horizon optimality for continuous-time Markov decision processes with unbounded transition rates. Adv Appl Probab 47:1064–1087
Guo XP, Zhang WZ (2014) Convergence of controlled models and finite-state approximation for discounted continuous-time Markov decision processes with constraints. Euro J Oper Res 238:486–496
Kitaev MY, Rykov VV (1995) Controlled queueing systems. CRC Press, Boca Raton
Miller BL (1968) Finite state continuous time Markov decision processes with finite planning horizon. SIAM J Control 6:266–280
Pliska SR (1975) Controlled jump processes. Stoch Process Appl 3:259–282
Puterman ML (1994) Markov decision processes: discrete stochastic dynamic programming. Wiley, New York
van Dijk NM (1988) On the finite horizon Bellman equation for controlled Markov jump models with unbounded characteristics: existence and approximation. Stoch Process Appl 28:141–157
van Dijk NM (1989) A note on constructing \(\varepsilon \)-optimal policies for controlled Markov jump models with unbounded characteristics. Stochastics 27:51–58
Wei QD, Chen X (2014) Strong average optimality criterion for continuous-time Markov decision processes. Kybernetika 50:950–977
Yushkevich AA (1977) Controlled Markov models with countable state and continuous time. Theory Probab Appl 22:215–235
Acknowledgments
I am greatly indebted to the anonymous referees and the associate editor for many valuable comments and suggestions that have greatly improved the presentation. The research was supported by National Natural Science Foundation of China (Grant No. 11526092).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
I declare that no conflict of interest exists in this paper.
Rights and permissions
About this article
Cite this article
Wei, Q. Finite approximation for finite-horizon continuous-time Markov decision processes. 4OR-Q J Oper Res 15, 67–84 (2017). https://doi.org/10.1007/s10288-016-0321-3
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10288-016-0321-3
Keywords
- Continuous-time Markov decision processes
- Finite-horizon expected total cost criterion
- Unbounded transition rates
- Finite approximation