Skip to main content
SpringerLink
Log in

Markov control processes with randomized discounted cost

  • Original Article
  • Published:
Mathematical Methods of Operations Research Aims and scope Submit manuscript

Abstract

In this paper we consider Markov Decision Processes with discounted cost and a random rate in Borel spaces. We establish the dynamic programming algorithm in finite and infinity horizon cases. We provide conditions for the existence of measurable selectors. And we show an example of consumption-investment problem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abbad M, Daoui C (2003) Hierarchical algorithms for discounted Markov decision processes. Math Methods Oper Res 58:237–245

    Article  MATH  MathSciNet  Google Scholar 

  • Altman E (1999) Constrained Markov decision processes. Chapman & Hall/CRC, Boca Raton

    MATH  Google Scholar 

  • Bertsekas DP (1995) Dynamic programming and optimal control. Athena Scientific, Belmont

    MATH  Google Scholar 

  • Berument H, Kilinc Z, Ozlale U (2004) The effects of different inflation risk prepius on interest rate spreads. Physica A 333:317–324

    Article  MathSciNet  Google Scholar 

  • Borkar VS (1991) Topics in controlled Markov chains. Longman, New York

    MATH  Google Scholar 

  • Cai J, Dickson D, David CM (2004) Ruin probabilities with a Markov chain interest model. Insur Math Econ 35:513–525

    Article  MATH  Google Scholar 

  • Feinberg EA, Shwartz A (1994) Markov decision models with weighted discounted criteria. Math Oper Res 19:152–168

    MATH  MathSciNet  Google Scholar 

  • Feinberg EA, Shwartz A (1995) Constrained Markov decision models with weighted discounted rewards. Math Oper Res 20:302–320

    Article  MATH  MathSciNet  Google Scholar 

  • Feinberg EA, Shwartz A (1999) Constrained dynamic programming with two discount factors: applications and an algorithm. IEEE Trans Autom Control 42:628–631

    Article  MathSciNet  Google Scholar 

  • Gil A, Luis A (2004) Modelling the U.S. interest rate in terms of I(d) statistical models. Q Rev Econ Finance 44:475–486

    Article  Google Scholar 

  • Haberman S, Sung J (2005) Optimal pension funding dynamics over infinite control horizon when stochastic rates of return are stationary. Insur Math Econ 36:103–116

    Article  MATH  MathSciNet  Google Scholar 

  • Hernández-Lerma O, González-Hernández J (2000) Constrained Markov control processes in Borel spaces: the discounted case. Math Meth Oper Res, 52:271–285

    Article  MATH  Google Scholar 

  • Hernández-Lerma O, Lasserre JB (1996) Discrete-time Markov control processes: basic optimality criteria. Springer, Berlin Heidelberg New York

    Google Scholar 

  • Hinderer K (1970) Foundations of non-stationary dynamical programming with discrete time parameter Lecture notes in operation research, vol 33. Springer, Berlin Heidelberg New York

    Google Scholar 

  • Hu Q (2003) Analysis for some properties of discrete time Markov decision processes. Optimization 52:495–505

    Article  MATH  MathSciNet  Google Scholar 

  • Kurano M, Song J, Hosaka M, Huang Y (1998) Controlled Markov set-chains with discounting. J Appl Prob 35:293–302

    Article  MATH  MathSciNet  Google Scholar 

  • Lee P, Rosenfield DB (2005) When to refinance a mortgage: a dynamic programming approach. Eur J Oper Res 166:266–277

    Article  MATH  Google Scholar 

  • Lettau M, Uhlig H (1999) Rules of thumb versus dynamic programming. Am Econ Rev 89(1):148–174

    Article  Google Scholar 

  • Liu K (1999) Weighted discounted Markov decision processes with perturbation. Acta Math Appl Sinica 15:183–189

    Google Scholar 

  • López-Martínez RR, Hernández-Lerma O (2003) The Lagrange approach to constrained Markov processes: a survey and extension of results. Morfismos 7:1–26

    Google Scholar 

  • Newell RG, Pizer WA (2003) Discounting the distant future: how much do uncertain rates increase valuation?. J Environ Econ Manage 46:52–71

    Article  MATH  Google Scholar 

  • Mao X, Piunovskiy AB (2000) Strategic measure in optimal control problems for stochastic sequences. Stoch Anal Appl 18:755–776

    MATH  MathSciNet  Google Scholar 

  • Michael K Ng (1999) A note on policy algorithms for discounted Markov decision problems. Oper Res Lett 25:195–197

    Article  MATH  MathSciNet  Google Scholar 

  • Ogaki M, Santaella JA (2000) The exchange rate and the term structure of interest rates in México. J Dev Econ 63:135–155

    Article  Google Scholar 

  • Piunovskiy AB (1997) Optimal control of random sequences in problems with constraints. Kluwer, Boston

    MATH  Google Scholar 

  • Puterman ML (1994) Markov decision processes: discrete stochastic dynamic programming. Wiley, New York

    MATH  Google Scholar 

  • Sack B, Wieland V (2000) Interest-rate smoothing and optimal monetary policy: a review of recent empirical evidence. J Econ Bus 52:205–228

    Article  Google Scholar 

  • Shwartz A (2001) Death and discounting. IEEE Trans Autom Control 46:644–647

    Article  MATH  MathSciNet  Google Scholar 

  • Stockey NL, Lucas RE Jr (1989) Recursive methods in economic dynamics. Harvard University Press, Cambridge

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento de Probabilidad y Estadística, IIMAS-UNAM, A. Postal 20–726, Mexico D.F., 01000, Mexico

    Juan González-Hernández

  2. Facultad de Matemáticas UV, A. Postal 270, Xalapa, Ver.91090, Mexico

    Raquiel R. López-Martínez

  3. Departamento de Ciencias Básicas, UPIITA-IPN, Mexico, Mexico

    J. Rubén Pérez-Hernández

Authors
  1. Juan González-Hernández
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raquiel R. López-Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Rubén Pérez-Hernández
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Raquiel R. López-Martínez.

Additional information

This research was partially supported by the PROMEP grant 103.5/05/40.

Rights and permissions

Reprints and permissions

About this article

Cite this article

González-Hernández, J., López-Martínez, R.R. & Pérez-Hernández, J.R. Markov control processes with randomized discounted cost. Math Meth Oper Res 65, 27–44 (2007). https://doi.org/10.1007/s00186-006-0092-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00186-006-0092-2

Mathematics Subject Classification (2000)

Keywords

Search

Navigation