Skip to main content
Springer Nature Link
Log in

A poisson equation for the risk-sensitive average cost in semi-markov chains

  • Published:
Discrete Event Dynamic Systems Aims and scope Submit manuscript

Abstract

This work concerns semi-Markov chains evolving on a finite state space. The system development generates a cost when a transition is announced, as well as a holding cost which is incurred continuously during each sojourn time. It is assumed that these costs are paid by an observer with positive and constant risk-sensitivity, and the overall performance of the system is measured by the corresponding (long-run) risk-sensitive average cost criterion. In this framework, conditions are provided under which the average index does not depend on the initial state and is characterized in terms of a single Poisson equation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alanis-Duran A, Cavazos-Cadena R (2012) An optimality system for finite average Markov decision chains under risk-aversion. Kybernetika 48:83–104

    MathSciNet  MATH  Google Scholar 

  • Bauerle N, Rieder U (2011) Markov Decision Processes with Applications to Finance. Springer, New York

    Book  MATH  Google Scholar 

  • Bauerle N, Rieder U (2013) More risk-sensitive Markov decision processes. Math Oper Res 39:105–120

    Article  MathSciNet  MATH  Google Scholar 

  • Baykal-Gursoy M (2010) Semi-Markov decision processes. Wiley Encyclopedia of Operations Research and Management Sciences, Wiley, New York

    MATH  Google Scholar 

  • Cavazos-Cadena R (2009) Solutions of the average cost optimality equation for finite Markov decision chains: risk-sensitive and risk-neutral criteria. Math Method Oper Res 70:541–566

    Article  MathSciNet  MATH  Google Scholar 

  • Cavazos-Cadena R, Fernandez-Gaucherand E (1999) Controlled Markov chains with risk-sensitive criteria: Average cost, optimality equations, and optimal solutions. Math Method Oper Res 49:299–324

    MathSciNet  MATH  Google Scholar 

  • Cavazos-Cadena R, Fernandez-Gaucherand E (2002) Risk-sensitive control in communicating average Markov decision chains. In: Dror M, L’Ecuyer P, Szidarovsky F (eds) Modelling Uncertainty: An examination of Stochastic Theory, Methods and Applications. Kluwer, Boston, pp 525–544

  • Cavazos-Cadena R, Hernandez-Hernandez D (2004) A characterization of the average cost in finite controlled Markov chains. Ann Appl Probab 15:175–212

    Article  MathSciNet  MATH  Google Scholar 

  • Di Masi GB, Stettner L (1999) Risk-sensitive control of discrete time Markov processes with infinite horizon. SIAM J Control Optim 38:61–78

    Article  MathSciNet  MATH  Google Scholar 

  • Di Masi GB, Stettner L (2000) Infinite horizon risk sensitive control of discrete time Markov processes with small risk. SIAM J Control Optim 40:15–20

    MathSciNet  MATH  Google Scholar 

  • Di Masi GB, Stettner L (2007) Infinite horizon risk sensitive control of discrete time Markov processes under minorization property. SIAM J Control Optim 46:231–252

    Article  MathSciNet  MATH  Google Scholar 

  • Hernandez-Hernandez D, Marcus SI (1996) Risk sensitive control of Markov processes in countable state space. Syst Control Lett 29:147–155

    Article  MathSciNet  MATH  Google Scholar 

  • Hernandez-Hernandez D, Marcus SI (1999) Existence of risk sensitive optimal stationary polices for controlled Markov processes. Appl Math Opt 40:273–285

    Article  MathSciNet  MATH  Google Scholar 

  • Howard RA, Matheson JE (1972) Risk-sensitive Markov decision processes. Manage Sci 18:356–369

    Article  MathSciNet  MATH  Google Scholar 

  • Hu Q, Yue W (2003) Optimal replacement of a system according to a semi-Markov decision process in a semi-Markov environment. Optim Method Softw 18:181–196

    Article  MathSciNet  MATH  Google Scholar 

  • Jaskiewicz A (2007) Average optimality for risk sensitive control with general state space. Ann Appl Probab 17:654–675

    Article  MathSciNet  MATH  Google Scholar 

  • Jianyong L, Xiaobo Z (2004) On average reward semi-Markov decision processes with a general multichain structure. Math Oper Res 29:339–352

    Article  MathSciNet  MATH  Google Scholar 

  • Luque-Vasquez F, Hernandez-Lerma O (1999) Semi-Markov control models with average costs. Applicationes Mathematicae 26:315–331

    MathSciNet  MATH  Google Scholar 

  • Pinedo M (2008) Scheduling: Theory, Algorithms and Systems. Springer, New York

    MATH  Google Scholar 

  • Schweitzer PJ (1971) Iterative solutions of the functional equations of undiscounted Markov renewal programming. J Math Anal Appl 34:495–501

    Article  MathSciNet  MATH  Google Scholar 

  • Sennott LI (1999) Stochastic Dynamic Programming and the Control of Queueing Systems. Wiley, New York

    MATH  Google Scholar 

  • Stidham Jr S, Weber RR (1993) A survey of Markov decision models for control of networks of queues. Queueing Syst 13:291–314

    Article  MathSciNet  MATH  Google Scholar 

  • Tijms HC (2003) A first course in stochastic models. Wiley, New York

    Book  MATH  Google Scholar 

Download references

Acknowledgments

The author is deeply grateful to the reviewers and the Associate Editor for their careful reading of the original manuscript, and for their helpful suggestions to improve the paper.

Author information

Authors and Affiliations

  1. Departamento de Estadísitca y Cálculo, Universidad Autónoma Agraria Antonio Narro, Boulevard Antonio Narro 1923, Buenavista, Saltillo, COAH, 25315, México

    Rolando Cavazos-Cadena

Authors
  1. Rolando Cavazos-Cadena
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rolando Cavazos-Cadena.

Additional information

Dedicated to Professor Onésimo Hernández-Lerma, on the occasion of his seventieth birthday.

This work was partially supported by the PSF Organization under Grant No. 300-01-15, and by PRODEP under Grant No. 17332-CA-23.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cavazos-Cadena, R. A poisson equation for the risk-sensitive average cost in semi-markov chains. Discrete Event Dyn Syst 26, 633–656 (2016). https://doi.org/10.1007/s10626-015-0224-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10626-015-0224-z

Keywords