Skip to main content

Advertisement

Springer Nature Link
Log in

Risk measurement and risk-averse control of partially observable discrete-time Markov systems

  • Original Article
  • Published:
Mathematical Methods of Operations Research Aims and scope Submit manuscript

Abstract

We consider risk measurement in controlled partially observable Markov processes in discrete time. We introduce a new concept of conditional stochastic time consistency and we derive the structure of risk measures enjoying this property. We prove that they can be represented by a collection of static law invariant risk measures on the space of function of the observable part of the state. We also derive the corresponding dynamic programming equations. Finally we illustrate the results on a machine deterioration problem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  • Arlotto A, Gans N, Steele JM (2014) Markov decision problems where means bound variances. Oper Res 62(4):864–875

    Article  MathSciNet  Google Scholar 

  • Artzner P, Delbaen F, Eber J-M, Heath D, Ku H (2007) Coherent multiperiod risk adjusted values and Bellman’s principle. Ann Oper Res 152:5–22

    Article  MathSciNet  Google Scholar 

  • Aubin J-P, Frankowska H (2009) Set-valued analysis. Birkhäuser, Boston

    Book  Google Scholar 

  • Bäuerle N, Rieder U (2011) Markov decision processes with applications to finance. Universitext. Springer, Heidelberg

    Book  Google Scholar 

  • Bäuerle N, Rieder U (2013) More risk-sensitive Markov decision processes. Math Oper Res 39(1):105–120

    Article  MathSciNet  Google Scholar 

  • Bäuerle N, Rieder U (2017) Partially observable risk-sensitive Markov decision processes. Math Oper Res 42:1180–1196

    Article  MathSciNet  Google Scholar 

  • Bertsekas DP, Shreve SE (1978) Stochastic optimal control, volume 139 of mathematics in science and engineering. Academic Press, New York

    Google Scholar 

  • Çavus Ö, Ruszczyński A (2014a) Computational methods for risk-averse undiscounted transient Markov models. Oper Res 62(2):401–417

    Article  MathSciNet  Google Scholar 

  • Çavus Ö, Ruszczyński A (2014b) Risk-averse control of undiscounted transient Markov models. SIAM J Control Optim 52(6):3935–3966

    Article  MathSciNet  Google Scholar 

  • Chen Z, Li G, Zhao Y (2014) Time-consistent investment policies in Markovian markets: a case of mean-variance analysis. J Econ Dyn Control 40:293–316

    Article  MathSciNet  Google Scholar 

  • Cheridito P, Delbaen F, Kupper M (2006) Dynamic monetary risk measures for bounded discrete-time processes. Electron J Probab 11:57–106

    Article  MathSciNet  Google Scholar 

  • Cheridito P, Kupper M (2011) Composition of time-consistent dynamic monetary risk measures in discrete time. Int J Theor Appl Finance 14(01):137–162

    Article  MathSciNet  Google Scholar 

  • Chu S, Zhang Y (2014) Markov decision processes with iterated coherent risk measures. Int J Control 87(11):2286–2293

    MathSciNet  MATH  Google Scholar 

  • Coraluppi SP, Marcus SI (1999) Risk-sensitive and minimax control of discrete-time, finite-state Markov decision processes. Automatica 35(2):301–309

    Article  MathSciNet  Google Scholar 

  • Dai Pra P, Meneghini L, Runggaldier WJ (1998) Explicit solutions for multivariate, discrete-time control problems under uncertainty. Syst Control Lett 34(4):169–176

    Article  MathSciNet  Google Scholar 

  • Denardo EV, Rothblum UG (1979) Optimal stopping, exponential utility, and linear programming. Math Program 16(2):228–244

    Article  MathSciNet  Google Scholar 

  • Di Masi GB, Stettner Ł (1999) Risk-sensitive control of discrete-time Markov processes with infinite horizon. SIAM J Control Optim 38(1):61–78

    Article  MathSciNet  Google Scholar 

  • Fan J (2017) Process-based risk measures and risk-averse control of observable and partially observable discrete-time systems. Ph.D. Dissertation, Rutgers University

  • Fan J, Ruszczyński A (2016) Process-based risk measures and risk-averse control of discrete-time systems. arXiv:1411.2675

  • Feinberg EA, Kasyanov PO, Zgurovsky MZ (2016) Partially observable total-cost Markov decision processes with weakly continuous transition probabilities. Math Oper Res 41(2):656–681

    Article  MathSciNet  Google Scholar 

  • Fernández-Gaucherand E, Marcus SI (1997) Risk-sensitive optimal control of hidden Markov models: structural results. IEEE Trans Autom Control 42(10):1418–1422

    Article  MathSciNet  Google Scholar 

  • Filar JA, Kallenberg LCM, Lee H-M (1989) Variance-penalized Markov decision processes. Math Oper Res 14(1):147–161

    Article  MathSciNet  Google Scholar 

  • Föllmer H, Penner I (2006) Convex risk measures and the dynamics of their penalty functions. Stat Decis 24(1/2006):61–96

    MathSciNet  MATH  Google Scholar 

  • Hinderer K (1970) Foundations of non-stationary dynamic programming with discrete time parameter. Springer, Berlin

    Book  Google Scholar 

  • Howard RA, Matheson JE (1971/72) Risk-sensitive Markov decision processes. Manag Sci. 18:356–369

    Article  MathSciNet  Google Scholar 

  • James MR, Baras JS, Elliott RJ (1994) Risk-sensitive control and dynamic games for partially observed discrete-time nonlinear systems. IEEE Trans Autom Control 39(4):780–792

    Article  MathSciNet  Google Scholar 

  • Jaquette SC (1973) Markov decision processes with a new optimality criterion: discrete time. Ann Statist 1:496–505

    Article  MathSciNet  Google Scholar 

  • Jaśkiewicz A, Matkowski J, Nowak AS (2013) Persistently optimal policies in stochastic dynamic programming with generalized discounting. Math Oper Res 38(1):108–121

    Article  MathSciNet  Google Scholar 

  • Jobert A, Rogers LCG (2008) Valuations and dynamic convex risk measures. Math Finance 18(1):1–22

    Article  MathSciNet  Google Scholar 

  • Klöppel S, Schweizer M (2007) Dynamic indifference valuation via convex risk measures. Math Finance 17(4):599–627

    Article  MathSciNet  Google Scholar 

  • Kuratowski K, Ryll-Nardzewski C (1965) A general theorem on selectors. Bull Acad Polon Sci Ser Sci Math Astron Phys 13(1):397–403

    MathSciNet  MATH  Google Scholar 

  • Levitt S, Ben-Israel A (2001) On modeling risk in Markov decision processes. In: Rubinov A, Glover B (eds) Optimization and related topics . Applied Optimization, vol 47. Springer, Boston, MA, pp 27–40

    Google Scholar 

  • Lin K, Marcus SI (2013) Dynamic programming with non-convex risk-sensitive measures. In: American control conference (ACC), 2013, IEEE, pp 6778–6783

  • Mannor S, Tsitsiklis JN (2013) Algorithmic aspects of mean-variance optimization in Markov decision processes. Eur J Oper Res 231(3):645–653

    Article  MathSciNet  Google Scholar 

  • Marcus, SI, Fernández-Gaucherand E, Hernández-Hernández D, Coraluppi S, Fard P (1997) Risk sensitive Markov decision processes. In: Byrnes CI, Datta BN, Martin CF, Gilliam DS (eds) Systems and control in the twenty-first century. Systems & Control: Foundations & Applications, vol 22. Birkhäuser, Boston, MA, pp 263–279

    Chapter  Google Scholar 

  • Ogryczak W, Ruszczyński A (1999) From stochastic dominance to mean-risk models: semideviations as risk measures. Eur J Oper Res 116(1):33–50

    Article  Google Scholar 

  • Ogryczak W, Ruszczyński A (2001) On consistency of stochastic dominance and mean-semideviation models. Math Program 89(2):217–232

    Article  MathSciNet  Google Scholar 

  • Pflug ChG, Römisch W (2007) Modeling, measuring and managing risk. World Scientific, Singapore

    Book  Google Scholar 

  • Riedel F (2004) Dynamic coherent risk measures. Stoch Process Their Appl 112:185–200

    Article  MathSciNet  Google Scholar 

  • Roorda B, Schumacher JM, Engwerda J (2005) Coherent acceptability measures in multiperiod models. Math Finance 15(4):589–612

    Article  MathSciNet  Google Scholar 

  • Runggaldier WJ (1998) Concepts and methods for discrete and continuous time control under uncertainty. Insur Math Econ 22(1):25–39

    Article  MathSciNet  Google Scholar 

  • Ruszczyński A (2010) Risk-averse dynamic programming for Markov decision processes. Math Program 125(2, Ser. B):235–261

    Article  MathSciNet  Google Scholar 

  • Ruszczyński A, Shapiro A (2006a) Optimization of convex risk functions. Math Oper Res 31:433–542

    Article  MathSciNet  Google Scholar 

  • Ruszczyński A, Shapiro A (2006b) Conditional risk mappings. Math Oper Res 31:544–561

    Article  MathSciNet  Google Scholar 

  • Scandolo G (2003) Risk measures in a dynamic setting. Ph.D. thesis, Università degli Studi di Milano

  • Shen Y, Stannat W, Obermayer K (2013) Risk-sensitive Markov control processes. SIAM J Control Optim 51(5):3652–3672

    Article  MathSciNet  Google Scholar 

  • White DJ (1988) Mean, variance, and probabilistic criteria in finite Markov decision processes: a review. J Optim Theory Appl 56(1):1–29

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

Funding was provided by the National Science Foundation, Division of Mathematical Sciences (Grant No. 1312016).

Author information

Authors and Affiliations

  1. RUTCOR, Rutgers University, Piscataway, NJ, 08854, USA

    Jingnan Fan

  2. Department of Management Science and Information Systems, Rutgers University, Piscataway, NJ, 08854, USA

    Andrzej Ruszczyński

Authors
  1. Jingnan Fan
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Andrzej Ruszczyński
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Andrzej Ruszczyński.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, J., Ruszczyński, A. Risk measurement and risk-averse control of partially observable discrete-time Markov systems. Math Meth Oper Res 88, 161–184 (2018). https://doi.org/10.1007/s00186-018-0633-5

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00186-018-0633-5

Keywords