Risk-Sensitive Optimality Criteria in Markov Decision Processes

Abstract

The usual optimization criteria for Markov decision processes (e.g. total discounted reward or mean reward) can be quite insufficient to fully capture the various aspects for a decision maker. It may be preferable to select more sophisticated criteria that also reflect variability-risk features of the problem. To this end we focus attention on risk-sensitive optimality criteria (i.e. the case when expectation of the stream of rewards generated by the Markov processes evaluated by an exponential utility function is considered) and their connections with mean-variance optimality (i.e. the case when a suitable combination of the expected total reward and its variance, usually considered per transition, is selected as a reasonable optimality criterion). The research of risk-sensitive optimality criteria in Markov decision processes was initiated in the seminal paper by Howard and Matheson [6] and followed by many other researchers (see e.g. [1, 2, 3, 5, 4, 8, 9, 14]). In this note we consider a Markov decision chain X = X _n, n = 0,1, ... with finite state space $\mathcal{I}$ = 1,2, ..., N and a finite set $\mathcal{A}_i$ = 1,2, ..., K _i of possible decisions (actions) in state i ∈ $\mathcal{I}$. Supposing that in state i ∈ $\mathcal{I}$ action k ∈ $\mathcal{A}_i$ is selected, then state j is reached in the next transition with a given probability p ^k _ij and one-stage transition reward r _ij will be accrued to such transition.