A policy improvement method for constrained average Markov decision processes

Abstract

This brief paper presents a policy improvement method for constrained Markov decision processes (MDPs) with average cost criterion under an ergodicity assumption, extending Howard's policy improvement for MDPs. The improvement method induces a policy iteration-type algorithm that converges to a local optimal policy.

Introduction

Consider a Markov decision process [5], [10] (MDP) $M=(X,A,P,C)$ with a finite state set X, a finite admissible action set $A(x),x\in X$, a nonnegative cost function $C:X\times A(X)\to {\mathbb{R}}_{+}$, and a transition function P that maps $\{(x,a)|x\in X,a\in A(x)\}$ to the set of probability distributions over X. We denote the probability of making a transition to state $y\in X$ when taking action $a\in A(x)$ at state $x\in X$ by $P_{\mathit{xy}}^a$. We impose the following ergodicity assumption on P:

Assumption 1.1

Define $K≔\{(x,a)|x\in X,a\in A(x)\}$ and $P(y|k)≔P_{\mathit{xy}}^a$ for all $(x,a)\in K,y\in X$. There exists a positive number $\alpha <1$ such that

$$\underset{k,k^{\prime }\in K}{\max }\sum _{y\in X}|P(y|k)-P(y|k^{\prime })|⩽2\alpha \text{.}$$

Let $\Pi$ be the set of all Markovian (history-independent) stationary deterministic policies $\pi :X\to A(X)$. Define the objective function value of a policy $\pi \in \Pi$ with an initial state $x\in X$:

$$V^{\pi }(x)=\underset{H\to \infty }{\lim }\frac{1}{H}E\left[\left.\sum _{t=0}^{H-1}C(X_t,\pi (X_t))\right|X_0=x\right]\text{,}x\in X\text{,}$$

where $X_t$ is a random variable denoting state at time t. The MDP M is associated with a constant $\kappa \in \mathbb{R}$ and a constraint cost function $D:X\times A(X)\to {\mathbb{R}}_{+}$, where each feasible policy $\phi \in \Pi$ needs to satisfy the constraint inequality of $J^{\phi }(x)⩽\kappa ,x\in X$, where the constraint function value of $\phi$ is defined such that

$$J^{\phi }(x)=\underset{H\to \infty }{\lim }\frac{1}{H}E\left[\left.\sum _{t=0}^{H-1}D(X_t,{\phi }_t(X_t))\right|X_0=x\right]\text{,}x\in X\text{.}$$

The ergodicity assumption implies that for any policy $\pi \in \Pi$, $J^{\pi }(x)$ is independent of the starting state x [5, Lemma 3.3 (b.ii)], from which we write $J^{\pi }(x)$ as a constant $J^{\pi }$ omitting x (similarly for $V^{\pi })$. Throughout the paper, we further assume that the following feasibility condition holds:

Assumption 1.2

$\kappa \in [{\min }_{\pi \in \Pi }{J}^{\pi },\infty )$.

This paper considers the problem of designing a policy $\tilde{\phi }$ that improves a given feasible base-policy $\phi$ such that $V^{\tilde{\phi }}⩽V^{\phi }$ while satisfying $J^{\tilde{\phi }}⩽\kappa$. For the unconstrained case, Howard provided an improvement method [10] that induces the well-known policy iteration algorithm for solving MDPs. Based on his proof idea in unichain MDPs, we provide a simple proof on the policy improvement method presented below for the constrained case.

Even though there exists an extensive list of works that deal with solving/analyzing constrained MDPs via “direct method,” linear programming, the Lagrange approach, and the Pareto approach, (see [6] for the discussion and the references therein), to the author's best knowledge, there are few works on the policy improvement approach. Recently, Chang [1] studied policy improvement methods in constrained finite-horizon MDPs and infinite-horizon discounted MDPs that induce local optimal policy-iteration algorithms. This paper is a counterpart work on the average reward case.