The control of a two-level Markov decision process by time aggregation

Abstract

The solution of Markov Decision Processes (MDPs) often relies on special properties of the processes. For two-level MDPs, the difference in the rates of state changes of the upper and lower levels has led to limiting or approximate solutions of such problems. In this paper, we solve a two-level MDP without making any assumption on the rates of state changes of the two levels. We first show that such a two-level MDP is a non-standard one where the optimal actions of different states can be related to each other. Then we give assumptions (conditions) under which such a specially constrained MDP can be solved by policy iteration. We further show that the computational effort can be reduced by decomposing the MDP. A two-level MDP with M upper-level states can be decomposed into one MDP for the upper level and M to $M(M-1)$ MDPs for the lower level, depending on the structure of the two-level MDP. The upper-level MDP is solved by time aggregation, a technique introduced in a recent paper [Cao, X.-R., Ren, Z. Y., Bhatnagar, S., Fu, M., & Marcus, S. (2002). A time aggregation approach to Markov decision processes. Automatica, 38(6), 929–943.], and the lower-level MDPs are solved by embedded Markov chains.

Introduction

In Markov Decision Processes (MDPs) of two-level hierarchical structures, the states are formed by the status of both the upper and the lower levels and state changes can be caused by the status changes at either level. Decisions are also made at each level. Such decisions affect both the state transitions of the two levels and the reward of the MDPs. The existing solutions of such MDPs often rely on the difference in the time scales of the two levels, i.e., the rate of state changes in the lower level is faster than that of the upper level by multiple orders of magnitude. Between two upper-level state changes, on average the lower level has already gone through so many state changes that one can make use of the long-run sum or average from the lower level to make decision at an upper-level state change. See Chang, Fard, Marcus, and Shayman (2003) and its references for examples of MDPs with multiple time scales.

In a different context and for a different purpose, singularly perturbed MDPs also make use of two time scales (Abbad et al., 1992, Bielecki and Filar, 1991). Reducible transition probability matrices of policies in the form of disjoint, closed communicating classes are made positive by perturbing with a policy dependent matrix $\varepsilon D$, where $\varepsilon >0$ is a constant. By controlling $\varepsilon$, the inter-class state transitions can be less frequent than the intra-class. One main result of singularly perturbed MDPs is that the optimal control policy for the “limit control MDPs”, the cases as $\varepsilon \to 0$, is a good control policy for singularly perturbed MDPs of a sufficiently small $\varepsilon$.

The idea of two time scales also occurs in hybrid stochastic systems (Filar et al., 2001, Filar and Haurie, 2001). The operation modes in the upper level are modeled by Markov jump processes and the characteristics of the lower level are modeled either by deterministic functions (Filar et al., 2001) or by diffusion processes (Filar & Haurie, 2001), with both the upper-level operations modes and the lower-level characteristics controllable.

In this paper, we study a class of the two-level MDPs under the long-run average reward criterion. Our two-level MDPs take a structure similar to MDPs of two time scales except that our two-level model is an atypical MDP: an upper-level decision at a state can affect the state transitions of a group of states with the same upper-level state. The various decisions, within the lower levels and spanning across both levels, are coupled, i.e., we cannot single out and solve the decisions level by level, purpose by purpose. Such coupling effect increases the computational burden to solve the two-level MDP, making it practically infeasible for real-life problems. We then show that the coupling effect disappears if (i) the sojourn times of each upper-level state—the duration between entering and leaving an upper-level state—are uncontrollable, and (ii) the set of the initial lower-level state distributions after an upper-level state change is independent of the lower-level states before the upper-level state change. With both of these assumptions, the decisions are decoupled; the computational effort for the optimal policy becomes manageable; and it is possible to implement the centralized control scheme in a decentralized fashion.

To further reduce the computational effort, we show that the whole problem can be decomposed into smaller MDPs, one for the upper level, and a number of MDPs for the lower level, where the number depends on the problem structure. Our solution of the upper-level MDP uses time aggregation (Cao, Ren, Bhatnagar, Fu, & Marcus, 2002), and our lower-level of embedded Markov chains. Combining the algorithms of the two levels solves the two-level MDP.

Our model is related to that in Chang et al. (2003), though the two models have two critical differences. First, the number of lower-level state changes between two upper-level state changes is fixed in Chang et al. (2003) but random here. With constant time between two upper-level state changes, it is conceptually straightforward to embed on upper-level state changes and make use of the total rewards in upper-level sojourn times to look for optimal decisions. Nonetheless, the computational effort for all possible nonstationary policies is too large to be materialized. Consequently, Chang et al. (2003) looks for approximations, and bounds the performance of approximations in the same spirit of those in single-level MDPs. As for our model, the calculation of the total rewards of sojourn times is made involved by randomness. We still find computationally simple close-form expressions for such quantities, based on which later we give an exact analysis of the two-level MDP. Second, the upper- and lower-level decisions in Chang et al. (2003) are not coupled as ours. There, any consideration for a state, e.g., its upper-level decision, can be made purely based on the cost and benefit of the state. However, in our model, as shown in (1) below, states with the same upper level must take the same upper-level decisions, a constraint that makes our MDP unconventional.

Our contributions are as follows. First, we tackle a two-level MDP from a perspective that does not rely on multiple time scales. Our analysis is exact, and we allow for random number of lower-level state changes between two upper-level state changes. Second, our two-level control problem is not a standard MDP because each action at the upper level applies to a group of states. We solve such a specially constrained MDP under different assumptions. Third, other than satisfying with an algorithm that solves the two-level MDP, we go further to find algorithms that take less computational effort, which is a continuation of one of the authors’ previous work on time aggregation (Cao et al., 2002). Such algorithms can be implemented as the optimal decentralized control. Finally, our approach sheds light on hybrid systems where the lower level is modeled as continuous systems.