Computation of weighted sums of rewards for concurrent MDPs

Abstract

We consider sets of Markov decision processes (MDPs) with shared state and action spaces and assume that the individual MDPs in such a set represent different scenarios for a system’s operation. In this setting, we solve the problem of finding a single policy that performs well under each of these scenarios by considering the weighted sum of value vectors for each of the scenarios. Several solution approaches as well as the general complexity of the problem are discussed and algorithms that are based on these solution approaches are presented. Finally, we compare the derived algorithms on a set of benchmark problems.

Appendices

Appendix A: Action dependent rewards

Assume that rewards of a concurrent MDP depend on actions and successor states. Then ${\varvec{R}}_k^a(s,s')$ is the reward that is obtained in state s of MDP k if action a is chosen and $s'$ is the successor state. Let

$$\begin{aligned} \left( {\mathcal {S}}, \varvec{\alpha }, \left( {\varvec{P}}^a_k\right) _{a \in {\mathcal {A}}}, \left( {\varvec{R}}^a_k\right) _{a \in {\mathcal {A}}}\right) \end{aligned}$$

(27)

be a concurrent MDP with action-dependent rewards. We assume that this MDP should be analyzed for the discounted reward with discount factor $\gamma \in ({0,1})$. The transformation into an MDP with rewards that do not depend on the action results in the following MDP which depends on $\gamma$.

$$\begin{aligned} \begin{array}{l} \left( {\tilde{{\mathcal {S}}}}, \varvec{{\tilde{\alpha }}}, \left( {\varvec{{\tilde{P}}}}^a_k\right) _{a \in {\mathcal {A}}}, \varvec{{\tilde{r}}}\right) , \text{ where } \\ {\tilde{{\mathcal {S}}}} = {\mathcal {S}}\cup \left\{ (s,a,s') | s, s' \in {\mathcal {S}}, a \in {\mathcal {A}}, {\varvec{P}}_k^a(s,a,s') > 0\right\} \\ \varvec{{\tilde{\alpha }}}\in {{\mathbb {R}}}_{\ge 0}^{|{\tilde{{\mathcal {S}}}}|\times 1}, \varvec{{\tilde{\alpha }}}\mathbb {1}= 1 \text{ with } \varvec{{\tilde{\alpha }}}(\varvec{\sigma }) = \left\{ \begin{array}{ll} \varvec{\alpha }(s) &{} \text{ if } \varvec{\sigma } = s, \\ 0 &{} \text{ otherwise, } \end{array}\right. \\ {\varvec{{\tilde{P}}}}_k^a \in {{\mathbb {R}}}^{|{\tilde{{\mathcal {S}}}}| \times |{\tilde{{\mathcal {S}}}}|}_{\ge 0}, {\varvec{{\tilde{P}}}}_k^a\mathbb {1}= \mathbb {1} \text{ with } {\varvec{{\tilde{P}}}}_k^a(\varvec{\sigma },\varvec{\sigma '}) = \left\{ \begin{array}{ll} {\varvec{P}}_k^a(s,s') &{} \text{ if } \varvec{\sigma } = s \text{ and } \varvec{\sigma '} = (s,a,s'),\\ 1.0 &{} \text{ if } \varvec{\sigma } = (s, a, s') \text{ and } \varvec{\sigma '} = s', \\ 0 &{} \text{ otherwise, } \end{array}\right. \\ \varvec{{\tilde{r}}}\in {{\mathbb {R}}}^{1 \times |{\tilde{{\mathcal {S}}}}|} \text{ with } \varvec{{\tilde{r}}}(\varvec{\sigma }) = \left\{ \begin{array}{ll} \frac{{\varvec{R}}^a_k(s,s')}{\sqrt{\gamma }} &{} \text{ if } \varvec{\sigma } = (s, a, s'),\\ 0 &{} \text{ otherwise. } \end{array}\right. \end{array}\nonumber \\ \end{aligned}$$

(28)

Observe that the state space ${\tilde{{\mathcal {S}}}}$ consists of states $s \in {\mathcal {S}}$ and states $(s,a,s')$. The modified MDP alternates between states $s \in {\tilde{{\mathcal {S}}}} \cap {\mathcal {S}}$ and states $(s,a,s') \in {\tilde{{\mathcal {S}}}} {\setminus } {\mathcal {S}}$. Rewards are only gained in the latter states.

For some policy ${\varvec{{\varPi }}}$ defined on ${\mathcal {S}}$ we define a policy ${\varvec{{\tilde{{\varPi }}}}}$ on ${\tilde{{\mathcal {S}}}}$ as follows

$$\begin{aligned} \varvec{{\tilde{\pi }}}_{\varvec{\sigma }}(a) = \left\{ \begin{array}{ll} \varvec{\pi }_s(a) &{}\quad \text{ if } \varvec{\sigma }=s \in {\mathcal {S}}, \\ 1 &{} \quad \text{ if } \varvec{\sigma } = (s, a, s'),\\ 0 &{} \quad \text{ otherwise. } \end{array}\right. \end{aligned}$$

Thus, a policy for the MDP with action-dependent rewards can be uniquely translated into a policy for the MDP with action-independent rewards. Define the following sequences of vectors

$$\begin{aligned} \begin{array}{lll} \varvec{g}^0 = {\mathbf {0}}\in {{\mathbb {R}}}^{|{\mathcal {S}}| \times 1}, &{} \varvec{g}^h(s) = \sum \limits _{s' \in {\mathcal {S}}} \sum \limits _{a \in {\mathcal {A}}} \varvec{\pi }_s(a){\varvec{P}}_k^a(s,s')\left( {\varvec{R}}_k^a(s,s')+ \gamma \varvec{g}^{h-1}(s')\right) \\ \varvec{{\tilde{g}}}^0 = {\mathbf {0}}\in {{\mathbb {R}}}^{|{\tilde{{\mathcal {S}}}}| \times 1}, &{} \varvec{{\tilde{g}}}^h(\varvec{\sigma }) = \sum \limits _{\varvec{\sigma '} \in {\tilde{{\mathcal {S}}}}} \sum \limits _{a \in {\mathcal {A}}} \varvec{{\tilde{\pi }}}_s(a){\varvec{{\tilde{P}}}}_k^a(\varvec{\sigma },\varvec{\sigma '})\left( \varvec{{\tilde{r}}}(\varvec{\sigma })+ \sqrt{\gamma } \varvec{{\tilde{g}}}^{h-1}(\varvec{\sigma '})\right) \end{array} \end{aligned}$$

The vectors $\varvec{g}^h$ and $\varvec{{\tilde{g}}}^h$ contain the expected discounted rewards over the course of h transitions under the policy ${\varvec{{\varPi }}}$ resp. ${\varvec{{\tilde{{\varPi }}}}}$. Now we show the main relation between $\varvec{g}^h$ and $\varvec{{\tilde{g}}}^h$.

Theorem 6

For all $s \in {\mathcal {S}}$ and all $h \in {{\mathbb {N}}}_0$ it is $\varvec{g}^h(s) = \varvec{{\tilde{g}}}^{2h}(s) = \varvec{{\tilde{g}}}^{2h+1}(s)$.

Proof

We prove the correspondence of the values in the vectors for h and 2h by induction. For $h=0$ we have by definition of the vectors $\varvec{g}^0(s) = \varvec{{\tilde{g}}}^0(s) = 0$. Now assume that the correspondence holds for $h \ge 0$, then

$$\begin{aligned} \begin{array}{lll} \varvec{{\tilde{g}}}^{2h+2}(s) &{} = \sum \limits _{s'\in {\mathcal {S}}}\sum \limits _{a \in {\mathcal {A}}} \varvec{{\tilde{\pi }}}_s(a){\varvec{{\tilde{P}}}}_k^a(s,(s,a,s')) \left( \varvec{{\tilde{r}}}(s) + \sqrt{\gamma } \varvec{{\tilde{g}}}^{2h+1}(s,a,s')\right) \\ &{} = \sum \limits _{s'\in {\mathcal {S}}}\sum \limits _{a \in {\mathcal {A}}} \varvec{\pi }_s(a){\varvec{P}}_k^a(s,s')\left( \sum \limits _{s''\in {\mathcal {S}}}\sum \limits _{a' \in {\mathcal {A}}} \varvec{{\tilde{\pi }}}_{(s,a,s')}(a'){\varvec{{\tilde{P}}}}_k^{a'}((s,a,s'), s'') \right. \\ &{} \quad \left. \times \left( \sqrt{\gamma } \varvec{{\tilde{r}}}(s,a,s') + \gamma \varvec{{\tilde{g}}}^{2h}(s'') \right) \right) \\ &{} = \sum \limits _{s'\in {\mathcal {S}}}\sum \limits _{a \in {\mathcal {A}}} \varvec{\pi }_s(a){\varvec{P}}_k^a(s,s') \left( \sqrt{\gamma } \varvec{{\tilde{r}}}(s,a,s') + \gamma \varvec{{\tilde{g}}}^{2h}(s') \right) \\ &{} = \sum \limits _{s'\in {\mathcal {S}}}\sum \limits _{a \in {\mathcal {A}}} \varvec{\pi }_s(a){\varvec{P}}_k^a(s,s')\left( \sqrt{\gamma } \frac{{\varvec{R}}_k^a(s,s')}{\sqrt{\gamma }} + \gamma \varvec{g}^h(s')\right) \\ &{} = \varvec{g}^{h+1}(s) \end{array} \end{aligned}$$

which implies that it holds for $h+1$. Now we consider $\varvec{{\tilde{g}}}^{2h+1}(s)$. For $h = 0$ we have

$$\begin{aligned} \begin{array}{lll} \varvec{{\tilde{g}}}^1(s) = \sum \limits _{s'\in {\mathcal {S}}}\sum \limits _{a \in {\mathcal {A}}} \varvec{\pi }_s(a){\varvec{{\tilde{P}}}}_k^a(s,(s,a,s')) \left( \varvec{{\tilde{r}}}(s) + \sqrt{\gamma }\varvec{{\tilde{g}}}^{0}(s,a,s')\right) = 0 \end{array} \end{aligned}$$

because $\varvec{{\tilde{r}}}(s) = 0$ and $\varvec{{\tilde{g}}}^0={\mathbf {0}}$. Now assume that the result holds for $2h+1$, we show that it holds for $2h+3$.

$$\begin{aligned} \begin{array}{lll} \varvec{{\tilde{g}}}^{2h+3}(s) &{} = \sum \limits _{s'\in {\mathcal {S}}}\sum \limits _{a \in {\mathcal {A}}} \varvec{{\tilde{\pi }}}_s(a){\varvec{{\tilde{P}}}}_k^a(s,(s,a,s')) \left( \varvec{{\tilde{r}}}(s) + \sqrt{\gamma }\varvec{{\tilde{g}}}^{2h+2}(s,a,s')\right) \\ &{} = \sum \limits _{s'\in {\mathcal {S}}}\sum \limits _{a \in {\mathcal {A}}} \varvec{\pi }_s(a){\varvec{P}}_k^a(s,s')\left( \sum \limits _{s''\in {\mathcal {S}}}\sum \limits _{a' \in {\mathcal {A}}} \varvec{{\tilde{\pi }}}_{(s,a,s')}(a'){\varvec{{\tilde{P}}}}_k^{a'}((s,a,s'), s'')\right. \\ &{} \quad \times \left. \left( \sqrt{\gamma }\varvec{{\tilde{r}}}(s,a,s') + \gamma \varvec{{\tilde{g}}}^{2h+1}(s'') \right) \right) \\ &{} = \sum \limits _{s'\in {\mathcal {S}}}\sum \limits _{a \in {\mathcal {A}}} \varvec{\pi }_s(a){\varvec{P}}_k^a(s,s')\left( \sqrt{\gamma }\frac{{\varvec{R}}_k^a(s,s')}{\sqrt{\gamma }} +\gamma \varvec{g}^h(s')\right) \\ &{} = \varvec{g}^{h+1}(s) \end{array} \end{aligned}$$

$\square$

Let $G^{\varvec{{\varPi }}}(\gamma ) = \varvec{\alpha }\lim _{h \rightarrow \infty } \varvec{g}^h$ and ${\tilde{G}}^{\varvec{{\tilde{{\varPi }}}}}(\sqrt{\gamma }) = \varvec{{\tilde{\alpha }}}\lim _{h \rightarrow \infty } \varvec{{\tilde{g}}}^h$ the discounted gains for the two MDPs with discount factors $\gamma \in ({0,1})$ and $\sqrt{\gamma }$, respectively.

Theorem 7

For any policy ${\varvec{{\varPi }}}$ and any discount factor $\gamma \in ({0,1})$ the relation

$$\begin{aligned} G^{\varvec{{\varPi }}}(\gamma ) = {\tilde{G}}^{\varvec{{\tilde{{\varPi }}}}}(\sqrt{\gamma }) \end{aligned}$$

holds.

Proof

First, we show the existence of $G^{\varvec{{\varPi }}}(\gamma )$. Since ${\varvec{R}}^a_k(s, s')$ is finite for all $a \in {\mathcal {A}}$ and for all $s, s' \in {\mathcal {S}}$ and can be bounded by a real value $R \ge \left| {{\varvec{R}}^a_k(s, s')}\right|$, the sequence $\left( \varvec{g}^h\right) _{h \in {{\mathbb {N}}}_0}$ adheres to

$$\begin{aligned} \left| {\varvec{g}^{h+i}(s) - \varvec{g}^i(s)}\right|&= \sum \limits _{s' \in {\mathcal {S}}} \sum \limits _{a \in {\mathcal {A}}} \varvec{\pi }_s(a) {\varvec{P}}^a_k(s, s') \left| {\varvec{R}}^a_k(s, s') + \gamma \varvec{g}^{h+i - 1}(s') - {\varvec{R}}^a_k(s, s') \right. \\&\quad \left. - \gamma \varvec{g}^{i - 1}(s') \right| \\&\le \gamma \max \limits _{s' \in {\mathcal {S}}} \left| { \varvec{g}^{h+i - 1}(s') - \varvec{g}^{i - 1}(s') }\right| . \end{aligned}$$

By induction, it follows that

$$\begin{aligned} \left| {\varvec{g}^{h+i}(s) - \varvec{g}^i(s)}\right|&\le \gamma ^i \max \limits _{s' \in {\mathcal {S}}} \left| {\varvec{g}^h(s')}\right| \\&\le \gamma ^i R \sum \limits _{j = 0}^h \gamma ^j \\&= \gamma ^i R \frac{1 - \gamma ^h}{1 - \gamma } \\&\le \gamma ^i \frac{R}{1 - \gamma } \end{aligned}$$

for $i \in {{\mathbb {N}}}$ which implies the Cauchy convergence criterion. Then Theorem 6 implies that $\varvec{g}(s) = \lim _{h \rightarrow \infty } \varvec{g}^h(s) = \lim _{h \rightarrow \infty } \varvec{{\tilde{g}}}^{2h}(s) = \lim _{h \rightarrow \infty } \varvec{{\tilde{g}}}^h(s) = \varvec{{\tilde{g}}}(s)$ for all $s \in {\mathcal {S}}$ and we have

$$\begin{aligned} G^{\varvec{{\varPi }}}(\gamma )&= \sum \limits _{s \in {\mathcal {S}}} \varvec{\alpha }(s) \varvec{g}(s) \\&= \sum \limits _{s \in {\mathcal {S}}} \varvec{{\tilde{\alpha }}}(s) \varvec{{\tilde{g}}}(s)&\text {since }\varvec{{\tilde{\alpha }}}(s) = \varvec{\alpha }(s)\text { for }s \in {\mathcal {S}}\\&= \sum \limits _{\sigma \in {\tilde{{\mathcal {S}}}}} \varvec{{\tilde{\alpha }}}(\sigma ) \varvec{{\tilde{g}}}(\sigma )&\text {since }\varvec{{\tilde{\alpha }}}(\sigma ) = 0\text { for }\sigma \not \in {\mathcal {S}}\\&= {\tilde{G}}^{\varvec{{\varPi }}}(\sqrt{\gamma }). \end{aligned}$$

$\square$

Analogously the following result can be shown.

Theorem 8

For any policy ${\varvec{{\tilde{{\varPi }}}}}$ in the action-independent MDP $\left( {{\tilde{{\mathcal {S}}}}, \varvec{{\tilde{\alpha }}}, \left( {{\varvec{{\tilde{P}}}}^a_k}\right) _{a \in {\mathcal {A}}}, \varvec{{\tilde{r}}}}\right)$, the policy ${\varvec{{\varPi }}}$ in the action-dependent MDP derived by projection of ${\varvec{{\tilde{{\varPi }}}}}$ on ${\mathcal {S}}$ is subject to

$$\begin{aligned} G^{\varvec{{\varPi }}}(\gamma ) = {\tilde{G}}^{\varvec{{\tilde{{\varPi }}}}}(\sqrt{\gamma }) \end{aligned}$$

This implies that every policy (including the optimal policy) in the MDP with action-independent rewards yields the same expected discounted reward as its projection in the MDP with action-dependent rewards. Together with Theorem 7 this means that there exists a one-to-one correspondence between policies in both MDPs which yields the same expected discounted reward, and optimal policies in one MDP are also optimal in the other one. This completes the transformation from the action-dependent reward model.

Appendix B: Proof of Theorem 1

Theorem 1

The decision problem defined in Definition 1 is NP-complete.

Fig. 3

The variable gadget

We perform a reduction from 3-SAT. Given a 3-SAT instance with n variables $x_1, \ldots , x_n$ and m clauses $C_1, \ldots , C_m$ where each clause contains three literals, we construct a concurrent MDP ${\mathcal {M}}= \{1, 2\}$ consisting of two MDPs, a vector $\varvec{w}\in {{\mathbb {R}}}^2$ and a real number $g \in {{\mathbb {R}}}$ such that the instance will be satisfiable if and only if there is a policy ${\varvec{{\varPi }}}$ for the concurrent MDP that yields the value g.

The first part of our construction are the states. They are arranged in three groups.

• First, we create a specially designated sink state $s_0$ which yields 0 reward in both MDPs.

• Then, we transfer the variables of the Boolean satisfiability problem into states of the MDPs: for each variable x we create two states $s_{x}, s_{x}'$ in both MDPs. The reward is 0 in states of the form $s_{x}$ and 1 in states of the form $s_{x}'$.

• Last, we create states for clauses: for each clause C, a state $s_C$ is created. Again, the reward in $s_C$ is zero for all clauses.

The second part of the construction are the actions. We create three actions ${\mathcal {A}}= \{1, 2, 3\}$ with the following semantics.

• In sink state $s_0$, it is ${\varvec{P}}^a_k(s_0, s_0) = 1$ for all $a \in {\mathcal {A}}$ and $k \in {\mathcal {M}}$.

• In the variable states, we define ${\varvec{P}}^1_1(s_x, s_{x}') = {\varvec{P}}^2_1(s_x, s_0) = {\varvec{P}}^3_1(s_x, s_0) = 1$ and ${\varvec{P}}^1_2(s_x, s_0) = {\varvec{P}}^2_2(s_x, s_{x}') = {\varvec{P}}^3_2(s_x, s_{x}') = 1$, that is, we define actions in $s_x$ to lead to different outcomes in the MDPs. The motivation is to force a mutually exclusive choice of values for the Boolean variables in the concurrent MDP. In the auxiliary variable states, it is ${\varvec{P}}^a_k(s_x', s_x) = 1$ for all actions $a \in {\mathcal {A}}$ and MDPs $k \in {\mathcal {M}}$; the idea behind these states is to exploit non-linearity of the problem. The construction is visualized in Fig. 3 where the upper part corresponds to the first MDP and the lower part corresponds to the second MDP in ${\mathcal {M}}$.

• In the clause states, we define actions as follows. In a clause $C = L_1 \vee L_2 \vee L_3$, the chosen action represents the literal that evaluates to true. Hence, we define ${\varvec{P}}^a_k(C, s)$ by setting ${\varvec{P}}^a_k(C, s) = 1$ in the cases

  • $L_a = x, k = 1, s = s_x$

  • $L_a = \lnot x, k = 1, s = s_0$

  • $L_a = \lnot x, k = 2, s = s_x$

  • $L_a = x, k = 2, s = s_0$

A graphical sketch of this setup can be seen in Fig. 4. Again, the upper part of the drawing corresponds to the first MDP in ${\mathcal {M}}$ while the lower part corresponds to the second MDP.

The idea behind this construction is to infer functions $\beta :\{1, \ldots , n\} \rightarrow \{0, 1\}$ that map variables to values and $\nu :\{1, \ldots , m\} \rightarrow \{1, 2, 3\}$ that map the clauses to the satisfying variables. This is done to create a mapping from policies to variable assignments in the SAT problem. Furthermore, we define the initial distribution $\alpha$ with $\alpha (s_C) = {1}/{m}$ for all clauses C for both MDPs and weights $\varvec{w}= ({1}/{2}, {1}/{2})$. Concerning the value, we set an auxiliary constant $q := \frac{1}{1 - \gamma ^2}$ and the required value $g := \frac{\gamma ^2 q}{2}$ where $\gamma$ is a non-zero discount factor in the concurrent MDP.

Fig. 4

The clause gadget

We prove the validity of the reduction. First, we show that if there is an assignment $\beta :\{1, \ldots , n\} \rightarrow \{0, 1\}$ that satisfies the SAT instance, then there also exists a policy $\pi$ such that $\sum _{k=1}^{K} \varvec{w}(k) G^{{\varvec{{\varPi }}}}_k \ge g$. We construct the policy in two steps. In the first step, we set $\pi _{s_x}(1) = 1 \Leftrightarrow \beta (x) = 1$ for all variables x. In the second step, it follows from the existence of a satisfying assignment that in each clause, a literal is satisfied, defining a function $\nu :\{1, \ldots , m\} \rightarrow \{1, 2, 3\}$ that defines the number of a satisfied literal in every clause. Thus, we set $\pi _{s_C}(a) = 1 \Leftrightarrow \nu (C) = a$.

We verify that the constructed policy yields the given value. As in each clause the satisfying literal is chosen, the value of this state will be 0 in one MDP and $\gamma ^2 q$ in the other one.

Now we show that if there is no satisfying assignment, then the value of the concurrent MDP will be lower than g. Given any assignment $\beta$ and any assignment $\nu$, the induced policy will lead from at least one clause state to the sink state $s_0$ with nonzero probability in both MDPs, yielding a lower value. However, we must take care of stationary but not pure policies that still might induce the desired value. One can observe that if the stationary policy is not pure in a state $s_x$ for a variable x, then the cumulative discounted reward in this state is $\frac{p\gamma }{1 - p^2\gamma ^2}$ for some real $0< p < 1$. Deriving the value of a clause state from which one can arrive to this variable state, we get, summing over both MDPs, a summand

$$\begin{aligned} \frac{1}{2} \left( \frac{p \gamma ^2}{1 - p^2 \gamma ^2} + \frac{(1 - p) \gamma ^2}{1 - (1 - p)^2 \gamma ^2} \right) \end{aligned}$$

Let $f(p) = \frac{p}{1 - p^2 \gamma ^2} + \frac{1 - p}{1 - (1 - p)^2 \gamma ^2}$. Computing the derivative, we obtain

$$\begin{aligned} f'(p) = \frac{2 \gamma ^2 p}{ \left( 1 -\gamma ^2 p^2 \right) ^2 } - \frac{2 \gamma ^2 (1 - p)^2}{ \left( 1 -\gamma ^2 (1 - p)^2 \right) ^2 } + \frac{1}{1 - \gamma ^2 p^2} - \frac{1}{1 - \gamma ^2 (1 - p)^2} \end{aligned}$$

which has its roots at

$$\begin{aligned} \frac{1}{2}, \frac{1 \pm \sqrt{1 - 4\gamma ^{-2} + 4\gamma ^{-2} \sqrt{4 - \gamma ^{2}} }}{2}, \frac{1 \pm i \sqrt{1 - 4\gamma ^{-2} + 4\gamma ^{-2} \sqrt{4 - \gamma ^{2}} }}{2}. \end{aligned}$$

The only roots of interest are the real ones, and thus, we investigate the pair

$$\begin{aligned} \frac{1 \pm \sqrt{1 - 4\gamma ^{-2} + 4\gamma ^{-2} \sqrt{4 - \gamma ^{2}} } }{2}. \end{aligned}$$

(29)

It can be seen that for $0< \gamma < 1$, the value $\sqrt{4 - \gamma ^2}$ is at least $\sqrt{3} > 1$, and the root term in (29) is thus greater than one. This means that the whole term (29) is either greater than one or negative. Hence, the possible extreme points of f in [0, 1] may lie at 0, 1, or ${1}/{2}$. We can see that $f(0) = f(1) = q$ while $f({1}/{2}) = \frac{1}{1 - {1}/{4} \gamma ^2} < q$. Hence, a non-pure policy in a variable state will have a lower cumulative discounted reward. Concerning the clause states, we observe that a non-pure policy cannot yield higher rewards than a pure one, as the expected discounted reward in a clause state is linear in the expected discounted rewards in the following variable states; the clause states are not visited again.