Cooperative Dynamic Games with Control Lags

Abstract

Controls with lags are control strategies with prolonged effects lasting for more than one stage of the game after the controls had been executed. Lags in controls yielding adverse effects often make the negative impacts more significant. Cooperation provides an effective means to alleviate the problem and obtains an optimal solution. This paper extends the existing paradigm in cooperative dynamic games by allowing the existence of controls with lag effects on the players’ payoffs in subsequent stages. A novel dynamic optimization theorem with control lags is developed to derive the Pareto optimal cooperative controls. Subgame consistent solutions are derived to ensure sustainable cooperation. In particular, subgame consistency guarantees that the optimality principle agreed upon at the outset will remain effective throughout the game and, hence, there is no incentive for any player to deviate from cooperation scheme. A procedure for imputation distribution is provided to formulate a dynamically stable cooperative scheme under control lags. An application in cooperative environmental management is presented. This is the first time that cooperative dynamic games with control lags are studied.

Appendices

Appendix A: Proof of Theorem 2.1

According to (2.4) in Theorem 2.1

$$ \begin{aligned} W\left( {k,x;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right) & = \mathop {\hbox{max} }\limits_{{\mu_{k}^{(0)} ,\mu_{k}^{(T)} }} \left\{ {g_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} ;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)\,\delta^{k - 1} } \right. \\ & \quad \left. { + \,W\left[ {\left. {k + 1,f_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} } \right);\mu_{k}^{(T)} ,\mu_{k - 1}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)} \right]} \right\}. \\ \end{aligned} $$

(A.1)

We prove the validity of (A.1) by contradiction. Suppose

$$ \begin{aligned} & W\left( {k,x;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right) < \mathop {\hbox{max} }\limits_{{\mu_{k}^{(0)} ,\mu_{k}^{(T)} }} \left\{ {g_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} ;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)\,\delta^{k - 1} } \right. \\ & \quad \left. { + \,W\left[ {\left. {k + 1,f_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} } \right);\mu_{k}^{(T)} ,\mu_{k - 1}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)} \right]} \right\}. \\ \end{aligned} $$

(A.2)

This is not possible because on the right-hand-side of (A.2), we choose controls $u_{k}^{(0)}$ and $\mu_{k}^{(T)}$ to maximize the value inside the curly brackets, and this maximized value would at best equal the maximal payoff on the right-hand-side.Now suppose

$$ \begin{aligned} & W\left( {k,x;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right) > \mathop {\hbox{max} }\limits_{{\mu_{k}^{(0)} ,\mu_{k}^{(T)} }} \left\{ {g_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} ;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)\,\delta^{k - 1} } \right. \\ & \quad \left. { + \,W\left[ {\left. {k + 1,f_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} } \right);\mu_{k}^{(T)} ,\mu_{k - 1}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)} \right]} \right\}. \\ \end{aligned} $$

(A.3)

From (A.1) we obtain $W(k,x;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} )$ which is the maximal of the payoffs

$$ \begin{aligned} & \left\{ {g_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} ;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)} \right. \\ & \quad \left. { + \sum\limits_{t = k + 1}^{T} {g_{t}^{{}} \left( {x_{t}^{{}} ,\mu_{t}^{(0)} ,\mu_{t}^{(T)} ;\mu_{t - 1}^{(T)} ,\mu_{t - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)} \,\delta^{k - 1} + q_{T + 1}^{{}} (x_{T + 1}^{{}} )\delta^{T} } \right\} \\ & = g_{k}^{{}} \left( {x,\tilde{\mu }_{k}^{(0)} ,\tilde{\mu }_{k}^{(T)} ;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right) \\ & \quad + \sum\limits_{t = k + 1}^{T} {g_{t}^{{}} \left( {x_{t}^{{}} ,\tilde{\mu }_{t}^{(0)} ,\tilde{\mu }_{t}^{(T)} ;\mu_{t - 1}^{(T)} ,\mu_{t - 2}^{(T)} ,v,\mu_{1}^{(T)} } \right)} \,\delta^{k - 1} \; + q_{T + 1}^{{}} (x_{T + 1}^{{}} )\;\delta^{T} . \\ \end{aligned} $$

(A.4)

By definition, the term $W(k + 1,x;\mu_{k}^{(T)} ,\mu_{k - 1}^{(T)} , \ldots ,\mu_{1}^{(T)} )$ is the maximal of the payoffs

$$ \left\{ {\sum\limits_{t = k + 1}^{T} {g_{t}^{{}} \left( {x_{t}^{{}} ,\mu_{t}^{(0)} ,\mu_{t}^{(T)} ;\mu_{t - 1}^{(T)} ,\mu_{t - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)} \,\delta^{k - 1} + q_{T + 1}^{{}} (x_{T + 1}^{{}} )\delta^{T} } \right\} $$

which is no less than

$$ \left\{ {\sum\limits_{t = k + 1}^{T} {g_{t}^{{}} \left( {x_{t}^{{}} ,\tilde{\mu }_{t}^{(0)} ,\tilde{\mu }_{t}^{(T)} ;\mu_{t - 1}^{(T)} ,\mu_{t - 2}^{(T)} , \cdots ,\mu_{1}^{(T)} } \right)} \,\delta^{k - 1} + q_{T + 1}^{{}} (x_{T + 1}^{{}} )\delta^{T} } \right\}. $$

(A.5)

Hence,

$$ \begin{aligned} & W\left( {k,x;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \cdots ,\mu_{1}^{(T)} } \right) \le \mathop {\hbox{max} }\limits_{{\mu_{k}^{(0)} ,\mu_{k}^{(T)} }} \left\{ {\begin{array}{*{20}c} {} \\ {} \\ \end{array} } \right.g_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} ;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \cdots ,\mu_{1}^{(T)} } \right)\,\delta^{k - 1} \\ & \quad \left. { +\, W\left[ {\left. {k + 1,f_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} } \right);\mu_{k}^{(T)} ,\mu_{k - 1}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)} \right]} \right\}. \\ \end{aligned} $$

(A.6)

(A.6) contradicts with (A.3). Hence the validity of Eq. (2.4) stands.

In addition, to solve the problem (2.3)–(2.4), we adopt the technique of backward induction. Consider first the last operational stage $T$, invoking Theorem 2.1 we have

$$ \begin{aligned} & W\left( {T,x;\mu_{T - 1}^{(T)} ,\mu_{T - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right) \\ & = \mathop {\hbox{max} }\limits_{{\mu_{T}^{(0)} ,\mu_{T}^{(T)} }} \left\{ {g_{T}^{{}} \left( {x,\mu_{T}^{(0)} ,\mu_{T}^{(T)} ;\mu_{T - 1}^{(T)} ,\mu_{T - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)\,\delta^{T - 1} } \right. \\ & \quad \left. { + \,q_{T + 1}^{{}} \left[ {f_{T}^{{}} (x,\mu_{T}^{(0)} ,\mu_{T}^{(T)} } \right]\delta^{T} } \right\}. \\ \end{aligned} $$

(A.7)

The maximization operator in stage $T$ involves $\mu_{T}^{(0)}$ and $\mu_{T}^{(T)}$ only. However, the current state $x$ and the previous determined controls $(\mu_{T - 1}^{(T)} ,\mu_{T - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} )$ appear in the stage $T$ payoff function. If the first order conditions of the maximization problem in (A.7) satisfy the implicit function theorem, one can obtain the optimal controls $\mu_{T}^{(0)}$ and $\mu_{T}^{(T)}$ as functions of $x$ and $(\mu_{T - 1}^{(T)} ,\mu_{T - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} )$. Substituting these optimal controls into the function on the right-hand-side of (A.7) yields the function $W(T,x;\mu_{T - 1}^{(T)} ,\mu_{T - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} )$, which satisfies the optimal conditions of a maximum for given $x$ and $(\mu_{T - 1}^{(T)} ,\mu_{T - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} )$.

Consider the second last operational stage $T - 1$, invoking Theorem 2.1 we have

$$ \begin{aligned} &W\left( {T - 1,x;\mu_{T - 2}^{(T)} ,\mu_{T - 3}^{(T)} , \cdots ,\mu_{1}^{(T)} } \right) \\ & = \mathop {\hbox{max} }\limits_{{\mu_{T - 1}^{(1)} }} \left\{ {g_{T - 1}^{{}} \left( {x,\mu_{T - 1}^{(0)} ,\mu_{T - 1}^{(T)} ;\mu_{T - 2}^{(T)} ,\mu_{T - 3}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)\,\delta^{T - 2} } \right. \\ & \quad \left. { + \,W\left[ {T,f_{T - 1}^{{}} \left( {x,\mu_{T - 1}^{(0)} ,\mu_{T - 1}^{(T)} } \right);\mu_{T - 1}^{(T)} ,\mu_{T - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right]} \right\}. \\ \end{aligned} $$

(A.8)

The maximization operator in stage $T - 1$ involves $\mu_{T - 1}^{(0)}$ and $\mu_{T - 1}^{(T)}$. The current state $x$ and the previous determined controls $(\mu_{T - 2}^{(T)} ,\mu_{T - 3}^{(T)} , \ldots ,\mu_{1}^{(T)} )$ appear in the stage $T - 1$ payoff function. If the first order conditions of the maximization problem in (A.8) satisfy the implicit function theorem, one can obtain the optimal controls $\mu_{T - 1}^{(0)}$ and $\mu_{T - 1}^{(T)}$ as functions of $x$ and previously determined controls $(\mu_{T - 2}^{(T)} ,\mu_{T - 3}^{(T)} , \ldots ,\mu_{1}^{(T)} )$. Substituting these optimal controls into the function on the right-hand-side of (A.8) yields the function $W(T - 1,x;\mu_{T - 2}^{(T)} ,\mu_{T - 3}^{(T)} , \ldots ,\mu_{1}^{(T)} )$.

Now consider stage $k \in \{ T - 2,T - 3, \ldots ,2,1\}$, invoking Theorem 2.1 we have

$$ \begin{aligned} &W\left( {k,x;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right) = \mathop {\hbox{max} }\limits_{{\mu_{k}^{(T)} }} \left\{ {\begin{array}{*{20}c} {} \\ {} \\ \end{array} } \right.g_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} ;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right)\,\delta^{k - 1} \\ & \quad + W\left[ {k + 1,f_{k}^{{}} \left( {x,\mu_{k}^{(0)} ,\mu_{k}^{(T)} } \right);\mu_{k}^{(T)} ,\mu_{k - 1}^{(T)} , \ldots ,\mu_{1}^{(T)} } \right]\left. {\begin{array}{*{20}c} {} \\ {} \\ \end{array} } \right\}. \\ \end{aligned} $$

(A.9)

The maximization operator involves $\mu_{k}^{(0)}$ and $\mu_{k}^{(T)}$. Again, the current state $x$ and the previous determined controls $(\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} )$ appear in the stage $k$ payoff function. If the first order conditions of the maximization problem in (A.9) satisfy the implicit function theorem, one can obtain the optimal controls $\mu_{k}^{(0)}$ and $\mu_{k}^{(T)}$ as functions of $x$ and $(\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} )$. Substituting these optimal controls into the function on the right-hand-side of (A.9) yields the function $W(k,x;\mu_{k - 1}^{(T)} ,\mu_{k - 2}^{(T)} , \ldots ,\mu_{1}^{(T)} )$. □

Appendix B: Proof of Theorem 3.1

The conditions in (3.4)–(3.5) satisfy the optimal conditions of the dynamic optimization technique with control lags in Theorem 2.1 and hence an optimal solution to the control problem results. □

Appendix C: Proof of Theorem 3.2

Conditions (3.9)–(3.10) show that $V_{{}}^{i} \left( {k,x;\underline{\mu }_{k - 1}^{(T)**} ,\underline{\mu }_{k - 2}^{(T)**} , \ldots ,\underline{\mu }_{1}^{(T)**} } \right)$ is the maximized payoff of player $i \in N$ according to Theorem 2.1 given the game equilibrium strategies of the other $n - 1$ players. Hence a Nash equilibrium results. □

Appendix D: Proof of Theorem 4.1

Using (4.2) one can obtain

$$ \begin{aligned} &\xi_{{}}^{i} \left( {k + 1,x_{k + 1}^{*} ;\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \ldots ,\underline{\mu }_{1}^{(T)*} } \right) = B_{k + 1}^{i} \left( {x_{k + 1}^{*} ;\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \ldots ,\underline{\mu }_{1}^{(T)*} } \right)\;\delta^{k} \\ & \quad + \left\{ {\sum\limits_{\zeta = k + 2}^{T} {B_{\zeta }^{i} \left( {x_{\zeta }^{*} ;\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \cdots ,\underline{\mu }_{1}^{(T)*} } \right)\delta^{\zeta - 1} + q_{T + 1}^{i} (x_{T + 1}^{{}} )\;\delta_{{}}^{T} } } \right\}, \\ \end{aligned} $$

(D.1)

Upon substituting (D.1) into (4.2) yields

$$ \begin{aligned} &\xi_{{}}^{i} \left( {k,x_{k}^{*} ;\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \ldots ,\underline{\mu }_{1}^{(T)*} } \right) = B_{k}^{i} \left( {x_{k}^{*} ;\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \ldots ,\underline{\mu }_{1}^{(T)*} } \right)\delta^{k - 1} \\ & \quad + \xi_{{}}^{i} \left( {k + 1,x_{k + 1}^{*} ;\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \ldots ,\underline{\mu }_{1}^{(T)*} } \right), \\ \end{aligned} $$

which can be expressed as

$$ \begin{aligned} &\xi_{{}}^{i} \left( {k,x_{k}^{*} ;\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \ldots ,\underline{\mu }_{1}^{(T)*} } \right) = B_{k}^{i} \left( {x_{k}^{*} ;\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \ldots ,\underline{\mu }_{1}^{(T)*} } \right)\delta^{k - 1} \\ & \quad + \xi_{{}}^{i} \left( {k + 1,f_{k}^{{}} \left( {x_{k}^{*} ,\underline{\mu }_{k}^{(0)*} ,\underline{\mu }_{k}^{(T)*} } \right);\underline{\mu }_{k - 1}^{(T)*} ,\underline{\mu }_{k - 2}^{(T)*} , \ldots ,\underline{\mu }_{1}^{(T)*} } \right). \\ \end{aligned} $$

(D.2)

From (D.2) one can obtain Theorem 4.1. □

Appendix E: Proof of Proposition 5.1

First, we consider the last operating stage $T = 10$. Using (5.5) and Proposition 5.1 we can obtain the optimal cooperative strategies:

$$ \mu_{10}^{(0)i*} = - A_{11}^{{}} \frac{{b_{{}}^{i} (x)^{1/2} \delta }}{{2\gamma_{10}^{i} }}, $$

$$ \mu_{10}^{(10)i*} = \frac{{\alpha_{10}^{i} - \varepsilon_{10}^{i(10)i} - \omega_{10}^{\ell (10)i} + \delta A_{11}^{{}} a_{{}}^{i} }}{{2c_{10}^{i} }},\quad i,\ell \in \{ 1,2\} \;{\text{and}}\;\ell \ne i. $$

(E.1)

Substituting (E.1) into the stage 10 equation in (5.5) we obtain:

$$ \begin{aligned} &W\left( {10,x;\mu_{9}^{(10)} ,\mu_{8}^{(10)} , \ldots ,\mu_{1}^{(10)} } \right) = (A_{10}^{{}} x + C_{10}^{{}} )\delta^{9} \\ & = \sum\limits_{j = 1}^{2} {\left[ {\alpha_{10}^{j} \frac{{\alpha_{10}^{j} - \varepsilon_{10}^{j(10)j} - \omega_{10}^{\ell (10)j} + \delta A_{11}^{{}} a_{{}}^{j} }}{{2c_{10}^{j} }} - c_{10}^{j} \left( {\frac{{\alpha_{10}^{j} - \varepsilon_{10}^{j(10)j} - \omega_{10}^{\ell (10)j} + \delta A_{11}^{{}} a_{{}}^{j} }}{{2c_{10}^{j} }}} \right)^{2} } \right.} \\ & \quad - \gamma_{10}^{j} \left( {\frac{{\delta A_{11}^{{}} b_{{}}^{j} }}{{2\gamma_{10}^{j} }}} \right)^{2} x - h_{10}^{j} x - \varepsilon_{10}^{j(10)j} \frac{{\alpha_{10}^{j} - \varepsilon_{10}^{j(10)j} - \omega_{10}^{\ell (10)j} + \delta A_{11}^{{}} a_{{}}^{j} }}{{2c_{10}^{j} }} \\ & \quad \left. { -\, \omega_{10}^{j(10)\ell } \frac{{\alpha_{10}^{\ell } - \varepsilon_{10}^{\ell (10)\ell } - \omega_{10}^{j(10)\ell } + \delta A_{11}^{{}} a_{{}}^{\ell } }}{{2c_{10}^{\ell } }} - \sum\limits_{t = 1}^{9} {\varepsilon_{10}^{j(t)j} \mu_{t}^{(10)j} } - \sum\limits_{t = 1}^{9} {\omega_{10}^{j(t)\ell } \mu_{t}^{(10)\ell } } } \right]\delta^{9} \\ & \quad + \left[ {A_{11}^{{}} \left( {\sum\limits_{j = 1}^{2} {a_{{}}^{j} \frac{{\alpha_{10}^{j} - \varepsilon_{10}^{j(10)j} - \omega_{10}^{\ell (10)j} + \delta A_{11}^{{}} a_{{}}^{j} }}{{2c_{10}^{j} }}} + \sum\limits_{j = 1}^{2} {b_{{}}^{j} } \frac{{\delta A_{11}^{{}} b_{{}}^{j} x}}{{2\gamma_{10}^{j} }} - \lambda x} \right) + C_{11}^{{}} } \right]\delta^{10} . \\ \end{aligned} $$

(E.2)

Both the LHS and RHS of (E.2) are linear functions of $x$. For (E.2) to hold it is required that:

$$ A_{10}^{{}} = \sum\limits_{j = 1}^{2} {\left[ { - \gamma_{10}^{j} \left( {\frac{{\delta A_{11}^{{}} b_{{}}^{j} }}{{2\gamma_{10}^{j} }}} \right)^{2} - h_{10}^{j} } \right] + \left[ {A_{11}^{{}} \left( {\sum\limits_{j = 1}^{2} {b_{{}}^{j} } \frac{{\delta A_{11}^{{}} b_{{}}^{j} }}{{2\gamma_{10}^{j} }} - \lambda } \right)} \right]\delta ,} $$

(E.3)

and $C_{10}^{{}}$ being equal to the (undiscounted) expression not involving $x$ on the right-hand-side of (E.3). Note that the term $C_{10}^{{}}$ contains previously executed strategies in the form of $- \sum\nolimits_{t = 1}^{9} {\varepsilon_{10}^{j(t)j} \mu_{t}^{(10)j} } - \sum\nolimits_{t = 1}^{9} {\omega_{10}^{j(t)\ell } \mu_{t}^{(10)\ell } }$.

Invoking Proposition 5.1 and performing similar operations in stage $k \in \{ 9,8, \ldots ,1\}$ in (5.5), we obtain the optimal cooperative strategies:$\mu_{k}^{(0)i*} = - A_{k + 1}^{{}} \frac{{b_{{}}^{i} (x)^{1/2} \delta }}{{2\gamma_{k}^{i} }}$,

$$ \begin{aligned} \mu_{k}^{(10)i*} & = \frac{{\alpha_{k}^{i} - \varepsilon_{k}^{i(k)i} - \omega_{k}^{\ell (k)i} + \delta A_{k + 1}^{{}} a_{{}}^{i} + \delta \left( {\partial C_{k + 1}^{{}} /\partial \mu_{k}^{(10)i} } \right)}}{{2c_{k}^{i} }}, \\ & \quad i,\ell \in \{ 1,2\} \;{\text{and}}\;\ell \ne i, \\ \end{aligned} $$

(E.4)

where $\partial C_{k + 1}^{{}} /\partial u_{k}^{(10)i} = - \sum\nolimits_{\tau = k + 1}^{10} {} \delta_{{}}^{\tau - (k + 1)} (\varepsilon_{\tau }^{i(k)i} + \omega_{\tau }^{\ell (k)i} )$. Substituting (E.4) into Eq. (5.5) for $k \in \{ 8,7, \cdots ,1\}$ we obtain:

$$ \begin{aligned} &W\left({k,x;\mu_{k - 1}^{(10)},\mu_{k - 2}^{(10)}, \ldots,\mu_{1}^{(10)}} \right) = (A_{k}^{{}} x + C_{k}^{{}})\delta^{k - 1} \hfill \\ &= \sum\limits_{j = 1}^{2} {\left[{\alpha_{k}^{j} \frac{{\alpha_{k}^{j} - \varepsilon_{k}^{j(k)j} - \omega_{k}^{\ell (k)j} + \delta A_{k + 1}^{{}} a_{{}}^{j}}}{{2c_{k}^{j}}} - c_{k}^{j} \left({\frac{{\alpha_{k}^{j} - \varepsilon_{k}^{j(k)j} - \omega_{k}^{\ell (k)j} + \delta A_{k + 1}^{{}} a_{{}}^{j}}}{{2c_{k}^{j}}}} \right)^{2}} \right.} \hfill \\ & \quad - \gamma_{k}^{j} \left({\frac{{\delta A_{k + 11}^{{}} b_{{}}^{j}}}{{2\gamma_{k}^{j}}}} \right)^{2} x - h_{k}^{j} x - \varepsilon_{k}^{j(k)j} \frac{{\alpha_{k}^{j} - \varepsilon_{k}^{j(k)j} - \omega_{k}^{\ell (k)j} + \delta A_{k + 1}^{{}} a_{{}}^{j}}}{{2c_{k}^{j}}} \hfill \\ & \left. \quad {- \,\omega_{k}^{j(k)\ell} \frac{{\alpha_{k}^{\ell} - \varepsilon_{k}^{\ell (k)\ell} - \omega_{k}^{j(k)\ell} + \delta A_{k + 1}^{{}} a_{{}}^{\ell}}}{{2c_{10}^{\ell}}} - \sum\limits_{t = 1}^{k - 1} {\varepsilon_{k}^{j(t)j} \mu_{t}^{(10)j} - \sum\limits_{t = 1}^{k - 1} {\omega_{k}^{j(t)\ell} \mu_{t}^{(10)\ell}}}} \right]\delta^{k - 1} \hfill \\ & \quad + \left[{A_{k + 1}^{{}} \left({\sum\limits_{j = 1}^{2} {a_{{}}^{j} \frac{{\alpha_{k}^{j} - \varepsilon_{k}^{j(k)j} - \omega_{k}^{\ell (k)j} + \delta A_{k + 1}^{{}} a_{{}}^{j}}}{{2c_{9}^{j}}}} + \sum\limits_{j = 1}^{2} {b_{{}}^{j}} \frac{{\delta A_{k + 1}^{{}} b_{{}}^{j} x}}{{2\gamma_{k}^{j}}} - \lambda x} \right) + C_{k + 1}^{{}}} \right]\delta^{k}. \hfill \\ \end{aligned} $$

(E.5)

Both the LHS and RHS of (E.5) are linear functions of $x$. For (E.5) to hold it is required that:

$$ A_{k}^{{}} = \sum\limits_{j = 1}^{2} {\left[ { - \gamma_{k}^{j} \left( {\frac{{\delta A_{k + 1}^{{}} b_{{}}^{j} }}{{2\gamma_{k}^{j} }}} \right)^{2} - h_{k}^{j} } \right]} + \left[ {A_{k + 1}^{{}} \left( {\sum\limits_{j = 1}^{2} {b_{{}}^{j} } \frac{{\delta A_{k + 1}^{{}} b_{{}}^{j} }}{{2\gamma_{k}^{j} }} - \lambda } \right)} \right]\delta , $$

(E.6)

and $C_{k}^{{}}$ being equal to the (undiscounted) expression not involving $x$ on the right-hand-side of (E.5). Note that the term $- \sum\nolimits_{t = 1}^{k - 1} {\varepsilon_{k}^{j(t)j} \mu_{t}^{(10)j} } - \sum\nolimits_{t = 1}^{9} {\omega_{k}^{j(t)\ell } \mu_{t}^{(10)\ell } }$ appears in $C_{k}^{{}}$ as given parameters.

Finally substituting the optimal cooperative controls

$$ \mu_{10}^{(10)i*} = \frac{{\alpha_{10}^{i} - \varepsilon_{10}^{i(10)i} - \omega_{10}^{\ell (10)i} + \delta A_{11}^{{}} a_{{}}^{i} }}{{2c_{10}^{i} }},\quad i,\ell \in \{ 1,2\} \;{\text{and}}\;\ell \ne i. $$

$$ \begin{aligned} \mu_{k}^{(10)i*} & = \frac{{\alpha_{k}^{i} - \varepsilon_{k}^{i(k)i} - \omega_{k}^{\ell (k)i} + \delta A_{k + 1}^{{}} a_{{}}^{i} - \delta \sum\nolimits_{\tau = k + 1}^{10} {\delta_{{}}^{\tau - (k + 1)} \left( {\varepsilon_{\tau }^{i(k)i} + \omega_{\tau }^{\ell (k)i} } \right)} }}{{2c_{k}^{i} }}, \\ & \quad i,\ell \in \{ 1,2\} ,\;\ell \ne i \;{\text{and}}\;k \in \{ 1,2, \ldots ,9\} , \\ \end{aligned} $$

(E.7)

into $C_{k}^{{}}$ for $k \in \{ 1,2, \ldots ,10\}$, we can express $C_{k}^{{}}$ in terms of the model parameters explicitly. □