Decentralized optimality conditions of stochastic differential decision problems via Girsanov’s measure transformation

Abstract

In this paper, we apply two methods to derive necessary and sufficient decentralized optimality conditions for stochastic differential decision problems with multiple Decision Makers (DMs), which aim at optimizing a common pay-off, based on the notions of decentralized global optimality and decentralized person-by-person (PbP) optimality. Method 1: We utilize the stochastic maximum principle to derive necessary and sufficient conditions which consist of forward and backward Stochastic Differential Equations (SDEs), and conditional variational Hamiltonians, conditioned on the information structures of the DMs. The sufficient conditions for decentralized PbP optimality are local conditions, closely related to the necessary conditions for decentralized PbP optimality. However, under certain convexity condition on the Hamiltonian, and a global version of the sufficient conditions for decentralized PbP optimality, we show decentralized global optimality. Method 2: We utilize the value processes of decentralized PbP optimal policies, we relate them to solutions of backward SDEs, we identify sufficient conditions for decentralized PbP optimality, and we show these are precisely those derived via the maximum principle. For both methods, as usual, we utilize Girsanov’s theorem to transform the initial decentralized stochastic optimal decision problems, to equivalent decentralized stochastic optimal decision problems on a reference probability space, in which the controlled process and the information processes which generate part of the information structures of the DMs, are independent of any of the decisions.

Appendix

Proof of Theorem 1

Statement (2), ${\mathbb {E}} \Lambda ^u(t)=1, \forall t \in [0,T]$.

First, we show that ${\mathbb {E}}\{\Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2\}<K$. By applying the Itô differential rule

$$\begin{aligned} \mathrm{d} |x(t)|_{{\mathbb {R}}^n}^2= & {} 2\langle x(t), \sigma (t,x(t))\mathrm{d}W(t) \rangle + \hbox {tr} ( a(t,x(t)))\mathrm{d}t, \quad a(t,x) \mathop {=}\limits ^{\triangle }\sigma (t,x) \sigma ^*(t,x) \\ \mathrm{d} \Big (\Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2\Big )= & {} \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2 f^{*}(t,x(t),u(t)) (a(t,x(t)))^{-1} \mathrm{d}x(t) \\&+\, 2 \Lambda ^u(t) \langle x(t), \sigma (t,x(t))\mathrm{d}W(t) \rangle \\&+\, 2 \Lambda ^u(t) \langle x(t), f(t,x(t),u(t)) \rangle \mathrm{d}t + \Lambda ^u(t) \hbox {tr} (a(t,x(t)))\mathrm{d}t. \end{aligned}$$

Then by applying the Itô differential rule once more we have

$$\begin{aligned}&\mathrm{d} \frac{ \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2}{ 1+ \epsilon \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2} = \frac{1}{\Big ( 1+ \epsilon \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2\Big )^2}\nonumber \\&\qquad \times \Big \{ \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2 f^{*}(t,x(t))(a(t,x(t)))^{-1} \mathrm{d}x(t)+2 \Lambda ^u(t) \langle x(t), \sigma (t,x(t))\mathrm{d}W(t) \rangle \nonumber \\&\qquad +\, 2 \Lambda ^u(t) \langle x(t), f(t,x(t),u(t)) \rangle \mathrm{d}t+\Lambda ^u(t) \hbox {tr} ( a(t,x(t)) )\mathrm{d}t \Big \} \\&\qquad -\, \frac{\epsilon (\Lambda ^u(t))^2}{( 1+ \epsilon \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2)^3}\Big \{ \langle ( ( a(t,x(t)))^{-1} f(t,x(t),u(t)) |x(t)|_{{\mathbb {R}}^n}^2\\&\qquad +\,2x(t), a(t,x(t)) ( ( a(t,x(t)))^{-1} f(t,x(t),u(t)) |x(t)|_{{\mathbb {R}}^n}^2+2x(t) )\rangle \Big \}\mathrm{d}t \end{aligned}$$

Integrating over [0, T] and taking the expectation with respect to ${\mathbb {P}}$, and using the fact that ${\mathbb {E}} ( \Lambda ^u(t)) \le 1, \forall t \in [0,T]$, yields

$$\begin{aligned} \frac{\mathrm{d}}{\mathrm{d}t} {\mathbb {E}} \left\{ \frac{ \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2}{ 1+ \epsilon \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2} \right\} \le {\mathbb {E}} \left\{ \frac{\Lambda ^u(t) \Big [ 2\langle x(t), f(t,x(t),u(t)) \rangle + \hbox {tr} \Big ( a(t,x(t)) \Big ) \Big ] }{ 1+ \epsilon \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2} \right\} .\nonumber \\ \end{aligned}$$

(159)

By Assumptions 1, (A1), (A2), there exists $K>0$ such that

$$\begin{aligned} \frac{\mathrm{d}}{\mathrm{d}t} {\mathbb {E}} \left\{ \frac{ \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2}{ 1+ \epsilon \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2} \right\} \le K \left( {\mathbb {E}} \left\{ \frac{\Lambda ^u(t) \Big [ |x(t)|_{{\mathbb {R}}^n}^2 + |u(t)|_{{\mathbb {R}}^d}^2\Big ] }{ 1+ \epsilon \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2} \right\} +1 \right) \end{aligned}$$

Since for any $u \in {\mathbb {U}}_A^{(K)}[0, T]$, we have ${\mathbb {E}} \int _{0}^T \Lambda ^u(t) |u(t)|_{{\mathbb {R}}^d}^2 \mathrm{d}t$ is finite, then it follows from Gronwall inequality that

$$\begin{aligned} {\mathbb {E}}\left\{ \frac{ \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2}{ 1+ \epsilon \Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2} \right\} \le C, \quad \forall t \in [0,T]. \end{aligned}$$

(160)

By Fatou’s lemma we obtain ${\mathbb {E}} \{\Lambda ^u(t) |x(t)|_{{\mathbb {R}}^n}^2\}< C, \forall t \in [0,T]$. Consider

$$\begin{aligned} \mathrm{d} \frac{\Lambda ^u(t)}{ 1 + \epsilon \Lambda ^u(t)}= & {} \frac{\Lambda ^u(t) f^*(t,x(t))(a(s,x(s)))^{-1} \mathrm{d}x(t)}{( 1 + \epsilon \Lambda ^u(t))^2} \\&-\, \frac{\epsilon (\Lambda ^u(t))^2 f^*(t,x(t),u(t))(a(t,x(t)))^{-1}f(t,x(t),u(t))}{(1 + \epsilon \Lambda ^u(t))^3}; \end{aligned}$$

then

$$\begin{aligned} {\mathbb {E}} \frac{\Lambda ^u(t)}{ 1 + \epsilon \Lambda ^u(t)} = 1- {\mathbb {E}} \int _{0}^t \frac{\epsilon (\Lambda ^u(s))^2 f^*(s,x(s),u(s))(a(t,x(t)))^{-1} f(s,x(s),u(s))}{(1 + \epsilon \Lambda ^u(s))^3}\mathrm{d}s.\nonumber \\ \end{aligned}$$

(161)

Since

$$\begin{aligned}&\frac{\epsilon (\Lambda ^u(t))^2 f^*(t,x(t),u(t))( a(t,x(t)))^{-1}f(t,x(t),u(t))}{( 1 + \epsilon \Lambda ^u(t))^3} \longrightarrow 0,\\&\quad \hbox {a.e. }t \in [0,T],{\mathbb {P}}\text {-a.s.} \quad \hbox {as }\epsilon \longrightarrow 0 \end{aligned}$$

and by (A7) there exists a constant $C>0$ such that it is bounded by $C \Lambda ^u(t) ( 1+|x(t)|_{{\mathbb {R}}^n}^2+ |u(t)|_{{\mathbb {R}}^d}^2)$, by the Lebesgue’s dominated convergence theorem we have

$$\begin{aligned} {\mathbb {E}} \int _{0}^t \frac{\epsilon (\Lambda ^u(s))^2 f^*(s,x(s),u(s))(a(s,x(s)))^{-1}f(s,x(s),u(s))}{(1 + \epsilon \Lambda ^u(s))^3} \!\longmapsto \! 0 \quad \hbox {as }\epsilon \!\longrightarrow \! 0.\nonumber \\ \end{aligned}$$

(162)

Since ${\mathbb {E}} (\Lambda ^u(t))\le 1, \forall t \in [0,T]$ by using (161) into (162), we obtain ${\mathbb {E}} \frac{\Lambda ^u(t)}{ 1 + \epsilon \Lambda ^u(t)} \longrightarrow {\mathbb {E}} \Lambda ^u(t)$, as $\epsilon \longrightarrow 0$. Hence, we must have ${\mathbb {E}} \Lambda ^u(t)=1, \forall t \in [0,T]$. This completes the derivation.