On a Constrained Infinite-Time Horizon Linear Quadratic Game

Abstract

A linear quadratic differential game on an infinite-time horizon is studied in the case when the controls of the minimizing player are subject to constraints. A sufficient condition for a saddle point equilibrium is provided based on the conversion of the infinite-time horizon game to a game on a finite-time horizon. The method is applied to a simple monetary policy model as an illustrative example.

Appendix

Proof

For simplicity, we present the proof for $t_0= 0$. The case $t_0>0$ can be considered in the same way. We set $\Vert z\Vert _\Theta ^2:= z^\top \Theta z$, where $z^\top$ is the transpose of the vector z and $\Theta$ is an arbitrary symmetric positive definite square matrix of suitable dimension. Because $\int _{0}^\infty \Vert w(t)\Vert ^2 dt < \infty$ and the matrices

$$\begin{aligned} A, A - B R^{-1}B^\top P,~ A + E S^{-1} E^\top P, ~A - B R^{-1}B^\top P + E S^{-1} E^\top P \end{aligned}$$

are asymptotically stable, we have that $x_{u,w}(t)\rightarrow 0$, $x_{{\bar{u}},w}(t)\rightarrow 0$, $x_{u, {\bar{w}}}(t) \rightarrow 0$ and $x_{{\bar{u}}, {\bar{w}}}(t) \rightarrow 0$ as $t \rightarrow \infty$. We consider the case of open-loop controls and present the proof in full detail. (The cases, where one or two closed-loop controls are used, are simpler and can be studied in the same way.) Let $u \in \mathcal U$ and $w\in \mathcal W$ be arbitrary admissible open-loop controls, and let $x_{u,w}(T)$, $T >0$, be the corresponding trajectory. Then for each $\tau \in (0, T)$, we have that

$$\begin{aligned}{} & {} x_{u,v}(T)=e^{AT}x_0 +\int _0^T e^{A(T-s)}\left( Bu(s) +Ew(s)\right) ds \\{} & {} =e^{AT}x_0 +\int _0^\tau e^{A(T-s)}\left( Bu(s) +Ew(s)\right) ds +\int _\tau ^T e^{A(T-s)}\left( Bu(s) +Ew(s)\right) ds. \end{aligned}$$

Because the eigenvalues of the matrix A have negative real parts, there exist a constant $c>0$ and a real number $\alpha >0$ such that $\left\| e^{At}\right\| \le c e^{-\alpha t}$ for each $t\ge 0$. Hence,

Applying the H$\ddot{\text{ o }}$lder inequality, one can check that

and then to obtain the following two estimates:

$$\begin{aligned}{} & {} \left( \int _0 ^\tau \left\| e^{A(T-s)} \right\| ^2 \left\| B\right\| ^2 ds\right) ^{1/2} \left( \int _0 ^\tau \Vert u(s)\Vert ^2 ds\right) ^{1/2} \\{} & {} \quad \le \left\| B\right\| \left( \int _0 ^\tau c^2 e^{-2\alpha (T-s)} ds \right) ^{1/2} \Vert u\Vert _{L^2} \\{} & {} \quad =c \left\| B\right\| \left( \frac{e^{-2\alpha (T-\tau )}-e^{-2\alpha T} }{2\alpha }\right) ^{1/2} \Vert u\Vert _{L^2} \\{} & {} \quad \le c \left\| B\right\| \left( \frac{e^{-2\alpha (T-\tau )} }{2\alpha }\right) ^{1/2} \Vert u\Vert _{L^2} \end{aligned}$$

and

$$\begin{aligned}{} & {} \left( \int _0 ^\tau \left\| e^{A(T-s)} \right\| ^2 \left\| E\right\| ^2 ds\right) ^{1/2} \left( \int _0 ^\tau \Vert w(s)\Vert ^2 ds\right) ^{1/2} \\{} & {} \quad \le \left\| E\right\| \left( \int _0 ^\tau c^2 e^{-2\alpha (T-s)} ds \right) ^{1/2} \Vert w\Vert _{L^2} \\{} & {} \quad =c \left\| E\right\| \left( \frac{e^{-2\alpha (T-\tau )}-e^{-2\alpha T} }{2\alpha }\right) ^{1/2} \Vert w\Vert _{L^2} \\ {}{} & {} \quad \le c \left\| E\right\| \left( \frac{e^{-2\alpha (T-\tau )} }{2\alpha }\right) ^{1/2} \Vert w\Vert _{L^2}. \end{aligned}$$

Hence,

$$\begin{aligned} \int _0 ^\tau \left\| e^{A(T-s)}\Big ( Bu(s) + Ew(s)\Big )\right\| ds \le \frac{c}{\sqrt{2\alpha }} \Big (\left\| B\right\| \Vert u\Vert _{L^2} + \left\| E\right\| \Vert w\Vert _{L^2}\Big ) e^{- \alpha (T-\tau )}. \end{aligned}$$

Similarly, the following estimates holds true:

$$\begin{aligned}{} & {} \int _ \tau ^T \left\| e^{A(T-s)}\left( Bu(s) +Ew(s)\right) \right\| ds \\{} & {} \quad \le \left( \int _ \tau ^T \left\| e^{A(T-s)} \right\| ^2\left\| B\right\| ^2 ds\right) ^{1/2} \left( \int _\tau ^T \left\| u(s) \right\| ^2 ds\right) ^{1/2} \\{} & {} \qquad + \left( \int _ \tau ^T \left\| e^{A(T-s)} \right\| ^2\left\| E\right\| ^2 ds\right) ^{1/2} \left( \int _\tau ^T \left\| w(s) \right\| ^2 ds\right) ^{1/2} \\{} & {} \quad \le \left( \int _ \tau ^T c^2 e^{-2\alpha (T-s)} \left\| B\right\| ^2 ds\right) ^{1/2} \left( \int _\tau ^T \left\| u(s) \right\| ^2 ds\right) ^{1/2} \\{} & {} \qquad + \left( \int _ \tau ^T c^2 e^{-2\alpha (T-s)} \left\| E\right\| ^2 ds\right) ^{1/2} \left( \int _\tau ^T \left\| w(s) \right\| ^2 ds\right) ^{1/2} \\{} & {} \quad = \left\| B\right\| c \left( \frac{1- e^{-2\alpha (T-\tau )} }{2\alpha }\right) ^{1/2} \left( \int _\tau ^T \left\| u(s) \right\| ^2 ds\right) ^{1/2} \\{} & {} \qquad + \left\| E\right\| c \left( \frac{1- e^{-2\alpha (T-\tau )} }{2\alpha }\right) ^{1/2} \left( \int _\tau ^T \left\| w(s) \right\| ^2ds\right) ^{1/2} \\{} & {} \le \left\| B\right\| c \frac{1 }{\sqrt{2\alpha }} \left( \int _\tau ^T \left\| u(s) \right\| ^2 ds\right) ^{1/2} + \left\| E\right\| c \frac{1 }{\sqrt{2\alpha }} \left( \int _\tau ^T \left\| w(s) \right\| ^2 ds\right) ^{1/2} \\{} & {} \quad = \frac{c }{\sqrt{2\alpha }}\left( \left\| B\right\| \left( \int _\tau ^T \left\| u(s) \right\| ^2 ds\right) ^{1/2}+\left\| E\right\| \left( \int _\tau ^T \left\| w(s) \right\| ^2 ds\right) ^{1/2}\right) . \end{aligned}$$

Since $\tau$ is an arbitrary element of (0, T), we obtain that

$$\begin{aligned} \begin{aligned} \Vert x_{u,v}(T)\Vert&\le ce^{-\alpha T}\Vert x_0\Vert + \inf _{\tau \in (0,T)} \left\{ e^{- \alpha (T-\tau )} \frac{c }{\sqrt{2\alpha }} \Big (\left\| B\right\| \Vert u\Vert _{L^2} + \left\| E\right\| \Vert w\Vert _{L^2} \Big )\right. \\&\quad + \frac{c }{\sqrt{2\alpha }} \left( \left. \left\| B\right\| \left( \int _\tau ^T \left\| u(s) \right\| ^2 ds\right) ^{1/2} +\left\| E\right\| \left( \int _\tau ^T \left\| w(s) \right\| ^2 ds\right) ^{1/2} \right) \right\} , \end{aligned} \end{aligned}$$

and hence

$$\begin{aligned} \begin{aligned} 0 \le \lim _{T\rightarrow \infty }\Vert x_{u,v}(T)\Vert&\le \lim _{T\rightarrow \infty }\inf _{\tau \in (0,T)} \left\{ \frac{c e^{-\alpha (T-\tau )}}{\sqrt{2\alpha }} \Big (\left\| B\right\| \Vert u\Vert _{L^2} + \left\| E\right\| \Vert w\Vert _{L^2}\Big ) \right. \\&\quad + \frac{c }{\sqrt{2\alpha }} \left( \left\| B\right\| \left( \int _\tau ^T \left\| u(s) \right\| ^2 ds\right) ^{1/2} + \left. \left\| E\right\| \left( \int _\tau ^T \left\| w(s) \right\| ^2 ds\right) ^{1/2} \right) \right\} . \end{aligned} \end{aligned}$$

Because

$$\begin{aligned} \int _0^\infty \Vert u(s)\Vert ^2 ds< \infty \text{ and } \int _0^\infty \Vert w(s)\Vert ^2<\infty , \end{aligned}$$

for each $\varepsilon > 0$, there exists $\tau _\varepsilon$ such that

$$\begin{aligned} \int _{\tau _\varepsilon }^\infty \Vert u(s)\Vert ^2 ds< \varepsilon ^2 \text{ and } \int _{\tau _\varepsilon }^\infty \Vert w(s)\Vert ^2 ds <\varepsilon ^2. \end{aligned}$$

Moreover, there exists $T_\varepsilon >\tau _\varepsilon$ such that for each $T>T_\varepsilon$

$$\begin{aligned} e^{- \alpha (T-\tau _\varepsilon )} <\varepsilon . \end{aligned}$$

From here, it follows that

$$\begin{aligned}{} & {} \lim _{T\rightarrow \infty }\inf _{\tau \in (0,T)} \left\{ \frac{c e^{-\alpha (T-\tau )} }{\sqrt{2\alpha }} \Big (\left\| B\right\| \Vert u\Vert _{L^2} + \left\| E\right\| \Vert w\Vert _{L^2}\Big ) \right. \\{} & {} \quad +\frac{c }{\sqrt{2\alpha }} \left( \left\| B\right\| \left( \int _\tau ^T \left\| u(s) \right\| ^2 ds\right) ^{1/2} + \left. \left\| E\right\| \left( \int _\tau ^T \left\| w(s) \right\| ^2 ds\right) ^{1/2}\right) \right\} =0, \end{aligned}$$

and hence

$$\begin{aligned} \lim _{T\rightarrow \infty } x_{u,v}(T)=0. \end{aligned}$$

Then,

$$\begin{aligned} \begin{aligned} I (x_0, t_0, u,w)&= \int _{t_0}^\infty \left[ x_{u,w}^\top (t) Q x_{u,w}(t) + u^\top (t) R u(t)- w^\top (t) S w(t) \right] dt \\&=\int _{t_0}^\infty \left[ x_{u,w}^\top (t)\left( Q + P E S^{-1} E^\top P - P B R^{-1} B^\top P + A^\top P + P A \right) x_{u,w} (t) \right] dt \\&\quad +\int _{t_0}^\infty \left[ u^\top (t) R u(t) + u^\top (t)B^\top P x_{u,w}(t) \right] dt\\&\quad +\int _{t_0}^\infty \left[ x^\top _{u,w} (t) P B R^{-1} B ^\top P x_{u,w}(t) + x_{u,w}^\top (t) P B u(t) \right] dt \\&\quad -\int _{t_0}^\infty \left[ w^\top (t) S w(t) - w^\top (t)E^\top P x_{u,w}(t) \right] dt \\&\quad -\int _{t_0}^\infty \left[ x_{u,w}^\top (t) P E S^{-1} E ^\top P x_{u,w}(t) - x_{u,w}^\top (t) P E w(t) \right] dt \\&\quad -\int _{t_0}^\infty \left[ u^\top (t)B^\top P x_{u,w}(t) + x_{u,w}^\top (t) P B u(t) \right] dt\\&\quad -\int _{t_0}^\infty \left[ w^\top (t)E^\top P x_{u,w}(t) + x_{u,w}^\top (t) P E w(t) \right] dt \\&\quad -\int _{t_0}^\infty \left[ x^\top (t)\left( A^\top P + P A \right) x_{u,w}(t) \right] dt \\&=\int _{t_0}^\infty \left[ \left\| u(t) + R^{-1} B^\top P x_{u,w}(t) \right\| _R^2 - \left\| w(t)-S^{-1}E^\top P x_{u,w}(t) \right\| _S^2\right] dt \\&\quad -\int _{t_0}^\infty \frac{d}{dt} \left[ x_{u,w}^\top (t) P x_{u,w}(t)\right] dt \\&= x_0^\top P x_0 +\int _{t_0}^\infty \left\| u(t)+R^{-1}B^\top P x_{u,w}(t) \right\| _R ^2 dt \\&- \int _{t_0}^\infty \left\| w(t)-S^{-1}E^\top P x_{u,w}(t) \right\| _S^2 dt, \end{aligned} \end{aligned}$$

i.e.,

$$\begin{aligned} I (x_0, t_0, u,w) = x_0^\top P x_0+ \end{aligned}$$

(13)

$$\begin{aligned} \int _{t_0}^\infty \left\| u(t)+R^{-1}B^\top P x_{u,w}(t) \right\| _R ^2 dt - \int _{t_0}^\infty \left\| w(t)-S^{-1}E^\top P x_{u,w}(t) \right\| _S^2 dt. \end{aligned}$$

The definition of ${\bar{u}}$ and ${\bar{w}}$ (according to (4)) implies that

$$\begin{aligned} {\bar{u}}(x_{{\bar{u}}, {\bar{w}}}(t) )=-R^{-1}B^\top P x_{{\bar{u}}, {\bar{w}}}(t) \text{ and } {\bar{w}}(x_{{\bar{u}}, {\bar{w}}}(t) )=S^{-1}E^\top P x_{\bar{u},{\bar{w}}}(t). \end{aligned}$$

Then,

$$\begin{aligned} I(x_0, t_0, {\bar{u}}, {\bar{w}})= x_0^\top P x_0. \end{aligned}$$

Also,

$$\begin{aligned} I(x_0, t_0, u, {\bar{w}}) = \int _{t_0}^\infty \Vert u(t)+R^{-1}B^\top P x_{u,{\bar{w}}}(t) \Vert _R^2 dt + x_0^\top P x_0 \ge x_0^\top P x_0 \end{aligned}$$

and

$$\begin{aligned} I(x_0, t_0, {\bar{u}}, w)= - \int _{t_0}^\infty \Vert w(t)- S^{-1}E^\top P x_{{\bar{u}},w}(t) \Vert _S ^2 dt + x_0^\top P x_0 \le x_0^\top P x_0. \end{aligned}$$

Hence, we have obtained that

$$\begin{aligned} I(x_0, t_0, {\bar{u}}, w) \le I(x_0, t_0, {\bar{u}}, {\bar{w}}) \le I(x_0, t_0, u, {\bar{w}}) \end{aligned}$$

for each control function pair $u\in \mathcal U$ and $w \in \mathcal W$, i.e., $({\bar{u}}, {\bar{w}})$ is a saddle point (Nash equilibrium) for the considered differential game. $\square$

Proof

Since U is a convex neighborhood of the origin in ${\mathbb {R}}^k$, there exists $\varepsilon >0$ such that the closed ball $\varepsilon \bar{\textbf{ B }}_k \subset {\mathbb {R}}^k$ is a subset of U. We can choose the positive real $\delta _0>0$ to be so small that the norm of the vector $-R^{-1}B^\top Px$ is less than $\varepsilon$ for each $x \in \delta _0 \bar{\textbf{ B }}_n$, and hence $-R^{-1}B^\top Px\in U$. Thus, i) holds true for each $0<\delta \le \delta _0$.

Let

$$\begin{aligned} \bar{\alpha }:= \min \{x^\top P x:\ \Vert x\Vert =\delta _0\}. \end{aligned}$$

Because the matrix P is positive definite, the real $\bar{\alpha }$ is positive. We choose an arbitrary $\alpha \in (0, \bar{\alpha })$ and set

$$\begin{aligned} \Omega := \{x \in \delta _0\bar{\textbf{ B }}_n: \ x^\top P x \le \alpha \}. \end{aligned}$$

Clearly, the set $\Omega$ is a compact neighborhood of the origin contained in the interior of the ball $\delta _0\bar{\textbf{ B}}_n \subset {\mathbb {R}}^n$.

Let x be an arbitrary point from $\Omega$. The definition of the set $\Omega$ implies that $x \in \delta _0\bar{\textbf{ B }}_n$ and $x^\top P x \le \alpha$. Then, the relation $x \in \delta _0\bar{\textbf{ B}}_n$ implies that $-R^{-1}B^\top Px \in U$.

Let us fix an arbitrary point y from $\Omega$ and a real $\tau \ge t_0$. Then, the trajectory ${\bar{x}}_{{\bar{u}}, \bar{w}}(\cdot ,y,\tau )$ corresponding to the controls ${\bar{u}}$ and $\bar{w}$ and starting from y at the moment of time $\tau$ is the solution of the following Cauchy problem:

$$\begin{aligned} \dot{x}(t)= {\bar{A}} x(t), ~~x(\tau )=y, \end{aligned}$$

where ${\bar{A}}=A -BR^{-1}B^\top P + ES^{-1}E^\top P$. Because P is a symmetric positive definite solution of the matrix algebraic Riccati equation

$$\begin{aligned} Q + XES^{-1}E^\top X - XBR^{-1}B^\top X + A^\top X + X A=0, \end{aligned}$$

one can check that

$$\begin{aligned} P{\bar{A}} + {\bar{A}}^\top P = - {\bar{Q}}. \end{aligned}$$

Indeed, we have that

$$\begin{aligned}{} & {} P(A -BR^{-1}B^\top P + ES^{-1}E^\top P) + (A -BR^{-1}B^\top P + ES^{-1}E^\top P)^\top P \\{} & {} \quad =PA -PBR^{-1}B^\top P + PES^{-1}E^\top P + A^\top P - P B R^{-1}B^\top P + PES^{-1}E^\top P \\{} & {} \quad =P A + A^\top P -2PBR^{-1}B^\top P + 2PES^{-1}E^\top P \\{} & {} \quad = -Q -PBR^{-1}B^\top P + PES^{-1}E^\top P= - {\bar{Q}}. \end{aligned}$$

Because ${\bar{Q}}$ is a symmetric positive definite matrix, the last equality implies that the function $V_L:{\mathbb {R}}^n \rightarrow {\mathbb {R}}$ defined as $V_L(x) = x^\top Px$ is a Lyapunov function for the system

$$\begin{aligned} \dot{x} = {\bar{A}} x, \end{aligned}$$

(14)

and because the function $V _L({\bar{x}}(\cdot ))$ is decreasing, we obtain that

$$\begin{aligned} V_L( {\bar{x}}(t)) \le V_L( x_0)\le \alpha \end{aligned}$$

for each $t \ge t_0$, i.e., ${\bar{x}}_{{\bar{u}},{\bar{w}} }(t,y,\tau ) \in \Omega$ for each $t\ge t_0$. Thus, ii) also holds true.

The positive definiteness of matrices P and Q implies that the system (14) is asymptotically stable at the origin, and hence ${\bar{x}}_{{\bar{u}},{\bar{w}} }(t,y,\tau )$ tends to zero as $t \rightarrow +\infty$. This completes the proof. $\square$

Proof

Let us assume that $x_{{\hat{u}}, {\hat{w}}}( t)$ does not belong to $\delta \bar{\textbf{ B }}_n$ for each $t\ge t_0$, i.e., for each $t\ge t_0$, we have that $\Vert x_{{\hat{u}}, {\hat{w}}}( t)\Vert >\delta$. The positive definiteness of the matrices P and Q implies the existence of positive reals p and q such that $x^\top P x \ge p \Vert x\Vert ^2$ and $x^\top Q x \ge q \Vert x\Vert ^2$ for each $x \in \mathbb R^n$. Then,

$$\begin{aligned}{} & {} I_T(x_0,t_0, {\hat{u}},{\hat{w}}):= \int _{t_0}^T \left( x^\top _{{\hat{u}}, {\hat{w}}} (t) Q x_{{\hat{u}}, {\hat{w}}}(t) + {\hat{u}}^\top (t) R {\hat{u}}(t)- {\hat{w}}^\top (t) S {\hat{w}}(t) \right) dt + x^\top _{{\hat{u}}, {\hat{w}}} (T) P x_{{\hat{u}}, {\hat{w}}}(T) \\{} & {} >(q (T-t_0)+ p)\delta ^2 - \int _{t_0}^T \left( {\hat{w}}^\top (t) S {\hat{w}}(t) \right) dt \ge (q (T-t_0)+ p)\delta ^2 - \Vert S\Vert \int _{t_0}^T \Vert {\hat{w}} (t)\Vert ^2 dt. \end{aligned}$$

Then,

$$\begin{aligned}{} & {} {\bar{V}}_U(x_0, t_0)=\lim _{T\rightarrow \infty } I_T(x_0,t_0, {\hat{u}},{\hat{w}}) \\{} & {} \ge \lim _{T\rightarrow \infty } \left( (q (T-t_0)+ p)\delta ^2 - \Vert S\Vert \int _{t_0}^T \Vert {\hat{w}} (t)\Vert ^2 dt \right) . \end{aligned}$$

The last inequality is impossible because $\int _{t_0}^{\infty } \Vert {\hat{w}} (t)\Vert ^2<\infty$. The obtained contradiction shows that there exists $T\ge t_0$ such that $x_{{\hat{u}}, {\hat{w}}}( T) \in \delta \bar{\textbf{ B }}_n$. According to Proposition 2, we obtain that $x_{{\hat{u}}, {\hat{w}}}( t) \in \delta \bar{\textbf{ B }}_n$ for each $t\ge T$ and $x_{{\hat{u}}, {\hat{w}}}( t) \rightarrow 0$ as $t\rightarrow \infty$. This completes the proof. $\square$

Proof

Because $\Vert x_{{\hat{u}}, {\hat{w}}} (T)\Vert \le \delta$, Proposition 2 implies that

$$\begin{aligned}{} & {} {\bar{V}}_{U } (x _{{\hat{u}}, {\hat{w}}} (T), T) =x^\top _{{\hat{u}}, {\hat{w}}} (T) P x_{{\hat{u}}, {\hat{w}}}(T)= {\bar{V}} (x _{{\hat{u}}, {\hat{w}}} (T), T) \\{} & {} =\int _{T}^\infty \left( x^\top _{{\bar{u}}, {\bar{w}}} (t) Q x_{{\bar{u}}, {\bar{w}}}(t) + {\bar{u}}^\top (t) R {\bar{u}}(t)- {\bar{w}}^\top (t) S \bar{w}(t) \right) dt. \end{aligned}$$

From here, we obtain that

$$\begin{aligned} V_{U, T} (x_0, t_0 ){} & {} =\int _{t_0}^{T} \left( x^\top _{{\hat{u}}, {\hat{w}}} (t) Q x_{{\hat{u}}, {\hat{w}}}(t) + {\hat{u}}^\top (t) R {\hat{u}}(t)- {\hat{w}}^\top (t) S \hat{w}(t) \right) dt + x^\top _{{\hat{u}}, {\hat{w}}} (T) P x_{{\hat{u}}, \hat{w}}(T)\\{} & {} =\int _{t_0}^T \left( x^\top _{{\hat{u}}_\infty , {\hat{w}}_\infty } (t) Q x_{{\hat{u}}_\infty , {\hat{w}}_\infty }(t) + {\hat{u}}^\top _\infty (t) R {\hat{u}}_\infty (t)- {\hat{w}}^\top _\infty (t) S {\hat{w}}_\infty (t) \right) dt \\{} & {} \quad + \int _{T}^\infty \left( x^\top _{{\hat{u}}_\infty , {\hat{w}}_\infty } (t) Q x_{{\hat{u}}_\infty , {\hat{w}}_\infty }(t) + \hat{u}^\top _\infty (t) R {\hat{u}}_\infty (t)- {\hat{w}}^\top _\infty (t) S {\hat{w}}_\infty (t) \right) dt \\{} & {} =I(x_0,t_0, {\hat{u}}_\infty , {\hat{w}}_\infty ) \end{aligned}$$

Let u be an arbitrary element of $\mathcal{U}_U$ defined on $[t_0, +\infty )$. We fix an arbitrary $\varepsilon >0$ and choose $\hat{w}_{u,\varepsilon } \in \mathcal W$ so that

$$\begin{aligned} \sup _{w \in \mathcal W} I_T (x_0,t_0, u, w ) <I_T (x_0,t_0, u, {\hat{w}}_{u,\varepsilon }) + \varepsilon . \end{aligned}$$

Then,

$$\begin{aligned} V_{U, T} (x_0, t_0 ){} & {} = I_{ T} (x_0,t_0, {\hat{u}}, {\hat{w}}) \le \sup _{w \in \mathcal W} I_T (x_0,t_0, u, w ) < I_{ T} (x_0,t_0, u, {\hat{w}}_{u,\varepsilon }) + \varepsilon \\{} & {} =\int _{t_0}^{T} \left( x^\top _{ u, {\hat{w}}_{u,\varepsilon }} (t) Q x_{ u, {\hat{w}}_{u,\varepsilon }}(t) + u^\top (t) R u(t)- \hat{w}_{u,\varepsilon }^\top (t) S {\hat{w}}_{u,\varepsilon }(t) \right) dt \\{} & {} \quad +x^{ \top }_{ u, {\hat{w}}_{u,\varepsilon }} (T) P x_{ u, \hat{w}_{u,\varepsilon }}(T)+ \varepsilon \\{} & {} =\int _{t_0}^{T} \left( x^\top _{ u, {\hat{w}}_{u,\varepsilon }} (t) Q x_{ u, {\hat{w}}_{u,\varepsilon }}(t) + u^\top (t) R u(t)- \hat{w}_{u,\varepsilon }^\top (t) S {\hat{w}}_{u,\varepsilon }(t) \right) dt\\{} & {} \quad +{\bar{V}} (x_{ u, {\hat{w}}_{u,\varepsilon }}(T), T) + \varepsilon \le \end{aligned}$$

(here we take into account (5))

$$\begin{aligned}{} & {} \int _{t_0}^{T} \left( x^\top _{ u, {\hat{w}}_{u,\varepsilon }} (t) Q x_{ u, {\hat{w}}_{u,\varepsilon }}(t) + u^\top (t) R u(t)- \hat{w}_{u,\varepsilon }^\top (t) S {\hat{w}}_{u,\varepsilon }(t) \right) dt\\{} & {} \quad +\int _{T}^\infty \left( x^\top _{ u, {\bar{w}} } (t) Q x_{ u, {\bar{w}} }(t) + u^\top (t) R u(t)- {\bar{w}}^\top (t) S {\bar{w}} (t) \right) dt + \varepsilon \\{} & {} \le \sup _{w \in \mathcal W} I (x_0, t_0,u, w )+ \varepsilon . \end{aligned}$$

If we let $\varepsilon$ to tend to zero, then we obtain that

$$\begin{aligned} I (x_0,t_0, {\hat{u}}_\infty , {\hat{w}}_\infty )= V_{U, T} (x_0, t_0 ) \le \sup _{w \in \mathcal W} I (x_0, t_0,u, w ). \end{aligned}$$

Because u is an arbitrary element of $\mathcal{U}_U$, the last inequality implies that

$$\begin{aligned} I (x_0,t_0, {\hat{u}}_\infty , {\hat{w}}_\infty ) \le \inf _{u\in \mathcal{U}_U}\sup _{w \in \mathcal W} I (x_0, t_0,u, w ) \end{aligned}$$

This completes the proof. $\square$

Proof

The corresponding proof is similar to the proof of Proposition 4. $\square$