On the stochastic linear quadratic control problem with piecewise constant admissible controls

doi:10.1016/j.jfranklin.2019.10.036

Journal of the Franklin Institute

Volume 357, Issue 3, February 2020, Pages 1532-1559

https://doi.org/10.1016/j.jfranklin.2019.10.036 Get rights and content

Abstract

A linear quadratic optimal control problem for a system described by Itô differential equations with state and control dependent white noise under the assumption that the set of admissible controls consists of a class of piecewise constant stochastic processes is considered. The considered LQ optimal control problem is converted into a LQ optimization problem for a stochastic controlled system with finite jumps and multiplicative white noise perturbations. One of the original contribution of this work is the proof of the equivalence between the solvability of the considered optimal control problem and the solvability of the problem with given terminal values associated to a matrix linear differential equation (MLDE) with finite jumps and constraints. Another original contribution consists in the proof of the global existence of the solution of the problem with given terminal value of the MLDE if the cost weights matrices are positive semidefinite. The results obtained in the case of a LQ optimal control for systems with finite jumps are then applied to derive explicit formulae of the optimal controls for the optimization problem under piecewise constant controls.

Introduction

One of the most popular optimal control problem in both deter ministic and stochastic framework is the so called linear quadratic (LQ) optimal control problem. In the time domain setting there are two main approaches of the solution to a LQ optimization problem. The first approach known as the open loop optimal control problem is based on the direct application of the Pontryagin’s minimal principle [21] leading to a set of necessary conditions expressed in terms of solvability of a two points boundary value problem with linear constraints that must be satisfied by the optimal control.

The other approach known as the closed loop optimal control problem provides a set of sufficient conditions for the existence of an optimal control in a state feedback form. Starting with the pioneering work of [17] the gain matrices of the optimal state feedback are computed based on the solution to a matrix Riccati differential equation (MRDE). A new kind of Riccati differential equation called matrix Riccati differential equations of stochastic control was introduced in [29].

The solution with given terminal value (TVP) of this type of MRDEs was involved in the designing of the gain matrices of the optimal control in a LQ optimal control problem associated to a controlled system modeled by Itô differential equations with state multiplicative white noise perturbations. Since the general theory of differential equations applied in the special case of the solutions with given terminal values of a MRDE guarantees only the local existence of these solutions, it is of interest the study of the prolongability of the solution to a MRDE on the whole interval [t₀, τ] where the optimal control problem is considered. This may be viewed as a challenging problem with interest in itself and it was intensively studied in the literature. Here we refer only to the monographes [1], [6], [7] and their references.

In [5] one shows that unlike the deterministic framework, in the stochastic case when the controlled system is described by Itô differential equations with control dependent of the diffusion part, the LQ optimization problem is still well possed in the case when the cost weights matrices of the states and controls are allowed to be indefinite. In [23] a new type of MRDEs named generalized MRDEs was introduced and was proved that the solvability of a LQ optimization problem is equivalent to the solvability of this new type of generalized MRDEs. Since a generalized MRDE involves algebraic equalities / inequalities which must be satisfied for any t ∈ [t₀, τ] it is hard to prove the global existence of the solution with given terminal values of this kind of MRDE even in the case of cost weights matrices with definite sign.

In the present work we show that modifying the class of admissible controls considered in [23] but preserving the controlled system and the quadratic functional we obtain a new LQ optimization problem whose solvability is equivalent to a set of conditions more relax than the ones from afore mentioned reference.

In our approach the set of admissible controls consists of a class of piecewise constant stochastic processes. The problem of LQ optimal control in the class of piecewise constant controls can be converted into a LQ optimal control problem for a system with finite jumps That is why the whole Section 3 is devoted to the solution to a LQ optimal control problem for a stochastic system with finite jumps and multiplicative white noise perturbations. We show that a LQ optimization problem for a stochastic system with finite jumps is equivalent to the global existence on the whole interval [t₀, τ] of the solution to the TVP associated to a matrix linear differential equation (MLDE) with finite jumps. We prove that if the weights cost matrices are positive semidefinite then, in the absence of any other additional assumptions, the TVP associated to the involved MLDE with finite jumps has a global solution. The results obtained in the general case of a LQ optimization problem for a linear stochastic system with finite jumps and multiplicative white noise perturbations are then specialized to derive necessary and sufficient conditions for solvability of a stochastic LQ optimization problem in the class of piecewise constant admissible controls. Since the obtained optimal controls are in a state feedback form involving only values x(t_k) of the states measured at the discrete-time instances t_k it follows that the results inclosed in Section 4 can be viewed as solution of a LQ optimization problem by sampling for a controlled system modeled by Itô differential equation with state dependent and control dependent diffusion part.

Sampled-data systems with periodic sampling have scored a great success in the literature. In the deterministic framework there are numerous works dealing with various robust control problems by sampling, see e.g. [3], [12], [13], [14], [15], [18], [24] to cite only few of them. In the stochastic framework we refer to [3], [4], [10], [16], [22], [25], [26].

The rest of the paper is organized as follows: in Section 2 the LQ optimization problem under consideration is stated and it is shown how it can be converted in a LQ optimization problem for a stochastic system with finite jumps. The main results of the paper can be found in Section 3 and Section 4. So, in Section 3, the solution to a general LQ optimization problem for a stochastic system with finite jumps and multiplicative white noise perturbations is presented. In Section 4, the solution to LQ optimization problem by piecewise constant controls is derived. Section 5 includes some numerical experiments to show the feasibility of the obtained results. The paper end to some conclusion and topics for future developments.

Section snippets

Stochastic linear quadratic optimization problem revisited

Let us consider the optimal control problem described by the controlled system: $\begin{matrix} d x (t) & = & [A_{0} (t) x (t) + B_{0} (t) u (t)] d t + [A_{1} (t) x (t) + B_{1} (t) u (t)] d w (t), \\ t & \geq & t_{0} \geq 0, \\ x (t_{0}) & = & x_{0}, \end{matrix}$ and the quadratic performance criterion $J (t_{0}, x_{0}; u (\cdot)) = E [x_{u}^{T} (τ) G x_{u} (τ) + \int_{t_{0}}^{τ} (x_{u}^{T} (t) M (t) x_{u} (t) + u^{T} (t) R (t) u (t)) d t],$ where x_u(t), t₀ ≤ t ≤ τ is the solution to the initial value problem (IVP) (1) corresponding to the input u( · ). In (1), $x (t) \in R^{n}$ is the state vector, $u (t) \in R^{m}$ is the vector of control parameters, and {w(t)}_t ≥ 0 is a 1-dimensional standard

Model description. Setting of the optimization problem

Let us consider the controlled system with finite jumps described by $\begin{matrix} d x (t) = A_{0} (t) x (t) d t + A_{1} (t) x (t) d w (t), \\ t_{k} \leq t \leq t_{k + 1} \\ x (t_{k}^{+}) = A_{0 d} (k) x (t_{k}) + B_{0 d} (k) u_{k} + w_{d} (t_{k}) (A_{1 d} (k) x (t_{k}) + B_{1 d} (k) u_{k}), \\ 0 \leq k \leq N - 1, \\ x (t_{0}) = x_{0}, \end{matrix}$ where $x (t) \in R^{n_{x}}$ is the state vector at the instance time t and $u_{k} \in R^{n_{u}}$ is the vector of control parameters at the instance $t_{k}, 0 = t_{0} < t_{1} < \dots < t_{N - 1} < t_{N} = τ,$ being a partition of the interval $[t_{0}, τ] \subset R_{+}$ .

In (17) {w(t)}_t ≥ 0 is a 1-dimensional standard Wiener process and $w_{d} (t_{0}), w_{d} (t_{1}), \dots, w_{d} (t_{N - 1})$ are independent random variables

The solution of the linear quadratic optimization problem by piecewise constant controls

In this section we use the results proved in the previous section to obtain conditions that guarantee the existence of controls that minimizes the quadratic functional (2) over the solution of the system (1) determined by piecewise constant controls of type (7). To this end, we shall apply the result proved in Theorem 1 in the special case of the system (11)–(12) and of the performance criterion (13)–(14).

To derive the main result of this section we shall take into account that the system (11)

Numerical experiments

Let us consider the classic example of a fourth-order model representing a nominal model for the CE150 helicopter model described by Yoneyama et al. [26]. The matrices occurring in Eq. (1) are shown in the following. $\begin{matrix} A_{0} (t) = (\begin{matrix} 0 & 0 & 10 & 0 \\ 0 & 0 & 0 & 10 \\ - 14.5076 & 68.5210 & - 2.0568 & 0 \\ 0 & - 25 & 0 & - 10 \end{matrix}), \\ B_{0} (t) = (\begin{matrix} 0 \\ 15 \\ 0 \\ 25 \end{matrix}), \\ A_{1} (t) = 0.1 A_{0} (t), B_{1} (t) = 0.1 B_{0} (t) . \end{matrix}$ It should be noted that 10% of the magnitudes of the state and input matrices can be represented by the Wiener process based on stochastic perturbations as the state and control dependent

Conclusion

In this paper a linear quadratic optimal control problem for a system described by Itô differential equations with state and control dependent white noise under the assumption that the set of admissible controls consists of a class of piecewise constant stochastic processes was considered. The considered LQ optimal control problem was converted into a LQ optimization problem for a stochastic controlled system with finite jumps and multiplicative white noise perturbations.

The two main original

References (30)

C. Briat
Convex conditions for robust stability analysis and stabilization of linear aperiodic impulsive and sampled-data systems under dwell-time constraints
Automatica
(2013)
C. Briat
Stability analysis and stabilization of stochastic linear impulsive, switched and sampled-data systems under dwell-time constraints
Automatica
(2016)
A. Friedman
Stochastic Differential Equations and Applications, Vol. I
(1975)
R.E. Kalman
Contribution to the theory of optimal control
Boletin de la Sociedad Matematica Mexicana
(1960)
M.A. Rami et al.
Indefinite stochastic linear quadratic control and generalized differential Riccati equation
SIAM J. Control Optim.
(2001)
B. Wang et al.
Stability analysis of semi-Markov switched stochastic systems
Automatica
(2018)
W.M. Wonham
On a matrix Riccati equation of stochastic control
SIAM J. Control Optim.
(1968)
H. Abou-Kandil et al.
Matrix Riccati Equations in Control Systems Theory
(2003)
A. Albert
Conditions for positive and nonnegative definiteness in terms of pseudo-inverses
SIAM, J. Appl. Math.
(1969)
S. Chen et al.
Stochastic linear quadratic regulators with indefinite control weight costs
SIAM J. Control Optim.
(1998)

T. Damm

Rational matrix equations in stochastic control

Lecture Notes in Control and Information Sciences

(2004)

V. Dragan et al.

Mathematical Methods in Robust Control of Linear Stochastic Systems

(2013)

V. Dragan et al.

On the mean square minimization of the final value of an output of a linear stochastic controled system

Math. Appl. / Ann. AOSR

(2018)

V. Dragan et al.

A stochastic linear quadratic optimization problem with sampled measurements

Innovat. Model. Anal. J. of Res.

(2018)

V. Dragan et al.

Optimal H₂ filtering for periodic linear stochastic systems with multiplicative white noise perturbations and sampled measurements

J. Frankl. Inst.

(2015)

Cited by (11)

Weight splitting iteration methods to solve quadratic nonlinear matrix equation MY<sup>2</sup>+NY+P=0
2023, Journal of the Franklin Institute
The quadratic nonlinear matrix equationoccurs in many applications such as the Quasi-Birth-Death processes, pseudo-spectra for quadratic eigenvalue problems and the quadratic eigenvalue problems. In this work, we propose efficient parametric iterative methods for finding the solution of this quadratic matrix equation based on weight splitting (WS) on matrices $M$ and $N$ , separately. We show that the proposed methods converge to the solution of the quadratic nonlinear matrix equation under conditions. Every iteration in proposed methods requires the solution of one or two linear matrix equations. Finally, various numerical examples indicate that the obtained methods in terms of CPU time, accuracy and computational cost are superior to the famous methods.
Piecewise parameterization for multifactor uncertain system and uncertain inventory-promotion optimization
2022, Knowledge-Based Systems
Citation Excerpt :
Wu et al. [3] obtained an analytical optimal control of stochastic LQ optimal control problem. Drăgan and Ivanovc [4] introduced a new feasible region of optimal control for stochastic LQ optimal control problem. Yaghobipour and Yarahmadi [5] designed a solving algorithm for the quantum stochastic LQ optimal control problem.
The analytic solution provides a great deal of convenience for linear quadratic (LQ) model in industrial implementation. But, it usually dependents on the solution of Riccati differential equation. In many cases, this will increase the complexity and design cost of controller because we are not able to find the analytical solution of Riccati differential equation. In addition, the dynamic systems may be disturbed by more than one uncertain factor. Here, we discuss the piecewise parameterization for multifactor uncertain system and an uncertain inventory promotion optimization problem. First, we obtain an analytic optimal control of multifactor uncertain LQ model. For simplifying the expression of optimal control, a parametric multifactor uncertain LQ model is formulated. Moreover, a control variable piecewise parameterization method is presented for obtaining the optimal control parameters. Finally, an inventory promotion optimization problem under uncertain environment is considered for demonstrating the effectiveness of the formulated multifactor uncertain model and presented method.
On the stochastic linear quadratic optimal control problem by piecewise constant controls: The infinite horizon time case
2024, Mathematical Methods in the Applied Sciences
Developing HSS iteration schemes for solving the quadratic matrix equation AX2+BX+C=0
2024, IET Control Theory and Applications
APPROXIMATION OF LINEAR CONTROLLED DYNAMICAL SYSTEMS WITH SMALL RANDOM NOISE AND FAST PERIODIC SAMPLING
2023, Mathematical Control and Related Fields
Stochastic Linear Quadratic Game for Discrete-time Systems Based-on Adaptive Dynamic Programming
2022, 2022 4th International Conference on Control and Robotics, ICCR 2022

View all citing articles on Scopus

View full text

On the stochastic linear quadratic control problem with piecewise constant admissible controls

Abstract

Introduction

Section snippets

Stochastic linear quadratic optimization problem revisited

Model description. Setting of the optimization problem

The solution of the linear quadratic optimization problem by piecewise constant controls

Numerical experiments

Conclusion

Automatica

Automatica

Boletin de la Sociedad Matematica Mexicana

SIAM J. Control Optim.

Automatica

SIAM J. Control Optim.

Matrix Riccati Equations in Control Systems Theory

Conditions for positive and nonnegative definiteness in terms of pseudo-inverses

SIAM, J. Appl. Math.

Stochastic linear quadratic regulators with indefinite control weight costs

SIAM J. Control Optim.

Rational matrix equations in stochastic control

Lecture Notes in Control and Information Sciences

Mathematical Methods in Robust Control of Linear Stochastic Systems

On the mean square minimization of the final value of an output of a linear stochastic controled system

Math. Appl. / Ann. AOSR

A stochastic linear quadratic optimization problem with sampled measurements

Innovat. Model. Anal. J. of Res.

Optimal H2 filtering for periodic linear stochastic systems with multiplicative white noise perturbations and sampled measurements

J. Frankl. Inst.

Optimal H₂ filtering for periodic linear stochastic systems with multiplicative white noise perturbations and sampled measurements