Approximate finite-horizon optimal control without PDEs☆
Introduction
The problem of controlling the state of a system during a desired time interval, which is generally set a priori, and to minimize, along the trajectory of the resulting closed-loop system, a criterion of optimality is crucial in control system applications [1], [2], [3]. The problem informally defined above is commonly referred to as finite-horizon optimal control problem.
Two different approaches are available in the literature to solve the problem, namely the Minimum Principle and the Dynamic Programming approach [4], [2], [5], [6], [7], [8]. The former hinges upon the definition of the Hamiltonian associated to the optimal control problem that must be minimized by the optimal control law. The latter is based on the principle of optimality [1], which formalizes the intuitive requirement that a truncation of the optimal control law must be optimal with respect to the resulting truncated problem. Obviously, each approach has its own advantages and drawbacks.
The solution relying on the Minimum Principle–which provides only necessary conditions for optimality–yields in general an open-loop control law defined in terms of an adjoint state, or costate, satisfying an ordinary differential equation with a boundary condition imposed on the value of the costate at the terminal time. Therefore, determining the trajectories of the state and the costate consists in finding the solution of a two-point boundary value problem.
The Dynamic Programming approach, on the other hand, provides necessary and sufficient conditions for optimality and the resulting control policy is a memoryless time-varying feedback. This methodology hinges upon the computation of the solution of the Hamilton–Jacobi–Bellman partial differential equation, solution which may be in general difficult or impossible to compute in closed-form.
The main contribution of this paper is a method to construct dynamically, i.e. by means of a dynamic extension, an exact solution of a (modified) HJB pde for input-affine nonlinear systems without actually solving any partial differential equation. The extended closed-loop system is a system of ordinary differential equations with two-point boundary conditions. However, differently from the Minimum Principle, if an approximate solution is sought then the solution of the two-point boundary value problem can be avoided and the cost of this approximation can be explicitly quantified. A preliminary version of this work has been published in [9].
The rest of the paper is organized as follows. In Section 2 the definition of the problem is given and the basic notation is introduced. A notion of solution of the Hamilton–Jacobi–Bellman partial differential equation is provided in Section 3. The design of a dynamic time-varying state feedback that approximates the solution of the finite-horizon optimal control problem is presented in Section 4. The paper is completed by a numerical example and by some conclusions in Sections 5 Numerical example, 6 Conclusions, respectively.
Section snippets
Definition of the problem
Consider a nonlinear system described by equations of the form where is the state of the system, while denotes the control input. The mappings and are assumed to be sufficiently smooth. Moreover suppose that , hence there exists a non-unique continuous matrix-valued function such that for all . The finite-horizon optimal control problem consists in determining a control input that minimizes the cost functional1
Algebraic solution and value function
In this section a notion of solution of the Hamilton–Jacobi–Bellman equation (3) is presented, see [10], [11], [12] for a similar notation in the case . To this end, consider the augmented system with , and . Note that the partial differential equation (3) can be rewritten as Following [11], [12], consider the HJB equation (13) and suppose that it can be solved algebraically, as
Main results
We first present the main ideas of the proposed approach in the case of linear systems. To this end consider the system (10) and the cost (2) with . From the definition of algebraic solution of the Eq. (13), we expect to approximate (in the sense defined in (14)) the partial derivative of the value function with respect to the state , whereas represents the partial derivative of with respect to time. Therefore, in the linear case an algebraic solution is given by
Numerical example
To illustrate the results of the paper consider the nonlinear system with , and the cost Note that no running cost is imposed on the state of the system, hence only the position of the state at the terminal time is penalized, together with the control effort. Let be the solution of the differential Riccati equation (11) with the boundary condition . Note that the
Conclusions
The finite-horizon optimal control problem for input-affine nonlinear systems is studied within the framework of dynamic programming. It is shown that the explicit computation of the solution of the Hamilton–Jacobi–Bellman partial differential equation can be avoided provided an additional cost is paid. The methodology makes use of a dynamic extension yielding a dynamic time-varying control law that solves the approximate regional dynamic finite-horizon optimal control problem. The closed-loop
References (13)
Dynamic Programming
(1957)Dynamic Programming and Optimal Control—Volume I
(2005)- et al.
Applied Optimal Control: Optimization, Estimation, and Control
(1975) - et al.
Optimal Control: Linear Quadratic Methods
(1989) An Introduction to Applied Optimal Control
(1981)- et al.
The Mathematical Theory of Optimal Processes
(1962)
Cited by (14)
Approximate finite-horizon optimal control for input-affine nonlinear systems with input constraints
2013, IFAC Proceedings Volumes (IFAC-PapersOnline)A new modified polynomial-based optimal control design approach
2021, Proceedings of the Institution of Mechanical Engineers. Part I: Journal of Systems and Control EngineeringAn algebraic approach to dynamic optimisation of nonlinear systems: a survey and some new results
2019, Journal of Control and DecisionApproximate Infinite-Horizon Optimal Control for Stochastic Systems
2018, Proceedings of the IEEE Conference on Decision and ControlDynamic demand and mean-field games
2017, IEEE Transactions on Automatic ControlA game theoretic approach to distributed control of homogeneous multi-Agent systems
2017, 2017 IEEE 56th Annual Conference on Decision and Control, CDC 2017
- ☆
This work is partially supported by the Austrian Center of Competence in Mechatronics (ACCM), by the EPSRC Programme Grant Control for Energy and Sustainability EP/G066477 and by the MIUR under PRIN Project Advanced Methods for Feedback Control of Uncertain Nonlinear Systems.