Bounded robust control of nonlinear systems using neural network–based HJB solution

Abstract

In this paper, a Hamilton–Jacobi–Bellman (HJB) equation–based optimal control algorithm for robust controller design is proposed for nonlinear systems. The HJB equation is formulated using a suitable nonquadratic term in the performance functional to tackle constraints on the control input. Utilizing the direct method of Lyapunov stability, the controller is shown to be optimal with respect to a cost functional, which includes penalty on the control effort and the maximum bound on system uncertainty. The bounded controller requires the knowledge of the upper bound of system uncertainty. In the proposed algorithm, neural network is used to approximate the solution of HJB equation using least squares method. Proposed algorithm has been applied on the nonlinear system with matched and unmatched type system uncertainties and uncertainties in the input matrix. Necessary theoretical and simulation results are presented to validate proposed algorithm.

Appendices

Appendix 1

The dynamics of robot manipulator is given by

$$ M(q)\ddot{q} + V(q,\dot{q}) + F(\dot{q}) + G(q) = \tau $$

where q consists of joint variables, τ is the generalized forces, M(q) the inertia matrix, $V(q,\dot{q})$ the centripetal vector, G(q) the gravity vector and $F(\dot{q})$ the friction vector. For simplicity, we denote $N(q,\dot{q}) = V(q,\dot{q}) + F(\dot{q}) + G(q)$. There are uncertainties in M(q) and $N(q,\dot{q})$ due to, say, unknown load on the manipulator and unmodeled frictions.

The robot manipulator has two joint variables: two angles q ₁ and q ₂. The inertia matrix is

$$ M(q) = \left[ {\begin{array}{*{20}c} {M_{11} } & {M_{12} } \\ {M_{21} } & {M_{22} } \\ \end{array} } \right] $$

where

$$ \begin{aligned} M_{11} &=\,J_{1} + m_{1} r_{1}^{2} + J_{2} + m_{2} (l_{1}^{2} + r_{2}^{2} + 2l_{1} r_{2} \cos q_{2} ) + J_{L} + m_{L} (l_{1}^{2} + r_{2}^{2} + 2l_{1} r_{2} \cos q_{2} ) \\ M_{12} &=\,M_{21} = J_{2} + m_{2} (r_{2}^{2} + l_{1} r_{2} \cos q_{2} ) + J_{L} + m_{L} l_{2}^{2} \\ M_{22} &=\,J_{2} + m_{2} r_{2}^{2} + J_{L} + m_{L} l_{2}^{2} . \\ \end{aligned} $$

For simplicity, we denote $N(q,\dot{q}) = V(q,\dot{q}) + F(\dot{q}) + G(q)$. There are uncertainties in M(q) and $N(q,\dot{q})$ due to, say, unknown load on the manipulator and unmodeled frictions.

The centripetal vector is

$$ V(q,\dot{q}) = \left[ {\begin{array}{*{20}c} {V_{1} } \\ {V_{2} } \\ \end{array} } \right] $$

where

$$ V_{1} = (m_{2} l_{1} r_{2} + m_{L} l_{1} l_{2} )(\dot{q}_{2}^{2} + 2\dot{q}_{1} \dot{q}_{2} )\sin q_{2} \quad {\text{and}}\quad V_{2} = (m_{2} l_{1} r_{2} + m_{L} l_{1} l_{2} )\dot{q}_{1}^{2} \sin q_{2} . $$

The friction vector

$$ F(q,\dot{q}) = \left[ {\begin{array}{*{20}c} {b_{1} \dot{q}_{1} } \\ {b_{2} \dot{q}_{2} } \\ \end{array} } \right]. $$

Without loss of generality, we assume that gravity vector G(q) = 0; otherwise, it is easy to calculate the additional torque to balance the gravity. The value of the parameters is as follows:

$$ \begin{aligned} m_{1} =\,& 13.86{\text{ oz}}; \quad m_{2} = 3.33{\text{ oz}}; \quad J_{1} = 62.39{\text{ oz-in}}/{\text{rad}}/{\text{s}}^{2} ;J_{2} = 16.70{\text{ oz-in}}/{\text{rad}}/{\text{s}}^{2} ; \quad r_{1} =\,6.12{\text{ in}}; \\ r_{2} = & 3.22{\text{ in}}; \quad l_{1} = 8{\text{ in}}; \quad l_{2} = 6{\text{ in}}; \quad b_{1} = 0.2{\text{ oz-in}}/{\text{rad}}/{\text{s}};b_{1} = 0.5{\text{ oz - in}}/{\text{rad}}/{\text{s}} \\ \end{aligned} $$

The load unknown, so we assume

$$ m_{L} = 10\varepsilon {\text{ oz}}; \quad J_{L} = 60\varepsilon^{2} {\text{ oz-in}}/{\text{rad}}/{\text{s}}^{2} $$

where ε ∈ [0, 1].

With these values, we can calculate M(q) and $N(q,\dot{q})$ as follows:

$$ \begin{aligned} M(q) = & \left[ {\begin{array}{*{20}c} {846 + 60\varepsilon^{2} + 1000\varepsilon + 172\cos q_{2} + 960\varepsilon \cos q_{2} } & {51 + 60\varepsilon^{2} + 360\varepsilon + 86\cos q_{2} } \\ {51 + 60\varepsilon^{2} + 360\varepsilon + 86\cos q_{2} } & {51 + 60\varepsilon^{2} + 360\varepsilon } \\ \end{array} } \right] \\ N(q,\dot{q}) = & \left[ {\begin{array}{*{20}c} {(86 + 480\varepsilon )(\dot{q}_{2}^{2} + 2\dot{q}_{1} \dot{q}_{2} )\sin q_{2} + 0.2\dot{q}_{1} } \\ {(86 + 480\varepsilon )\dot{q}_{1}^{2} \sin q_{2} + 0.5\dot{q}_{2} } \\ \end{array} } \right] \\ \end{aligned} $$

Assuming ε = 1 for the simplicity, we can select the following M ₀(q) and $N_{0} (q,\dot{q})$:

$$ \begin{aligned} M_{0} (q) = & \left[ {\begin{array}{*{20}c} {1906 + 1132\cos q_{2} } & {471 + 86\cos q_{2} } \\ {471 + 86\cos q_{2} } & {471} \\ \end{array} } \right] \\ N_{0} (q,\dot{q}) = & \left[ {\begin{array}{*{20}c} {566(\dot{q}_{2}^{2} + 2\dot{q}_{1} \dot{q}_{2} )\sin q_{2} + 0.2\dot{q}_{1} } \\ {566\dot{q}_{1}^{2} \sin q_{2} + 0.5\dot{q}_{2} } \\ \end{array} } \right]. \\ \end{aligned} $$

Appendix 2

In Sects. 5.1 and 5.2, we applied proposed algorithm on two different class of nonlinear systems. Now we would like to show that proposed algorithm will also work for the linear system:Consider a linear uncertain system

$$ \dot{x} = Ax + Bu + Bfx $$

It is assumed that f is bounded by a known term, $f_{\max }$, i.e., $\left\| f \right\| \le f_{\max }$.

For the given open-loop system, our aim is to find a feedback control law u = Kx such that the closed-loop system is global asymptotically stable for all admissible uncertainties f. With the performance function $\int_{0}^{\infty } {\left( {x^{T} \mu f_{\max }^{2} I_{n \times n} x + x^{T} Qx + u^{T} Ru} \right)} {\text{d}}t,$ where μ > 0 is design parameter, this problem can be solved using following HJB equation:

$$ {\text{HJB(}}V^{ * } (x)) = x^{T} \mu f_{\max }^{2} I_{n \times n} x + x^{T} Qx + u^{T} Ru + V_{x}^{ * T} (Ax + Bu) = 0 $$

where $V^{ * } (x) = x^{T} Px$. It gives optimal control law $u^{ * } = Kx = - {\frac{1}{2 \, }}R^{ - 1} B^{T} V_{x}^{ * }$ and corresponding algebraic Reccati equation (ARE) is

$$ A^{T} P + PA - P^{T} BR^{ - 1} B^{T} P + Q + \mu f_{\max }^{2} I_{n \times n} = 0 $$

(38)

u* becomes a robust control law if

$$ \left( {\mu f_{\max }^{2} I_{n \times n} - PBf_{\max } } \right) \ge 0. $$

(39)

It is similar to condition (11) proposed for nonlinear system.

We have considered following system:

$$ \left[ {\begin{array}{*{20}c} {\dot{x}_{1} } \\ {\dot{x}_{2} } \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} 0 & 1 \\ 1 & 0 \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {x_{1} } \\ {x_{2} } \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} 0 \\ 1 \\ \end{array} } \right]u + \left[ {\begin{array}{*{20}c} 0 \\ 1 \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} p & p \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {x_{1} } \\ {x_{2} } \\ \end{array} } \right] $$

where $p \in \left[ { - 10,1} \right]$ is the uncertain term. We would like to design a robust control law u = Kx such that the closed-loop system is stable for all $p \in \left[ { - 10,1} \right].$

Since f = p, it gives $f_{\max }^{2} = \left\| p \right\|^{2} \le 100$.

The corresponding ARE (I) is solved with μ = 2 and identity matrices Q and R. It gives solution of ARE as

$$ P = \left[ {\begin{array}{*{20}c} { 1 6. 2 1 2 7} & { 1 5. 2 1 2 7} \\ { 1 5. 2 1 2 7} & { 1 5. 2 1 2 7} \\ \end{array} } \right] $$

Condition (II) is always satisfied as

$$ \left( {\mu f_{\max }^{2} I_{n \times n} - PBf_{\max } } \right) = \left[ {\begin{array}{*{20}c} {47.873} & {47.873} \\ {47.873} & {47.873} \\ \end{array} } \right] \ge 0. $$

Hence, we have shown that condition similar to (11) is always satisfied for the linear system.