Skip to main content
SpringerLink
Log in

Numerical Solutions to the Bellman Equation of Optimal Control

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

In this paper, we present a numerical algorithm to compute high-order approximate solutions to Bellman’s dynamic programming equation that arises in the optimal stabilization of discrete-time nonlinear control systems. The method uses a patchy technique to build local Taylor polynomial approximations defined on small domains, which are then patched together to create a piecewise smooth approximation. The numerical domain is dynamically computed as the level sets of the value function are propagated in reverse time under the closed-loop dynamics. The patch domains are constructed such that their radial boundaries are contained in the level sets of the value function and their lateral boundaries are constructed as invariant sets of the closed-loop dynamics. To minimize the computational effort, an adaptive subdivision algorithm is used to determine the number of patches on each level set depending on the relative error in the dynamic programming equation. Numerical tests in 2D and 3D are given to illustrate the accuracy of the method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Laub, A.J.: A Schur method for solving algebraic Riccati equations. IEEE Trans. Autom. Control 24, 913–921 (1979)

    Article  MATH  MathSciNet  Google Scholar 

  2. Crandall, M.G., Lions, P.L.: Viscosity solutions of Hamilton–Jacobi equations. Trans. Am. Math. Soc. 277, 1–42 (1983)

    Article  MATH  MathSciNet  Google Scholar 

  3. Albrekht, E.G.: On the optimal stabilization of nonlinear systems. J. Appl. Math. Mech. 25, 1254–1266 (1961)

    Article  MathSciNet  Google Scholar 

  4. Leake, R.J., Liu, R.-W.: Construction of suboptimal control sequences. SIAM J. Control 5, 54–63 (1967)

    Article  MATH  MathSciNet  Google Scholar 

  5. Lukes, D.L.: Optimal regulation of nonlinear dynamical systems. SIAM J. Control 7, 75–100 (1969)

    Article  MATH  MathSciNet  Google Scholar 

  6. Capuzzo-Dolcetta, I., Ishii, H.: Approximate solutions of the Bellman equation of deterministic control theory. Appl. Math. Optim. 11, 161–181 (1984)

    Article  MathSciNet  Google Scholar 

  7. Falcone, M., Ferretti, R.: Discrete time high-order schemes for viscosity solutions of Hamilton–Jacobi–Bellman equations. Numer. Math. 67, 315–344 (1994)

    Article  MATH  MathSciNet  Google Scholar 

  8. Beard, R.W., Sardis, G.N., Wen, J.T.: Galerkin approximations of the generalized Hamilton–Jacobi–Bellman equation. Automatica 33, 2159–2177 (1997)

    Article  MATH  Google Scholar 

  9. Mracek, C.P., Cloutier, J.R.: Control designs for the nonlinear benchmark problem via the state-dependent Riccati equation method. Int. J. Robust Nonlinear Control 8, 401–433 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  10. Sethian, J.A.: Level Set Methods and Fast Marching Methods. Cambridge University Press, Cambridge (1999)

    MATH  Google Scholar 

  11. Markman, J., Katz, I.N.: An iterative algorithm for solving Hamilton–Jacobi type equations. SIAM J. Sci. Comput. 22, 312–329 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  12. Navasca, C., Krener, A.J.: Patchy solution of the HJB PDE. In: Chiuso, A., Ferrante, A., Pinzoni, S. (eds.) Modeling, Estimation and Control. Lecture Notes in Control and Information Sciences, vol. 364, pp. 251–270 (2007)

    Chapter  Google Scholar 

  13. Sakamoto, N., van der Schaft, A.J.: Analytical approximation methods for the stabilizing solution of the Hamilton–Jacobi equation. IEEE Trans. Autom. Control 53, 2335–2350 (2008)

    Article  Google Scholar 

  14. Beeler, S.C., Tran, H.T., Banks, H.T.: Feedback control methodologies for nonlinear systems. J. Optim. Theory Appl. 107, 1–33 (2000)

    MATH  MathSciNet  Google Scholar 

  15. Cacace, S., Cristiani, E., Falcone, M., Picarelli, A.: A patchy dynamic programming scheme for a class of Hamilton–Jacobi–Bellman equations. SIAM J. Sci. Comput. 34, 2625–2649 (2012)

    Article  MathSciNet  Google Scholar 

  16. Garrard, W.L., Jordan, J.M.: Design of nonlinear automatic flight control systems. Automatica 13, 497–505 (1977)

    Article  MATH  Google Scholar 

  17. Yoshida, T., Loparo, K.A.: Quadratic regulatory theory for analytic non-linear systems with additive controls. Automatica 25, 531–544 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  18. Spencer, B.F., Timlin, T.L., Sain, M.K., Dyke, S.J.: Series solution of a class of nonlinear optimal regulators. J. Optim. Theory Appl. 91, 321–345 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  19. Ancona, F., Bressan, A.: Patchy vector fields and asymptotic stabilization. ESAIM Control Optim. Calc. Var. 4, 445–471 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  20. Hunt, T.: A proof of the higher order accuracy of the patchy method for solving the Hamilton–Jacobi–Bellman equation. PhD thesis, University of California, Davis, CA (2011)

  21. Nešić, D., Teel, A.R.: Backstepping on the Euler approximate model for stabilization of sampled-data nonlinear systems. In: Proc. 40th IEEE Conference on Decision and Control, pp. 1737–1742 (2001)

    Google Scholar 

  22. Neśić, D., Teel, A.R., Kokotović, P.V.: Sufficient conditions for the stabilization of sampled-data nonlinear systems via discrete-time approximations. Syst. Control Lett. 38, 259–270 (1999)

    Article  MATH  Google Scholar 

  23. Grüne, L., Nešić, D.: Optimization-based stabilization of sampled-data nonlinear systems via their approximate discrete-time models. SIAM J. Control Optim. 42(42), 98–122 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  24. Keerthi, S.S., Gilbert, E.G.: Optimal infinite-horizon feedback laws for a general class of constrained discrete-time systems: Stability and moving-horizon approximations. J. Optim. Theory Appl. 57, 265–293 (1988)

    Article  MATH  MathSciNet  Google Scholar 

  25. Bellman, R.: Introduction to the Mathematical Theory of Control Processes. Academic Press, New York (1971)

    MATH  Google Scholar 

  26. Navasca, C.: Local solutions of the dynamic programming equations and the Hamilton–Jacobi–Bellman PDEs. PhD Thesis, University of California, Davis (2002)

  27. Lewis, F.: Optimal Control. Wiley-Interscience, New York (1986)

    MATH  Google Scholar 

  28. Baumgardner, J.R., Frederickson, P.O.: Icosahedral discretization of the two-sphere. SIAM J. Numer. Anal. 22, 1107–1115 (1985)

    Article  MATH  MathSciNet  Google Scholar 

  29. Guckenheimer, J., Holmes, P.: Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields. Springer, New York (1983)

    Book  MATH  Google Scholar 

  30. Witkin, A., Heckbert, P.: Using particles to sample and control implicit surfaces. Comput. Graph. 28, 269–278 (1994)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their valuable comments and suggestions that improved the presentation of this paper. The first author acknowledges the support of the Natural Research Council Postdoctoral Associateship program and the Naval Postgraduate School.

Author information

Authors and Affiliations

  1. Department of Mathematics, California State University, Bakersfield, CA, USA

    Cesar O. Aguilar

  2. Department of Applied Mathematics, Naval Postgraduate School, Monterey, CA, USA

    Arthur J. Krener

Authors
  1. Cesar O. Aguilar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arthur J. Krener
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cesar O. Aguilar.

Additional information

Communicated by Lars Grüne.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Aguilar, C.O., Krener, A.J. Numerical Solutions to the Bellman Equation of Optimal Control. J Optim Theory Appl 160, 527–552 (2014). https://doi.org/10.1007/s10957-013-0403-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-013-0403-8

Keywords

Search

Navigation