Skip to main content
SpringerLink
Log in

Numerical solution to optimal feedback control by dynamic programming approach: A local approximation algorithm

  • Published:
Journal of Systems Science and Complexity Aims and scope Submit manuscript

Abstract

This paper considers optimal feedback control for a general continuous time finite-dimensional deterministic system with finite horizon cost functional. A practically feasible algorithm to calculate the numerical solution of the optimal feedback control by dynamic programming approach is developed. The highlights of this algorithm are: a) It is based on a convergent constructive algorithm for optimal feedback control law which was proposed by the authors before through an approximation for the viscosity solution of the time-space discretization scheme developed by dynamic programming method; b) The computation complexity is significantly reduced since only values of viscosity solution on some local cones around the optimal trajectory are calculated. Two numerical experiments are presented to illustrate the effectiveness and fastness of the algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Sussmann H J and Willems J C, 300 years of optimal control: From the brachystochrone to the maximum principle, IEEE Controls Systems Magazine, 1997, 17: 32–44.

    Article  Google Scholar 

  2. Stoer J and Bulirsch R, Introduction to Numerical Analysis, 2nd ed., Springer-Verlag, New York, 1993.

    Book  MATH  Google Scholar 

  3. Brysun Jr A E, Optimal Control — 1950 to 1985, IEEE Control Systems Magazine, 1996, 13: 26–33.

    Article  Google Scholar 

  4. Bardi M and Dolcetta I C, Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations, Birkhäuser, Boston, 1997.

    Book  MATH  Google Scholar 

  5. Crandall M G and Lions P L, Viscosity solutions of Hamilton-Jacobi equations, Tran. Amer. Math. Soc., 1983, 277: 1–42.

    Article  MathSciNet  MATH  Google Scholar 

  6. Crandall M G and Lions P L, Two approximations of solutions of Hamilton-Jacobi equations, Math. Comp., 1984, 43: 1–19.

    Article  MathSciNet  MATH  Google Scholar 

  7. Carlini E, Falcone M, and Ferretti R, An efficient algorithm for Hamilton-Jacobi equations in high dimension, Comput. Vis. Sci., 2004, 7: 15–29.

    Article  MathSciNet  MATH  Google Scholar 

  8. Osher S and Shu C W, High order essentially non-oscillatory schemes for Hamilton-Jacobi equations, SIAM J. Numer. Anal., 1991, 28: 907–922.

    Article  MathSciNet  MATH  Google Scholar 

  9. Zhang Y T and Shu C W, Third and fourth order weighted ENO schemes for Hamilton-Jacobi equations on 2D unstructured meshes, “Hyperbolic Problems: Theory, Numerics, Applications”, Eds. by Hou T Y and Tadmor E, Springer-Verlag, Berlin, 2003, 941–950.

    Chapter  Google Scholar 

  10. Hu C and Shu C W, A discontinuous Galerkin finite element method for Hamilton-Jacobi equations, SIAM J. Sci. Comput., 1999, 21: 666–690.

    Article  MathSciNet  MATH  Google Scholar 

  11. Cheng Y and Shu C W, A discontinuous Galerkin finite element method for directly solving the Hamilton-Jacobi equations, J. Comput. Phys., 2007, 223: 398–415.

    Article  MathSciNet  MATH  Google Scholar 

  12. Aubin J P and Frankowska H, The viability kernel algorithm for computing value functions of infinite horizon optimal control problems, J. Math. Anal. Appl., 1996, 201: 555–576.

    Article  MathSciNet  MATH  Google Scholar 

  13. Carlini E, Cristiani E, and Forcadel N, A non-monotone fast marching scheme for a Hamilton-Jacobi Equation modelling dislocation dynamics, “Numerical Mathematics and Advanced Applications”, Springer-Verlag, Berlin, 2006, 723–731.

    Chapter  Google Scholar 

  14. Tsai Y, Cheng L T, Osher S, et al., Fast sweeping algorithms for a class of Hamilton-Jacobi equations, SIAM J. Numer. Anal., 2003, 41: 673–694.

    Article  MathSciNet  MATH  Google Scholar 

  15. Sethian J A and Vladimirsky A, Ordered upwind methods for static Hamilton-Jacobi equations, Proc. Natl. Acad. Sci. USA, 2001, 98: 11069–11074.

    Article  MathSciNet  MATH  Google Scholar 

  16. Dupuis P and Szpiro A, Convergence of the optimal feedback policies in a numerical method for a class of determinstic optimal control problems, SIAM J. Control and Optim., 2001, 40: 393–420.

    Article  MathSciNet  MATH  Google Scholar 

  17. Falcone M, Some remarks on the synthesis of feedback controls via numerical methods, “Optimal Control and Partial Differntial Equations”, Eds. by Menaldi J L, Rofman E, and Sulem A, IOS Press, 2001, 456–465.

    Google Scholar 

  18. Kushner H J and Dupuis P G, Numerical Methods for Stochastic Control Problems in Continuous Time, Springer-Verlag, Berlin, 1992.

    Book  MATH  Google Scholar 

  19. Barron E N, Application of viscosity solutions of infinite-dimensional Hamilton-Jacobi-Bellman equations to some problems in distributed optimal control, J. Optim. Theory and Appl., 1990, 64: 245–268.

    Article  MathSciNet  MATH  Google Scholar 

  20. Guo B Z and Sun B, Numerical solution to the optimal birth feedback control of a population dynamics: Viscosity solution approach, Optim. Control Appl. Meth., 2005, 26: 229–254.

    Article  MathSciNet  Google Scholar 

  21. Kocan M and Soravia P, A viscosity approach to infinite-dimensional Hamilton-Jacobi equations arising in optimal control with state constraints, SIAM J. Control Optim., 1998, 36: 1348–1357.

    Article  MathSciNet  MATH  Google Scholar 

  22. Yong J M and Zhou X Y, Stochastic Controls: Hamiltonian Systems and HJB Equations, Springer, New York, 1999.

    Book  MATH  Google Scholar 

  23. McEneaney W M, Max-Plus Methods for Nonlinear Control and Estimation, Birkhauser, Boston, 2006.

    MATH  Google Scholar 

  24. McEneaney WM, Convergence rate for a curse-of-dimensionality-free method for Hamilton-Jacobi- Bellman PDEs represented as maxima of quadratic forms, SIAM J. Control Optim., 2009, 48: 2651–2685.

    Article  MathSciNet  MATH  Google Scholar 

  25. McEneaney W M and Kluberg L J, Convergence rate for a curse-of-dimensionality-free method for a class of HJB PDEs, SIAM J. Control Optim., 2009, 48: 3052–3079.

    Article  MathSciNet  MATH  Google Scholar 

  26. Guo B Z and Wu T T, Approximation of optimal feedback control: A dynamic programming approach, J. Global Optim., 2010, 46: 395–422.

    Article  MathSciNet  MATH  Google Scholar 

  27. Peyret R and Taylor T D, Computational Methods for Fluid Flow, Springer-Verlag, New York, 1983.

    Book  MATH  Google Scholar 

  28. Wang S, Gao F, and Teo K L, An upwind finite-difference method for the approximation of viscosity solutions to Hamilton-Jacobi-Bellman equations, IMA J. Math. Control and Inform., 2000, 17: 167–178.

    Article  MathSciNet  MATH  Google Scholar 

  29. Guo B Z and Sun B, Numerical solution to the optimal feedback control of continuous casting process, J. Global Optim., 2007, 39: 171–195.

    Article  MathSciNet  MATH  Google Scholar 

  30. Guo B Z and Sun B, A new algorithm for finding numerical solutions of optimal feedback control, IMA J. Math. Control and Inform., 2009, 26: 95–104.

    Article  MathSciNet  MATH  Google Scholar 

  31. Crandall M G, Ishii H, and Lions P L, User’s guide to viscosity solutions of second order partial differntial equations, Bull. Amer. Math. Soc., 1992, 27: 1–67.

    Article  MathSciNet  MATH  Google Scholar 

  32. Falcone M and Ferretti R, Discrete time high-order schemes for viscosity solutions of Hamilton- Jacobi-Bellman equations, Numer. Math., 1994, 67: 315–344.

    Article  MathSciNet  MATH  Google Scholar 

  33. Drake D, Xin M, and Balakrishnan S N, A new nonlinear control technique for ascent phase of reusable launch vehicles, AIAA J. Guidance, Control, and Dynamics, 2004, 27: 938–948.

    Google Scholar 

  34. Wu T T and Guo B Z, A neighborhood approximation algorithm for the numerical solution of optimal feedback control, Proc. 4th Int. Conf. Optimization and Control with Applications, June 6–11, Harbin and Wudalianchi, China, 2009, 515–525.

    Google Scholar 

  35. Yong J M, Dynamic Programming Principle and Hamilton-Jacobi-Bellman Equations, Shanghai Scientific and Technical Publishers, Shanghai, 1992 (in Chinese).

    Google Scholar 

  36. Knowles G, An Introduction to Applied Optimal Control, Academic Press, New York, 1981.

    MATH  Google Scholar 

  37. Guo B Z and Sun B, Numerical solution of the optimal control for two types of drug therapies of HIV/AIDS, Optim. Eng., 2014, 15: 119–136.

    Article  MathSciNet  MATH  Google Scholar 

  38. Westphal L C, Handbook of Control Systems Engineering, Kluwer Academic Publishers, Boston, 2001.

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, China

    Bao-Zhu Guo & Tao-Tao Wu

  2. School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China

    Bao-Zhu Guo

Authors
  1. Bao-Zhu Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tao-Tao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bao-Zhu Guo.

Additional information

This paper was recommended for publication by Editor CHEN Jie.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, BZ., Wu, TT. Numerical solution to optimal feedback control by dynamic programming approach: A local approximation algorithm. J Syst Sci Complex 30, 782–802 (2017). https://doi.org/10.1007/s11424-017-5149-1

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11424-017-5149-1

Keywords

Search

Navigation