Skip to main content
SpringerLink
Log in

Optimal Control of Nonlinear Time-Delay Systems with Input Constraints Using Reinforcement Learning

  • Conference paper
  • First Online:
Neural Computing for Advanced Applications (NCAA 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1265))

Included in the following conference series:

Abstract

In this paper, input-constrained optimal control policy for nonlinear time delay system is proposed in virtue of Lyapunov theories and adaptive dynamic programming method. The stability on delayed nonlinear systems is investigated based on linear matrix inequalities, upon which a sufficient stability condition is proposed. To implement the feedback control synthesis, a single neural network is constructed to work as critic and actor network simultaneously, which consequently reduces the computation complexity and storage occupation in programs. The weights of NN are online tuned and the weight estimate errors are proved to be convergent. Finally, simulation results are demonstrated to illustrate our results.

This research was supported in part by the National Natural Science Foundation of China under Grant 61603179, and in part by the China Postdoctoral Science Foundation under Grant 2016M601805, 2019T120427.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Cao, Y.-Y., Lin, Z.: Stability analysis of discrete-time systems with actuator saturation by a saturation-dependent Lyapunov function. Automatica 39(7), 1235–1241 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  2. Chen, M., Ge, S.S., Ren, B.: Adaptive tracking control of uncertain MIMO nonlinear systems with input constraints. Automatica 47(3), 452–465 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  3. Saberi, A., Lin, Z., Teel, A.R.: Control of linear systems with saturating actuators. IEEE Trans. Autom. Control 41(3), 368–378 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bensoussan, A.: Maximum principle and dynamic programming approaches of the optimal control of partially observed diffusions. Stoch.: Int. J. Probab. Stoch. Process. 9(3), 169–222 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  5. Himmelberg, C.J., Parthasarathy, T., VanVleck, F.S.: Optimal plans for dynamic programming problems. Math. Oper. Res. 1(4), 390–394 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  6. Angelov, V.G.: A converse to a contraction mapping theorem in uniform spaces. Nonlinear Anal.: Theory Methods Appl. 12(10), 989–996 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  7. Branicky, M.S., Borkar, V.S., Mitter, S.K.: A unified framework for hybrid control: model and optimal control theory. IEEE Trans. Autom. Control 43(1), 31–45 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  8. Ross, I.M., Karpenko, M.: A review of pseudospectral optimal control: from theory to flight. Annu. Rev. Control 36(2), 182–197 (2012)

    Article  Google Scholar 

  9. Shen, J., Lam, J.: On the algebraic Riccati inequality arising in cone-preserving time-delay systems. Automatica 113, 108820 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  10. Wu, Z., Li, Q., Wu, W., Zhao, M.: Crowdsourcing model for energy efficiency retrofit and mixed-integer equilibrium analysis. IEEE Trans. Ind. Inform. 16(7), 4512–4524 (2019)

    Article  Google Scholar 

  11. Manousiouthakis, V., Chmielewski, D.J.: On constrained infinite-time nonlinear optimal control. Chem. Eng. Sci. 57(1), 105–114 (2002)

    Article  MATH  Google Scholar 

  12. Huang, Y., Lu, W.-M.: Nonlinear optimal control: alternatives to Hamilton-Jacobi equation. In: Proceedings of 35th IEEE Conference on Decision and Control, vol. 4, pp. 3942–3947. IEEE (1996)

    Google Scholar 

  13. Li, R., Chen, M., Qingxian, W.: Adaptive neural tracking control for uncertain nonlinear systems with input and output constraints using disturbance observer. Neurocomputing 235, 27–37 (2017)

    Article  Google Scholar 

  14. Kurtz, M.J., Henson, M.A.: Feedback linearizing control of discrete-time nonlinear systems with input constraints. Int. J. Control 70(4), 603–616 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  15. Gu, K., Chen, J., Kharitonov, V.L.: Stability of Time-Delay Systems. Springer, Cham. https://doi.org/10.1007/978-1-4612-0039-0

  16. Kamalapurkar, R., Rosenfeld, J.A., Dixon, W.E.: Efficient model-based reinforcement learning for approximate online optimal control. Automatica 74, 247–258 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  17. Dierks, T., Thumati, B.T., Jagannathan, S.: Optimal control of unknown affine nonlinear discrete-time systems using offline-trained neural networks with proof of convergence. Neural Netw. 22(5–6), 851–860 (2009)

    Article  MATH  Google Scholar 

  18. Liu, D., Wei, Q., Wang, D., Yang, X., Li, H.: Adaptive Dynamic Programming with Applications in Optimal Control—Value Iteration ADP for Discrete-Time Nonlinear Systems. AIC. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-50815-3

    Book  Google Scholar 

  19. Abu-Khalaf, M., Lewis, F.L.: Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach. Automatica 41(5), 779–791 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  20. Zhu, J., Chen, J.: Stability of systems with time-varying delays: an \(\cal{L}_1\) small-gain perspective. Automatica 52, 260–265 (2015)

    MathSciNet  MATH  Google Scholar 

  21. Seuret, A., Gouaisbaut, F., Fridman, E.: Stability of systems with fast-varying delay using improved Wirtinger’s inequality. In: 2013 IEEE 52nd Annual Conference on Decision and Control (CDC) (2013)

    Google Scholar 

  22. Zhu, J., Hou, Y., Li, T.: Optimal control of nonlinear systems with time delays: an online ADP perspective. IEEE Access 7, 145574–145581 (2019)

    Article  Google Scholar 

  23. Li, Y.U.: Optimal guaranteed cost control of linear uncertain system: an LMI approach. Control Theory Appl. 3 (2000)

    Google Scholar 

  24. Zhang, M.-Y., Lu, Z.-D.: Lyapunov-based analyse of weights’ convergence on backpropagation neural networks algorithm. Mini-Micro Syst. 1, 93–95 (2004)

    Google Scholar 

  25. Reddy, K.N.: Integral inequalities and applications. Bull. Aust. Math. Soc. 21(1), 13–20 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  26. Cohen, M.B., Madry, A., Tsipras, D., Vladu, A.: Matrix scaling and balancing via box constrained newton’s method and interior point methods. In: 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) (2017)

    Google Scholar 

  27. Rodriguez-Guerrero, L., Santos-Sanchez, O., Mondie, S.: A constructive approach for an optimal control applied to a class of nonlinear time delay systems. J. Process Control 40, 35–49 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Jing Zhu, Peng Zhang & Yijing Hou

  2. Key Laboratory of Navigation, Control and Health-Management Technologies of Advanced Aerocraft, Ministry of Industry and Information Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Jing Zhu

Authors
  1. Jing Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yijing Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing Zhu .

Editor information

Editors and Affiliations

  1. Harbin Institute of Technology, Shenzhen, China

    Haijun Zhang

  2. Hefei University of Technology, Hefei, China

    Zhao Zhang

  3. Chongqing University, Chongqing, China

    Zhou Wu

  4. South China Normal University, Guangzhou, China

    Tianyong Hao

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhu, J., Zhang, P., Hou, Y. (2020). Optimal Control of Nonlinear Time-Delay Systems with Input Constraints Using Reinforcement Learning. In: Zhang, H., Zhang, Z., Wu, Z., Hao, T. (eds) Neural Computing for Advanced Applications. NCAA 2020. Communications in Computer and Information Science, vol 1265. Springer, Singapore. https://doi.org/10.1007/978-981-15-7670-6_28

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-7670-6_28

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-7669-0

  • Online ISBN: 978-981-15-7670-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation