Skip to main content
SpringerLink
Log in

Reinforcement learning-based robust optimal tracking control for disturbed nonlinear systems

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper concludes a robust optimal tracking control law for a class of nonlinear systems. A characteristic of this paper is that the designed controller can guarantee both robustness and optimality under nonlinearity and mismatched disturbances. Optimal controllers for nonlinear systems are difficult to obtain, hence a reinforcement learning method is adopted with two neural networks (NNs) approximating the cost function and optimal controller, respectively. We designed weight update laws for critic NN and actor NN based on gradient descent and stability, respectively. In addition, matched and mismatched disturbances are estimated by fixed-time disturbance observers and an artful transformation based on backstepping method is employed to convert the system into a filtered error nonlinear system. Through a rigorous analysis using the Lyapunov method, we demonstrate states and estimation errors remain uniformly ultimately bounded. Finally, the effectiveness of the proposed method is verified through two illustrative examples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

Data sharing not applicable as no new data were generated in this study.

References

  1. Tang L, Gao Y, Liu YJ (2014) Adaptive near optimal neural control for a class of discrete-time chaotic system. Neural Comput Appl 25:1111–1117

    Article  Google Scholar 

  2. Na J, Lv Y, Zhang K, Zhao J (2020) Adaptive identifier-critic-based optimal tracking control for nonlinear systems with experimental validation. IEEE Trans Syst Man Cybern Syst 52(1):459–472

    Article  Google Scholar 

  3. Fan ZX, Li S, Liu R (2022) ADP-based optimal control for dystems with mismatched disturbances: a PMSM application. IEEE Trans Circ Syst II Express Briefs 70(6):2057–2061

    Google Scholar 

  4. Fan ZX, Adhikary AC, Li S, Liu R (2020) Anti-disturbance inverse optimal control for systems with disturbances. Optim Control Appl Methods 44(3):1321–1340

    Article  MathSciNet  Google Scholar 

  5. Chen J, Li K, Li K, Yu PS (2021) Dynamic bicycle dispatching of dockless public bicycle-sharing systems using multi-objective reinforcement learning. ACM Trans Cyber-Phys Syst 5(4):1–24

    Article  Google Scholar 

  6. Lewis FL, Vrabie DL, Syrmos VL (2012) Optimal control. Wiley

    Book  MATH  Google Scholar 

  7. Werbos PJ (1992) Approximate dynamic programming for real-time control and neural modeling. handbook of intelligent control neural fuzzy and adaptive approaches, 1992

  8. Wei Q, Zhu L, Song R, Zhang P, Liu D, Xiao J (2022) Model-free adaptive optimal control for unknown nonlinear multiplayer nonzero-sum game. IEEE Trans Neural Netw Learn Syst 33(2):879–892

    Article  MathSciNet  Google Scholar 

  9. Gao W, Jiang ZP (2016) Adaptive dynamic programming and adaptive optimal output regulation of linear systems. IEEE Trans Autom Control 61(12):4164–4169

    Article  MathSciNet  MATH  Google Scholar 

  10. Gao W, Jiang ZP, Lewis FL, Wang Y (2018) Leader-to-formation stability of multiagent systems: an adaptive optimal control approach. IEEE Trans Autom Control 63(10):3581–3587

    Article  MathSciNet  MATH  Google Scholar 

  11. Krstic M, Tsiotras P (1999) Inverse optimal stabilization of a rigid spacecraft. IEEE Trans Autom Control 44(5):1042–1049

    Article  MathSciNet  MATH  Google Scholar 

  12. Fan ZX, Adhikary AC, Li S, Liu R (2022) Disturbance observer based inverse optimal control for a class of nonlinear systems. Neurocomputing 500:821–831

    Article  Google Scholar 

  13. Ming X, Balakrishnan SN (2005) A new method for suboptimal control of a class of non-linear systems. Optim Control Appl Methods 26(2):55–83

    Article  MathSciNet  Google Scholar 

  14. Do TD, Choi HH, Jung WJ (2015) \(\theta\)-D approximation technique for nonlinear optimal speed control design of surface-mounted PMSM drives. IEEE/ASME Trans Mechatron 20(4):1822–1831

    Article  Google Scholar 

  15. Zhang H, Cui L, Zhang X, Luo Y (2011) Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method. IEEE Trans Neural Netw 22(12):2226–2236

    Article  Google Scholar 

  16. Qin C, Zhang H, Luo Y (2014) Optimal tracking control of a class of nonlinear discrete-time switched systems using adaptive dynamic programming. Neural Comput Appl 24:531–538

    Article  Google Scholar 

  17. Wang D, Liu D, Zhao D, Huang Y, Zhang D (2013) A neural-network-based iterative GDHP approach for solving a class of nonlinear optimal control problems with control constraints. Neural Comput Appl 22(2):219–227

    Article  Google Scholar 

  18. Vamvoudakis KG, Lewis FL (2010) Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5):878–888

    Article  MathSciNet  MATH  Google Scholar 

  19. Yang W, Li K, Li K (2019) A pipeline computing method of SpTV for three-order tensors on CPU and GPU. ACM Trans Knowl Discov Data 13(6):1–27

    Article  MathSciNet  Google Scholar 

  20. Zhong K, Yang Z, Xiao G, Li X, Yang W, Li K (2022) An efficient parallel reinforcement learning approach to cross-layer defense mechanism in industrial control systems. IEEE Trans Parallel Distrib Syst 3(11):2979–2990

    Google Scholar 

  21. Liu C, Tang F, Hu Y, Li K, Tang Z, Li K (2021) Distributed task migration optimization in MEC by extending multi-agent deep reinforcement learning approach. IEEE Trans Parallel Distrib Syst 32(7):1603–1614

    Article  Google Scholar 

  22. Jiang Y, Jiang ZP (2012) Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica 48(10):2699–2704

    Article  MathSciNet  MATH  Google Scholar 

  23. Bian T, Jiang Y, Jiang ZP (2014) Adaptive dynamic programming and optimal control of nonlinear nonaffine systems. Automatica 50(10):2624–2632

    Article  MathSciNet  MATH  Google Scholar 

  24. Wang D (2020) Robust policy learning control of nonlinear plants with case studies for a power system application. IEEE Trans Industr Inf 16(3):1733–1741

    Article  Google Scholar 

  25. Zhao J, Yang C, Gao W, Modares H, Chen X, Dai W (2023) Linear quadratic tracking control of unknown systems: a two-phase reinforcement learning method. Automatica 148:110761

    Article  MathSciNet  MATH  Google Scholar 

  26. Modares H, Lewis FL (2014) Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning. Automatica 50(7):1780–1792

    Article  MathSciNet  MATH  Google Scholar 

  27. Chen WH (2004) Disturbance observer based control for nonlinear systems. IEEE/ASME Trans Mechatron 9(4):706–710

    Article  Google Scholar 

  28. Yu B, Du H, Ding L, Wu D, Li H (2022) Neural network-based robust finite-time attitude stabilization for rigid spacecraft under angular velocity constraint. Neural Comput Appl 34:5107–5117

    Article  Google Scholar 

  29. Zhou K, Doyle J, Glover K (1995) Robust and optimal control. Prentice Hall, New Jersey

    MATH  Google Scholar 

  30. Utkin V (2003) Variable structure systems with sliding modes. IEEE Trans Autom Control 22(2):212–222

    Article  MathSciNet  MATH  Google Scholar 

  31. Levant A (2003) Higher-order sliding modes, differentiation and output-feedback control. Int J Control 76(9–10):924–941

    Article  MathSciNet  MATH  Google Scholar 

  32. Huang J (2004) Nonlinear output regulation- theory and applications. SIAM

  33. Ohishi K, Nakao M, Ohnishi K et al (1987) Microprocessor-controlled DC motor for load-insensitive position servo system. IEEE Trans Industr Electron 34(1):44–49

    Article  Google Scholar 

  34. Han J (2009) From PID to active disturbance rejection control. IEEE Trans Industr Electron 56(3):900–906

    Article  Google Scholar 

  35. Li S, Yang J, Chen WH, Chen X (2014) Disturbance observer-based control: methods and applications. CRC Press, Inc., Boca Raton

    Google Scholar 

  36. Li S, Yang J, Chen WH, Chen X (2012) Generalized extended state observer based control for systems with mismatched uncertainties. IEEE Trans Industr Electron 59(12):4792–4802

    Article  Google Scholar 

  37. Sun H, Guo L (2017) Neural network-based DOBC for a class of nonlinear systems with unmatched disturbances. IEEE Trans Neural Netw Learn Syst 28(2):482–489

    Article  MathSciNet  Google Scholar 

  38. Cui B, Zhang L, Xia Y, Zhang J (2022) Continuous distributed fixed-time attitude controller design for multiple spacecraft systems with a directed graph. IEEE Trans Circ Syst II- Express Briefs 69(11):478–4482

    Google Scholar 

  39. Li X, Ma L, Mei K, Ding S, Pan T (2023) Fixed-time adaptive fuzzy SOSM controller design with output constraint. Neural Comput Appl 35(13):9893–9905

    Article  Google Scholar 

  40. Liu W, Chen M, Shi P (2022) Fixed-time disturbance observer-based control for quadcopter suspension transportation system. IEEE Trans Circ Syst I- Regul Pap 69(11):4632–4642

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Automation, Southeast University, Sipailou, Nanjing, 210096, Jiangsu, China

    Zhong-Xin Fan & Shihua Li

  2. Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Southeast University, Sipailou, Nanjing, 210096, Jiangsu, China

    Zhong-Xin Fan & Shihua Li

  3. Department of Statistics, Florida State University, Tallahassee, FL, 32304, USA

    Lintao Tang & Rongjie Liu

Authors
  1. Zhong-Xin Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lintao Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shihua Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rongjie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rongjie Liu.

Ethics declarations

Conflict of interest

All authors declare that there are no conflicts of interest in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, ZX., Tang, L., Li, S. et al. Reinforcement learning-based robust optimal tracking control for disturbed nonlinear systems. Neural Comput & Applic 35, 23987–23996 (2023). https://doi.org/10.1007/s00521-023-08993-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08993-0

Keywords

Search

Navigation