Skip to main content

Reinforcement-learning-based decentralized event-triggered control of partially unknown nonlinear interconnected systems with state constraints

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In many applications with great potential, safety is critical as it needs to meet strict safety specifications within physical constraints. This paper studies the decentralized event-triggered control problem of a class of partially unknown nonlinear interconnected systems with state constraints under the reinforcement learning approach. First, by introducing a control barrier function into the performance function of each auxiliary subsystem with state constraints, the system state can be operated within a user-defined safe set. And then, the original control problem can be translated equivalently into finding or searching optimal event-triggered control policies that combine to form the desired decentralized controller, resulting in significant savings in communication resources. Compared with the traditional actor-critic network structure approach, the proposed identifier-critic network structure can loosen the constraints on the system dynamics and eliminate the errors arising from approximating the actor network. Updating the weight vectors in the critic network by gradient descent and concurrent learning techniques removes the need for the traditional persistence of excitation conditions. Furthermore, it is rigorously proved that all the signals of the interconnected nonlinear system are bound according to the Lyapunov stability theory. Last, the effectiveness of the proposed control scheme is verified by simulation examples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The authors can confirm that all relevant data are included in the article.

References

  1. Wang S, Shang M, Levin MW, Stern R (2023) A general approach to smoothing nonlinear mixed traffic via control of autonomous vehicles. Transp Res C Emerg Technol 146:103967

  2. Shao X, Ye D (2022) Event-based adaptive fuzzy fixed-time control for nonlinear interconnected systems with non-affine nonlinear faults. Fuzzy Sets Syst 432:1–27

    Article  MathSciNet  MATH  Google Scholar 

  3. Yu L, Wang R (2022) Researches on adaptive cruise control system: A state of the art review. Proc IME D J Automob Eng 236(2–3):211–240

    Article  MATH  Google Scholar 

  4. Ames AD, Xu X, Grizzle JW, Tabuada P (2016) Control barrier function based quadratic programs for safety critical systems. IEEE Trans Autom Control 62(8):3861–3876

    Article  MathSciNet  MATH  Google Scholar 

  5. Zhao Y, He Y, Feng Z, Shi P, Du X (2018) Relaxed sum-of-squares based stabilization conditions for polynomial fuzzy-model-based control systems. IEEE Trans Fuzzy Syst 27(9):1767–1778

    Article  MATH  Google Scholar 

  6. He Y, Chang X-H, Wang H, Zhao X (2023) Command-filtered adaptive fuzzy control for switched mimo nonlinear systems with unknown dead zones and full state constraints. Int J Fuzzy Syst 25(2):544–560

    Article  MATH  Google Scholar 

  7. Marvi Z, Kiumarsi B (2021) Safe reinforcement learning: A control barrier function optimization approach. Int J Robust Nonlinear Control 31(6):1923–1940

    Article  MathSciNet  MATH  Google Scholar 

  8. Qin C, Zhang Z, Shang Z, Zhang J, Zhang D (2023) Adaptive optimal safety tracking control for multiplayer mixed zero-sum games of continuous-time systems. Appl Intell 1–16

  9. Xu J, Wang J, Rao J, Zhong Y, Wang H (2022) Adaptive dynamic programming for optimal control of discrete-time nonlinear system with state constraints based on control barrier function. Int J Robust Nonlinear Control 32(6):3408–3424

    Article  MathSciNet  MATH  Google Scholar 

  10. Tang F, Niu B, Zong G, Zhao X, Xu N (2022) Periodic event-triggered adaptive tracking control design for nonlinear discrete-time systems via reinforcement learning. Neural Netw 154:43–55

    Article  MATH  Google Scholar 

  11. Roman R-C, Precup R-E, Hedrea E-L, Preitl S, Zamfirache IA, Bojan-Dragos C-A, Petriu EM (2022) Iterative feedback tuning algorithm for tower crane systems. Procedia Comput Sci 199:157–165

    Article  MATH  Google Scholar 

  12. Zamfirache IA, Precup R-E, Petriu EM (2024) Adaptive reinforcement learning-based control using proximal policy optimization and slime mould algorithm with experimental tower crane system validation. Appl Soft Comput 160:111687

  13. Yao D, Li H, Shi Y (2022) Adaptive event-triggered sliding-mode control for consensus tracking of nonlinear multiagent systems with unknown perturbations. IEEE Trans Cybern 53(4):2672–2684

    Article  MATH  Google Scholar 

  14. Huang F, Xu J, Yin L, Wu D, Cui Y, Yan Z, Chen T (2022) A general motion control architecture for an autonomous underwater vehicle with actuator faults and unknown disturbances through deep reinforcement learning. Ocean Eng 263:112424

  15. Elhaki O, Shojaei K (2022) Output-feedback robust saturated actor-critic multi-layer neural network controller for multi-body electrically driven tractors with n-trailer guaranteeing prescribed output constraints. Robot Auton Syst 154:104106

  16. Dou L, Cai S, Zhang X, Su X, Zhang R (2022) Event-triggered-based adaptive dynamic programming for distributed formation control of multi-uav. J Frankl Inst 359(8):3671–3691

    Article  MathSciNet  MATH  Google Scholar 

  17. Wang X, Li Y, Quan Z, Wu J (2023) Optimal trajectory-tracking guidance for reusable launch vehicle based on adaptive dynamic programming. Eng Appl Artif Intell 117:105497

  18. Precup R-E, Roman R-C, Safaei A (2021) Data-driven Model-free Controllers. CRC Press, Boca Raton, FL

    Book  MATH  Google Scholar 

  19. Yang X, Liu D, Huang Y (2013) Neural-network-based online optimal control for uncertain non-linear continuous-time systems with control constraints. IET Control Theory Appl 7(17):2037–2047

    Article  MathSciNet  MATH  Google Scholar 

  20. Bakule L (2008) Decentralized control: An overview. Annu Rev Control 32(1):87–98

    Article  MathSciNet  MATH  Google Scholar 

  21. Wang H, Liu X, Liu K (2015) Robust adaptive neural tracking control for a class of stochastic nonlinear interconnected systems. IEEE Trans Neural Netw Learn Syst 27(3):510–523

    Article  MathSciNet  MATH  Google Scholar 

  22. Zhao Y, Zhang H, Chen Z, Wang H, Zhao X (2022) Adaptive neural decentralised control for switched interconnected nonlinear systems with backlash-like hysteresis and output constraints. Int J Syst Sci 53(7):1545–1561

    Article  MathSciNet  MATH  Google Scholar 

  23. Sun J, Liu C (2019) Decentralised zero-sum differential game for a class of large-scale interconnected systems via adaptive dynamic programming. Int J Control 92(12):2917–2927

    Article  MathSciNet  MATH  Google Scholar 

  24. Aggarwal RK, Dave M (2012) Filterbank optimization for robust asr using ga and pso. Int J Speech Technol 15:191–201

    Article  MATH  Google Scholar 

  25. Saberi A (1988) On optimality of decentralized control for a class of nonlinear interconnected systems. Automatica 24(1):101–104

    Article  MathSciNet  MATH  Google Scholar 

  26. Wang T, Wang H, Xu N, Zhang L, Alharbi KH (2023) Sliding-mode surface-based decentralized event-triggered control of partially unknown interconnected nonlinear systems via reinforcement learning. Inf Sci 641:119070

  27. Chen Y (2022) Interconnected backlash inverse compensation in neural decentralized control for switched nonlinear systems. Appl Intell 52(9):10135–10147

    Article  MATH  Google Scholar 

  28. Tang F, Wang H, Chang X-H, Zhang L, Alharbi KH (2023) Dynamic event-triggered control for discrete-time nonlinear markov jump systems using policy iteration-based adaptive dynamic programming. Nonlinear Anal Hybrid Syst 49:101338

  29. Shi Z, Zhou C (2022) Distributed optimal consensus control for nonlinear multi-agent systems with input saturation based on event-triggered adaptive dynamic programming method. Int J Control 95(2):282–294

    Article  MathSciNet  MATH  Google Scholar 

  30. Xu L-X, Wang Y-L, Wang X, Peng C (2022) Decentralized event-triggered adaptive control for interconnected nonlinear systems with actuator failures. IEEE Trans Fuzzy Syst 31(1):148–159

    Article  MATH  Google Scholar 

  31. Hentout A, Maoudj A, Kouider A (2024) Shortest path planning and efficient fuzzy logic control of mobile robots in indoor static and dynamic environments. Sci Technol 27(1):21–36

    Google Scholar 

  32. Liu K, Ji Z (2022) Dynamic event-triggered consensus of general linear multi-agent systems with adaptive strategy. IEEE Trans Circ Syst II Express Briefs 69(8):3440–3444

    MATH  Google Scholar 

  33. Huo Y, Wang D, Qiao J (2022) Adaptive critic optimization to decentralized event-triggered control of continuous-time nonlinear interconnected systems. Optim Control Appl Methods 43(1):198–212

    Article  MathSciNet  MATH  Google Scholar 

  34. Tan LN, Tran H-T, Tran T-T (2022) Event-triggered observers and distributed \(h_{\infty }\) control of physically interconnected nonholonomic mechanical agents in harsh conditions. IEEE Trans Syst Man Cybern Syst 52(12):7871–7884

    Article  MATH  Google Scholar 

  35. Yang X, He H (2019) Decentralized event-triggered control for a class of nonlinear-interconnected systems using reinforcement learning. IEEE Trans Cybern 51(2):635–648

    Article  MATH  Google Scholar 

  36. Zhao Y, Wang H, Xu N, Zong G, Zhao X (2023) Reinforcement learning-based decentralized fault tolerant control for constrained interconnected nonlinear systems. Chaos, Solitons Fractals 167:113034

  37. Qin C, Wang J, Zhu H, Zhang J, Hu S, Zhang D (2022) Neural network-based safe optimal robust control for affine nonlinear systems with unmatched disturbances. Neurocomputing 506:228–239

    Article  MATH  Google Scholar 

  38. Yang X, He H (2019) Adaptive critic learning and experience replay for decentralized event-triggered control of nonlinear interconnected systems. IEEE Trans Syst Man Cybern Syst 50(11):4043–4055

    Article  MATH  Google Scholar 

  39. Xue S, Luo B, Liu D, Yang Y (2020) Constrained event-triggered \(h_{\infty }\) control based on adaptive dynamic programming with concurrent learning. IEEE Trans Syst Man Cybern Syst 52(1):357–369

    Article  MATH  Google Scholar 

  40. Xu J, Wang J, Rao J, Zhong Y, Wang H (2022) Adaptive dynamic programming for optimal control of discrete-time nonlinear system with state constraints based on control barrier function. Int J Robust Nonlinear Control 32(6):3408–3424

    Article  MathSciNet  MATH  Google Scholar 

  41. Huo X, Karimi HR, Zhao X, Wang B, Zong G (2021) Adaptive-critic design for decentralized event-triggered control of constrained nonlinear interconnected systems within an identifier-critic framework. IEEE Trans Cybern 52(8):7478–7491

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by science and technology research project of the Henan province (222102240014).

Author information

Authors and Affiliations

Authors

Contributions

C.Q. and D.Z. provided methodology, validation, and writing—original draft preparation; Y.W. and T.Z. provided conceptualization and writing—review; J.K. provided supervision; C.Q. provided funding support. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Dehua Zhang.

Ethics declarations

Ethical and informed consent for data used

This study did not involve ethical approval.

Conflicts of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

A Appendix A

A Appendix A

A Proof of Exclusion of Zeno Behavior

According to the (19), the event-triggered error \(\mathcal {Z} _{a,k}(t)\) has the derivative with respect to t as

$$\begin{aligned} \frac{\textrm{d} \mathcal {Z} _{a,k}(t)}{\textrm{d} t} =\dot{\tilde{x}} _{a,k}-\dot{x} _{a}(t),t\in [t_{k},t_{k+1}). \end{aligned}$$
(a1)

By utilizing the optimal DETC policy and the optimal event-triggered auxiliary control provided by (54) and (55), we obtain

$$\begin{aligned} \left\| \dot{\mathcal {Z}} _{a,k} \right\| =&\left\| \dot{\tilde{x}} _{a,k}-\dot{x} _{a} \right\| =\left\| \dot{x} _{a} \right\| \\ =&\left\| f_{a} \left( x_{a} \right) +g_{a} \left( x_{a} \right) \hat{u} _{a}(\tilde{x}_{a,k} )+\eta _{a}\left( x_{a} \right) \hat{v} _{a}(\tilde{x}_{a,k}) \right\| \nonumber \\\le&\left\| f_{a}\left( x_{a}\right) \right\| +\left\| \frac{1}{2}R_{a}^{-1}g_{a}(x_{a})g_{a}^T(\tilde{x}_{a,k}) \nabla \sigma _{c_{a}}^{T}(\tilde{x}_{a,k})\hat{W} _{c_{a}} \right\| \nonumber \\ &+\left\| \frac{1}{2\xi _{a}}\eta _{a}(x_{a})\eta _{a}^{T}(\tilde{x}_{a,k})\nabla \sigma _{c_{a}}^{T}(\tilde{x}_{a,k})\hat{W} _{c_{a}} \right\| \nonumber . \end{aligned}$$
(a2)

Note that the function \(f_{a}(x_{a})\) is locally Lipschitz continuous. That is, there exists a non-negative constant \(K_{f,a}\) such that \(\left\| f_{a}(x_{a}) \right\| \le K_{f,a}\left\| x_{a} \right\| \) holds. In addition, under Assumptions 2 and 7, we have

$$\begin{aligned} \left\| \dot{\mathcal {Z}} _{a,k} \right\| \le&\left\| f_{a}\left( x_{a}\right) \right\| +\left\| \frac{1}{2}R_{a}^{-1}g_{a}(x_{a})g_{a}^T(\tilde{x}_{a,k}) \nabla \sigma _{c_{a}}^{T}(\tilde{x}_{a,k})\hat{W} _{c_{a}} \right\| \\&+\left\| \frac{1}{2\xi _{a}}\eta _{a}(x_{a})\eta _{a}^{T}(\tilde{x}_{a,k})\nabla \sigma _{c_{a}}^{T}(\tilde{x}_{a,k})\hat{W} _{c_{a}} \right\| \nonumber \\ \le&K_{f,a}\left\| x_{a} \right\| +\frac{1}{2R_{a}}d^{2}_{g_{a}}d_{\sigma _{a}}\left\| \hat{W} _{c_{a}} \right\| \nonumber \\ \le&K_{f,a}\left\| \tilde{x}_{a,k}-x_{a,k} \right\| +\mathfrak {D}_{a} \nonumber \\ \le&K_{f,a}\left\| \tilde{x}_{a,k} \right\| +K_{f,a}\left\| x_{a,k} \right\| +\mathfrak {D}_{a}\nonumber , \end{aligned}$$
(a3)

where \(\mathfrak {D}_{a}=\frac{1}{2R_{a}}d^{2}_{g_{a}}d_{\sigma _{a}}\left\| \hat{W} _{a} \right\| + \frac{1}{2\xi _{a}}d^{2}_{\eta _{a}}d_{\sigma _{a}}\left\| \hat{W} _{a} \right\| \). According to [27,28], we can obtain the inequality related to (a3), i.e.

$$\begin{aligned} \left\| \mathcal {Z}_{a,k} \right\| \le \frac{K_{f,a}\left\| \tilde{x}_{a,k}) \right\| +\mathfrak {D}_{a}}{K_{f,a}}\left( exp^{K_{f,a}(t-t_{k})}-1 \right) ,\;\forall t\in \left[ t_{k},t_{k+1} \right) . \end{aligned}$$
(a4)

Thus, the kth sampling period can be denoted as

$$\begin{aligned} t_{k+1}-t_{k}\ge \frac{1}{K_{f,a}}\ln {\left( 1+\frac{\left\| \mathcal {Z} _{a,k} \right\| }{\mathfrak {D}_{m_{a}} } \right) } >0, \end{aligned}$$
(a5)

where \(\mathfrak {D}_{m_{a}}=\frac{K_{f,a}\left\| \tilde{x}_{a,k}) \right\| +\mathfrak {D}_{a}}{K_{f,a}}\). From the above discussion, we can find the minimum sampling period \(\left( \triangle t_{k} \right) _{min}>0,\;k\in \mathbb {N}\). Thus, Zeno behavior will not happen.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, C., Wu, Y., Zhu, T. et al. Reinforcement-learning-based decentralized event-triggered control of partially unknown nonlinear interconnected systems with state constraints. Appl Intell 55, 164 (2025). https://doi.org/10.1007/s10489-024-06072-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-06072-y

Keywords