Skip to main content
Springer Nature Link
Log in

Neural-network-based safe learning control for non-zero-sum differential games of nonlinear systems with asymmetric input constraints

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This paper primarily investigates a neural-network-based safe control scheme for solving the optimal control problem of continuous-time (CT) nonlinear systems with asymmetric input constraints under non-zero-sum (NZS) differential game scenarios. Initially, by constructing a novel non-quadratic function, the issue of asymmetric input constraints in the non-zero-sum differential game controllers is addressed. Subsequently, the safe Hamilton-Jacobi-Bellman (HJB) equation is derived from the direct integration of the control barrier function (CBF) into the traditional cost function, ensuring that the system states remain within a safe region. Then, the safe learning control scheme based on single critic neural network (NN) and adaptive dynamic programming (ADP) is proposed to approximate the optimal control strategy, differing from the dual-network update method commonly used in traditional ADP. Based on the constructed neural network weight adjustment rules, the optimal solution to the HJB equation can be derived within the safe learning control framework. Following this, Lyapunov’s stability theory demonstrates that the errors in neural network weights and all signals within the closed-loop system are uniformly ultimately bounded (UUB). Finally, the effectiveness of the developed neural-network-based safe learning control method is validated through two simulation results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The authors can confirm that all relevant data are included in the article.

References

  1. Xie T, Xian B, Gu X (2023) Fixed-time convergence attitude control for a tilt trirotor unmanned aerial vehicle based on reinforcement learning. ISA Trans 132:477–489

    Google Scholar 

  2. Dou L, Cai S, Zhang X, Su X, Zhang R (2022) Event-triggered-based adaptive dynamic programming for distributed formation control of multi-uav. J Frank Inst 359(8):3671–3691

    MathSciNet  Google Scholar 

  3. Fang H, Zhu Y, Dian S, Xiang G, Guo R, Li S (2022) Robust tracking control for magnetic wheeled mobile robots using adaptive dynamic programming. ISA Trans 128:123–132

    Google Scholar 

  4. Limbasiya T, Teng KZ, Chattopadhyay S, Zhou J (2022) A systematic survey of attack detection and prevention in connected and autonomous vehicles. Veh Commu 100515

  5. Lin Z, Ma J, Duan J, Li SE, Ma H, Cheng B, Lee TH (2023) Policy iteration based approximate dynamic programming toward autonomous driving in constrained dynamic environment. IEEE Trans Intell Transp Syst 24(5):5003–5013

    Google Scholar 

  6. Ames AD, Xu X, Grizzle JW, Tabuada P (2016) Control barrier function based quadratic programs for safety critical systems. IEEE Trans Autom Control 62(8):3861–3876

    MathSciNet  Google Scholar 

  7. Chen Y, Ahmadi M, Ames AD (2020) Optimal safe controller synthesis: a density function approach. In: 2020 American control conference (ACC), IEEE, pp 5407–5412

  8. Wang L, Han D, Egerstedt M (2018) Permissive barrier certificates for safe stabilization using sum-of-squares. In: 2018 Annual american control conference (ACC), IEEE, pp 585–590

  9. Qin C, Wang J, Zhu H, Zhang J, Hu S, Zhang D (2022) Neural network-based safe optimal robust control for affine nonlinear systems with unmatched disturbances. Neurocomputing 506:228–239

    Google Scholar 

  10. Qin C, Zhang Z, Shang Z, Zhang J, Zhang D (2023) Adaptive optimal safety tracking control for multiplayer mixed zero-sum games of continuous-time systems. Appl Intell 53:17460–17475

    Google Scholar 

  11. Xu J, Wang J, Rao J, Zhong Y, Wang H (2022) Adaptive dynamic programming for optimal control of discrete-time nonlinear system with state constraints based on control barrier function. Int J Robust Nonlinear Control 32(6):3408–3424

    MathSciNet  Google Scholar 

  12. Qin C, Shang Z, Zhang Z, Zhang D, Zhang J (2024) Parallel learning-based security robust tracking control for nonlinear systems with uncertainties: an event-triggered design. Eng Appl Artif Intell 133:108077

    Google Scholar 

  13. Yang Y, Pan Y, Xu C-Z, Wunsch DC (2022) Hamiltonian-driven adaptive dynamic programming with efficient experience replay. IEEE Trans Neural Netw Learn Syst 35(3):3278–3290

    MathSciNet  Google Scholar 

  14. Liu D, Yang X, Wang D, Wei Q (2015) Reinforcement-learning-based robust controller design for continuous-time uncertain nonlinear systems subject to input constraints. IEEE Trans Cybern 45(7):1372–1385

    Google Scholar 

  15. Wang D, He H, Liu D (2017) Adaptive critic nonlinear robust control: a survey. IEEE Trans Cybern 47(10):3429–3451

    Google Scholar 

  16. Liu D, Xue S, Zhao B, Luo B, Wei Q (2020) Adaptive dynamic programming for control: a survey and recent advances. IEEE Trans Syst Man Cybern Syt 51(1):142–160

    Google Scholar 

  17. Akande T, Alabi O, Ajagbe S (2024) A deep learning-based CAE approach for simulating 3D vehicle wheels under real-world conditions. In: Artificial intelligence and applications. https://doi.org/10.47852/bonviewAIA42021882

  18. Wen G, Ge SS, Chen CP, Tu F, Wang S (2018) Adaptive tracking control of surface vessel using optimized backstepping technique. IEEE Trans Cybern 49(9):3420–3431

    Google Scholar 

  19. Vu VT, Pham TL, Dao PN (2022) Disturbance observer-based adaptive reinforcement learning for perturbed uncertain surface vessels. ISA Trans 130:277–292

    Google Scholar 

  20. Luo B, Wu Z, Zhou F, Wang B-C (2023) Human-in-the-loop reinforcement learning in continuous-action space. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2023.3289315

    Article  Google Scholar 

  21. Zhang Q, Zhao D, Wang D (2016) Event-based robust control for uncertain nonlinear systems using adaptive dynamic programming. IEEE Trans Neural Netw Learn Syst 29(1):37–50

    MathSciNet  Google Scholar 

  22. Xue S, Luo B, Liu D (2020) Event-triggered adaptive dynamic programming for unmatched uncertain nonlinear continuous-time systems. IEEE Trans Neural Netw Learn Syst 32(7):2939–2951

    MathSciNet  Google Scholar 

  23. Yang Y, Wunsch D, Yin Y (2017) Hamiltonian-driven adaptive dynamic programming for continuous nonlinear dynamical systems. IEEE Trans Neural Netw Learn Syst 28(8):1929–1940

    MathSciNet  Google Scholar 

  24. Yang Y, Zhu H, Zhang Q, Zhao B, Li Z, Wunsch DC (2022) Sparse online kernelized actor-critic learning in reproducing kernel hilbert space. Artif Intell Rev 55(1):23–58

    Google Scholar 

  25. Wang D, Li X, Zhao M, Qiao J (2024) Adaptive critic control design with knowledge transfer for wastewater treatment applications. IEEE Trans Ind Inform 20(2):1488–1497

    Google Scholar 

  26. Wei Q, Zhou T, Lu J, Liu Y, Su S, Xiao J (2023) Continuous-time stochastic policy iteration of adaptive dynamic programming. IEEE Trans Syst Man Cybern Syst 53(10):6375–6387

    Google Scholar 

  27. Werbos, P.: Advanced forecasting methods for global crisis warning and models of intelligence. General System Yearbook, 25–38 (1977)

  28. Chen H, Long H, Chen T, Song Y, Chen H, Zhou X, Deng W (2024) \(M^{3} \) FuNet: an unsupervised multivariate feature fusion network for hyperspectral image classification. IEEE Trans Geosci Remote Sens 62:5513015

    Google Scholar 

  29. Chen W, Liu B, Guan W (2024) ERNIE and Multi-feature fusion for news topic classification. In: Artificial intelligence and applications, vol 2, pp 149–154

  30. Zhao H, Wu Y, Deng W (2023) An interpretable dynamic inference system based on fuzzy broad learning. IEEE Trans Instrum Meas 72:2527412

    Google Scholar 

  31. Granado FM, Alkhaled L (2024) How GNNs can be used in the vehicle industry. In: Artificial intelligence and applications. https://doi.org/10.47852/bonviewAIA42021556

  32. Vamvoudakis KG, Lewis FL (2010) Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5):878–888

    MathSciNet  Google Scholar 

  33. Xue S, Luo B, Liu D, Yang Y (2020) Constrained event-triggered \({H\infty }\) control based on adaptive dynamic programming with concurrent learning. IEEE Trans Syst Man Cybern Syst 52(1):357–369

    Google Scholar 

  34. Zou H, Zhang G (2023) Dynamic event-triggered-based single-network adp optimal tracking control for the unknown nonlinear system with constrained input. Neurocomputing 518:294–307

    Google Scholar 

  35. Zhu L, Wei Q, Guo P (2024) Synergetic learning neuro-control for unknown affine nonlinear systems with asymptotic stability guarantees. IEEE Trans Neural Netw Learn Syst

  36. Yi J, Chen S, Zhong X, Zhou W, He H (2018) Event-triggered globalized dual heuristic programming and its application to networked control systems. IEEE Trans Ind Inform 15(3):1383–1392

    Google Scholar 

  37. Zhuo Y, Zhu J, Chen J, Wang Z, Ye H, Liu H, Liu M (2022) Rsm-based approximate dynamic programming for stochastic energy management of power systems. IEEE Trans Power Syst 38(6):5392–5405

    Google Scholar 

  38. Song R, Wei Q, Song B (2017) Neural-network-based synchronous iteration learning method for multi-player zero-sum games. Neurocomputing 242:73–82

    Google Scholar 

  39. Zhang Q, Zhao D (2018) Data-based reinforcement learning for nonzero-sum games with unknown drift dynamics. IEEE Trans Cybern 49(8):2874–2885

    Google Scholar 

  40. Zhang H, Su H, Zhang K, Luo Y (2019) Event-triggered adaptive dynamic programming for non-zero-sum games of unknown nonlinear systems via generalized fuzzy hyperbolic models. IEEE Trans Fuzzy Syst 27(11):2202–2214

    Google Scholar 

  41. Lu J, Wei Q, Wang Z, Zhou T, Wang F-Y (2022) Event-triggered optimal control for discrete-time multi-player non-zero-sum games using parallel control. Inf Sci 584:519–535

    Google Scholar 

  42. Vamvoudakis KG, Lewis FL (2011) Multi-player non-zero-sum games: Online adaptive learning solution of coupled hamilton-jacobi equations. Automatica 47(8):1556–1569

    MathSciNet  Google Scholar 

  43. Zhang H, Cui L, Luo Y (2012) Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using single-network adp. IEEE Trans Cybern 43(1):206–216

  44. Zhang Y, Zhao B, Liu D, Zhang S (2022) Adaptive dynamic programming-based event-triggered robust control for multiplayer nonzero-sum games with unknown dynamics. IEEE Trans Cybern 53(8):5151–5164

    Google Scholar 

  45. Mu C, Wang K, Ni Z (2021) Adaptive learning and sampled-control for nonlinear game systems using dynamic event-triggering strategy. IEEE Trans Neural Netw Learn Syst 33(9):4437–4450

    MathSciNet  Google Scholar 

  46. Huo Y, Wang D, Qiao J, Li M (2023) Adaptive critic design for nonlinear multi-player zero-sum games with unknown dynamics and control constraints. Nonlinear Dyn 111:11671–11683

  47. Song R, Liu L, Xia L, Lewis FL (2022) Online optimal event-triggered \({H\infty }\) control for nonlinear systems with constrained state and input. IEEE Trans Syst Man Cybern Syst 53(1):131–141

    Google Scholar 

  48. Yang X, Zhou Y, Dong N, Wei Q (2021) Adaptive critics for decentralized stabilization of constrained-input nonlinear interconnected systems. IEEE Trans Syst Man Cybern Syst 52(7):4187–4199

    Google Scholar 

  49. Kong L, He W, Dong Y, Cheng L, Yang C, Li Z (2019) Asymmetric bounded neural control for an uncertain robot by state feedback and output feedback. IEEE Trans Syst Man Cybern Syst 51(3):1735–1746

    Google Scholar 

  50. Yang X, Zhao B (2020) Optimal neuro-control strategy for nonlinear systems with asymmetric input constraints. IEEE/CAA J Autom Sin 7(2):575–583

    MathSciNet  Google Scholar 

  51. Xue S, Luo B, Liu D, Gao Y (2022) Event-triggered integral reinforcement learning for nonzero-sum games with asymmetric input saturation. Neural Netw 152:212–223

    Google Scholar 

  52. Zou H, Zhang G, Liu W, Yan Z (2024) Dynamic event-triggered robust optimal tracking control for multi-player nonzero-sum games with mismatched uncertainties and asymmetric constrained inputs. Inf Sci 662:120177

    Google Scholar 

  53. Zhao Y, Wang H, Xu N, Zong G, Zhao X (2023) Reinforcement learning-based decentralized fault tolerant control for constrained interconnected nonlinear systems. Chaos Solit Fractals 167:113034

    MathSciNet  Google Scholar 

  54. Abu-Khalaf M, Lewis FL (2005) Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network hjb approach. Automatica 41(5):779–791

    MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported by science and technology research project of the Henan province (222102240014).

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Henan University, Zhengzhou, 450000, China

    Chunbin Qin, Tianzeng Zhu, Kaijun Jiang & Yinliang Wu

  2. School of Software, Henan University, Kaifeng, 475000, China

    Jishi Zhang

Authors
  1. Chunbin Qin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Tianzeng Zhu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Kaijun Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Yinliang Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Jishi Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

C.Q. and T.Z. provided methodology, validation, and writing–original draft preparation; K.J. and Y.W. provided conceptualization, writing–review; J.Z. provided supervision; C.Q. provided funding support. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Jishi Zhang.

Ethics declarations

Conflicts of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, C., Zhu, T., Jiang, K. et al. Neural-network-based safe learning control for non-zero-sum differential games of nonlinear systems with asymmetric input constraints. Appl Intell 54, 7810–7828 (2024). https://doi.org/10.1007/s10489-024-05593-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05593-w

Keywords