Skip to main content
SpringerLink
Log in

Switching-aware multi-agent deep reinforcement learning for target interception

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This paper investigates the multi-agent interception problem under switching topology based on deep reinforcement learning. Due to communication restrictions or network attacks, the connectivity between every two intercepting agents may change during the entire tracking process before the successful interception. That is, the topology of the multi-agent system is switched, which leads to a partial missing or dynamic jump of each agent’s observation. To solve this issue, a novel multi-agent level-fusion actor-critic (MALFAC) approach is proposed with a direction assisted (DA) actor and a dimensional pyramid fusion (DPF) critic. Besides, an experience adviser (EA) function is added to the learning process of the actor. Furthermore, a reward factor is proposed to balance the relationship between individual reward and shared reward. Experimental results show that the proposed method performs better than recent algorithms in the multi-agent interception scenarios with switching topologies, which achieves the highest successful interception with the least average steps. The ablation study also verifies the effectiveness of the innovative components in the proposed method. The extensive experimental results demonstrate the scalability of our method in different scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Nguyen TT, Nguyen ND, Nahavandi S (2020) Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE Trans Cybern 50(9):3826–3839

    Article  Google Scholar 

  2. Mahmoud M S (2020) Multiagent systems: Introduction and coordination control. CRC Press, Boca Raton, FL, USA

    Book  MATH  Google Scholar 

  3. Ji G, Yan J, Du J, Yan W, Chen J, Lu Y, Rojas J, Cheng SS (2021) Towards safe control of continuum manipulator using shielded multiagent reinforcement learning. IEEE Robot Autom Lett 6(4):7461–7468

    Article  Google Scholar 

  4. Perrusqu’ia A, Yu W, Li X (2021) Multi-agent reinforcement learning for redundant robot control in task-space. Int J Mach Learn Cybern 12:231–241

    Article  Google Scholar 

  5. Kim H, Kim D, Kim H, Shin JU, Myung H (2016) An extended any-angle path planning algorithm for maintaining formation of multi-agent jellyfish elimination robot system. Int J Control Autom Syst 14(2):598–607

    Article  Google Scholar 

  6. Zhou W, Liu Z, Li J, Xu X, Shen L (2021) Multi-target tracking for unmanned aerial vehicle swarms using deep reinforcement learning. Neurocomputing 466:285–297

    Article  Google Scholar 

  7. Kim J (2020) Cooperative localization and unknown currents estimation using multiple autonomous underwater vehicles. IEEE Robot Autom Lett 5(2):2365–2371

    Article  Google Scholar 

  8. Chen Y-J, Chang D-K, Zhang C (2020) Autonomous tracking using a swarm of uavs: A constrained multi-agent reinforcement learning approach. IEEE Trans Veh Technol 69(11):13702–13717

    Article  Google Scholar 

  9. Shi Y, Hu Q (2021) Observer-based spacecraft formation coordinated control via a unified event-triggered communication. IEEE Trans Aerosp Electron Syst 57(5):3307–3319

    Article  Google Scholar 

  10. Dong Y, Chen J (2021) Nonlinear observer-based approach for cooperative control of networked rigid spacecraft systems. Automatica 128:109552

    Article  MathSciNet  MATH  Google Scholar 

  11. Zhang C, Wang J, Zhang D, Shao X (2018) Fault-tolerant adaptive finite-time attitude synchronization and tracking control for multi-spacecraft formation. Aerosp Sci Technol 73:197–209

    Article  Google Scholar 

  12. Duan P, Liu K, Huang N, Duan Z (2020) Event-based distributed tracking control for second-order multiagent systems with switching networks. IEEE Trans Syst Man Cybern Syst 50(9):3220–3230

    Article  Google Scholar 

  13. Dong L, Yu D, Yan H (2020) Stability analysis of nonlinear multi-agent relay tracking systems over a finite time interval. Int J Control 93(3):519–527

    Article  MathSciNet  MATH  Google Scholar 

  14. Wang Y-W, Lei Y, Bian T, Guan Z-H (2020) Distributed control of nonlinear multiagent systems with unknown and nonidentical control directions via event-triggered communication. IEEE Trans Cybern 50(5):1820–1832

    Article  Google Scholar 

  15. Liu C, Jiang B, Zhang K, Patton RJ (2021) Distributed fault-tolerant consensus tracking control of multi-agent systems under fixed and switching topologies. IEEE Transactions on Circuits and Systems I: Regular Papers 68(4):1646–1658

    Article  MathSciNet  Google Scholar 

  16. Zou W, Shi P, Xiang Z, Shi Y (2020) Finite-time consensus of second-order switched nonlinear multi-agent systems. IEEE Trans Neural Netw Learn Syst 31(5):1757–1762

    Article  MathSciNet  Google Scholar 

  17. Jiang J, Jiang Y (2020) Leader-following consensus of linear time-varying multi-agent systems under fixed and switching topologies. Automatica 113:108804

    Article  MathSciNet  MATH  Google Scholar 

  18. Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT Press, Cambridge, MA, USA

    MATH  Google Scholar 

  19. Hernandez-Leal P, Kartal B, Taylor M E (2019) A survey and critique of multiagent deep reinforcement learning. Auton Agent Multi-Agent Syst 33(6):750–797

    Article  Google Scholar 

  20. Zhang K, Yang Z, Başar T (2021) Multi-agent reinforcement learning: A selective overview of theories and algorithms. pp 321–384

  21. Gronauer S, Diepold K (2021) Multi-agent deep reinforcement learning: a survey. Artif Intell Rev, pp 1–49

  22. Gupta S, Singal G, Garg D (2021) Deep reinforcement learning techniques in diversified domains: A survey. Archives of Computational Methods in Engineering, pp 4715–4754

  23. Shang M, Zhou Y, Fujita H (2021) Deep reinforcement learning with reference system to handle constraints for energy-efficient train control. Inf Sci 570:708–721

    Article  MathSciNet  Google Scholar 

  24. Le N, Rathour VS, Yamazaki K, Luu K, Savvides M (2021) Deep reinforcement learning in computer vision: a comprehensive survey. Artif Intell Rev

  25. Zhou SK, Le HN, Luu K, V Nguyen H, Ayache N (2021) Deep reinforcement learning in medical imaging: A literature review. Med Image Anal 73:102193

  26. Silver D, Schrittwieser J, Simonyan K, et al. (2017) Mastering the game of go without human knowledge. Nature 550:354– 359

    Article  Google Scholar 

  27. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv:1707.06347v2

  28. Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2016) Continuous control with deep reinforcement learning. In: 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, May 2-4, 2016

  29. Silver D, Huang A, Maddison C, et al. (2016) Mastering the game of go with deep neural networks and tree search. Nature 529:484–489

    Article  Google Scholar 

  30. Mnih V, Kavukcuoglu K, Silver D, et al. (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

    Article  Google Scholar 

  31. Shoham Y, Leyton-Brown K (2009) Multiagent systems - algorithmic, game-theoretic, and logical foundations. Cambridge University Press, Cambridge, England

    MATH  Google Scholar 

  32. Sadhu AK, Konar A (2020) Multi-agent coordination: A reinforcement learning approach. John Wiley & Sons

  33. Zhang Y, Zhou Y, Lu H, Fujita H (2021) Cooperative multi-agent actor-critic control of traffic network flow based on edge computing. Futur Gener Comput Syst 123:128–141

    Article  Google Scholar 

  34. Ye Z, Chen Y, Jiang X, Song G, Yang B, Fan S (2021) Improving sample efficiency in multi-agent actor-critic methods. Appl Intell

  35. Cao D, Zhao J, Hu W, Ding F, Huang Q, Chen Z, Blaabjerg F (2021) Data-driven multi-agent deep reinforcement learning for distribution system decentralized voltage control with high penetration of pvs. IEEE Trans on Smart Grid 12(5):4137–4150

    Article  Google Scholar 

  36. Du W, Ding S, Zhang C, Du S (2021) Modified action decoder using bayesian reasoning for multi-agent deep reinforcement learning. Int J Mach Learn Cybern 12(10):2947–2961

    Article  Google Scholar 

  37. Xu C, Liu S, Zhang C, Huang Y, Lu Z, Yang L (2021) Multi-agent reinforcement learning based distributed transmission in collaborative cloud-edge systems. IEEE Trans Veh Technol 70(2):1658–1672

    Article  Google Scholar 

  38. Sunehag P, Lever G, Gruslys A, et al. (2018) Value-decomposition networks for cooperative multi-agent learning based on team reward. In: Proceedings of the 17th international conference on autonomous agents and multiagent systems (AAMAS), Stockholm, Sweden, July 10-15, 2018, pp 2085–2087

  39. Rashid T, Samvelyan M, Schroeder C, Farquhar G, Foerster J, Whiteson S (2018) QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In: Proceedings of the 35th international conference on machine learning (ICML), Stockholm Sweden, 10-15 Jul, 2018, vol 80, pp 4295–4304

  40. Son K, Kim D, Kang W J, Hostallero D, Yi Y (2019) QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In: Proceedings of the 36th international conference on machine learning (ICML), Long Beach, California, USA, 9-15 june 2019, vol 97, pp 5887–5896

  41. Foerster J N, Farquhar G, Afouras T, Nardelli N, Whiteson S (2018) Counterfactual multi-agent policy gradients. In: Proceedings of the 32nd AAAI conference on artificial intelligence (AAAI), New Orleans, Louisiana, USA, February 2-7, 2018, pp 2974–2982

  42. Lowe R, Wu Y, Tamar A, Harb J, Abbeel P, Mordatch I (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in neural information processing systems 30 (NIPS), Long Beach, CA, USA, 4-9 december 2017, pp 6379–6390

  43. Huang L, Fu M, Qu H, Wang S, Hu S (2021) A deep reinforcement learning-based method applied for solving multi-agent defense and attack problems. Expert Syst Appl 176:114896

    Article  Google Scholar 

  44. Chen X, Liu G (2021) Energy-efficient task offloading and resource allocation via deep reinforcement learning for augmented reality in mobile edge networks. IEEE Internet of Things Journal 8(13):10843–10856

    Article  Google Scholar 

  45. Yang Y, Li B, Zhang S, Zhao W, Zhang H (2021) Cooperative proactive eavesdropping based on deep reinforcement learning. IEEE Wirel Commun Lett 10(9):1857–1861

    Article  Google Scholar 

  46. Wang L, Wang K, Pan C, Xu W, Aslam N, Hanzo L (2021) Multi-agent deep reinforcement learning-based trajectory planning for multi-uav assisted mobile edge computing. IEEE Trans Cogn Commun Netw 7(1):73–84

    Article  Google Scholar 

  47. Wu T, Zhou P, Wang B, Li A, Tang X, Xu Z, Chen K, Ding X (2021) Joint traffic control and multi-channel reassignment for core backbone network in sdn-iot: A multi-agent deep reinforcement learning approach. IEEE Trans Netw Sci Eng 8(1):231–245

    Article  MathSciNet  Google Scholar 

  48. Gao A, Du C, Ng S X, Liang W (2021) A cooperative spectrum sensing with multi-agent reinforcement learning approach in cognitive radio networks. IEEE Commun Lett 25(8):2604– 2608

    Article  Google Scholar 

  49. Sun X, Qiu J (2021) Two-stage volt/var control in active distribution networks with multi-agent deep reinforcement learning method. IEEE Trans on Smart Grid 12(4):2903–2912

    Article  Google Scholar 

  50. Zhang F, Li J, Li Z (2020) A TD3-based multi-agent deep reinforcement learning method in mixed cooperation-competition environment. Neurocomputing 411:206–215

    Article  Google Scholar 

  51. Chaudhuri K, Salakhutdinov R (2019) Actor-attention-critic for multi-agent reinforcement learning. In: Proceedings of the 36th international conference on machine learning (ICML), 9-15 June 2019, Long Beach, California, USA

  52. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, USA, june 27-30, 2016

  53. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: 3rd international conference on learning representations (ICLR 2015). ICLR, San Diego, CA, USA

Download references

Author information

Authors and Affiliations

  1. School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, Beijing, 100044, People’s Republic of China

    Dongyu Fan, Haikuo Shen & Lijing Dong

  2. Key Laboratory of Vehicle Advanced Manufacturing, Measuring and Control Technology (Beijing Jiaotong University), Ministry of Education, Beijing, 100044, People’s Republic of China

    Haikuo Shen & Lijing Dong

  3. Beijing Advanced Innovation Center for Intelligent Robots and Systems, Beijing Institute of Technology, Beijing, 100081, China

    Lijing Dong

Authors
  1. Dongyu Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haikuo Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lijing Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haikuo Shen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the National Natural Science Foundation of China under Grant 61903022 and funded by the Beijing Advanced Innovation Center for Intelligent Robots and Systems under Grant 2019IRS11.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, D., Shen, H. & Dong, L. Switching-aware multi-agent deep reinforcement learning for target interception. Appl Intell 53, 7876–7891 (2023). https://doi.org/10.1007/s10489-022-03821-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03821-9

Keywords

Search

Navigation