Abstract
Modelling and planning as well as Machine Learning techniques such as Reinforcement Learning are often difficult in multi-agent problems. With increasing numbers of agents the decision space grows rapidly and is made increasingly complex through interacting agents. This paper is motivated by the question of if it is possible to train single-agent policies in isolation and without the need for explicit cooperation or coordination still successfully deploy them to multi-agent scenarios. In particular we look at the multi-agent Persistent Surveillance Problem (MAPSP), which is the problem of using a number of agents to continually visit and re-visit areas of a map to maximise a metric of surveillance.
We outline five distinct single-agent policies to solve the MAPSP: Reinforcement Learning (DDPG); Neuro-Evolution (NEAT); a Gradient Descent (GD) heuristic; a random heuristic; and a pre-defined ‘ploughing pattern’ (Trail). We will compare the performance and scalability of these single-agent policies to the Multi-Agent PSP. Importantly, in doing so we will demonstrate an emergent property which we call the Homogeneous-Policy Convergence Cycle (HPCC), whereby agents following homogeneous policies can get stuck together, continuously repeating the same action as other agents, significantly impacting performance. This paper will show that just a small amount of noise, at the state or action level, is sufficient to solve the problem, essentially creating artificially-heterogeneous policies for the agents.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Albani, D., Manoni, T., Nardi, D., Trianni, V.: Dynamic UAV swarm deployment for non-uniform coverage: robotics track. In: Proceedings of the International Joint Conference on Autonomous Agents and Multi-agent Systems, pp. 523–531. AAMAS (2018)
Arthur, W.B.: Complexity and the Economy. Science 284(5411), 107–109 (1999). https://doi.org/10.1126/science.284.5411.107
Brockman, G., et al.: OpenAI Gym, pp. 1–4 (2016). http://arxiv.org/abs/1606.01540
Butterworth, J., Tuyls, K., Broecker, B., Paoletti, P.: Evolving coverage behaviours for MAVs using NEAT. In: Proceedings of the International Joint Conference on Autonomous Agents and Multi-agent Systems, Vol. 3, pp. 1886–1888. AAMAS (2018)
Cabreira, T., Brisolara, L., Ferreira Jr., P.R.: Survey on Coverage Path Planning with Unmanned Aerial Vehicles, Vol. 3 (2019). https://doi.org/10.3390/drones3010004
Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzz. Knowl. Based Syst. 06(02), 107–116 (1998). https://doi.org/10.1142/S0218488598000094
LaValle, S.M.: Planning algorithms. Methods (2006). https://doi.org/10.1017/CBO9780511546877
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning, September 2015. http://arxiv.org/abs/1509.02971
Liu, Y., Nejat, G.: Robotic urban search and rescue: a survey from the control perspective. J. Intell. Robot. Syst. 72(2), 147–165 (2013). https://doi.org/10.1007/s10846-013-9822-x
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236
Mogili, U.R., Deepak, B.B.: Review on application of drone systems in precision agriculture. Procedia Comput. Sci. 133, 502–509 (2018). https://doi.org/10.1016/j.procs.2018.07.063
Nigam, N.: The multiple unmanned air vehicle persistent surveillance problem: a review. Machines 2(1), 13–72 (2014). https://doi.org/10.3390/machines2010013
Nigam, N., Kroo, I.: Persistent surveillance using multiple unmanned air vehicles. In: IEEE Aerospace Conference Proceedings, pp. 1–15 (2008). https://doi.org/10.1109/AERO.2008.4526242
Peters, J.R., Wang, S.J., Surana, A., Bullo, F.: Asynchronous and dynamic coverage control scheme for persistent surveillance missions, pp. 1–12 (2016). http://arxiv.org/abs/1609.05264
Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems, vol. 37. University of Cambridge, Department of Engineering Cambridge, England (1994)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolut. Comput. 10(2), 99–127 (2002). https://doi.org/10.1162/106365602320169811
Stone, P., Veloso, M.: Multi-agent systems: a survey from a machine learning perspective. Autonom. Robots 8(3), 345–383 (2000). https://doi.org/10.1023/A:1008942012299
Uhlenbeck, G.E., Ornstein, L.S.: On the theory of the Brownian motion. Phys. Rev. 36(5), 823–841 (1930)
Acknowledgments
This work was funded and delivered in partnership between the Thales Group and the University of Bristol, and with the support of the UK Engineering and Physical Sciences Research Council Grant Award EP/R004757/1 entitled ‘Thales-Bristol Partnership in Hybrid Autonomous Systems Engineering (T-B PHASE)’.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Kent, T., Richards, A., Johnson, A. (2020). Single-Agent Policies for the Multi-Agent Persistent Surveillance Problem via Artificial Heterogeneity. In: Bassiliades, N., Chalkiadakis, G., de Jonge, D. (eds) Multi-Agent Systems and Agreement Technologies. EUMAS AT 2020 2020. Lecture Notes in Computer Science(), vol 12520. Springer, Cham. https://doi.org/10.1007/978-3-030-66412-1_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-66412-1_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66411-4
Online ISBN: 978-3-030-66412-1
eBook Packages: Computer ScienceComputer Science (R0)