Abstract
Multi-agent patrolling has significant implications for addressing real-world security concerns. In multi-agent systems, the actions of agent directly influence those with whom it interacts. Traditional reinforcement learning-based multi-agent security patrolling methods overlook the role of these localized interactions in coordination among agents, thus failing to enhance the efficiency. To address this issue, this paper introduces a security patrolling approach based on Structured Coordinated Proximal Policy Optimization (PPO). The multi-agent patrolling task is modeled as a finite-time-step distributed partially observable semi-Markov decision process. This method, grounded in the Shapley Value, designs a multi-agent credit allocation function. The efficiency of this function is amplified using the structure of localized interactions. By accurately evaluating the contributions of each agent’s selected actions, this function fosters enhanced coordination among agents. Extensive experiments in various scenarios were conducted, and the results demonstrate that our algorithm outperforms benchmark algorithms in terms of convergence speed and patrolling performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ma, X., An, B., Zhao, M., et al.: Randomized security patrolling for link flooding attack detection. IEEE Trans. Depend. Secure Comput. 17(4), 795–812 (2019)
Brázdil, T., Kučera, A., Řehák, V.: Solving patrolling problems in the internet environment. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 121–127 (2018)
Savkin, A.V., Huang, H.: Asymptotically optimal deployment of drones for surveillance and monitoring. Sensors 19(9), 2068 (2019)
Yang, J., Ding, Z., Wang, L.: The programming model of air-ground cooperative patrol between multi-UAV and police car. IEEE Access 9, 134503–134517 (2021)
Huang, H., Savkin, A.V., Huang, C.: Decentralized autonomous navigation of a UAV network for road traffic monitoring. IEEE Trans. Aerosp. Electron. Syst. 57(4), 2558–2564 (2021)
Gu, H., Zhu, S., Cui, Y., et al.: Application of agent in security platform. In: 2019 IEEE/CIC International Conference on Communications Workshops in China (ICCC Workshops), pp. 233–238. IEEE (2019)
Yang, Q., Yindong, L., Wei, X.: Hierarchical planning for multiple AGVs in warehouse based on global vision. Simul. Model. Pract. Theory 104, 102124 (2020)
Elor, Y., Bruckstein, A.M.: Autonomous multi-agent cycle based patrolling. In: Swarm Intelligence: 7th International Conference, ANTS 2010, Brussels, Belgium. Proceedings 7, pp. 119–130. Springer, Heidelberg (2010)
Mao, T., Ray, L.: Frequency-based patrolling with heterogeneous agents and limited communication. arXiv preprint arXiv:1402.1757 (2014)
Sea, V., Sugiyama, A., Sugawara, T.: Frequency-based multi-agent patrolling model and its area partitioning solution method for balanced workload. In: van Hoeve, W.-J. (ed.) CPAIOR 2018. LNCS, vol. 10848, pp. 530–545. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93031-2_38
Wiandt, B., Simon, V.: Autonomous graph partitioning for multi-agent patrolling problems. In: 2018 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 261–268. IEEE (2018)
Almeida, A., Castro, P., Menezes, T., et al.: Combining idleness and distance to design heuristic agents for the patrolling task. In: II Brazilian Workshop in Games and Digital Entertainment, pp. 33–40 (2003)
Machado, A., Almeida, A., Ramalho, G., et al.: Multi-agent movement coordination in patrolling. In: Proceedings of the 3rd International Conference on Computer and Game. pp. 155–170. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-36483-8_11
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. MIT Press (2018)
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Luis, S.Y., Reina, D.G., Marín, S.L.T.: A multiagent deep reinforcement learning approach for path planning in autonomous surface vehicles: the Ypacaraí lake patrolling case. IEEE Access 9, 17084–17099 (2021)
Jana, M., Vachhani, L., Sinha, A.: A deep reinforcement learning approach for multi-agent mobile robot patrolling. Int. J. Intell. Robot. Appl. 6(4), 724–745 (2022)
Fatima, S.S., Wooldridge, M., Jennings, N.R.: A linear approximation method for the Shapley value. Artif. Intell. 172(14) (2008)
Schulman, J., Wolski, F., Dhariwal, P., et al.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Santana, H., Ramalho, G., Corruble, V., et al.: Multi-agent patrolling with reinforcement learning. In: International Joint Conference on Autonomous Agents and Multiagent Systems, IEEE Computer Society. vol. 4, pp. 1122–1129 (2004)
Lauri, F., Koukam, A.: Robust multi-agent patrolling strategies using reinforcement learning. In: Swarm Intelligence Based Optimization: First International Conference (ICSIBO 2014), Mulhouse, 13–14 May 2014, pp. 157–165, Springer (2014)
Rashid, T., Samvelyan, M., De Witt, C.S., et al.: Monotonic value function factorisation for deep multi-agent reinforcement learning. J. Mach. Learn. Res. 21(1), 7234–7284 (2020)
Li, J., Kuang, K., Wang, B., et al.: Shapley counterfactual credits for multi-agent reinforcement learning. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 934–942 (2021)
Han S., Wang H., Su S., et al.: Stable and efficient Shapley value-based reward reallocation for multi-agent reinforcement learning of autonomous vehicles. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 8765–8771. IEEE (2022)
Wang, J., Zhang, Y., Kim, T.K., et al.: Shapley Q-value: a local reward approach to solve global reward games. Proc. AAAI Conf. Artif. Intell. 34(05) 7285–7292 (2020)
Machado, A., Ramalho, G., Zucker, J.D., et al.: Multi-agent patrolling: an empirical analysis of alternative architectures. In: Multi-agent-Based Simulation II: Third International Workshop. MABS 2002 Bologna, Italy, pp. 155–170. Springer, Heidelberg (2003)
Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol.30, no. 1 (2016)
Fatemi, M., Wu, M., Petch, J., et al.: Semi-Markov offline reinforcement learning for healthcare. In: Conference on Health, Inference, and Learning, pp. 119–137. PMLR (2022)
Wang, Z., Schaul, T., Hessel, M., et al.: Dueling network architectures for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1995–2003. PMLR (2016)
Rosenfeld, A., Maksimov, O., Kraus, S.: When security games hit traffic: a deployed optimal traffic enforcement system. Artif. Intell. 289, 103381 (2020)
Weisburd, S.: Does Police Presence Reduce Car Accidents? Tel Aviv University, Pinhas Sapir Center for Development (2016)
Elliott, M.A., Broughton, J.: How Methods and Levels of Policing Affect Road Casualty Rates. TRL Limited, London (2005)
Acknowledgement
This research was supported by the National Natural Science Foundation of China (62076060, 62072099, 61932007, 61806053).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, Y., Che, Q., Wang, F., Zhang, H., Wang, W., Jiang, Y. (2024). A Method for Security Traffic Patrolling Based on Structural Coordinated Proximal Policy Optimization. In: Sun, Y., Lu, T., Wang, T., Fan, H., Liu, D., Du, B. (eds) Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2023. Communications in Computer and Information Science, vol 2013. Springer, Singapore. https://doi.org/10.1007/978-981-99-9640-7_5
Download citation
DOI: https://doi.org/10.1007/978-981-99-9640-7_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-9639-1
Online ISBN: 978-981-99-9640-7
eBook Packages: Computer ScienceComputer Science (R0)
