Abstract
This paper introduces a deep reinforcement learning method to train an autonomous aerial agent acting as a shepherd to provide guidance for a swarm of ground vehicles. The learner is situated within a high-fidelity robotic-operating-system (ROS)-based simulation environment consisting of an Unmanned Aerial Vehicle (UAV) learning to guide a swarm of Unmanned Ground Vehicles (UGVs) to a target location. Our approach uses a combination of machine education, apprenticeship bootstrapping, and deep-learning-based methodologies to decompose the complex shepherding strategy into sub-problems requiring simpler skills that get fused to form the overall skills required for shepherding. The proposed methodology is effective in training the UAV agent with multiple reward designing schemes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Balch, T., Arkin, R.C.: Behavior-based formation control for multirobot teams. IEEE Trans. Robot. Autom. 14(6), 926–939 (1998)
Carelli, R., De la Cruz, C., Roberti, F.: Centralized formation control of non-holonomic mobile robots. Lat. Am. Appl. Res. 36(2), 63–69 (2006)
Chaimowicz, L., Kumar, V.: Aerial shepherds: coordination among UAVs and swarms of robots. In: Proceedings of DARS 2004. Citeseer (2004)
Clayton, N.R., Abbass, H.: Machine teaching in hierarchical genetic reinforcement learning: curriculum design of reward functions for swarm shepherding. arXiv preprint arXiv:1901.00949 (2019)
ClearpathRobotics: ROS husky robot (2017). http://wiki.ros.org/Robots/Husky. Accessed 20 June 2019
Gee, A., Abbass, H.: Transparent machine education of neural networks for swarm shepherding using curriculum design. arXiv preprint arXiv:1903.09297 (2019)
Guillet, A., Lenain, R., Thuilot, B., Rousseau, V.: Formation control of agricultural mobile robots: a bidirectional weighted constraints approach. J. Field Robot. 34, 1260–1274 (2017)
Huang, H., Sturm, J.: Tum simulator (2014). http://wiki.ros.org/tum_simulator. Accessed 20 June 2019
Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996)
Martinez, S., Cortes, J., Bullo, F.: Motion coordination with distributed information. IEEE Control Syst. Mag. 27(4), 75–88 (2007)
Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937 (2016)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)
Nguyen, H., et al.: Apprenticeship bootstrapping via deep learning with a safety net for UAV-UGV interaction. arXiv preprint arXiv:1810.04344 (2018)
Nguyen, H.T., Garratt, M., Bui, L.T., Abbass, H.: Supervised deep actor network for imitation learning in a ground-air UAV-UGVs coordination task. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8. IEEE (2017)
Nguyen, T., Nguyen, H., Debie, E., Kasmarik, K., Garratt, M., Abbass, H.: Swarm Q-learning with knowledge sharing within environments for formation control. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2018)
Oh, H., Shirazi, A.R., Sun, C., Jin, Y.: Bio-inspired self-organising multi-robot pattern formation: a review. Robot. Auton. Syst. 91, 83–100 (2017)
Oh, K.K., Park, M.C., Ahn, H.S.: A survey of multi-agent formation control. Automatica 53, 424–440 (2015)
Palmer, G., Tuyls, K., Bloembergen, D., Savani, R.: Lenient multi-agent deep reinforcement learning. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 443–451. International Foundation for Autonomous Agents and Multiagent Systems (2018)
Speck, C., Bucci, D.J.: Distributed UAV swarm formation control via object-focused, multi-objective SARSA. In: 2018 Annual American Control Conference (ACC), pp. 6596–6601. IEEE (2018)
Strömbom, D., et al.: Solving the shepherding problem: heuristics for herding autonomous, interacting agents. J. R. Soc. Interface 11(100), 20140719 (2014)
Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: AAAI, Phoenix, AZ, vol. 2, p. 5 (2016)
Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)
Xu, D., Zhang, X., Zhu, Z., Chen, C., Yang, P.: Behavior-based formation control of swarm robots. Math. Probl. Eng. 2014 (2014)
Yang, Z., Merrick, K., Jin, L., Abbass, H.A.: Hierarchical deep reinforcement learning for continuous action control. IEEE Trans. Neural Netw. Learn. Syst. 99, 1–11 (2018)
Acknowledgement
This material is based upon work supported by the Air Force Office of Scientific Research under award number FA2386-17-1-4054 and an Australian Research Council Discovery Grant DP160102037.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Nguyen, H.T. et al. (2019). A Deep Hierarchical Reinforcement Learner for Aerial Shepherding of Ground Swarms. In: Gedeon, T., Wong, K., Lee, M. (eds) Neural Information Processing. ICONIP 2019. Lecture Notes in Computer Science(), vol 11953. Springer, Cham. https://doi.org/10.1007/978-3-030-36708-4_54
Download citation
DOI: https://doi.org/10.1007/978-3-030-36708-4_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36707-7
Online ISBN: 978-3-030-36708-4
eBook Packages: Computer ScienceComputer Science (R0)