Abstract
Behavior Trees have gained widespread applications across diverse domains, facilitating the decomposition of complex tasks into manageable subtasks. However, an inherent challenge in maximizing the performance of BTs lies in dynamically allocating agents to subtasks as time progresses. This allocation predicament is compounded by the intricate nature of game states and the temporal variations in subtask activation. In this paper, we propose a novel approach that combines temporal task graphs with reinforcement learning to dynamically allocate agents among subtasks in BT. We employ a temporal task graph to model the dynamic activation of subtasks, where encoded vectors are multiplied by the agent’s encoded observation. This enables each agent to be assigned to a specific subtask while considering comprehensive information about all subtasks. Moreover, we aggregate the Q-values of selected subtasks for all agents, leveraging this information to compute a total loss for updating the entire network. To evaluate the efficacy of our approach, we conducted extensive experiments on the challenging benchmark provided by Google Research Football. The results clearly demonstrate a significant performance improvement in BTs when leveraging our proposed framework.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bitsakos, C., Konstantinou, I., Koziris, N.: DERP: a deep reinforcement learning cloud system for elastic resource provisioning. In: 2018 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 21–29. IEEE (2018)
Chen, X., Liu, G.: Energy-efficient task offloading and resource allocation via deep reinforcement learning for augmented reality in mobile edge networks. IEEE Internet Things J. 8(13), 10843–10856 (2021)
Choudhury, S., Gupta, J.K., Kochenderfer, M.J., Sadigh, D., Bohg, J.: Dynamic multi-robot task allocation under uncertainty and temporal constraints. Auton. Robot. 46, 231–247 (2020)
Dey, R., Child, C.: QL-BT: enhancing behaviour tree design and implementation with Q-learning. In: 2013 IEEE Conference on Computational Intelligence in Games (CIG), pp. 1–8. IEEE (2013)
Groß, J., Gros, T.P.: Using deep reinforcement learning to optimize assignment problems (2021)
Hoff, J.W., Christensen, H.J.: Evolving behaviour trees:-automatic generation of AI opponents for real-time strategy games. Master’s thesis, NTNU (2016)
Hu, H., Jia, X., Liu, K., Sun, B.: Self-adaptive traffic control model with behavior trees and reinforcement learning for AGV in industry 4.0. IEEE Trans. Ind. Inf. 17(12), 7968–7979 (2021)
Iovino, M., Scukins, E., Styrud, J., Ögren, P., Smith, C.: A survey of behavior trees in robotics and AI. Robot. Auton. Syst. 154, 104096 (2022)
Kurach, K., et al.: Google research football: a novel reinforcement learning environment. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 4501–4510 (2020)
Li, L., Wang, L., Li, Y., Sheng, J.: Mixed deep reinforcement learning-behavior tree for intelligent agents design. In: ICAART, vol. 1, pp. 113–124 (2021)
Liu, C., Chen, C.X., Chen, C.: META: a city-wide taxi repositioning framework based on multi-agent reinforcement learning. IEEE Trans. Intell. Transp. Syst. (2021)
Manessi, F., Rozza, A., Manzo, M.: Dynamic graph convolutional networks. Pattern Recogn. 97, 107000 (2020)
Marzinotto, A.: Flexible robot to object interactions through rigid and deformable cages. Ph.D. thesis, KTH Royal Institute of Technology (2017)
Neupane, A., Goodrich, M.A.: Designing emergent swarm behaviors using behavior trees and grammatical evolution. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2138–2140 (2019)
Paul, S., Ghassemi, P., Chowdhury, S.: Learning scalable policies over graphs for multi-robot task allocation using capsule attention networks. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 8815–8822 (2022)
Rossi, E., Chamberlain, B., Frasca, F., Eynard, D., Monti, F., Bronstein, M.: Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 (2020)
Rovida, F., Grossmann, B., Krüger, V.: Extended behavior trees for quick definition of flexible robotic tasks. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6793–6800. IEEE (2017)
Tokic, M., Palm, G.: Value-difference based exploration: adaptive control between epsilon-greedy and softmax. In: Bach, J., Edelkamp, S. (eds.) KI 2011. LNCS (LNAI), vol. 7006, pp. 335–346. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24455-1_33
Wang, J., Zhao, L., Liu, J., Kato, N.: Smart resource allocation for mobile edge computing: a deep reinforcement learning approach. IEEE Trans. Emerg. Top. Comput. 9(3), 1529–1541 (2019)
Acknowledgements
This work was financially supported by the National Natural Science Foundation of China Youth Science Foundation under Grants No. 61902425, No. 62102444.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, X., Li, Y., Zhang, J., Xu, X., Liu, D. (2024). Temporal Task Graph Based Dynamic Agent Allocation for Applying Behavior Trees in Multi-agent Games. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1962. Springer, Singapore. https://doi.org/10.1007/978-981-99-8132-8_10
Download citation
DOI: https://doi.org/10.1007/978-981-99-8132-8_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8131-1
Online ISBN: 978-981-99-8132-8
eBook Packages: Computer ScienceComputer Science (R0)