Abstract
Multi-agent cooperation is one of the attractive aspects in multi-agent systems. However, during the process of cooperation, communication among agents is limited by the distance or the bandwidth. Besides, the agents move around and their neighbors appear or vanish, which makes the agents hard to capture temporal dependences and to learn a stable policy. To address these issues, a Spatial-Temporal Graph Attentional Long Short-Term Memory (LSTM) Scheme (STGA-LSTM), which is composed of spatial capture network and spatiotemporal LSTM network, is proposed. The spatial capture network is designed based on graph attention network to enlarge the agents’ communication range and capture the spatial structure of the multi-agent system. Based on the standard LSTM, a spatiotemporal LSTM network, which is in combination with graph convolutional network and attention mechanism, is designed to capture the temporal evolutionary patterns while keeping the spatial structure learned by spatial capture network. The results of simulations including mixed cooperative and competitive tasks indicate that the agents can learn stable and complicated strategies with STGA-LSTM.
Supported by the National Key Research and Development Program of China under Grant 2018AAA0102402, and Innovation Academy for Light-duty Gas Turbine, Chinese Academy of Sciences, No. CXYJJ19-ZD-02.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agarwal, A., Kumar, S., Sycara, K.: Learning transferable cooperative behavior in multi-agent teams. arXiv preprint arXiv:1906.01202 (2019)
Cui, Z., Henrickson, K., Ke, R., Wang, Y.: Traffic graph convolutional recurrent neural network: a deep learning framework for network-scale traffic learning and forecasting. IEEE Trans. Intell. Transp. Syst. 21, 4883–4894 (2019)
Das, A., et al.: TarMAC: targeted multi-agent communication. In: International Conference on Machine Learning, pp. 1538–1546 (2019)
Foerster, J.N., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual multi-agent policy gradients. In: 32nd AAAI Conference on Artificial Intelligence (2018)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Huang, Y., Bi, H., Li, Z., Mao, T., Wang, Z.: STGAT: modeling spatial-temporal interactions for human trajectory prediction. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 6272–6281 (2019)
Iqbal, S., Sha, F.: Actor-attention-critic for multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 2961–2970 (2019)
Jiang, J., Dun, C., Lu, Z.: Graph convolutional reinforcement learning for multi-agent cooperation. arXiv preprint arXiv:1810.09202 (2018)
Jiang, J., Lu, Z.: Learning attentional communication for multi-agent cooperation. In: Advances in Neural Information Processing Systems, pp. 7254–7264 (2018)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Li, X., Zhang, J., Bian, J., Tong, Y., Liu, T.Y.: A cooperative multi-agent reinforcement learning framework for resource balancing in complex logistics network. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 980–988. International Foundation for Autonomous Agents and Multiagent Systems (2019)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O.P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in Neural Information Processing Systems, pp. 6379–6390 (2017)
Malysheva, A., Sung, T.T., Sohn, C.B., Kudenko, D., Shpilman, A.: Deep multi-agent reinforcement learning with relevance graphs. arXiv preprint arXiv:1811.12557 (2018)
Niepert, M., Ahmed, M., Kutzkov, K.: Learning convolutional neural networks for graphs. In: International Conference on Machine Learning, pp. 2014–2023 (2016)
Radhakrishnan, B.M., Srinivasan, D.: A multi-agent based distributed energy management scheme for smart grid applications. Energy 103, 192–204 (2016)
Shi, L., Zhang, Y., Cheng, J., Lu, H.: Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12026–12035 (2019)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=rJXMpikCZ
Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., Wang, J.: Mean field multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 5567–5576 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, H., Liu, Z., Pu, Z., Yi, J. (2020). STGA-LSTM: A Spatial-Temporal Graph Attentional LSTM Scheme for Multi-agent Cooperation. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Lecture Notes in Computer Science(), vol 12533. Springer, Cham. https://doi.org/10.1007/978-3-030-63833-7_56
Download citation
DOI: https://doi.org/10.1007/978-3-030-63833-7_56
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63832-0
Online ISBN: 978-3-030-63833-7
eBook Packages: Computer ScienceComputer Science (R0)