Abstract
While numbers of partially observable agents improve their policies throughout decentralized training, the performance of multi-agent systems under this setting suffers from severe non-cooperation and non-stationary. Most previous works attempt to introduce communication into the training and optimization to facilitate cooperation between agents, but the noise message brought by communication may lead to misunderstandings during complex tasks and even lead to catastrophic failure of long-term training. To alleviate the above dilemma, in this paper, we propose the Hierarchical Structure with Shared Attention Mechanism (HiSA), a novel communication-based approach, to facilitate the efficiency and robustness of coordination and cooperation in multi-agent reinforcement learning (MARL). HiSA can not only resist the negative impact of noise in communication, but also effectively utilize attention as communication tool to build efficient cooperative hierarchical policies. Experimental results demonstrate that HiSA significantly outperforms existing communication-based MARL methods especially in the long-term complex cooperation scenarios with isomorphic agents.
Supported by Science and Technology Innovation 2030 New Generation Artificial Intelligence Major Project (No.2018AAA0100905), the National Natural Science Foundation of China (No. 62192783), Primary Research & Developement Plan of Jiangsu Province (No. BE2021028), Shenzhen Fundamental Research Program (No. 2021Szvup056).
Z. Chen and Z. Zhu—Contribute equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bacon, P., Harb, J., Precup, D.: The option-critic architecture. In: Proceedings of the Thirty-first AAAI Conference on Artificial Intelligence, San Francisco, California, pp. 1726–1734. AAAI Press (2017)
Bengio, S., et al.: Learning Attentional Communication for Multi-Agent Cooperation. In: Proceedings of the Thirty-second International Conference on Neural Information Processing Systems, Montréal, Canada, pp. 7265–7275 (2018)
Carroll, M., et al.: On the utility of learning about humans for human-AI coordination. In: Proceedings of the Thirty-Third International Conference on Neural Information Processing Systems, Vancouver, BC, pp. 5175–5186 (2019)
Chen, L., Zhang, H., Xiao, J., et al.: SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, pp. 6298–6306. IEEE Computer Society (2017)
Foerster, J.N., Assael, Y.M., de Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning. In: Proceedings of the Thirtieth International Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 2137–2145 (2016)
Iqbal, S., Sha, F.: Actor-attention-critic for multi-agent reinforcement learning. In: Proceedings of the Thirty-sixth International Conference on Machine Learning. PMLR, Long Beach, California, pp. 2961–2970 (2019)
Kim, D., Moon, S., Hostallero, D., et al.: Learning to schedule communication in multi-agent reinforcement learning. In: Proceedings of the Seventh International Conference on Learning Representations, New Orleans, LA (2019)
Kim, W., Cho, M., Sung, Y.: Message-dropout: an efficient training method for multi-agent deep reinforcement learning. In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, pp. 6079–6086. AAAI Press (2019)
Kulkarni, T.D., Narasimhan, K., Saeedi, A., Tenenbaum, J.: Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation. In: Proceedings of the Thirtieth International Conference on Neural Information Processing Systems. Barcelona, Spain, pp. 3675–3683 (2016)
Liu, Y., Wang, W., Hu, Y., et al.: Multi-agent game abstraction via graph attention neural network. In: Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, pp. 7211–7218. AAAI Press (2020)
Mao, H., Zhang, Z., Xiao, Z., Gong, Z.: Modelling the dynamic joint policy of teammates with attention multi-agent DDPG. In: Proceedings of the Eighteenth International Conference on Autonomous Agents and MultiAgent Systems. IFAAMAS, Montreal, QC, pp. 1108–1116 (2019)
Niv, Y., Daniel, R., Geana, A., et al.: Reinforcement learning in multidimensional environments relies on attention mechanisms. J. Neurosci. 35(21), 8145–8157 (2015)
Pesce, E., Montana, G.: Improving coordination in small-scale multi-agent deep reinforcement learning through memory-driven communication. Mach. Learn. 109(9–10), 1727–1747 (2020)
Samvelyan, M., Rashid, T., de Witt, C.S., et al.: The StarCraft multi-agent challenge. In: Proceedings of the Eighteenth International Conference on Autonomous Agents and MultiAgent Systems. IFAAMAS, Montreal, QC, pp. 2186–2188(2019)
Song, Y., Wang, J., Lukasiewicz, T., et al.: Diversity-driven extensible hierarchical reinforcement learning. In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, Honolulu, Hawaii. AAAI Press, pp. 4992–4999 (2019)
Sukhbaatar, S., Szlam, A., Fergus, R.: Learning multiagent communication with backpropagation. In: Proceedings of the Thirtieth International Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 2244–2252 (2016)
Tomasello, M. (ed.): Origins of Human Communication. MIT Press, USA (2010)
Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Proceedings of the Thirty-first International Conference on Neural Information Processing Systems, Long Beach, CA pp. 5998–6008 (2017)
Vinyals, O., Ewalds, T., Bartunov, S., et al.: StarCraft II: a new challenge for reinforcement learning (2017)
Wang, T., Dong, H., Lesser, V.R., Zhang, C.: ROMA: multi-agent reinforcement learning with emergent roles. In: Proceedings of the Thirty-seventh International Conference on Machine Learning. PMLR, Virtual Event, pp. 9876–9886 (2020)
Wang, W., Yang, T., Liu, Y., et al.: Action semantics network: considering the effects of actions in multiagent systems. In: Proceedings of the Eighth International Conference on Learning Representations. OpenReview.net, Addis Ababa, Ethiopia (2020)
Acknowledgements
Thanks to Science,Technology and Innovation Commission of Shenzhen Municipality. This work is supported by Science and Technology Innovation 2030 New Generation Artificial Intelligence Major Project (No. 2018AAA0100905), the National Natural Science Foundation of China (No. 62192783), Primary Research & Developement Plan of Jiangsu Province (No. BE2021028), Shenzhen Fundamental Research Program (No. 2021Szvup056).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, Z., Zhu, Z., Yang, G., Gao, Y. (2022). HiSA: Facilitating Efficient Multi-Agent Coordination and Cooperation by Hierarchical Policy with Shared Attention. In: Khanna, S., Cao, J., Bai, Q., Xu, G. (eds) PRICAI 2022: Trends in Artificial Intelligence. PRICAI 2022. Lecture Notes in Computer Science, vol 13631. Springer, Cham. https://doi.org/10.1007/978-3-031-20868-3_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-20868-3_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20867-6
Online ISBN: 978-3-031-20868-3
eBook Packages: Computer ScienceComputer Science (R0)