Abstract
Multi-agent systems are taking great part in nowadays industries. Cooperative environments may help agents perform better, but the training process is usually time consuming and costly. Although the training process can be speeded up by communicating with important information (such as observations, action sequence, model data and other potential data) between agents, redundant information is still a disturbance term. In this paper, we present a method, called Featured Message Network, which uses a fixed random network to extract features of information from agents as featured message. By communicating with featured messages, agents can cooperate more efficiently while not suffering from useful information loss or redundant information disturbing. We optimized a traditional Deep Q-learning system with our method. We tested our method on Meeting in a Grid problem and experimental results show that training process becomes faster and more stable with our method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amato, C., Bernstein, D.S., Zilberstein, S.: Optimal fixed-size controllers for decentralized pomdps. In: Proceedings of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains (MSDM) (2006)
Amato, C., Zilberstein, S.: Achieving goals in decentralized POMDPs. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems, International Foundation for Autonomous Agents and Multiagent Systems, vol. 1, pp. 593–600 (2009)
Banerjee, B., Peng, J.: Adaptive policy gradient in multiagent learning. In: Proceedings of the Second International Joint Conference on Autonomous Agents and Multiagent Systems, ACM, pp. 686–692 (2003). https://doi.org/10.1145/860575.860686
Bernstein, D.S., Givan, R., Immerman, N., Zilberstein, S.: The complexity of decentralized control of markov decision processes. Math. Oper. Res. 27(4), 819–840 (2002). https://doi.org/10.1287/moor.27.4.819.297
Bernstein, D.S., Hansen, E.A., Zilberstein, S.: Bounded policy iteration for decentralized pomdps. In: Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence (IJCAI), pp. 52–57 (2005)
Bowling, M., Veloso, M.: An analysis of stochastic game theory for multiagent reinforcement learning. Technical report, Carnegie-Mellon Univ Pittsburgh Pa School of Computer Science (2000)
Bowling, M., Veloso, M.: Multiagent learning using a variable learning rate. Artif. Intell. 136(2), 215–250 (2002). https://doi.org/10.1016/s0004-3702(02)00121-2
Burda, Y., Edwards, H., Storkey, A., Klimov, O.: Exploration by random network distillation. arXiv preprint arXiv:1810.12894 (2018)
Cao, K., Lazaridou, A., Lanctot, M., Leibo, J.Z., Tuyls, K., Clark, S.: Emergent communication through negotiation. arXiv preprint arXiv:1804.03980 (2018)
de Cote, E.M., Lazaric, A., Restelli, M.: Learning to cooperate in multi-agent social dilemmas. In: AAMAS, vol. 6, pp. 783–785 (2006). https://doi.org/10.1145/1160633.1160770
Dolcetta, I.C., Ishii, H.: Approximate solutions of the bellman equation of deterministic control theory. Appl. Math. Optim. 11(1), 161–181 (1984). https://doi.org/10.1007/bf01442176
Ferber, J., Weiss, G.: Multi-agent Systems: An Introduction to Distributed Artificial Intelligence, vol. 1, Addison-Wesley Reading (1999)
Foerster, J., Assael, I.A., de Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning. In: Advances in Neural Information Processing Systems, pp. 2137–2145 (2016)
Foerster, J., Chen, R.Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., Mordatch, I.: Learning with opponent-learning awareness. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, International Foundation for Autonomous Agents and Multiagent Systems, pp. 122–130 (2018)
Foerster, J., et al.: Stabilising experience replay for deep multi-agent reinforcement learning. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70. pp. 1146–1155 (2017) JMLR.org
Hansen, E.A., Bernstein, D.S., Zilberstein, S.: Dynamic programming for partially observable stochastic games. In: AAAI, vol. 4, pp. 709–715 (2004)
Havrylov, S., Titov, I.: Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In: Advances in Neural Information Processing Systems, pp. 2149–2159 (2017)
Huang, X., Xiao, S.: Self-augmenting strategy for reinforcement learning. In: Proceedings of the 2017 International Conference on Computer Science and Artificial Intelligence, ACM, pp. 1–4 (2017). https://doi.org/10.1145/3168390.3168392
Ishii, S., Fujita, H., Mitsutake, M., Yamazaki, T., Matsuda, J., Matsuno, Y.: A reinforcement learning scheme for a partially-observable multi-agent game. Mach. Learn. 59(1–2), 31–54 (2005). https://doi.org/10.1007/s10994-005-0461-8
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kok, J. R., Vlassis, N.: Sparse cooperative q-learning. In: Proceedings of the Twenty-First International Conference on Machine Learning, ACM, p. 61 (2004). https://doi.org/10.1145/1015330.1015410
Kok, J.R., Vlassis, N.: Collaborative multiagent reinforcement learning by payoff propagation. J. Mach. Learn. Res. 7, 1789–1828 (2006)
Kuhnle, A., Schaarschmidt, M., Fricke, K.: Tensorforce: a tensorflow library for applied reinforcement learning (2017). https://github.com/tensorforce/tensorforce
Lazaridou, A., Peysakhovich, A., Baroni, M.: Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182 (2016)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
McCandlish, S., Kaplan, J., Amodei, D., Team, O.: An Empirical Model of Large-Batch Training. arXiv preprint arXiv:1812.06162 (2018)
Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)
Nair, R., Tambe, M., Yokoo, M., Pynadath, D., Marsella, S.: Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In: IJCAI, vol. 3, pp. 705–711 (2003)
Omidshafiei, S., Pazis, J., Amato, C., How, J.P., Vian, J.: Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 2681–2690 (2017). JMLR.org
Rashid, T., Samvelyan, M., de Witt, C.S., Farquhar, G., Foerster, J., Whiteson, S.: QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:1803.11485 (2018)
Schaarschmidt, M., Kuhnle, A., Ellis, B., Fricke, K., Gessert, F., Yoneki, E.: Lift: reinforcement learning in computer systems by learning from demonstrations. arXiv preprint arXiv:1808.07903 (2018)
Schulman, J., Levine, S., Abbeel, P., Jordan, M.I., Moritz, P.: Trust region policy optimization. In: ICML, vol. 37, pp. 1889–1897 (2015)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Sukhbaatar, S., Fergus, R.: Learning multiagent communication with backpropagation. In: Advances in Neural Information Processing Systems, pp. 2244–2252 (2016)
Sutton, R.S., Barto, A.G.: Reinforcement learning: an introduction (1998). https://doi.org/10.1109/tnn.1998.712192
Tampuu, A., et al.: Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE 12(4), e0172395 (2017). https://doi.org/10.1371/journal.pone.0172395
Varshavskaya, P., Kaelbling, L.P., Rus, D.: Efficient distributed reinforcement learning through agreement. In: Asama, H., Kurokawa, H., Ota, J., Sekiyama, K. (eds.) Distributed Autonomous Robotic Systems, vol. 8, pp. 367–378. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-00644-9_33
Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., Wang, J.: Mean field multi-agent reinforcement learning. arXiv preprint arXiv:1802.05438 (2018)
Yu, C., Zhang, M., Ren, F.: Emotional multiagent reinforcement learning in social dilemmas. In: Boella, G., Elkind, E., Savarimuthu, B.T.R., Dignum, F., Purvis, M.K. (eds.) PRIMA 2013. LNCS (LNAI), vol. 8291, pp. 372–387. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-44927-7_25
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Jiang, J., Xiao, S., Xun, L. (2019). FMNet: Multi-agent Cooperation by Communicating with Featured Message Network. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation. ICANN 2019. Lecture Notes in Computer Science(), vol 11727. Springer, Cham. https://doi.org/10.1007/978-3-030-30487-4_48
Download citation
DOI: https://doi.org/10.1007/978-3-030-30487-4_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30486-7
Online ISBN: 978-3-030-30487-4
eBook Packages: Computer ScienceComputer Science (R0)