Skip to main content
SpringerLink
Log in

Applying Deep Q-learning for Multi-agent Cooperative-Competitive Environments

  • Conference paper
  • First Online:
17th International Conference on Soft Computing Models in Industrial and Environmental Applications (SOCO 2022) (SOCO 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 531))

  • 550 Accesses

Abstract

Cooperative-competitive social group dynamics may be modelled with multi-agent environments with a large number of agents from a few distinct agent-types. Even the simplest games modelling social interactions are suitable to analyze emerging group dynamics. In many cases, the underlying computational problem is NP-complex, thus various machine learning techniques are implemented to accelerate the optimization process. Multi-agent reinforcement learning provides an effective framework to train autonomous agents with an adaptive nature. We analyze the performance of centralized and decentralized training featuring Deep Q-Networks on cooperative-competitive environments introduced in the MAgent library. Our experiments demonstrate that sensible policies may be constructed utilizing centralized and decentralized reinforcement learning methods by observing the mean rewards accumulated during training episodes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aymanns, C., Foerster, J., Georg, C.-P.: Fake news in social networks. SSRN Electr. J. Paper No. 2018/4 (2017)

    Google Scholar 

  2. Bloembergen, D., Tuyls, K., Hennes, D., Kaisers, M.: Evolutionary dynamics of multi-agent learning: a survey. J. Artif. Intell. Res. 53, 659–697 (2015)

    Article  MathSciNet  Google Scholar 

  3. Brooks, R.A.: Intelligence without representation. Artif. Intell. 47(1), 139–159 (1991)

    Article  Google Scholar 

  4. Cai, C., Yang, C., Zhu, Q., Liang, Y.: Collision avoidance in multi-robot systems. In: 2007 International Conference on Mechatronics and Automation, pp. 2795–2800 (2007)

    Google Scholar 

  5. Cai, P., Lee, Y., Luo, Y., Hsu, D.: Summit: A simulator for urban driving in massive mixed traffic. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4023–4029. IEEE (2020)

    Google Scholar 

  6. Carley, K., Martin, M., Hirshman, B.: The etiology of social change. Top. Cogn. Sci. 1, 621–650 (2009)

    Article  Google Scholar 

  7. Dechter, R., Meiri, I., Pearl, J.: Temporal constraint networks. Artif. Intell. 49(1), 61–95 (1991)

    Article  MathSciNet  Google Scholar 

  8. dos Santos, D.S., Bazzan, A.L.: Distributed clustering for group formation and task allocation in multiagent systems: a swarm intelligence approach. Appl. Soft Comput. 12(8), 2123–2131 (2012)

    Article  Google Scholar 

  9. Gupta, J.K., Egorov, M., Kochenderfer, M.: Cooperative multi-agent control using deep reinforcement learning. In: Sukthankar, G., Rodriguez-Aguilar, J.A. (eds.) AAMAS 2017. LNCS (LNAI), vol. 10642, pp. 66–83. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71682-4_5

    Chapter  Google Scholar 

  10. Herrera, M., Pérez-Hernández, M., Kumar Parlikad, A., Izquierdo, J.: Multi-agent systems and complex networks: review and applications in systems engineering. Processes 8(3), 312 (2020)

    Article  Google Scholar 

  11. Kossinets, G., Watts, D.J.: Empirical analysis of an evolving social network. Science 311, 88–90 (2006)

    Article  MathSciNet  Google Scholar 

  12. Leibo, J. Z., et al.: Scalable evaluation of multi-agent reinforcement learning with melting pot. In: International Conference on Machine Learning, pp. 6187–6199. PMLR (2021)

    Google Scholar 

  13. Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., Graepel, T.: Multi-agent reinforcement learning in sequential social dilemmas. In: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’17, pp. 464–473 (2017)

    Google Scholar 

  14. Li, J.-Q., Mirchandani, P.B., Borenstein, D.: The vehicle rescheduling problem: model and algorithms. Networks 50(3), 211–229 (2007)

    Article  MathSciNet  Google Scholar 

  15. Liang, E., et al.: Ray RLlib: A composable and scalable reinforcement learning library. arXiv preprint arXiv:abs/1712.09381 (2017)

  16. Liu, S., Lever, G., Merel, J., Tunyasuvunakool, S., Heess, N., Graepel, T.: Emergent coordination through competition. arXiv preprint arXiv:abs/1902.07151 (2019)

  17. Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

  18. Mordatch, I., Abbeel, P.: Emergence of grounded compositional language in multi-agent populations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  19. Nicosia, V., Tang, J., Musolesi, M., Russo, G., Mascolo, C., Latora, V.: Components in time-varying graphs. Chaos: Interdisc. J. Nonlinear. Sci. 22(2), 023101 (2012)

    Google Scholar 

  20. Nowé, A., Vrancx, P., Hauwere, Y.-M.D.: Game theory and multi-agent reinforcement learning. In: Wiering, M., van Otterlo, M. (eds) Reinforcement Learning. Adaptation, Learning, and Optimization, pp. 441–470. Springer, Berlin, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27645-3_14

  21. Papoudakis, G., Christianos, F., Schäfer, L., Albrecht, S.V.: Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS) (2021)

    Google Scholar 

  22. Rashid, T., Samvelyan, M., Witt, C.S.D., Farquhar, G., Foerster, J.N., Whiteson, S.: Monotonic value function factorisation for deep multi-agent reinforcement learning. J. Mach. Learn. Res. 21, 178:1–178:51 (2020)

    Google Scholar 

  23. Stehlé, J.: Simulation of an SEIR infectious disease model on the dynamic contact network of conference attendees. BMC Med. 9(1), 1–15 (2011)

    Article  MathSciNet  Google Scholar 

  24. Sunehag, P., et al.: Value-decomposition networks for cooperative multi-agent learning based on team reward. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’18, pp. 2085–2087 (2018)

    Google Scholar 

  25. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. The MIT Press (2018)

    Google Scholar 

  26. Tan, M.: Multi-agent reinforcement learning: Independent vs. cooperative agents. In: Proceedings of the Tenth International Conference on Machine Learning, pp. 330–337. Morgan Kaufmann (1993)

    Google Scholar 

  27. Terry, J.K., et al.: PettingZoo: Gym for multi-agent reinforcement learning. arXiv preprint arXiv:2009.14471 (2020)

  28. Törnquist, J., Persson, J., et al.: Train traffic deviation handling using tabu search and simulated annealing. In: Proceedings of the 38th Hawaii International Conference on System Sciences, pp. 1–10 (2005)

    Google Scholar 

  29. Vinyals, O., et al.: Starcraft II: A new challenge for reinforcement learning. arXiv preprint arXiv:abs/1708.04782 (2017)

  30. Vrancx, P., Verbeeck, K., Nowe, A.: Decentralized learning in Markov games. Part B, Cybern.: a publication of the IEEE Syst., Man, Cybern. Soc. 38, 976–981 (2008)

    Article  Google Scholar 

  31. Wang, L., Wang, Z., Hu, S., Liu, L.: Ant colony optimization for task allocation in multi-agent systems. China Commun. 10(3), 125–132 (2013)

    Article  Google Scholar 

  32. Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., Wang, J.: Mean field multi-agent reinforcement learning. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, vol. 80 of Proceedings of Machine Learning Research, pp. 5571–5580. PMLR (2018)

    Google Scholar 

  33. Zhang, T., Ye, Q., Bian, J., Xie, G., Liu, T.-Y.: MFVFD: a multi-agent Q-learning approach to cooperative and non-cooperative tasks. In: Zhou, Z.-H. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 500–506 (2021)

    Google Scholar 

  34. Zheng, L., et al.: MAgent: A many-agent reinforcement learning platform for artificial collective intelligence. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Mathematics and Computer Science, Babeş-Bolyai University, Cluj-Napoca, Romania

    Anikó Kopacz, Lehel Csató & Camelia Chira

Authors
  1. Anikó Kopacz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lehel Csató
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Camelia Chira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anikó Kopacz .

Editor information

Editors and Affiliations

  1. Faculty of Engineering, University of Deusto, Bilbao, Spain

    Pablo García Bringas

  2. University of León, León, Spain

    Hilde Pérez García

  3. Mechanical Engineering Department, University of La Rioja, Logroño, La Rioja, Spain

    Francisco Javier Martinez-de-Pison

  4. Inteligencia Artificial, University of Oviedo, A Coruña, La Coruña, Spain

    José Ramón Villar Flecha

  5. Data Science and Big Data Lab, Pablo de Olavide University, Sevilla, Spain

    Alicia Troncoso Lora

  6. University of Oviedo, Oviedo, Spain

    Enrique A. de la Cal

  7. Department of Civil Engineering, University of Burgos, Burgos, Spain

    Álvaro Herrero

  8. School of engineering, Pablo Olavide University, Seville, Spain

    Francisco Martínez Álvarez

  9. DIGIP, University of Bergamo, Dalmine, Bergamo, Italy

    Giuseppe Psaila

  10. Department of Industrial Engineering, University of A Coruña, Ferrol, Spain

    Héctor Quintián

  11. Department of Computing Science, University of Salamanca, Salamanca, Spain

    Emilio S. Corchado Rodriguez

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kopacz, A., Csató, L., Chira, C. (2023). Applying Deep Q-learning for Multi-agent Cooperative-Competitive Environments. In: García Bringas, P., et al. 17th International Conference on Soft Computing Models in Industrial and Environmental Applications (SOCO 2022). SOCO 2022. Lecture Notes in Networks and Systems, vol 531. Springer, Cham. https://doi.org/10.1007/978-3-031-18050-7_61

Download citation

Publish with us

Policies and ethics

Search

Navigation