Skip to main content
SpringerLink
Log in

ScaRLib: A Framework for Cooperative Many Agent Deep Reinforcement Learning in Scala

  • Conference paper
  • First Online:
Coordination Models and Languages (COORDINATION 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13908))

Included in the following conference series:

  • 253 Accesses

Abstract

Multi Agent Reinforcement Learning (MARL) is an emerging field in machine learning where multiple agents learn, simultaneously and in a shared environment, how to optimise a global or local reward signal. MARL has gained significant interest in recent years due to its successful applications in various domains, such as robotics, IoT, and traffic control. Cooperative Many Agent Reinforcement Learning (CMARL) is a relevant subclass of MARL, where thousands of agents work together to achieve a common coordination goal.

In this paper, we introduce ScaRLib, a Scala framework relying on state-of-the-art deep learning libraries to support the development of CMARL systems. The framework supports the specification of centralised training and decentralised execution, and it is designed to be easily extensible, allowing to add new algorithms, new types of environments, and new coordination toolchains.

This paper describes the main structure and features of ScaRLib and includes basic demonstrations that showcase binding with one such toolchain: ScaFi programming framework and Alchemist simulator can be exploited to enable learning of field-based coordination policies for large-scale systems.

Supported by Department of Computer Science and Engineering @ Alma Mater Studiorum - University of Bologna.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://alchemistsimulator.github.io/.

  2. 2.

    https://scafi.github.io/.

  3. 3.

    Tool available on GitHub at https://github.com/ScaRLib-group/ScaRLib.

  4. 4.

    Demo video at: https://github.com/ScaRLib-group/ScaRLib-demo-video.

  5. 5.

    https://pytorch.org/.

  6. 6.

    https://deeplearning4j.konduit.ai/.

  7. 7.

    https://github.com/microsoft/scala_torch.

  8. 8.

    Repository available at https://github.com/ScaRLib-group/ScaRLib-flock-demo.

  9. 9.

    https://github.com/mlii/mfrl.

References

  1. Aguzzi, G.: Research directions for aggregate computing with machine learning. In: 2021 IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C). IEEE (2021). https://doi.org/10.1109/acsos-c52956.2021.00078

  2. Aguzzi, G., Casadei, R., Pianini, D., Viroli, M.: Dynamic decentralization domains for the internet of things. IEEE Internet Comput. 26(6), 16–23 (2022). https://doi.org/10.1109/mic.2022.3216753

  3. Aguzzi, G., Casadei, R., Viroli, M.: Addressing collective computations efficiency: Towards a platform-level reinforcement learning approach. In: Casadei, R., et al. (eds.) IEEE International Conference on Autonomic Computing and Self-Organizing Systems, ACSOS 2022, Virtual, CA, USA, 19–23 September 2022, pp. 11–20. IEEE (2022). https://doi.org/10.1109/ACSOS55765.2022.00019

  4. Aguzzi, G., Casadei, R., Viroli, M.: Machine learning for aggregate computing: a research roadmap. In: 2022 IEEE 42nd International Conference on Distributed Computing Systems Workshops (ICDCSW). IEEE (2022). https://doi.org/10.1109/icdcsw56584.2022.00032

  5. Aguzzi, G., Casadei, R., Viroli, M.: Towards reinforcement learning-based aggregate computing. In: ter Beek, M.H., Sirjani, M. (eds) Coordination Models and Languages. COORDINATION 2022. IFIP Advances in Information and Communication Technology, vol. 13271, pp. 72–91. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-08143-9_5

  6. Baker, B., et al.: Emergent tool use from multi-agent autocurricula (2019). https://doi.org/10.48550/ARXIV.1909.07528. https://arxiv.org/abs/1909.07528

  7. Beal, J., Pianini, D., Viroli, M.: Aggregate programming for the internet of things. Computer 48(9), 22–30 (2015). https://doi.org/10.1109/mc.2015.261

  8. Bettini, M., Kortvelesy, R., Blumenkamp, J., Prorok, A.: VMAS: a vectorized multi-agent simulator for collective robot learning. The 16th International Symposium on Distributed Autonomous Robotic Systems (2022)

    Google Scholar 

  9. Busoniu, L., Babuska, R., Schutter, B.D.: A comprehensive survey of multiagent reinforcement learning. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 38(2), 156–172 (2008). https://doi.org/10.1109/tsmcc.2007.913919

  10. Casadei, R.: Macroprogramming: Concepts, state of the art, and opportunities of macroscopic behaviour modelling. ACM Computing Surveys (2023). https://doi.org/10.1145/3579353

  11. Casadei, R., Viroli, M., Aguzzi, G., Pianini, D.: ScaFi: a scala DSL and toolkit for aggregate programming. SoftwareX 20, 101248 (2022). https://doi.org/10.1016/j.softx.2022.101248

  12. Casadei, R., Viroli, M., Audrito, G., Pianini, D., Damiani, F.: Engineering collective intelligence at the edge with aggregate processes. Eng. Appl. Artif. Intell. 97, 104081 (2021). https://doi.org/10.1016/j.engappai.2020.104081

  13. Chu, T., Wang, J., Codecà, L., Li, Z.: Multi-agent deep reinforcement learning for large-scale traffic signal control (2019). https://doi.org/10.48550/ARXIV.1903.04527. https://arxiv.org/abs/1903.04527

  14. Du, W., Ding, S.: A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artif. Intell. Rev. 54(5), 3215–3238 (2020). https://doi.org/10.1007/s10462-020-09938-y

    Article  Google Scholar 

  15. Fey, M., Lenssen, J.E.: Fast graph representation learning with pyTorch geometric (2019)

    Google Scholar 

  16. He, K., Doshi, P., Banerjee, B.: Many agent reinforcement learning under partial observability (2021). https://doi.org/10.48550/ARXIV.2106.09825. https://arxiv.org/abs/2106.09825

  17. Hüttenrauch, M., Adrian, S., Neumann, G., et al.: Deep reinforcement learning for swarm systems. J. Mach. Learn. Res. 20(54), 1–31 (2019)

    MathSciNet  MATH  Google Scholar 

  18. Laddad, S., Sen, K.: ScalaPy: seamless python interoperability for cross-platform scala programs. In: Proceedings of the 11th ACM SIGPLAN International Symposium on Scala. ACM (2020). https://doi.org/10.1145/3426426.3428485

  19. Lei, L., Tan, Y., Zheng, K., Liu, S., Zhang, K., Shen, X.: Deep reinforcement learning for autonomous internet of things: Model, applications and challenges. IEEE Commun. Surv. Tutorials 22(3), 1722–1760 (2020). https://doi.org/10.1109/comst.2020.2988367

  20. Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: Cohen, W.W., Hirsh, H. (eds.) Machine Learning Proceedings 1994, pp. 157–163. Morgan Kaufmann, San Francisco (CA) (1994). https://doi.org/10.1016/B978-1-55860-335-6.50027-1. https://www.sciencedirect.com/science/article/pii/B9781558603356500271

  21. Long, P., Fanl, T., Liao, X., Liu, W., Zhang, H., Pan, J.: Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2018). https://doi.org/10.1109/icra.2018.8461113

  22. Mnih, V., et al.: Playing Atari with deep reinforcement learning (2013). https://doi.org/10.48550/ARXIV.1312.5602. https://arxiv.org/abs/1312.5602

  23. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236

  24. Moritz, P., et al.: Ray: a distributed framework for emerging AI applications (2017). https://doi.org/10.48550/ARXIV.1712.05889. https://arxiv.org/abs/1712.05889

  25. Pianini, D., Montagna, S., Viroli, M.: Chemical-oriented simulation of computational systems with ALCHEMIST. J. Simulation 7(3), 202–215 (2013). https://doi.org/10.1057/jos.2012.27

  26. Reynolds, C.W.: Flocks, herds and schools: a distributed behavioral model. In: Stone, M.C. (ed.) Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1987, Anaheim, California, USA, 27–31 July 1987, pp. 25–34. ACM (1987). https://doi.org/10.1145/37401.37406

  27. Richmond, P., Coakley, S., Romano, D.M.: A high performance agent based modelling framework on graphics card hardware with Cuda. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2, pp. 1125–1126. AAMAS 2009, International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2009)

    Google Scholar 

  28. Samvelyan, M., et al.: The starcraft multi-agent challenge (2019). https://doi.org/10.48550/ARXIV.1902.04043. https://arxiv.org/abs/1902.04043

  29. Samvelyan, M., et al.: The StarCraft Multi-Agent Challenge. CoRR abs/1902.04043 (2019)

    Google Scholar 

  30. Suarez, J., Du, Y., Isola, P., Mordatch, I.: Neural MMO: a massively multiagent game environment for training and evaluating intelligent agents (2019). https://doi.org/10.48550/ARXIV.1903.00784. https://arxiv.org/abs/1903.00784

  31. Terry, J., et al.: PettingZoo: Gym for multi-agent reinforcement learning. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 15032–15043. Curran Associates, Inc. (2021). https://proceedings.neurips.cc/paper/2021/file/7ed2d3454c5eea71148b11d0c25104ff-Paper.pdf

  32. Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3-4), 279–292 (1992). https://doi.org/10.1007/bf00992698

  33. Yang, Y.: Many-agent reinforcement learning, Ph. D. thesis, UCL (University College London) (2021)

    Google Scholar 

  34. Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., Wang, J.: Mean field multi-agent reinforcement learning (2018). https://doi.org/10.48550/ARXIV.1802.05438. https://arxiv.org/abs/1802.05438

  35. Yu, C., et al.: The surprising effectiveness of PPO in cooperative, multi-agent games (2021). https://doi.org/10.48550/ARXIV.2103.01955. https://arxiv.org/abs/2103.01955

  36. Šošić, A., KhudaBukhsh, W.R., Zoubir, A.M., Koeppl, H.: Inverse reinforcement learning in swarm systems (2016). https://doi.org/10.48550/ARXIV.1602.05450. https://arxiv.org/abs/1602.05450

Download references

Author information

Authors and Affiliations

  1. Alma Mater Studiorum – Università di Bologna, Cesena, Italy

    Davide Domini, Filippo Cavallari, Gianluca Aguzzi & Mirko Viroli

Authors
  1. Davide Domini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Filippo Cavallari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gianluca Aguzzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mirko Viroli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gianluca Aguzzi .

Editor information

Editors and Affiliations

  1. Open University of The Netherlands, Amsterdam, The Netherlands

    Sung-Shik Jongmans

  2. Universidade de Lisboa, Lisbon, Portugal

    Antónia Lopes

Rights and permissions

Reprints and permissions

Copyright information

© 2023 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Domini, D., Cavallari, F., Aguzzi, G., Viroli, M. (2023). ScaRLib: A Framework for Cooperative Many Agent Deep Reinforcement Learning in Scala. In: Jongmans, SS., Lopes, A. (eds) Coordination Models and Languages. COORDINATION 2023. Lecture Notes in Computer Science, vol 13908. Springer, Cham. https://doi.org/10.1007/978-3-031-35361-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35361-1_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35360-4

  • Online ISBN: 978-3-031-35361-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation