Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Reinforcement Learning for Multi-Agent Stochastic Resource Collection

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13716))

  • 545 Accesses

Abstract

Stochastic Resource Collection (SRC) describes tasks where an agent tries to collect a maximal amount of dynamic resources while navigating through a road network. An instance of SRC is the traveling officer problem (TOP), where a parking officer tries to maximize the number of fined parking violations. In contrast to vehicular routing problems, in SRC tasks, resources might appear and disappear by an unknown stochastic process, and thus, the task is inherently more dynamic. In most applications of SRC, such as TOP, covering realistic scenarios requires more than one agent. However, directly applying multi-agent approaches to SRC yields challenges considering temporal abstractions and inter-agent coordination. In this paper, we propose a novel multi-agent reinforcement learning method for the task of Multi-Agent Stochastic Resource Collection (MASRC). To this end, we formalize MASRC as a Semi-Markov Game which allows the use of temporal abstraction and asynchronous actions by various agents. In addition, we propose a novel architecture trained with independent learning, which integrates the information about collaborating agents and allows us to take advantage of temporal abstractions. Our agents are evaluated on the multiple traveling officer problem, an instance of MASRC where multiple officers try to maximize the number of fined parking violations. Our simulation environment is based on real-world sensor data. Results demonstrate that our proposed agent can beat various state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    cf. https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html.

  2. 2.

    https://www.openstreetmap.org.

  3. 3.

    https://data.melbourne.vic.gov.au/browse?tags=parking.

  4. 4.

    https://github.com/niklasdbs/masrc.

  5. 5.

    Notice though that even with joint rewards, our approach is able to beat baselines trained with individual rewards.

References

  1. Bono, G., Dibangoye, J.S., Simonin, O., Matignon, L., Pereyron, F.: Solving multi-agent routing problems using deep attention mechanisms. IEEE Trans. Intell. Transp. Syst. 22(12), 7804–7813 (2020)

    Article  Google Scholar 

  2. Chakravorty, J., et al.: Option-critic in cooperative multi-agent systems. arXiv preprint arXiv:1911.12825 (2019)

  3. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual multi-agent policy gradients. In: AAAI, vol. 32 (2018)

    Google Scholar 

  4. Gendreau, M., Laporte, G., Séguin, R.: Stochastic vehicle routing. Eur. J. Oper. Res. 88(1), 3–12 (1996)

    Article  MATH  Google Scholar 

  5. Han, D., Böhmer, W., Wooldridge, M., Rogers, A.: Multi-agent hierarchical reinforcement learning with dynamic termination. In: Nayak, A.C., Sharma, A. (eds.) PRICAI 2019. LNCS (LNAI), vol. 11671, pp. 80–92. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29911-8_7

    Chapter  Google Scholar 

  6. Hernandez-Leal, P., Kartal, B., Taylor, M.E.: A survey and critique of multiagent deep reinforcement learning. Auton. Agents Multi-Agent Syst. 33(6), 750–797 (2019). https://doi.org/10.1007/s10458-019-09421-1

    Article  Google Scholar 

  7. Hu, J., Wellman, M.P., et al.: Multiagent reinforcement learning: theoretical framework and an algorithm. In: ICML, vol. 98, pp. 242–250. Citeseer (1998)

    Google Scholar 

  8. Kim, J., Kim, K.: Optimizing large-scale fleet management on a road network using multi-agent deep reinforcement learning with graph neural network. In: ITSC, pp. 990–995. IEEE (2021)

    Google Scholar 

  9. Kool, W., Van Hoof, H., Welling, M.: Attention, learn to solve routing problems! arXiv preprint arXiv:1803.08475 (2018)

  10. Koutsoupias, E., Papadimitriou, C.: Worst-case equilibria. In: Meinel, C., Tison, S. (eds.) STACS 1999. LNCS, vol. 1563, pp. 404–413. Springer, Heidelberg (1999). https://doi.org/10.1007/3-540-49116-3_38

    Chapter  Google Scholar 

  11. Kumar, S.N., Panneerselvam, R.: A survey on the vehicle routing problem and its variants (2012)

    Google Scholar 

  12. Li, M., et al.: Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning. In: The world wide web conference, pp. 983–994 (2019)

    Google Scholar 

  13. Liu, Z., Li, J., Wu, K.: Context-aware taxi dispatching at city-scale using deep reinforcement learning. IEEE Trans. Intell. Transp. Syst. 99, 1–14 (2020)

    Google Scholar 

  14. Makar, R., Mahadevan, S., Ghavamzadeh, M.: Hierarchical multi-agent reinforcement learning. In: Proceedings of the fifth International Conference on Autonomous agents, pp. 246–253 (2001)

    Google Scholar 

  15. Mnih, V., et al.: Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

  16. Nazari, M., Oroojlooy, A., Snyder, L., Takác, M.: Playing Atari with deep reinforcement learning. In: Advance Neural Information Processing System, vol. 31 (2018)

    Google Scholar 

  17. Peng, B., Wang, J., Zhang, Z.: A deep reinforcement learning algorithm using dynamic attention model for vehicle routing problems. In: Li, K., Li, W., Wang, H., Liu, Y. (eds.) ISICA 2019. CCIS, vol. 1205, pp. 636–650. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-5577-0_51

    Chapter  Google Scholar 

  18. Qin, K.K., Shao, W., Ren, Y., Chan, J., Salim, F.D.: Solving multiple travelling officers problem with population-based optimization algorithms. Neural Comput. Appl. 32(16), 12033–12059 (2020)

    Article  Google Scholar 

  19. Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., Whiteson, S.: QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In: ICML, pp. 4295–4304. PMLR (2018)

    Google Scholar 

  20. Rohanimanesh, K., Mahadevan, S.: Learning to take concurrent actions. In: Advance Neural Information Processing System, vol. 15 (2002)

    Google Scholar 

  21. Schmoll, S., Schubert, M.: Semi-markov reinforcement learning for stochastic resource collection. In: IJCAI, pp. 3349–3355 (2021)

    Google Scholar 

  22. Shao, W., Salim, F.D., Gu, T., Dinh, N.T., Chan, J.: Traveling officer problem: managing car parking violations efficiently using sensor data. IEEE Internet Things J. 5(2), 802–810 (2017)

    Article  Google Scholar 

  23. Sukhbaatar, S., Fergus, R., et al.: Learning multiagent communication with backpropagation. In: Advance Neural Information Processing System, vol. 29 (2016)

    Google Scholar 

  24. Sunehag, P., et al.: Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 (2017)

  25. Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., Vicente, R.: Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE 12(4), e0172395 (2017)

    Article  Google Scholar 

  26. Tan, M.: Multi-agent reinforcement learning: Independent vs. cooperative agents. In: ICML, pp. 330–337 (1993)

    Google Scholar 

  27. Tang, H., et al.: Hierarchical deep multiagent reinforcement learning with temporal abstraction. arXiv preprint arXiv:1809.09332 (2018)

  28. Tang, X., et al.: A deep value-network based approach for multi-driver order dispatching. In: Proceedings of the 25th ACM SIGKDD, pp. 1780–1790 (2019)

    Google Scholar 

  29. Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: AAAI, vol. 30 (2016)

    Google Scholar 

  30. Vaswani, A., et al.: Attention is all you need. In: Advance Neural Information Processing System, vol. 30 (2017)

    Google Scholar 

  31. Zheng, L., et al.: Magent: a many-agent reinforcement learning platform for artificial collective intelligence. In: AAAI, vol. 32 (2018)

    Google Scholar 

  32. Zhou, M., et al.: Multi-agent reinforcement learning for order-dispatching via order-vehicle distribution matching. In: Proceedings of the 28th ACM Int’l Conf on Information and Knowledge Management, pp. 2645–2653 (2019)

    Google Scholar 

Download references

Acknowledgments

We thank the City of Melbourne, Australia, for providing the parking datasets used in this paper and Oliver Schrüfer for contributing the implementation of LERK. This work has been funded by the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. The authors of this work take full responsibilities for its content.

Author information

Authors and Affiliations

  1. LMU Munich, Munich, Germany

    Niklas Strauss, David Winkel, Max Berrendorf & Matthias Schubert

Authors
  1. Niklas Strauss
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Winkel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Max Berrendorf
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matthias Schubert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Niklas Strauss .

Editor information

Editors and Affiliations

  1. Grenoble Alpes University, Saint Martin d’Hères, France

    Massih-Reza Amini

  2. INSA Rouen Normandy, Saint Etienne du Rouvray, France

    Stéphane Canu

  3. Ruhr-Universität Bochum, Bochum, Germany

    Asja Fischer

  4. KU Leuven, Leuven, Belgium

    Tias Guns

  5. Central European University, Vienna, Austria

    Petra Kralj Novak

  6. Aristotle University of Thessaloniki, Thessaloniki, Greece

    Grigorios Tsoumakas

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 19 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Strauss, N., Winkel, D., Berrendorf, M., Schubert, M. (2023). Reinforcement Learning for Multi-Agent Stochastic Resource Collection. In: Amini, MR., Canu, S., Fischer, A., Guns, T., Kralj Novak, P., Tsoumakas, G. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2022. Lecture Notes in Computer Science(), vol 13716. Springer, Cham. https://doi.org/10.1007/978-3-031-26412-2_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-26412-2_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-26411-5

  • Online ISBN: 978-3-031-26412-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation