Skip to main content
SpringerLink
Log in

Self-learning Governance of Black-Box Multi-Agent Systems

  • Conference paper
  • First Online:
Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XV (COINE 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13549))

  • 275 Accesses

Abstract

Agents in Multi-Agent Systems (MAS) are not always built and controlled by the system designer, e.g., on electronic trading platforms. In this case, there is often a system objective which can differ from the agents’ own goals (e.g., price stability). While much effort has been put into modeling and optimizing agent behavior, we are concerned in this paper with the platform perspective. Our model extends Stochastic Games (SG) with dynamic restriction of action spaces to a new self-learning governance approach for black-box MAS. This governance learns an optimal restriction policy via Reinforcement Learning.

As an alternative to the two straight-forward approaches—fully centralized control and fully independent learners—, this novel method combines a sufficient degree of autonomy for the agents with selective restriction of their action spaces. We demonstrate that the governance, though not explicitly instructed to leave any freedom of decision to the agents, learns that combining the agents’ and its own capabilities is better than controlling all actions. As shown experimentally, the self-learning approach outperforms (w.r.t. the system objective) both “full control” where actions are always dictated without any agent autonomy, and “ungoverned MAS” where the agents simply pursue their individual goals.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abdunabi, T., Basir, O.: Holonic intelligent multi-agent algorithmic trading system (HIMAATS). Int. J. Comput. Appl. 21, 54–61 (2014)

    Google Scholar 

  2. Aires, J.P., Meneguzzi, F.: Norm conflict identification using deep learning. In: AAMAS Workshops (2017)

    Google Scholar 

  3. Aldewereld, H., Boissier, O., Dignum, V., Noriega, P., Padget, J. (eds.): Social Coordination Frameworks for Social Technical Systems. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33570-4

    Book  Google Scholar 

  4. Arcos, J.L., Esteva, M., Noriega, P., Rodríguez-Aguilar, J.A., Sierra, C.: Environment engineering for multiagent systems. In: Engineering Applications of Artificial Intelligence (2004)

    Google Scholar 

  5. Arcos, J.L., Rodríguez-Aguilar, J.A., Rosell, B.: Engineering autonomic electronic institutions. In: Weyns, D., Brueckner, S.A., Demazeau, Y. (eds.) EEMMAS 2007. LNCS (LNAI), vol. 5049, pp. 76–87. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85029-8_6

    Chapter  Google Scholar 

  6. Balke, T., et al.: Norms in MAS: definitions and related concepts, p. 31. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2013)

    Google Scholar 

  7. Barbuceanu, M.: Coordinating agents by role based social constraints and conversation plans. In: AAAI/IAAI (1997)

    Google Scholar 

  8. Boella, G., van der Torre, L., Verhagen, H.: Introduction to normative multiagent systems. Comput. Math. Organ. Theory 12(2), 71–79 (2006). https://doi.org/10.1007/s10588-006-9537-7

    Article  Google Scholar 

  9. Boella, G., van der Torre, L., Verhagen, H.: Introduction to the special issue on normative multiagent systems. Auton. Agents Multi-Agent Syst. 17(1), 1–10 (2008). https://doi.org/10.1007/s10458-008-9047-8

    Article  Google Scholar 

  10. Bou, E., López-Sánchez, M., Rodríguez-Aguilar, J.A.: Towards self-configuration in autonomic electronic institutions. In: Noriega, P., et al. (eds.) COIN 2006. LNCS (LNAI), vol. 4386, pp. 229–244. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74459-7_15

    Chapter  Google Scholar 

  11. Cacciamani, F., Celli, A., Ciccone, M., Gatti, N.: Multi-agent coordination in adversarial environments through signal mediated strategies. In: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2021)

    Google Scholar 

  12. Cheung, W.C., Simchi-Levi, D., Zhu, R.: Reinforcement learning for non-stationary markov decision processes: the blessing of (more) optimism. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, Virtual Event, 13–18 July 2020 (2020)

    Google Scholar 

  13. Conitzer, V., Sandholm, T.: AWESOME: a general multiagent learning algorithm that converges in self-play and learns a best response against stationary opponents. Mach. Learn. 67, 23–43 (2003). https://doi.org/10.1007/s10994-006-0143-1

    Article  MATH  Google Scholar 

  14. Conte, R., Falcone, R., Sartor, G.: Introduction: agents and norms: how to fill the gap? Artif. Intell. Law 7(1), 1–15 (1999). https://doi.org/10.1023/A:1008397328506

    Article  Google Scholar 

  15. Esteva, M., et al.: Electronic institutions development environment. In: AAMAS Demo Proceedings, vol. 3. International Foundation for Autonomous Agents and Multiagent Systems (2008)

    Google Scholar 

  16. Esteva, M., Rodríguez-Aguilar, J.-A., Sierra, C., Garcia, P., Arcos, J.L.: On the formal specification of electronic institutions. In: Dignum, F., Sierra, C. (eds.) Agent Mediated Electronic Commerce. LNCS (LNAI), vol. 1991, pp. 126–147. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44682-6_8

    Chapter  Google Scholar 

  17. Frantz, C., Pigozzi, G.: Modelling norm dynamics in multi-agent systems. J. Appl. Logic 5, 491–564 (2018)

    MATH  Google Scholar 

  18. Gomez-Sanz, J.J.: Ingenias. In: Aldewereld, H., Boissier, O., Dignum, V., Noriega, P., Padget, J. (eds.) Social Coordination Frameworks for Social Technical Systems. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33570-4_5

    Chapter  Google Scholar 

  19. Greenwald, A., Hall, K.: Correlated-Q learning. In: Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML2003. AAAI Press (2003)

    Google Scholar 

  20. Gronauer, S., Diepold, K.: Multi-agent deep reinforcement learning: a survey. Artif. Intell. Rev. 55, 895–943 (2021). https://doi.org/10.1007/s10462-021-09996-w

    Article  Google Scholar 

  21. Hwang, K., Jiang, W., Chen, Y.: Model learning and knowledge sharing for a multiagent system with Dyna-Q learning. IEEE Trans. Cybern. 45(5), 978–990 (2015)

    Article  Google Scholar 

  22. Liang, E., et al.: RLlib: abstractions for distributed reinforcement learning. In: ICML (2018)

    Google Scholar 

  23. Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: Proceedings of the Eleventh International Conference on International Conference on Machine Learning, ICML 1994. Morgan Kaufmann Publishers Inc., San Francisco (1994)

    Google Scholar 

  24. Lopes Cardoso, H., Urbano, J., Rocha, A., Castro, A.J.M., Oliveira, E.: ANTE: a framework integrating negotiation, norms and trust. In: ldewereld, H., Boissier, O., Dignum, V., Noriega, P., Padget, J. (eds.) Social Coordination Frameworks for Social Technical Systems, vol. 30. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33570-4_3

  25. Marín-Lora, C., Chover, M., Sotoca, J.M., García, L.A.: A game engine to make games as multi-agent systems. Adv. Eng. Softw. 140, 02732 (2020)

    Article  Google Scholar 

  26. McGroarty, F., Booth, A., Gerding, E., Chinthalapati, V.L.R.: High frequency trading strategies, market fragility and price spikes: an agent based model perspective. Ann. Oper. Res. 282(1), 217–244 (2019). https://doi.org/10.1007/s10479-018-3019-4

    Article  MathSciNet  MATH  Google Scholar 

  27. Mellema, R., Jensen, M., Dignum, F.: Social rules for agent systems. In: Aler Tubella, A., Cranefield, S., Frantz, C., Meneguzzi, F., Vasconcelos, W. (eds.) COIN/COINE 2017/2020. LNCS (LNAI), vol. 12298, pp. 175–180. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72376-7_10

    Chapter  Google Scholar 

  28. Morales, J.: On-line norm synthesis for open Multi-Agent systems. Ph.D. thesis, Universitat de Barcelona (2016)

    Google Scholar 

  29. Neufeld, E., Bartocci, E., Ciabattoni, A., Governatori, G.: A normative supervisor for reinforcement learning agents. In: Platzer, A., Sutcliffe, G. (eds.) CADE 2021. LNCS (LNAI), vol. 12699, pp. 565–576. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-79876-5_32

    Chapter  Google Scholar 

  30. Noriega, P.: Agent-mediated auctions: the fishmarket metaphor. Ph.D. thesis, Universitat Autonoma de Barcelona (1997)

    Google Scholar 

  31. Noriega, P., de Jonge, D.: Electronic institutions: the EI/EIDE framework. In: Aldewereld, H., Boissier, O., Dignum, V., Noriega, P., Padget, J. (eds.) Social Coordination Frameworks for Social Technical Systems. LGTS, vol. 30, pp. 47–76. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33570-4_4

    Chapter  Google Scholar 

  32. Nowé, A., Vrancx, P., De Hauwere, Y.M.: Game theory and multi-agent reinforcement learning. In: Wiering, M., van Otterlo, M. (eds.) Reinforcement Learning: State-of-the-Art. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27645-3_14

    Chapter  MATH  Google Scholar 

  33. Padakandla, S., K. J., P., Bhatnagar, S.: Reinforcement learning algorithm for non-stationary environments. Appl. Intell. 50(11), 3590–3606 (2020). https://doi.org/10.1007/s10489-020-01758-5

  34. Riad, M., Golpayegani, F.: Run-time norms synthesis in multi-objective multi-agent systems. In: Theodorou, A., Nieves, J.C., De Vos, M. (eds.) COINE 2021. LNCS, vol. 13239, pp. 78–93. Springer, Cham (2021). https://doi.org/10.1007/978-3-031-16617-4_6

    Chapter  Google Scholar 

  35. Rizk, Y., Awad, M., Tunstel, E.: Decision making in multi-agent systems: a survey. IEEE Trans. Cogn. Dev. Syst. 10, 514–529 (2018)

    Article  Google Scholar 

  36. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017)

    Google Scholar 

  37. Shoham, Y., Tennenholtz, M.: On social laws for artificial agent societies: off-line design. Artif. Intell. 73(1), 231–252 (1995)

    Article  Google Scholar 

  38. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. A Bradford Book, Cambridge (2018)

    MATH  Google Scholar 

  39. Wang, X., Sandholm, T.: Reinforcement learning to play an optimal nash equilibrium in team Markov games. In: NIPS (2002)

    Google Scholar 

  40. Weyns, D., Brückner, S., Demazeau, Y.: Engineering Environment-Mediated Multi-Agent Systems. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-85029-8

    Book  Google Scholar 

  41. Zaib, M., Sheng, Q.Z., Zhang, W.E.: A short survey of pre-trained language models for conversational AI-A NewAge in NLP (2021)

    Google Scholar 

  42. Zhang, K., Yang, Z., Başar, T.: Multi-agent reinforcement learning: a selective overview of theories and algorithms. In: Vamvoudakis, K.G., Wan, Y., Lewis, F.L., Cansever, D. (eds.) Handbook of Reinforcement Learning and Control. SSDC, vol. 325, pp. 321–384. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-60990-0_12

    Chapter  MATH  Google Scholar 

  43. Zinkevich, M., Greenwald, A., Littman, M.L.: Cyclic equilibria in Markov games. In: Proceedings of the 18th International Conference on Neural Information Processing Systems, NIPS 2005. MIT Press, Cambridge (2005)

    Google Scholar 

Download references

Acknowledgements

This work is supported by the German Federal Ministry for Economic Affairs and Energy (BMWi).

Author information

Authors and Affiliations

  1. Institute for Enterprise Systems (InES), University of Mannheim, Mannheim, Germany

    Michael Oesterle, Christian Bartelt & Stefan Lüdtke

  2. University of Mannheim, Mannheim, Germany

    Heiner Stuckenschmidt

Authors
  1. Michael Oesterle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Bartelt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Lüdtke
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Heiner Stuckenschmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Oesterle .

Editor information

Editors and Affiliations

  1. University of Bristol, Bristol, UK

    Nirav Ajmeri

  2. University of Bath, Bath, UK

    Andreasa Morris Martin

  3. University of Otago, Dunedin, New Zealand

    Bastin Tony Roy Savarimuthu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Oesterle, M., Bartelt, C., Lüdtke, S., Stuckenschmidt, H. (2022). Self-learning Governance of Black-Box Multi-Agent Systems. In: Ajmeri, N., Morris Martin, A., Savarimuthu, B.T.R. (eds) Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XV. COINE 2022. Lecture Notes in Computer Science(), vol 13549. Springer, Cham. https://doi.org/10.1007/978-3-031-20845-4_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20845-4_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20844-7

  • Online ISBN: 978-3-031-20845-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation