Skip to main content
Springer Nature Link
Log in

Emergent Dominance Hierarchies in Reinforcement Learning Agents

  • Conference paper
  • First Online:
Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XVII (COINE 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 15398))

  • 52 Accesses

Abstract

Modern Reinforcement Learning (RL) algorithms are able to outperform humans in a wide variety of tasks. Multi-agent reinforcement learning (MARL) settings present additional challenges, and successful cooperation in mixed-motive groups of agents depends on a delicate balancing act between individual and group objectives. Social conventions and norms, often inspired by human institutions, are used as tools for striking this balance.

We examine a fundamental, well-studied social convention that underlies cooperation in animal and human societies: dominance hierarchies.

We adapt the ethological theory of dominance hierarchies to artificial agents, borrowing the established terminology and definitions with as few amendments as possible. We demonstrate that populations of RL agents, operating without explicit programming or intrinsic rewards, can invent, learn, enforce, and transmit a dominance hierarchy to new populations. The dominance hierarchies that emerge have a similar structure to those studied in chickens, mice, fish, and other species.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The code and its manual are available at https://github.com/cool-RR/chicken-coop.

References

  1. Aroca-Ouellette, S., Aroca-Ouellette, M., Biswas, U., Kann, K., Roncone, A.: Hierarchical reinforcement learning for ad hoc teaming. In: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 2337–2339 (2023)

    Google Scholar 

  2. Bakker, M.A., et al.: Modelling cooperation in network games with spatio-temporal complexity (2021)

    Google Scholar 

  3. Bonanni, R., Cafazzo, S., Abis, A., Barillari, E., Valsecchi, P., Natoli, E.: Age-graded dominance hierarchies and social tolerance in packs of free-ranging dogs. Behav. Ecol. 28(4), 1004–1020 (2017)

    Article  Google Scholar 

  4. Bowles, S.: Did warfare among ancestral hunter-gatherers affect the evolution of human social behaviors? Science 324(5932), 1293–1298 (2009)

    Article  MATH  Google Scholar 

  5. Boyd, R., Richerson, P.J.: Culture and the evolution of human cooperation. Philos. Trans. R. Soc. B: Biol. Sci. 364(1533), 3281–3288 (2009)

    Article  MATH  Google Scholar 

  6. CGI Team: Learning robust real-time cultural transmission without human data. arXiv preprint arXiv:2203.00715 (2022)

  7. Chase, I., Bartolomeo, C., Dugatkin, L.: Aggressive interactions and inter-contest interval: how long do winners keep winning? Anim. Behav. 48(2), 393–400 (1994)

    Article  Google Scholar 

  8. Chase, I., Coelho, D., Lee, W., Mueller, K., Curley, J.: Networks never rest: an investigation of network evolution in three species of animals. Soc. Netw. 68, 356–373 (2022)

    Article  MATH  Google Scholar 

  9. Chase, I., Tovey, C., Spangler-Martin, D., Manfredonia, M.: Individual differences versus social dynamics in the formation of animal dominance hierarchies. Proc. Natl. Acad. Sci. 99(8), 5744–5749 (2002)

    Article  Google Scholar 

  10. Chen Zeng, T., Cheng, J.T., Henrich, J.: Dominance in humans. Philos. Trans. R. Soc. B 377(1845), 20200451 (2022)

    Article  MATH  Google Scholar 

  11. Couzin, I.D.: Collective cognition in animal groups. Trends Cogn. Sci. 13(1), 36–43 (2009)

    Article  MATH  Google Scholar 

  12. Dafoe, A., et al.: Open problems in cooperative AI (2020)

    Google Scholar 

  13. David, B.O., Stoffels, R.J.: Spatial organisation and behavioural interaction of giant kokopu (galaxias argenteus) in two stream pools differing in fish density. New Zealand Journal of Marine and Freshwater Research (2003)

    Google Scholar 

  14. De Vries, H.: An improved test of linearity in dominance hierarchies containing unknown or tied relationships. Anim. Behav. 50(5), 1375–1389 (1995)

    Article  MATH  Google Scholar 

  15. De Vries, H.: Finding a dominance order most consistent with a linear hierarchy: a new procedure and review. Anim. Behav. 55(4), 827–843 (1998)

    Article  MATH  Google Scholar 

  16. Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I.: Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325 (2023)

  17. Duéñez-Guzmán, E.A., et al.: Statistical discrimination in learning agents. arXiv preprint arXiv:2110.11404 (2021)

  18. Dugatkin, L.A.: Winner and loser effects and the structure of dominance hierarchies. Behav. Ecol. 8(6), 583–587 (1997)

    Article  MATH  Google Scholar 

  19. Dugatkin, L.A.: Bystander effects and the structure of dominance hierarchies. Behav. Ecol. 12(3), 348–352 (2001)

    Article  MATH  Google Scholar 

  20. Essler, J., Cafazzo, S., Marshall-Pescini, S., Virányi, Z., Kotrschal, K., Range, F.: Play behavior in wolves: Using the ‘50:50’ rule to test for egalitarian play styles. PLoS ONE 11(5), e0154150 (2016)

    Article  Google Scholar 

  21. Foerster, J., Chen, R.Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., Mordatch, I.: Learning with opponent-learning awareness. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 122–130, AAMAS ’18, International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2018)

    Google Scholar 

  22. Gavrilets, S., Duéñez-Guzmán, E.A., Vose, M.D.: Dynamics of alliance formation and the egalitarian revolution. PLoS ONE 3(10), e3293 (2008)

    Article  MATH  Google Scholar 

  23. Gordon, D.M.: The organization of work in social insect colonies. Nature 380(6570), 121–124 (1996)

    Article  MATH  Google Scholar 

  24. Halevy, N., Chou, E.Y., Cohen, T.R., Livingston, R.W.: Status conferral in intergroup social dilemmas: behavioral antecedents and consequences of prestige and dominance. J. Pers. Soc. Psychol. 102(2), 351 (2012)

    Article  Google Scholar 

  25. Hansen, E.A., Bernstein, D.S., Zilberstein, S.: Dynamic programming for partially observable stochastic games. In: AAAI, vol. 4, pp. 709–715 (2004)

    Google Scholar 

  26. Hemelrijk, C.: Dominance interactions, spatial dynamics and emergent reciprocity in a virtual world. In: The 4th International Conference on Simulation of Adaptive Behavior, vol. 4, pp. 545–552, Citeseer (1996)

    Google Scholar 

  27. Hemelrijk, C.: Social phenomena emerging by self-organisation in a competitive, virtual world (“domworld”). In: Learning to behave Workshop II: Internalising knowledge. Ieper, Belgium, pp. 11–19 (2000)

    Google Scholar 

  28. Henazi, S.P., Barrett, L.: The value of grooming to female primates. Primates 40, 47–59 (1999)

    Article  MATH  Google Scholar 

  29. Iverson, G.J., Sade, D.S.: Statistical issues in the analysis of dominance hierarchies in animal societies. J. Quant. Anthropol. 2(1), 61–83 (1990)

    MATH  Google Scholar 

  30. Kraus, S.: Negotiation and cooperation in multi-agent environments. Artif. Intell. 94(1), 79–97 (1997). ISSN 0004-3702, https://doi.org/10.1016/S0004-3702(97)00025-8, economic Principles of Multi-Agent Systems

  31. Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., Graepel, T.: Multi-agent reinforcement learning in sequential social dilemmas. In: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pp. 464–473, AAMAS ’17, International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2017)

    Google Scholar 

  32. Leimar, O.: The evolution of social dominance through reinforcement learning. Am. Nat. 197(5), 560–575 (2021)

    Article  MATH  Google Scholar 

  33. Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., Gao, J.: Multimodal foundation models: From specialists to general-purpose assistants (2023)

    Google Scholar 

  34. Liang, E., et al.: RLlib: Abstractions for distributed reinforcement learning (2018)

    Google Scholar 

  35. Liang, T., et al.: Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118 (2023)

  36. Lu, C., Willi, T., De Witt, C.A.S., Foerster, J.: Model-free opponent shaping. In: International Conference on Machine Learning, pp. 14398–14411, PMLR (2022)

    Google Scholar 

  37. Magee, J.C., Galinsky, A.D.: Social hierarchy: the self-reinforcing nature of power and status. Acad. Manag. Ann. 2, 351–398 (2008)

    Article  MATH  Google Scholar 

  38. Mordatch, I., Abbeel, P.: Emergence of grounded compositional language in multi-agent populations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  39. Muthukrishna, M., Henrich, J.: Innovation in the collective brain. Ph. Transactions of the Royal Society B: Biological Sciences 371(1690) (2016)

    Google Scholar 

  40. Przepiorka, W., Rutten, C., Buskens, V., Szekely, A.: How dominance hierarchies emerge from conflict: a game theoretic model and experimental evidence. Soc. Sci. Res. 86, 102393 (2020)

    Article  Google Scholar 

  41. Radke, D., Larson, K., Brecht, T., Tilbury, K.: Towards a better understanding of learning with multiagent teams. In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 271–279, ijcai.org (2023). https://doi.org/10.24963/ijcai.2023/31

  42. Ramchurn, S.D., Mezzetti, C., Giovannucci, A., Rodriguez-Aguilar, J.A., Dash, R.K., Jennings, N.R.: Trust-based mechanisms for robust and efficient task allocation in the presence of execution uncertainty. J. Artif. Intell. Res. 35, 119–159 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  43. Rapoport, A.: Outline of a probabilistic approach to animal sociology: I. Bull. Math. Biophys. 11, 183–196 (1949)

    Article  MATH  Google Scholar 

  44. Rapoport, A., Chammah, A.M.: The game of chicken. Am. Behav. Sci. 10(3), 10–28 (1966)

    Article  MATH  Google Scholar 

  45. Reeves, B., Nass, C.: The media equation: How people treat computers, television, and new media like real people. Cambridge, UK 10(10) (1996)

    Google Scholar 

  46. Samuels, A., Silk, J.B., Altmann, J.: Continuity and change in dominance relations among female baboons. Anim. Behav. 35(3), 785–793 (1987)

    Article  MATH  Google Scholar 

  47. Sanders, N.J., Gordon, D.M.: Resource-dependent interactions and the organization of desert ant communities. Ecology 84(4), 1024–1031 (2003)

    Article  MATH  Google Scholar 

  48. Schjelderup-Ebbe, T.: Beiträge zur sozialpsychologie des haushuhns. Zeitschrift für Psychologie und Physiologie der Sinnesorgane. Abt. 1. Zeitschrift für Psychologie 88, 225 (1922)

    Google Scholar 

  49. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

  50. Smith, J.M., Price, G.R.: The logic of animal conflict. Nature 246(5427), 15–18 (1973)

    Article  MATH  Google Scholar 

  51. Strauss, E.D., Shizuka, D.: The dynamics of dominance: open questions, challenges and solutions. Philos. Trans. R. Soc. B 377(1845), 20200445 (2022)

    Article  MATH  Google Scholar 

  52. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA (2018) 0262039249

    Google Scholar 

  53. Tomlinson, B., Blumberg, B., Rhodes, B.: How is an agent like a wolf?: Dominance and submission in multi-agent systems. In: International ICSC Symposium on Multi-Agents and Mobile Agents in Virtual Organizations and E-Commerce (MAMA’2000) (2000)

    Google Scholar 

  54. Tufano, M., Agarwal, A., Jang, J., Moghaddam, R.Z., Sundaresan, N.: AutoDev: Automated AI-driven development (2024)

    Google Scholar 

  55. Vezhnevets, A.S., et al.: Generative agent-based modeling with actions grounded in physical, social, or digital space using concordia (2023)

    Google Scholar 

  56. Vinitsky, E., et al.: A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings. Collective Intell. 2(2) (2023). https://doi.org/10.1177/26339137231162025

  57. Watson, J.R.: Dominance-subordination in caged groups of house sparrows. The Wilson Bulletin, pp. 268–278 (1970)

    Google Scholar 

  58. Williamson, C.M., Lee, W., Romeo, R.D., Curley, J.P.: Social context-dependent relationships between mouse dominance rank and plasma hormone levels. Physiol. Behavior 171, 110–119 (2017)

    Article  MATH  Google Scholar 

  59. Woolley, A.W., Chabris, C.F., Pentland, A., Hashmi, N., Malone, T.W.: Evidence for a collective intelligence factor in the performance of human groups. Science 330(6004), 686–688 (2010)

    Article  Google Scholar 

  60. Wu, Q., et al.: AutoGen: Enabling next-gen LLM applications via multi-agent conversation (2023)

    Google Scholar 

  61. Xi, Z., et al.: The rise and potential of large language model based agents: A survey (2023)

    Google Scholar 

  62. Xu, Y., Deng, Z., Wang, M., Xu, W., So, A.M.C., Cui, S.: Voting-based multi-agent reinforcement learning for intelligent IoT (2020)

    Google Scholar 

  63. Zhan, S., Bendapudi, N., Hong, Y.y.: Re-examining diversity as a double-edged sword for innovation process. J. Organ. Behav. 36(7), 1026–1049 (2015)

    Google Scholar 

Download references

Acknowledgments

We thank the following colleagues for their advice and support in writing this paper: David Aha, Ivan Chase, Edgar Duéñez-Guzmán, Errol King, Joel Leibo, Olof Leimar, David Manheim, Markov, Georg Ostrovski, Jérémy Perret, Saul Pwanson, Venkatesh Rao and Eli Strauss. This research was funded by The Association For Long Term Existence And Resilience (ALTER).

Author information

Authors and Affiliations

  1. Bar-Ilan University, Ramat Gan, Israel

    Ram Rachum & Reuth Mirsky

  2. Tel Aviv University, Tel Aviv, Israel

    Yonatan Nakar

  3. University of California, Irvine, USA

    Bill Tomlinson

  4. Max Planck Institute for Biological Cybernetics, Tübingen, Germany

    Nitay Alon

  5. The Hebrew University of Jerusalem, Jerusalem, Israel

    Nitay Alon

Authors
  1. Ram Rachum
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yonatan Nakar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bill Tomlinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nitay Alon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Reuth Mirsky
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ram Rachum .

Editor information

Editors and Affiliations

  1. University of Otago, Dunedin, New Zealand

    Stephen Cranefield

  2. IMT Mines Saint-Etienne, Saint‐Étienne, France

    Luis Gustavo Nardin

  3. Ontario Tech University, Ontario, ON, Canada

    Nathan Lloyd

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rachum, R., Nakar, Y., Tomlinson, B., Alon, N., Mirsky, R. (2025). Emergent Dominance Hierarchies in Reinforcement Learning Agents. In: Cranefield, S., Nardin, L.G., Lloyd, N. (eds) Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XVII. COINE 2024. Lecture Notes in Computer Science(), vol 15398. Springer, Cham. https://doi.org/10.1007/978-3-031-82039-7_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-82039-7_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-82038-0

  • Online ISBN: 978-3-031-82039-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics