Abstract
Ad hoc teamwork has been introduced as a general challenge for AI and especially multiagent systems [16]. The goal is to enable autonomous agents to band together with previously unknown teammates towards a common goal: collaboration without pre-coordination. A long-term vision for ad hoc teamwork is to enable robots or other autonomous agents to exhibit the sort of flexibility and adaptability on complex tasks that people do, for example when they play games of “pick-up” basketball or soccer. As a testbed for ad hoc teamwork, autonomous robots have played in pick-up soccer games, called “drop-in player challenges”, at the international RoboCup competition. An open question is how best to evaluate ad hoc teamwork performance—how well agents are able to coordinate and collaborate with unknown teammates—of agents with different skill levels and abilities competing in drop-in player challenges. This paper presents new metrics for assessing ad hoc teamwork performance, specifically attempting to isolate an agent’s coordination and teamwork from its skill level, during drop-in player challenges. Additionally, the paper considers how to account for only a relatively small number of pick-up games being played when evaluating drop-in player challenge participants.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
Full rules of the challenges can be found at http://www.cs.utexas.edu/~AustinVilla/sim/3dsimulation/2015_dropin_challenge/.
- 5.
Empirically we have found that the average goal difference when one team plays itself approaches 0 across many games.
References
Barrett, S., Stone, P.: An analysis framework for ad hoc teamwork tasks. In: AAMAS 2012, June 2012
Barrett, S., Stone, P., Kraus, S.: Empirical evaluation of ad hoc teamwork in the pursuit domain. In: AAMAS 2011, May 2011
Bowling, M., McCracken, P.: Coordination and adaptation in impromptu teams. In: AAAI (2005)
Elo, A.: The Rating of Chess Players, Past and Present. Arco Publishing, New York (1978)
Genter, K., Laue, T., Stone, P.: Benchmarking robot cooperation without pre-coordination in the robocup standard platform league drop-in player competition. In: Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2015), September 2015
Genter, K., Laue, T., Stone, P.: Three years of the robocup standard platform league drop-in player competition: creating and maintaining a large scale ad hoc teamwork robotics competition. Auton. Agents Multi-Agent Syst. (JAAMAS) 31(4), 790–820 (2017). Springer
Grosz, B., Kraus, S.: Collaborative plans for complex group actions. Artif. Intell. 86, 269–368 (1996)
Herbrich, R., Minka, T., Graepel, T.: Trueskill™: a bayesian skill rating system. In: Proceedings of the 19th International Conference on Neural Information Processing Systems, pp. 569–576. MIT Press (2006)
Horling, B., Lesser, V., Vincent, R., Wagner, T., Raja, A., Zhang, S., Decker, K., Garvey, A.: The TAEMS White Paper, January 1999
Jones, E., Browning, B., Dias, M.B., Argall, B., Veloso, M.M., Stentz, A.T.: Dynamically formed heterogeneous robot teams performing tightly-coordinated tasks. In: ICRA, pp. 570–575, May 2006
Liemhetcharat, S., Luo, Y.: Applying the synergy graph model to human basketball. In: Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 1695–1696 (2015)
Liemhetcharat, S., Veloso, M.: Modeling mutual capabilities in heterogeneous teams for role assignment. In: IROS 2011, pp. 3638–3644 (2011)
MacAlpine, P., Depinet, M., Liang, J., Stone, P.: UT Austin Villa: RoboCup 2014 3D simulation league competition and technical challenge champions. In: Bianchi, R.A.C., Akin, H.L., Ramamoorthy, S., Sugiura, K. (eds.) RoboCup 2014. LNCS (LNAI), vol. 8992, pp. 33–46. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18615-3_3
MacAlpine, P., Genter, K., Barrett, S., Stone, P.: The RoboCup 2013 drop-in player challenges: experiments in ad hoc teamwork. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September 2014
MacAlpine, P., Hanna, J., Liang, J., Stone, P.: UT Austin Villa: RoboCup 2015 3D simulation league competition and technical challenges champions. In: Almeida, L., Ji, J., Steinbauer, G., Luke, S. (eds.) RoboCup 2015. LNCS (LNAI), vol. 9513, pp. 118–131. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-29339-4_10
Stone, P., Kaminka, G.A., Kraus, S., Rosenschein, J.S.: Ad hoc autonomous agent teams: collaboration without pre-coordination. In: AAAI 2010, July 2010
Tambe, M.: Towards flexible teamwork. J. Artif. Intell. Res. 7, 81–124 (1997)
Wu, F., Zilberstein, S., Chen, X.: Online planning for ad hoc autonomous agent teams. In: IJCAI (2011)
Acknowledgments
This work has taken place in the Learning Agents Research Group (LARG) at UT Austin. LARG research is supported in part by NSF (CNS-1330072, CNS-1305287, IIS-1637736, IIS-1651089), ONR (21C184-01), AFOSR (FA9550-14-1-0087), Raytheon, Toyota, AT&T, and Lockheed Martin. Peter Stone serves on the Board of Directors of, Cogitai, Inc. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
MacAlpine, P., Stone, P. (2017). Evaluating Ad Hoc Teamwork Performance in Drop-In Player Challenges. In: Sukthankar, G., Rodriguez-Aguilar, J. (eds) Autonomous Agents and Multiagent Systems. AAMAS 2017. Lecture Notes in Computer Science(), vol 10642. Springer, Cham. https://doi.org/10.1007/978-3-319-71682-4_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-71682-4_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-71681-7
Online ISBN: 978-3-319-71682-4
eBook Packages: Computer ScienceComputer Science (R0)