1 Introduction

UT Austin Villa won the 2016 RoboCup 3D Simulation League for the fifth time in the past six years, having also won the competition in 2011 [1], 2012 [2], 2014 [3], and 2015 [4] while finishing second in 2013. During the course of the competition the team scored 88 goals and only conceded 1 along the way to winning all 14 games the team played. Many of the components of the 2016 UT Austin Villa agent were reused from the team’s successful previous years’ entries in the competition. This paper is not an attempt at a complete description of the 2016 UT Austin Villa agent, the base foundation of which is the team’s 2011 championship agent fully described in a team technical report [5], but instead focuses on changes made in 2016 that helped the team repeat as champions.

In addition to winning the main RoboCup 3D Simulation League competition, UT Austin Villa also won the RoboCup 3D Simulation League technical challenge by winning each of the three league challenges: free, keepaway, and Gazebo running challenge. This paper also serves to document these challenges and the approaches used by UT Austin Villa when competing in the challenges.

The remainder of the paper is organized as follows. In Sect. 2 a description of the 3D simulation domain is given. Section 3 details changes and improvements to the 2016 UT Austin Villa team (including those for marking, getting open, tuning kick selection for height, indirect kick set plays, and directional kicks), while Sect. 4 analyzes the contributions of these changes in addition to the overall performance of the team at the competition. Section 5 describes and analyzes the league challenges that were used to determine the winner of the technical challenge, and Sect. 6 concludes.

2 Domain Description

The RoboCup 3D simulation environment is based on SimSpark [6], a generic physical multiagent system simulator. SimSpark uses the Open Dynamics Engine (ODE) library for its realistic simulation of rigid body dynamics with collision detection and friction. ODE also provides support for the modeling of advanced motorized hinge joints used in the humanoid agents.

Games consist of 11 versus 11 agents playing two 5 min halves of soccer on a \(30 \times 20\,\text {m}\) field. The robot agents in the simulation are modeled after the Aldebaran Nao robot, which has a height of about 57 cm, and a mass of 4.5 kg. Each robot has \(22^\circ \) of freedom: six in each leg, four in each arm, and two in the neck. In order to monitor and control its hinge joints, an agent is equipped with joint perceptors and effectors. Joint perceptors provide the agent with noise-free angular measurements every simulation cycle (20 ms), while joint effectors allow the agent to specify the speed/direction in which to move a joint.

Visual information about the environment is given to an agent every third simulation cycle (60 ms) through noisy measurements of the distance and angle to objects within a restricted vision cone (\(120^\circ \)). Agents are also outfitted with noisy accelerometer and gyroscope perceptors, as well as force resistance perceptors on the sole of each foot. Additionally, agents can communicate with each other every other simulation cycle (40 ms) by sending 20 byte messages.

In addition to the standard Nao robot model, four additional variations of the standard model, known as heterogeneous types, are available for use. These variations from the standard model include changes in leg and arm length, hip width, and also the addition of toes to the robot’s foot. Teams must use at least three different robot types, no more than seven agents of any one robot type, and no more than nine agents of any two robot types.

The 2016 RoboCup 3D Simulation League competition included a couple key changes from the previous year’s competition. The first of these was a rule change to make previously direct kick-ins—awarded to the opposing team when a player is the last to touch the ball before it goes out of bounds over the sideline—indirect (another player other than the player who took the kick must touch the ball before a goal can be scored). This rule was put in place to encourage both passing and teamwork, and to make the rules for kick-ins more similar to their equivalent in human soccer of indirect throw-ins. Coupled with this rule change, and also to match human soccer, an indirect kick is awarded to the opposing team if a player taking a kick (direct or indirect) touches the ball twice in a row.

The second change to the RoboCup 3D Simulation League was to add an automated referee for calling charging fouls. Players are deemed to have committed a charging foul, and are beamed outside the field of play as punishment, if when colliding with an opposing player their force (velocity component in the direction of the opposing player) exceeds a tunable threshold. Charging is not called on a player if the player has recently touched the ball—thus deemed to be going for the ball—or if two players are charging into each other at the same time. Calling charging fouls improves play by reducing the number of collisions while at the same time better matching the rules of human soccer.

3 Changes for 2016

While many components contributed to the success of the UT Austin Villa team, including dynamic role assignment [7] and an optimization framework used to learn low level behaviors for walking and kicking via an overlapping layered learning approach [8], the following subsections focus only on those that are new for 2016. A performance analysis of these components is provided in Sect. 4.1.

3.1 Marking

During the 2015 RoboCup competition it was noticed that teams had developed good kicking/passing abilities, and then were exploiting these abilities through set plays by passing the ball to open teammates in good positions to take a shot on goal. To counteract set plays, UT Austin Villa developed and employed a marking system for the 2016 RoboCup competition to cover and defend against opponents in dangerous offensive positions from receiving passes.

The marking system implemented by the UT Austin Villa team is a sequential process encompassing the following three steps:

  1. 1.

    Decide which players to mark

  2. 2.

    Select which roles to use for marking purposes

  3. 3.

    Use prioritized role assignment to assign players to positions.

In the first step, hand-coded heuristics are used to decide which opponents are in dangerous offensive positions and should be marked. Next, from the team’s default formation positions, computed as offset positions from the ball using Delaunay triangulation [9], a set of formation positions are selected to be replaced with positions needed for marking. The set of formation positions to be replaced, computed by the Hungarian algorithm [10], is that which minimizes the sum of distances between positions for marking and the formation positions that are being replaced by the marking positions. Finally, prioritized role assignment [11], an extension of SCRAM role assignment [7], is used to assign agents to move to the team’s desired role positions (both those for formation and marking purposes). A full description of the marking system can be found in [11].

3.2 Getting Open

In 2015 UT Austin Villa optimized variable distance kicks [4] that allow for the ball to be kicked different distances in one meter increments. These kicks provide passing options as robots can select from many potential targets to kick the ball to. Each potential kick location is given a score according to Eq. 1, and the location with the highest score is chosen as the location to kick the ball to. Equation 1 penalizes kick locations that are farther from the opponent’s goal, penalizes kicks that have the ball end up near opponents, and also rewards kicks for landing near a teammate. All distances in Eq. 1 are measured in meters.

$$\begin{aligned} \texttt {score}(\textit{location}) = \begin{array}{lr} -\Vert \textit{centerOfOpponentGoal}-\textit{location}\Vert \\ \forall \textit{opp} \in \textit{Opponents}, -\max (25-\Vert \textit{opp}-\textit{location}\Vert ^2, 0) \\ +\max (10-\Vert \textit{closestTeammateToTarget}-\textit{location}\Vert , 0) \end{array} \end{aligned}$$
(1)

Ideal locations for kicking the ball are those near a teammate who is in an offensive and open (no opponents are close by) position on the field to receive a pass and take a shot on goal. New for the RoboCup 2016 competition, rather than waiting for a teammate to broadcast a location the ball is going to be kicked to, and only then moving to that position [3], a robot assuming the center forward or striker role [2] in the team’s formation moves toward the position that it calculates its teammate closest to the ball is anticipated to potentially kick the ball to using Eq. 1. As Eq. 1 gives higher values for locations away from opponents, this behavior results in the striker dynamically moving to open offensive positions on the field rather than the previous behavior of standing at a fixed location relative to the ball—possibly close to opponents marking that player—as it waits to hear from a teammate where the ball might next be kicked.

3.3 Tuning Kick Selection for Height

In addition to allowing for more precise passing, variable distance kicks [4] are also useful for taking shots on goal. Generally speaking, the greater the distance a kick travels the longer and higher the ball may travel in the air, and possibly over the goal, when shooting. During the 2015 RoboCup competition, to try and prevent accidentally shooting the ball over the goal, kicks for shooting on goal were limited to those traveling no more than 7 m in distance beyond the goal line. Despite this attempt at limiting the distance for kicks when shooting, a number of shots still missed their target by flying above the goal.

Table 1. The maximum forward distance measured across 100 kicks for different kick types (kicks for different distances) during which the ball is above the height of the goal. All values are in meters.
Full size table

For the 2016 RoboCup competition, the UT Austin Villa team took a more calibrated approach to preventing shots from going over the goal. Instead of estimating 7 m as the difference between a kick’s distance and the maximum forward distance the ball can travel at a height above the goal for all kick types, the maximum forward distance the ball is above the height of the goal was directly measured across 100 kick attempts for each kick type. Table 1 shows these measurements for different kicks for an agent with a type four body model (an agent with the standard Nao model but with toes added). Using these measurements, the kick with maximum power that will not fly above the goal—the kick with the maximum distance, but whose forward above goal height distance is less than the ball’s distance to the goal—is selected when shooting on goal.

From the data in Table 1 it can be seen that there is not a constant offset difference between the distance of a kick and the maximum forward distance that the ball can be above the height of the goal. Furthermore, as for many of the kick types this difference is less than the 7 m estimated value used during the 2015 competition, it is not surprising that shots went over the goal during last year’s competition.

3.4 Indirect Kick Set Plays

With this year’s rule changes (outlined in Sect. 2) making kick-ins indirect, and double touches on kicks result in an indirect free kick for the opposing team, it becomes practical to use set plays for indirect kicks. During indirect kicks the team switches to a formation shown in Fig. 1, where three players are spread in a line across the field near the halfway line, and another three players are spread in a line across the field in a more forward offensive position. The player taking the kick, now with more options from the formation as to which teammate to pass the ball to, chooses to kick the ball to the best location as decided by Eq. 1 in Sect. 3.2. The player taking the kick also waits 10 s from the start of an indirect kick before kicking the ball so as to give its teammates time to move to their assigned positions in the indirect kick formation.

Fig. 1.
figure 1

Indirect kick formation for set plays during a kick-in. Yellow lines represent potential passes to teammates in the formation. (Color figure online)

Full size image

3.5 Directional Kicks

Directional kicks incorporating inverse kinematics created for the 2011 competition [1], but whose use was later discontinued due to poor performance, were reintroduced and integrated into agents with a type four body model (the standard Nao model but with toes added) using overlapping layered learning [8]. Unlike a standard forward kick, where an agent must take the time to line up behind the ball in the direction that it is to be kicked, directional kicks allow for an agent to quickly approach and kick the ball from multiple angles and relative ball positions (example ball positions and kick directions are shown in Fig. 2).

Fig. 2.
figure 2

Possible directions to kick the ball with respect to the placement of the ball at a, b, and c (left image). Agent with its back to the goal taking a shot before scoring at RoboCup 2016 using a \(180^\circ \) “hook” kick where the ball is pulled backward and then kicked behind the agent (right images).

Full size image

4 Main Competition Results and Analysis

In winning the 2016 RoboCup competition UT Austin Villa finished with a perfect record of 14 wins and no losses.Footnote 1 During the competition the team scored 88 goals while only conceding 1. Despite finishing with a perfect record, the relatively few number of games played at the competition, coupled with the complex and stochastic environment of the RoboCup 3D simulator, make it difficult to determine UT Austin Villa being better than other teams by a statistically significant margin. At the end of the competition, however, all teams were required to release their binaries used during the competition. Results of UT Austin Villa playing 1000 games against each of the other eight teams’ released binaries from the competition are shown in Table 2.

Table 2. UT Austin Villa’s released binary’s performance when playing 1000 games against the released binaries of all other teams at RoboCup 2016. This includes place (the rank a team achieved at the 2016 competition), average goal difference (values in parentheses are the standard error), win-loss-tie record, and goals for/against.
Full size table

UT Austin Villa finished with at least an average goal difference greater than 1.8 goals against every opponent. Additionally UT Austin Villa only lost 4 games out of the 8000 that were played in Table 2 with a win percentage greater than 88% against all teams. This shows that UT Austin Villa winning the 2016 competition was far from a chance occurrence. The following subsection analyzes some of the components described in Sect. 3 that contributed to the team’s dominant performance.

4.1 Analysis of Components

1000 games were played between a version of the UT Austin Villa team with marking (Sect. 3.1) turned off and each of the RoboCup 2016 teams’ released binaries. Only the top three teams from the competition showed any appreciable difference in the number of goals scored against the team without marking when compared to the same team with marking (no other teams were able to score more than 6 goals even with marking turned off). Results against the top three teams are shown in Table 3. Against all three teams the average goal difference went down when not using marking. Additionally both the number of goals against, and the percentage of goals scored off set plays,Footnote 2 increased drastically without marking. These results show marking provides a substantial defensive improvement, and furthermore is very effective in defending against set plays.

Table 3. Average goal difference and goals against (with percentage of goals against scored off set plays in parentheses) achieved by versions of the UT Austin Villa team with and without marking when playing 1000 games against the top three teams at RoboCup 2016.
Full size table
Table 4. Average goal difference achieved by different versions of the UT Austin Villa team when playing 9000 games against all teams at RoboCup 2016 (1000 games against each opponent).
Full size table

Table 4 shows the average goal difference achieved by the following different versions of the UT Austin Villa team when playing 9000 games against all teams at RoboCup 2016 (1000 games against each opponent).

  • UTAustinVilla Released binary (no directional kicks).

  • NoGettingOpen Agent assigned to striker role does not try and get open.

  • NoKickHeightTuning Kick selection not tuned for height when shooting.

  • NoIndirectKickSetPlays No indirect kick set plays.

  • DirectionalKicks Uses directional kicks.

As the removal of most components in Table 4 decreased average goal difference, these components were beneficial to the team. Having the player assigned to the striker role move to get open (Sect. 3.2) slightly improves performance and additionally increased the average number of goals scored against all opponents from 5.476 to 5.526. Tuning which kick is selected when taking a shot based on height (Sect. 3.3) also increased the average goal difference. This improvement is likely due to no longer kicking the ball over the goal as the average shot scoring percentage against all opponents increased from 29.69% without tuning to 32.05% with kick height tuning: the ability to kick the ball with just the right amount of power such that it flies into the goal—but not over it—is a valuable skill during games. While using indirect kick set plays (Sect. 3.4) showed only a very slight increase in average goal difference, the average scoring percentage on indirect kicks jumped from 27.10% to 34.78% when using set plays.

The only change that hurts performance is the use of directional kicks. It was known during the competition that choosing to pass the ball with a directional kick instead of using a longer kick to take a shot on goal decreased performance, and so directional kicks were disabled during the semifinals and finals rounds. Directional kicks do however increase the average number of pass attempts in a game from 13.020 to 15.838, and they raise the average possession (a teammate is closer than any opponent to the ball) percentage time from 56.71% to 59.81%. Directional kicks may become more useful if teams’ goalie behaviors improve resulting in the need to be more selective when choosing when to shoot on goal.

Table 5. Average goal difference for each team at RoboCup 2016 (rows) when playing 1000 games against the released binaries of all other teams at RoboCup 2016 (columns). Teams are ordered from most to least dominant in terms of winning (positive goal difference) and losing (negative goal difference).
Full size table

4.2 Additional Tournament Competition Analysis

To further analyze the tournament competition, Table 5 shows the average goal difference for each team at RoboCup 2016 when playing 1000 games against all other teams at RoboCup 2016. It is interesting to note that the ordering of teams in terms of winning (positive goal difference) and losing (negative goal difference) is strictly dominant—every opponent that a team wins against also loses to every opponent that defeats that same team. Relative goal difference does not have this same property, however, as a team that does better against one opponent relative to another team does not always do better against a second opponent relative to that same team. UT Austin Villa is dominant in terms of relative goal difference, however, as UT Austin Villa has a higher goal difference against each opponent than all other teams against the same opponent.

5 Technical Challenges

For the third straight year there was an overall technical challenge consisting of three different league challenges: free, keepaway, and Gazebo running challenge. For each league challenge a team participated in points were awarded toward the overall technical challenge based on the following equation:

$$\begin{aligned} \texttt {points}(\textit{rank}) = 25 - 20*(\textit{rank}-1)/(\textit{numberOfParticipants}-1) \end{aligned}$$
Table 6. Overall ranking and points totals for each team participating in the RoboCup 2016 3D Simulation League technical challenge as well as ranks and points awarded for each of the individual league challenges that make up the technical challenge.
Full size table

Table 6 shows the ranking and cumulative team point totals for the technical challenge as well as for each individual league challenge. UT Austin Villa earned the most points and won the technical challenge by taking first in each of the league challenges. The following subsections detail UT Austin Villa’s participation in each league challenge.Footnote 3

5.1 Free Challenge

During the free challenge, teams give a five minute presentation on a research topic related to their team. Each team in the league then ranks the top five presentations with the best receiving 5 votes and the 5th best receiving 1 vote. Additionally several respected research members of the RoboCup community outside the league vote, with their votes being counted double. The winner of the free challenge is the team that receives the most votes. Table 7 shows the results of the free challenge in which UT Austin Villa was awarded first place.

Table 7. Results of the free challenge.
Full size table

UT Austin Villa’s free challenge submissionFootnote 4 was on the team’s marking system [11] discussed in Sect. 3.1. The FC Portugal team presented research on a learning algorithm, originally designed to learn a parameterized kick for kicking the ball different distances [12], to also allow for the ball being at different relative positions with respect to the robot. The magmaOffenburg team divulged work on learning a kick while walking in raw actuator space [13]. The BahiaRT team discussed an extension to their existing defensive system [14], and the FUT-K team talked about their approach to preventing long passes by the opponent.

5.2 Keepaway Challenge

In the course of the keepaway challenge,Footnote 5 a group of three agents on one team attempts to maintain possession and keep the ball away from a single opponent agent for as long as possible. Additionally, the ball must be kept within a slowly shrinking square area on the field, which is eventually reduced in size to zero after five minutes. During a keepaway task attempt, if the opponent agent touches the ball, or the ball leaves the allowed keepaway area on the field, a keepaway attempt is considered over, and the time that has passed from the beginning of the attempt is recorded as the score for that attempt.

Fig. 3.
figure 3

Keepaway challenge triangle (left image) and square (right image) formations. Red box is keepaway area. Yellow lines show where ball will next be passed. Blue line shows agent movement using second level kick anticipation (moving to new position in formation once pass has occurred). (Color figure online)

Full size image

UT Austin Villa created two formations for use in the keepaway challenge shown in Fig. 3. In the first formation, agents align themselves in a triangle shape. The agent on the ball waits until the opponent gets near, and then that agent passes the ball to the teammate for which the angle between the opponent and teammate as measured from the ball is the largest (the ball is passed to the teammate with the least chance of the opponent intercepting the pass). In the second formation, agents assume a square shape with teammates in position to receive a pass at \(90^\circ \) angles to the ball. The same passing strategy as used with the triangle formation is used with the square formation, but with the addition that kick anticipation [4] is extended to the second level—a player anticipates and runs to the location of a second pass from the location that the next (first) pass is broadcast as going to by the player kicking the ball—so that teammates will be in their designated formation positions after a pass has occurred. Which formation to use is chosen randomly at the beginning of a keepaway attempt.

Table 8. Time in seconds for each of the teams competing in the keepaway challenge.
Full size table

Table 8 shows the results of the keepaway challenge where teams were ranked by the average time of a team’s best two out of three attempts. UT Austin Villa won the challenge with an average time of over three minutes. Each of UT Austin Villa’s keepaway attempt times were greater than all other teams’ attempts.

5.3 Gazebo Running Challenge

Ongoing work within the RoboCup community is the development of a pluginFootnote 6 for the Gazebo [15] robotics simulator to support the RoboCup 3D Simulation League. As such, a challengeFootnote 7 was held where robots attempt to walk forward as fast as possible for 20 s in the Gazebo simulator without falling. In preparation for the challenge UT Austin Villa optimized fast walking parameters for the team’s omnidirectional walk engine [16] within the Gazebo simulator using the CMA-ES algorithm [17]. Walk engine parameters were optimized for 300 generations of CMA-ES with a population size of 150.

Table 9. Speed in meters per second for each of the teams competing in the Gazebo running challenge.
Full size table

Results of the Gazebo running challenge are shown in Table 9. Each participating team performed five running attempts and were scored by the average forward walking speed across all attempts. UT Austin Villa won the challenge with all five of the team’s runs having a speed of over 1 m/s. Each of UT Austin Villa’s running attempt speeds were greater than all other teams’ attempts.

6 Conclusion

UT Austin Villa won the 2016 RoboCup 3D Simulation League main competition as well as all technical league challenges.Footnote 8 Data taken using released binaries from the competition show that UT Austin Villa winning the competition was statistically significant. The 2016 UT Austin Villa team also improved dramatically from 2015 as it was able to beat the team’s 2015 champion binary by an average of 0.561 (±0.029) goals across 1000 games.

In an effort to both make it easier for new teams to join the RoboCup 3D Simulation League, and also provide a resource that can be beneficial to existing teams, the UT Austin Villa team has released their base code [18].Footnote 9 This code release provides a fully functioning agent and good starting point for new teams to the RoboCup 3D Simulation League (it was used by the new KgpKubs team at the 2016 competition). Additionally the code release—which was awarded second place for the HARTING Open Source Prize at RoboCup 2016—offers a foundational platform for conducting research in multiple areas including robotics, multiagent systems, and machine learning.