RoboCup — Multiagent Systems
Daniel Polani
Adaptive Systems Research Group
School of Computer Science
University of Hertfordshire
UK
August 2, 2013
Daniel Polani
RoboCup — Multiagent Systems
Thanks!
Gliders
Mikhail
Prokopenko
Oliver Obst
HELIOS
Hidehisa
Akiyama
Bold
Hearts
Sander van
Dijk
Drew
Noakes
Ismael
DuqueGarcia
Nicole
Hendrickson
Daniel
Barry
Oliver Olding
Alex
Metaxas
Valerio
Lattarulo
Sven Magg
Jamie
Hurst
Julian
Zoellner
Peter Snow
Vighnesh
Pindoria
Steve Hunt
Wong
Qiming
Shen
Chaohua
Zhu
Santiago
Franco
Michael
Snelling
Jenna Garner
Jayasudha
Selvaraj
Parham
HaghigiRad
Chin
Baltic
Lübeck
Thomas
Martinetz
Martin
Haker
André
Meyer
Behboud
Kalantary
Jan Balster
Kord Eickmeyer
Tobias
Kochems
Nima
Madershahian
Jan
Hendrik
Sauselin
Foo
Jürgen Perl
Michael
Wünstel
Christian
Christian
Bauer
Meyer
Michael
Erich
Kutschinski
Junges
Volker
Axel
Haas
Arnold
Marc Hell- Götz
wig
Schwandtner
Ulf Krebs
Oliver Labs Manuel
Mathias
Gauer
Maul
Birgit
Roman
Schappel
Tobias
Pelek
Jens Scheid Hummrich
Beate
Starck
Thomas
Uthmann
Mainz
Rolling
Brains
and the International RoboCup Community
Daniel Polani
RoboCup — Multiagent Systems
Sebastian
Oehm
Frank
Schulz
Ralf
Schmitt
Peter
Dauscher
Tobias
Jung
Achim
Liese
Michael
Hawlitzki
Peter Faiß
Part I
What is an Agent?
Daniel Polani
RoboCup — Multiagent Systems
What is an Agent? I
One Agent and a World
. . . Wt−3
//
//
//
St−3
A
/ Wt−2
G //
///
/
Gt−3
//
//
// . . . Mt−3
St−2
A
/ Wt−1
G //
///
/
//
Gt−2
//
// / Mt−2
St−1
Daniel Polani
A
/ Wt
I +
++
+++
//
Gt−1
//
// / Mt−1
St
/ Wt+1
F //
///
/
At
++
H
++ ++ / Mt
St+1
RoboCup — Multiagent Systems
A
/ Wt+2 . . .
G
//
Gt+1
//
// / Mt+1 . . .
What is an Agent? II
Agent with World (and Other Agent)
0
. . . Mt−3
H ..
...
.
0
St−3
G
A0t−3
S0
A
St−2
. . . Wt−3
//
//
//
St−3
/ M0
t−2
H ..
...
.
//
Gt−2
//
// / Wt−2
G ///
//
/
Gt−3
//
//
// . . . Mt−3
/ M0
t−1
H ..
...
.
A0t−2
S0
A
St−1
//
Gt−1
//
// / Wt−1
G ///
//
/
Gt−2
//
//
// / Mt−2
Daniel Polani
A0t−1
A
/ M0
t
I ++
+++
+
0
St
I
++
++ ++ / Wt
I +++
+++
Gt−1
//
//
// / Mt−1
St
A0t
S0
/ M0 . . .
t+1
H ..
...
.
00
Gt+1
00
00 / Wt+1
F ///
//
/
At
H
++
++ ++ / Mt
St+1
RoboCup — Multiagent Systems
A0t+1
A
//
//
//
/ Wt+2 . . .
G
Gt+1
//
//
// / Mt+1 . . .
What is an Agent? III
Initial Observations
Purely Passive World:
a passive world has a dynamics
runs according to fixed dynamics
“reacts” to agent’s actions
World with Active Agent:
strictly spoken, world with agent has dynamics
however, dynamics of these agents looks like dictated
by a “purpose”
Daniel Polani
RoboCup — Multiagent Systems
Braitenberg Vehicles
[Braitenberg, 1984]
Purposeful Behaviour
fleeing the light
Daniel Polani
RoboCup — Multiagent Systems
Braitenberg Vehicles
[Braitenberg, 1984]
Purposeful Behaviour through
Simple Dynamics
fleeing the light
seeking the light
Daniel Polani
RoboCup — Multiagent Systems
Braitenberg Vehicles
[Braitenberg, 1984]
Purposeful Behaviour through
Simple Dynamics
fleeing the light
seeking the light
Daniel Polani
RoboCup — Multiagent Systems
Braitenberg Vehicles
[Braitenberg, 1984]
Purposeful Behaviour through
Simple Dynamics
fleeing the light
seeking the light
Daniel Polani
RoboCup — Multiagent Systems
Notes
Passive Objects and Agents
not always distinguishable
sometimes by virtue of “camouflage”
sometimes by simple lack of ability
Daniel Polani
RoboCup — Multiagent Systems
Notes
Passive Objects and Agents
not always distinguishable
sometimes by virtue of “camouflage”
sometimes by simple lack of ability
Do not attribute to malice what is equally
explained by incompetence.
Napoleon
The “Pizza Tower” Lesson
Daniel Polani
RoboCup — Multiagent Systems
Notes
Passive Objects and Agents
not always distinguishable
sometimes by virtue of “camouflage”
sometimes by simple lack of ability
Do not attribute to malice what is equally
explained by incompetence.
Napoleon
The “Pizza Tower” Lesson
Are those agents standing around waiting to spring a trap?
Daniel Polani
RoboCup — Multiagent Systems
Notes
Passive Objects and Agents
not always distinguishable
sometimes by virtue of “camouflage”
sometimes by simple lack of ability
Do not attribute to malice what is equally
explained by incompetence.
Napoleon
The “Pizza Tower” Lesson
Are those agents standing around waiting to spring a trap
or are they just lost?
Daniel Polani
RoboCup — Multiagent Systems
Recap
World with Another Active Agent
world with agent has dynamics
looking like dictated by a “purpose”
may or may be not consistent with one’s own “purpose”
Daniel Polani
RoboCup — Multiagent Systems
Mottos of Edification and Purpose
Goldfinger’s Motto
1
Once is happenstance.
2
Twice is bad luck.
3
Three times is enemy action
Daniel Polani
RoboCup — Multiagent Systems
Mottos of Edification and Purpose
Goldfinger’s Motto
1
Once is happenstance.
2
Twice is bad luck.
3
Three times is enemy action
“Kafka’s Motto”
The fact that you are paranoid
does not mean they are not after you.
Daniel Polani
RoboCup — Multiagent Systems
Slightly More Formal: Single Agents
Properties
single entity controls decisions
single mind
single goal
external world may be noisy
challenge: “optimal” ways of coping with external dynamics
constraints and noise
Daniel Polani
RoboCup — Multiagent Systems
Transition to Multiagent Systems
Agents
“interests”
shared goals
antagonisms
Motto
multiple agents have
inconsistent/conflicting
agenda
but even if consistent
agenda, multiple brains
crisscross interaction
Daniel Polani
RoboCup — Multiagent Systems
Types of Scenarios
Classification
single agent
2-agent
multiagent
cooperative
antagonistic
something in-between (real life, economy)
Daniel Polani
RoboCup — Multiagent Systems
Multiagent Systems
In General
multiagent (> 2)-systems can produce intricate strategy
balances
even fully antagonistic scenarios can be temporarily
cooperative
rich set of strategies, even for simple agents/dynamics
Daniel Polani
RoboCup — Multiagent Systems
Introductory Example: Ant Colony Scenario
[Polani and Uthmann, 1998]
Scenario
competition between
ant colonies
feeding
transporting food
signaling
fighting
Variations
1 XRaptor (1997–)
2
Daniel Polani
Google AI
Challenge (2011)
RoboCup — Multiagent Systems
RoboCup as Multiagent System
Notes
comparatively “simple” case
clear cooperation/antagonism structure
We will now visit the different levels of multiagenthood
Daniel Polani
RoboCup — Multiagent Systems
Part II
Behaviour Analysis
Daniel Polani
RoboCup — Multiagent Systems
Motivation
Analysis
of processes
of agent behaviors
of multi-agent systems
of RoboCup
Goal
automated analysis
behavior-based (no internal knowledge)
state-space trajectories
analysis of:
“micro”-behavior of a single player
player-ball interaction
Daniel Polani
RoboCup — Multiagent Systems
Self-Organizing Maps for Analysis
[Wünstel et al., 2001]
What are SOMs?
Properties
high-to-low dimension
mapping
clustering
topology preservation
sequence detection and
identification
Daniel Polani
RoboCup — Multiagent Systems
Trajectory Representation
Steps
SOM Representation:
vector space
metrics
Task: transform trajectory to a SOM representation
Problem: space of complete trajectories too large
Solution: consider trajectory slices
Daniel Polani
RoboCup — Multiagent Systems
Spatially Focused Representation
ut = ∆pt−1
pt
pt−1


ut−1
 ut 


 ut+1 
..
SOM Training
RoboCup game yields sequence of positions
conversion to u representation giving
vector space with
Euclidean distance
Daniel Polani
RoboCup — Multiagent Systems
Results SFR
I
IV
V
II
VI
I
III
VII
I
IV
V
V
II
VI
IV
III
VII
MRB 1999
Daniel Polani
II
VI
III
VII
CMU 1999
RoboCup — Multiagent Systems
Enhanced SFR (ESFR)
uballt = ∆pballt−1
pballt−1
pballt
uplayert−1
pplayert−1

Daniel Polani

uplayert−1
 uball 
t


 uplayer 
t 

 uball 
t+1 

 uplayer 
.. t+1
RoboCup — Multiagent Systems
Results ESFR
II
I
VII
VIII
VI
V
III
IV
I
I
II
II
VII
VIII
VII
VIII
V
V
IV
IV
III
MRB 1999
pass to right side
pass forward
pass backward
pass to left side
near-ball game
Dribbling
Dribbling
Dribbling
VI
VI
III
I
II
III
IV
V
VI
VII
VIII
CMU 1999
Daniel Polani
RoboCup — Multiagent Systems
Results ESFR II (Details)
Daniel Polani
RoboCup — Multiagent Systems
Results ESFR III (Details)
Daniel Polani
RoboCup — Multiagent Systems
Intermediate Bottom Line
Observations
analysis of micro-behavior by SOMs
trajectory characteristics made visible and transparent
implicit representations
usefulness for particularly for reactive analysis
More to do
higher level analysis of trajectories
semantic analysis
Daniel Polani
RoboCup — Multiagent Systems
Part III
Perception, Prediction and (Antagonistic)
Action
Daniel Polani
RoboCup — Multiagent Systems
World Model
agent control
filtered data
actions
world model
raw data
communication
sensor values filtered via world model
consistent view of past and future
match between assumptions and observations to identify
present
Daniel Polani
RoboCup — Multiagent Systems
Ball Position Filtering
[Haker et al., 2002]
Simulator
state sensor data are noisified and quantized
Filtering
improvement of state information by
additional evidence
object movement
related to particle filtering
Daniel Polani
RoboCup — Multiagent Systems
Ball Position Filtering II
vx -axis
vxmax
x-axis
observation
1
2
Daniel Polani
RoboCup — Multiagent Systems
Ball Position Filtering II
However
observing another agent
introduces significantly
more variation and
unpredictability
vx -axis
vxmax
in fact: try to be as
unpredictable as possible!
x-axis
observation
1
2
Daniel Polani
RoboCup — Multiagent Systems
Example: Optimal Goal Scoring
[Kok et al., 2002]
Task
simplest example of an antagonistic RoboCup problem
contains all basic ingredients relevant to the RoboCup scenario
4
Je
Observations/Assumptions
ball shot in straight direction will deviate by Gaussian with
deviation σ(d) after travelling d
dr
3"#(l r )
p
lr
!3"#(l )
p
y1
l
y2
3"#(l l )
dl
ll
(a)
(b)
Figure
2: Two
situationsRoboCup
of shooting
to the goalSystems
(light gray) together with the associate
Daniel
Polani
— Multiagent
Example: Optimal Goal Scoring II
[Kok et al., 2002]
Observations
probability of hitting goal can be computed
via probability of
Jelle Kok
missing it left and right
dr
3"#(l r )
p
lr
!3"#(l )
p
y1
y2
3"#(l l )
dl
l
ll
(a)
(b)
igure 2: Two situations of shooting to the goal (light gray) together with the associated probbility distributions. (a) Shooting perpendicularly. (b) Shooting at an angle.
Daniel Polani
RoboCup — Multiagent Systems
Goal Scoring
Scoring Success
use given goal keeper for generating tests
classification problem:
given player/goalie positions
determine class (interception or not)
record experiments of interception
Daniel Polani
RoboCup — Multiagent Systems
Ball
Interception
6
Jelle Kok
Jelle Kok
6
Jelle Kok
parametrization: angle goalie/shooting point and distance
6
Jelle Kok
player/goalie
6
passed
350
not passed
passed
not passed
80
80
300
250
70
70
not passed
passed
angle player sees
goalie−shooting
point
angle
player sees goalie−shooting
point
angle player sees angle
goalie−shooting
point
player sees
goalie−shooting point
80
60
60
70 80
50
50
70
60
40
40
60
50
30
50
30
40
20
40
20
350
not passed
passed
300
250
200
150
100
50
300
30
10
20
0
250
4
6
4
10
6
10
0
0
4
6
4
8
10
distance
8 player−goalie
10
distance player−goalie
8
6
(a)
(a)
12
200
14
12
14
150
50
10
12
12
not passed (normalized)
passed
(normalized)
not passed
(normalized)
passed (normalized)
0
0.5
−0.5
0
1
0.5
1.5
2
2.5
1
1.5
2
2.5
1
1.5
2
2.5
2
2.5
2.5
2
2.5
not passed
−1.5
−1
−1
−0.5
−0.5
50
150
0
−2
0
100 −2
0
−1.5
0.5
0.5
1
1.5
2
2.5
not passed
not passed
−1.5
−1.5
−1
−1
−0.5
−0.5
0
0
0.5
0.5
1
1
1.5
1.5
2
(b)
(b)
50
0
−2
0
−1.5
−1
−1
−0.5
−0.5
0
0
0.5
1
0.5
1.5
1
2
1.5
2.5
(b) (b)
probabiliy of passing the goalie
probabiliy of passing the goalie
0.9
0.9
not passed (normalized)
not passed
(normalized)
passed
(normalized)
passed (normalized)
0.70.9
0.7
0.60.8
0.8
0.6
1
0.8
1
0.8
0.9
0.7
0.9
0.7
0.8
0.6
0.8
0.6
0.7
0.7
0.5
0.6
0.6
probabiliy ofprobabiliy
passing the
ofgoalie
passing the goalie
Bayes
sigmoid
Bayes
sigmoid
Bayes
sigmoid
0.5
0.7
0.5
Bayes
sigmoid
0.4
0.6
0.4
0.5
0.4
0.5
0.3
0.5 0.3
0.4
0.2
0.4 0.2
0.3
0.1
0.3 0.1
0.2
0.4
0.3
0.2
−2
−0.5
−1
1
1
0.1 0
−1
−1.5
200
300
150
150
250
100
100
200
50
0
−2
0.9
0
0.2−2
0.10
−2
−1.5
not passed
300
0
250 −2
1
0.8 1
1
0.8
0.4
0.6
passed
50
300
14
14
(a)
(a)
0.9
0.5
0.7
passed
100
8
10
distance
player−goalie
distance player−goalie
1
0.9
250
200
350
200
150
300
150
100
250
100
50
200
50
0
150
−2
0
100 −2
250
0
−2 200 −1.5
30
10
20
0
passed
350
300
passed
−1.5
−1.5
−1.5
−1
−1
−1
−0.5
−0.5
−0.5
0
0
0
0.5
0.5
(c)
(c)
0.5
1
1.5
1.5
1
2
2.5
2.5
2
0.1
0
−2
1
1.5
2
0.3
0.5
0.3
0.2
0.4
0.2
0.1
0.3
0.1
0
−2 0
0.2
−2
0.1
Daniel Polani0
2.5
−1.5
−1.5
−1.5
−1
−1
−1
−0.5
−0.5
0
0.5
1−d
0 feature 0.5
1−d feature
1
1
1.5
1.5
2
2
(d)
RoboCup — Multiagent
Systems
(d)
−0.5
0
0.5
1−d feature
1
1.5
2
2.5
2.5
2.5
Goal Score Probability
Consider
goal hitting and interception are independent
1
1
0.9
0.9
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.4
0.3
0.3
0.3
0.2
prob. to pass goalkeeper
prob. ball enters goal
prob. for scoring
0.1
0
−10
0.1
0
−8
−6
−4
−2
0
2
y−coordinate of shooting point
4
6
8
10
(b)
−10
0.5
0.4
0.2
0.1
10
0.5
0.4
prob. to pass goalkeeper
prob. ball enters goal
prob. for scoring
probability
1
0.2
8
Results
0.9
probability
probability
unprotected versus well-defended
goal
7
Section 3
0
−8
−6
−4
−2
0
2
y−coordinate of shooting point
4
6
8
10
−1
(a)
(a) Player is shooting from the right while the goalkeeper
ting from straight in front of the goal while the goalkeeper
Daniel Polani
Figure 4: Scoring probability curves. (a) Player is sho
covers his goal well. (b) Player is shooting from straigh
RoboCup — Multiagent Systems
Part IV
Multiagent Strategies
Daniel Polani
RoboCup — Multiagent Systems
Some General Principles
[Almeida et al., 2010]
Challenges
simultaneous multimodal information: difficult to process
unpredictable environment
unreliable message reception
low bandwidth limits conveyance of meaningful knowledge in
messages
uncertainty in perceived world information may lead to
conflicting/inconsistent behaviours
[Penders, 2001]
Daniel Polani
RoboCup — Multiagent Systems
Concrete Problems
[Almeida et al., 2010]
Perception
Where, when and how to use vision?
Whom to listen to?
How to estimate information of others?
Communication
What, when and how to exchange information?
How to use exchanged information?
Action
Which action of player is best for the team?
How to evaluate different types of actions (e.g. pass vs
dribble)?
How to execute a given elementary (e.g. kick) or compound
action (e.g. dribble)?
Daniel Polani
RoboCup — Multiagent Systems
Coordination
[Almeida et al., 2010]
Types
Ball-centered: react to ball velocity changes (e.g. after kick)
Active: consider target location of desired action (e.g. a
pass to perform)
Strategic: consider strategic location (e.g. find open space for
pass)
Global: locker-room agreements
[Stone, 2000]
Time Range
Approach
ball-centered
active
strategic
Usage Scope
individual
individual or collective
collective
Daniel Polani
Inf. Validity Period
short
short to medium
medium to long
RoboCup — Multiagent Systems
Part V
Meditation: Limits on Cooperation
Daniel Polani
RoboCup — Multiagent Systems
Principled Limits of Multiagent Coordination
[Harder et al., 2010]
Question
What’s the best two agents can do in terms of coordination?
How does it compare to “two agents with one brain”?
Daniel Polani
RoboCup — Multiagent Systems
Principled Limits of Multiagent Coordination
[Harder et al., 2010]
Question
What’s the best two agents can do in terms of coordination?
How does it compare to “two agents with one brain”?
Separate Action Selection
Shared Action Selection
(1)
AAt (1)
t
a)
a)
St
S
t
b) b)
SS
t+1
t+1
A A
St S
t
St+1S
(2)
At (2)
At
Daniel Polani
RoboCup — Multiagent Systems
t+1
T
stat
wh
the
ap
var
by
the
orating the action chosen at state s and then following
Two Agents: One Goal
Prototypical Scenario
Daniel Polani
RoboCup — Multiagent Systems
t
e
-
s
=
5
o
ordination still increases for β → 0, but by a significantly
smaller amount: In a 6 × 7 grid world the difference be0
-5
E[U ]
h
.
y
r
-
Figure 6). The spikes in the graph are due to convergence
problems
certain values
of β. For larger worlds the coUtility
vs.for
Relevant
Information
-10
-15
-20
-25
0
0
0.5
1.5
1
2
I(A; S) - 1 × 6 Field
2.5
3
-2
-4
Daniel Polani
RoboCup — Multiagent Systems
Shared vs. Individual Controllers
Shared Controllers
I(A(1) ; A(2) )
1
0.8
0.6
0.4
0.2
0
0
0.5
1
1.5
I(A; S)
2
2.5
3
na
–
Figure 5: Solid line – coordination of agents with individual
controllers on an 6×1 field, dotted line – limit given by each
controllers relevant information.
is
ce
he
=
an
ns
he
er
c0
he
efn,
constraints.
(1)
(2)
The results show that agents use intrinsic coordination to
overcome limitations of their environment. This coordina(1)
tion is not needed in the optimal case where every agent can
get all the relevant information from the environment that it
needs to choose an optimal action. Though plausible, this
is not entirely obvious a priori. One could think of various
scenarios where the controllers are stochastic and the precise
knowledge of the others agent action would lead to a better
performance.
Now, large agent collectives will usually perform suboptimal policies as each agents’ abilities will be limited: In real
environments, the size of the agent and its supply of energyDaniel Polani
Bottom Line
coordination I (A
;A
1
1
0.8
0.8
I(A(1) ; A(2) )
I(A(1) ; A(2) ), I(A(1) ; A(2) |S)
Individual Controllers
0.6
0.4
0.2
0
0.5
1
1.5
I(A; S)
2
2.5
3
0
0
0
Figure 4: Coordination of agents with shared controller on a
6 × 1 field, dotted line – intrinsic coordination, solid line –
coordination.
Figure 5: Solid line
controllers on an 6×
controllers relevant
tween the coordination for small and large values of β is
constraints.
The results show
overcome limitatio
tion is not needed i
get all the relevant
needs to choose an
is not entirely obvi
scenarios where the
knowledge of the o
performance.
Now, large agent
mal policies as each
environments, the s
is ≈ 1.54 bit. For very narrow worlds (size n × 1) the
coordination even reached its maximum max H(A(1) ) =
max H(A(2) ) = 1 bit. It may seem unintuitive that this can
happen while the relevant information is positive, as it means
that one action fully determines the other and each of the
two possible actions is chosen with probability 12 . However
the coordination takes the expectation over all states: the actions can be totally synchronised, that is, H(A1 |A2 ) = 0
while H(A1 |S) is not maximal. Thus the distribution of the
possible two synchronous actions is not uniform, but this effect can vanish when the expectation over all states is taken,
RoboCup
— Multiagent
which can also
be seen by thatSystems
fact that the intrinsic coor-
; A(2) |S) vs.
coordination via environment
0.4
0.2
0
) distinguished
by in a 4 × 5 world the difference
only ≈ 0.05 bit whereas
intrinsic coordination I (A
0.6
Part VI
Tactics and Strategy: Case Studies
Daniel Polani
RoboCup — Multiagent Systems
tional competition games, but they have been demonstrated in
RoboCup’2008 MSL Free Technical Challenge and a similar mechaTactics
nismand
has beenStrategy:
used for corner kicksPassing
(see below). In the free challenge, two robots alternately took on the roles of passer and
[Lau et al., 2011]
receiver until one of them was in a position to score a goal (Fig. 5).
The sequence of actions on both players is described in Table 1.
They start from their own side of the field and, after each pass, the
Pass Coordination
passer moves forward in the field, then becoming the receiver of
the next pass. The coordinationRoleReceiver
between passer and receiver is
RolePasser
based TRYING
on passing flags,
one for each player, which can take the folPassFlag
TO PASS
lowing values: READY, TRYING_TO_PASS and BALL_PASSED. In the
Aligncase
to of
receiver
Align to Passer
a normal game, another pass coordination variable would
PassFlag READY
identify the receiver.
Kick the ball
5.2. SetBALL
plays PASSED
PassFlag
Move to
next position
Catch in
ball
Another
methodology implemented
CAMBADA is the use of
coordinated procedures for set plays, i.e. situations when the ball
the ball is engaged by the replacer, then with
to the replacer. The replacer also moves tow
waits that the toucher moves away and the
nent goal. It should be noted that both the to
position themselves on the shoot line, so th
cher moves away, the replacer is ready to s
a similar procedure is followed, but without
line, since the involved robots must be in
field.
This scheme has been updated in 2009 to
rule that only allows one robot of the tea
piece (and none from the opponent team)
around the ball and obliges the ball to b
and to roll free on the field for at least 0.5 m
placer passes the ball to one of, possibly m
as RoleReceiver. Before passing, an eva
corridors is performed jointly by the Repl
and results are shared through the RTDB.
of the Replacer to choose the destination of
communicated through the RTDB before pa
Finally, in the case of set pieces against CAM
is used to protect the goal from a direct sho
Fig. 5. Sequence of passes demonstrated in the free challenge of RoboCup’2008.
Daniel Polani
RoboCup — Multiagent Systems
Fig. 6. Replacer role algorithm for co
Tactics and Strategy: Goal Defense
[Lau et al., 2011]
450
N. Lau et al. / Mechatroni
Goal Defense
line ball—goal
one player on this line, as
close as possible to ball
two players near penalty
area
one player near ball, 45o
from above line to observe
ball and report to
teammates
Fig. 7. Placement of RoleBarrier players.
the ball to the own goal defines the barrier positions. One player
places itself on this line, as close to the ball as it is allowed. Two players place themselves near the penalty area. One player is placed near
the ball, 45 degrees from the mentioned line, so that it can observe
the ball coming into play and report that to teammates. Finally,
one player positions itself in such a way that it can oppose to the progression of the ball through the closest side of the field. The placement of players is illustrated in Fig. 7.
The assignment of the RoleBarrier, RoleReceiver,
Daniel PolaniRoleReplacer
RoboCup —
Systems
andMultiagent
RoleToucher
roles is executed by sorting
one player to oppose
progression of ball through
closest side of field
Optimization of Opponent Marking
[Kyrylov and Hou, 2007, Kyrylov and Hou, 2010] I
Problem Description
Collaborative Defensive Positioning:
multi-criteria assignment problem
n defenders are assigned to m attackers
each defender must mark at most one attacker
each attacker must be marked by no more than one
defender
Pareto Optimization:
improve the usefulness of the assignments
simultaneously minimizing required time
to execute an action and
prevent threat by an attacker
Daniel Polani
RoboCup — Multiagent Systems
Optimization of Opponent Marking
[Kyrylov and Hou, 2007, Kyrylov and Hou, 2010] II
Parameters
Angular size of own goal from the opponent’s location
Distance from the opponent’s location to own goal;
Distance between the ball and opponent’s location
Daniel Polani
RoboCup — Multiagent Systems
Optimization of Opponent Marking
[Kyrylov and Hou, 2007, Kyrylov and Hou, 2010] III
Criticisms
[Almeida et al., 2010]
Outnumbered Defenders:
should not mark specific attackers
should position themselves to prevent ball/attackers’
progression towards goal’s center
Outnumbered Attackers:
more than one defender should mark attacker (e.g. ball
owner)
pursue strategy to quickly intercept the ball
or compel the opponent to make bad decision/lose the ball
Daniel Polani
RoboCup — Multiagent Systems
Bold Hearts Example
Formations
different formations
depending on game
situations
e.g. trying to get 2 players
around ball
Coordination
visual
goalie decides roles
according to freed positions
and required roles
crowding rules
jitter suppression:
both go, one decides
reinforces decisions
Daniel Polani
Ball
1 or 2 positions fixed to the
ball: supporting players
field/ball equilibrium
Opponent Harassment
predicting opponent’s
behaviour
putting obstacles in
opponent’s plan
Passing
dribble
attack
pass
panic kick
RoboCup — Multiagent Systems
Part VII
Influence
Daniel Polani
RoboCup — Multiagent Systems
Who gets the Ball?
Simplest Case
both agents move immediately and with same speed
Daniel Polani
RoboCup — Multiagent Systems
Who gets the Ball?
Simplest Case
both agents move immediately and with same speed
Daniel Polani
RoboCup — Multiagent Systems
Who gets the Ball?
Simplest Case
both agents move immediately and with same speed
Daniel Polani
RoboCup — Multiagent Systems
Who gets the Ball?
Simplest Case
both agents move immediately and with same speed
Voronoi Cells/Delaunay Triangulation
[Almeida et al., 2010, Prokopenko et al., 2012, Akiyama et al., 2013]
Daniel Polani
RoboCup — Multiagent Systems
Turn to the ball
Task
Goal: turn to the ball and go there
Assume:
agent looks along x-axis
turning is elementary action in 2D simulator
of course, not in humanoids
(not necessary in PythoCup)
Ball
∆y
φ
∆x
Agent
Daniel Polani
RoboCup — Multiagent Systems
Getting to the Ball
Task
Goal: go to the ball
Assume: ball is not moving
Steps
1
assume we have angle φ
2
elementary turn by φ
3
move to the ball
duration:
4
d: distance
v : maximum velocity
t = d/v
Daniel Polani
RoboCup — Multiagent Systems
Getting to the Ball
Task
Goal: go to the ball
Assume: ball is not moving
Steps
1
assume we have angle φ
2
elementary turn by φ
3
move to the ball
duration:
4
d: distance
v : maximum velocity
t = d/v −
1
|{z}
time for turning
Daniel Polani
RoboCup — Multiagent Systems
Getting to the Ball — Ball is Moving I
Task
Goal: go to the ball
Assume: ball is moving in given direction
Approach
movement of ball
movement of agent
could compute contact point directly
Daniel Polani
RoboCup — Multiagent Systems
Getting to the Ball — Ball is Moving II
Steps
however, easier to do
step-wise
st∗
consider circle of radius
dt = vplayer · t for
t = 0, 1, 2, 3 . . .
consider st∗ = s0 + vball · t
for t = 0, 1, 2, 3 . . .
if kst∗ k ≤ dt , agent can — in
principle — catch ball at
this position if agent moves
in relevant direction
Daniel Polani
s0
RoboCup — Multiagent Systems
Getting to the Ball — Ball is Moving III
Notes
allows handling of
slowing-down ball
16
’ball_move.dat’
14
12
10
allows handling of
turn delay
8
6
if ball fast, consider
catch to fail
4
2
may need to consider
running after the ball,
until slower
0
-2
0
Daniel Polani
5
10
15
20
25
30
RoboCup — Multiagent Systems
35
40
45
50
Influence Regions: “Grass-Chess”
Daniel Polani
RoboCup — Multiagent Systems
Influence Regions II: “Grass-Chess”
Daniel Polani
RoboCup — Multiagent Systems
Example Insights III: “Grass-Chess”
Daniel Polani
RoboCup — Multiagent Systems
Example Insights IV: “Grass-Chess”
Daniel Polani
RoboCup — Multiagent Systems
Pass Optimization
Pass Value Iteration

(n)
(n) 
 max pj Vj + (1 − pj )Vĵ
j∈N(i)
(n+1)
Vi
=
(n)
(n) 
 min pj Vj + (1 − pj )Vĵ
j∈N(i)
Daniel Polani
if i friend
if i foe
RoboCup — Multiagent Systems
Part VIII
References
Daniel Polani
RoboCup — Multiagent Systems
Akiyama, H., Nakashima, T., and Yamashita, K. (2013).
Helios2013 team description paper.
Team Description Paper.
Almeida, F., Lau, N., and Reis, L. P. (2010).
A survey on coordination methodologies for simulated robotic
soccer teams.
In MAS&S@Mallow ’2010 — 4th Int. Workshop on
Multi-Agent Systems and Simulation (MAS&S), held at
MALLOW - Multi-Agent Logics, Languages, and Organisations
Federated Workshops, Lyon, France.
Braitenberg, V. (1984).
Vehicles: Experiments in Synthetic Psychology.
MIT Press, Cambridge.
Haker, M., Meyer, A., Polani, D., and Martinetz, T. (2002).
A method for incorporation of new evidence to improve world
state estimation.
Daniel Polani
RoboCup — Multiagent Systems
In Birk, A., Coradeschi, S., and Tadokoro, S., editors,
RoboCup-2001: Robot Soccer World Cup V, Berlin. Springer.
Harder, M., Polani, D., and Nehaniv, C. L. (2010).
Two agents acting as one.
In Proc. Artificial Life, Odense, Denmark, pages 599–606.
Kok, J. R., de Boer, R., and Vlassis, N. (2002).
Towards an optimal scoring policy for simulated soccer agents.
In Gini, M., Shen, W., Torras, C., and Yuasa, H., editors,
Proc. 7th Int. Conf. on Intelligent Autonomous Systems, pages
195–198, Marina del Rey, California. IOS Press.
Kyrylov, V. and Hou, E. (2007).
While the ball in the digital soccer is rolling, where the
non-player characters should go in a defensive situation?
In Kapralos, B., Katchabaw, M., , and Rajnovich, J., editors,
Future Play, pages 90–96. ACM, Toronto, Canada.
Kyrylov, V. and Hou, E. (2010).
Daniel Polani
RoboCup — Multiagent Systems
Pareto-optimal collaborative defensive player positioning in
simulated soccer.
In Baltes, J., Lagoudakis, M., Naruse, T., and Shiry, S.,
editors, RoboCup 2009: Robot Soccer World Cup XIII, volume
5949 of LNAI, Berlin. Springer.
Lau, N., Lopes, L. S., Corrente, G., Filipe, N., and Sequeira,
R. (2011).
Robot team coordination using dynamic role and positioning
assignment and role based setplays.
Mechatronics, 21:445–454.
Penders, J. (2001).
Conflict-based behaviour emergence in robot teams.
In Conflicting Agents: Conflict Management in Multi-Agent
Systems, Multiagent Systems, Artificial Societies, and
Simulated Organizations, International Book Series, pages
169–202. Kluwer Academic Publishers, Norwell.
Polani, D. and Uthmann, T. (1998).
Daniel Polani
RoboCup — Multiagent Systems
Survival strategies for ant-like agents in a competitive
environment.
In Wilke, C., Altmeyer, S., and Martinetz, T., editors, Proc.
Third German Workshop on Artificial Life, pages 185–196.
Harri Deutsch.
Prokopenko, M., Obst, O., Wang, P., and Held, J. (2012).
Gliders2012: Tactics with action-dependent evaluation
functions.
Team Description Paper.
Stone, P. (2000).
Layered Learning in Multiagent Systems: A Winning Approach
to Robotic Soccer.
MIT Press.
Wünstel, M., Polani, D., Uthmann, T., and Perl, J. (2001).
Behavior classification with self-organizing maps.
In Stone, P., Balch, T., and Kraetzschmar, G., editors,
RoboCup-2000: Robot Soccer World Cup IV, pages 108–118.
Springer Verlag, Berlin.
Daniel Polani
RoboCup — Multiagent Systems
Winner of the RoboCup 2000 Scientific Challenge Award.
Daniel Polani
RoboCup — Multiagent Systems