### Kilian`s slides

```Learning in Games
Fictitious Play
Notation!
For n Players we have:




n Finite Player’s Strategies Spaces S1, S2, …, Sn
n Opponent’s Strategies Spaces S-1, S-2, …, S-n
n Payoff Functions u1, u2,…, un
For each i and each s-i in S-i a set of
Best Responses BRi (s-i)
What is Fictitious Play?
Each player creates an assessment about the
opponent’s strategies in form of a weight
function:
 0i : S i   
1
i
i
i
i
 t (s )   t 1 (s )  
0
if sti1  s i
if sti1  s i
Prediction
Probability of player i assigning to player –i playing s-i at
time t:
 (s )
 (s ) 
i ~ i
t (s )
i
t
i
i
t
~
s i S i
i
Fictious Play is …

… any rule
ti ( ti )
that assigns
NOT UNIQUE!
 ti ( ti )  BR i ( ti )
Further Definitions
In 2 Player games:
Marginal empirical distributions of j’s play
(j=-i)
dt (s ) 
j
j
 t (s )   0 (s )
j
j
t
Asymptotic Behavior

Propositions:
 Strict
Nash equilibria are absorbing for the
process of fictitious play.
 Any pure-strategy steady state of fictitous play
must be a Nash equilibrium
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
1.5
T
2
Col Player
H
2
T
1.5
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
H
1.5
1.5
T
T
2
3
Col Player
H
H
2
2
T
T
H
T
H
T
H
T
H
T
1.5
2.5
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
H
1.5
1.5
T
T
2
3
Col Player
H
H
2
2
T
T
H
T
H
T
H
T
H
T
1.5
2.5
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
H
H
H
1.5
1.5
2.5
T
T
T
T
2
3
3
Col Player
H
H
H
H
2
2
2
T
T
T
T
1.5
2.5
3.5
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
H
H
H
1.5
1.5
2.5
T
T
T
T
2
3
3
Col Player
H
H
H
H
2
2
2
T
T
T
T
1.5
2.5
3.5
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
H
H
H
1.5
1.5
2.5
3.5
T
T
T
T
2
3
3
3
Col Player
H
H
H
H
2
2
2
2
T
T
T
T
1.5
2.5
3.5
4.5
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
H
H
H
1.5
1.5
2.5
3.5
T
T
T
T
2
3
3
3
Col Player
H
H
H
H
2
2
2
2
T
T
T
T
1.5
2.5
3.5
4.5
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
H
H
H
1.5
1.5
2.5
3.5
T
T
T
T
2
3
3
3
Col Player
H
H
H
H
2
2
2
2
T
T
T
T
1.5
2.5
3.5
4.5
Example “matching pennies”
H
T
H
1,-1
-1,1
T
-1,1
1,-1
Weights:
Row Player
H
H
H
H
1.5
1.5
2.5
1.5
3.5
T
T
T
T
2
3
3
2
3
Col Player
H
H
H
H
2
2
2
2
T
T
T
T
1.5
2.5
3.5
4.5
Weights:
Row Player
Col Player
1.5
2
2
1.5
1.5
3
2
2.5
2.5
3
2
3.5
3.5
3
2
4.5
4.5
3
3
4.5
5.5
3
4
4.5
6.5
6.5
3
4
5
4.5
6
4.5
Convergence?
Strategies cycle and do not converge …
…but the marginal empirical distributions?
 t (s )   0 (s )
j
j
t
1


2
t 
MATLAB Simulation - Pennies
Game Play
Weight / Time
Payoff
Proposition
Under fictitious play, if the empirical
distributions over each player’s
choices converge, the strategy profile
corresponding to the product of these
distributions is a Nash equilibrium.
Rock-Paper-Scissors
Game Play
Weight / Time
A
B C
A 12 , 12 1,0 0,1
1 1
B 0,1 2 , 2 1,0
C 1,0 0,1 12 , 12
Payoff
Rock-Paper-Scissors
Game Play
Weight / Time
1 1
,
2 2
1,0 0,1
1 1
0,1 2 , 2 1,0
1 1
1,0 0,1 2 , 2
Payoff
Shapley Game
Game Play
Weight / Time
0,0 1,0 0,1
0,1 0,0 1,0
1,0 0,1 0,0
Payoff
Persistent miscoordination
Game Play
Weight / Time
A
B
A 0,0
1,1
B 1,1 0,0
Initial weights:
1
1
Payoff
1.4
1.4
Nash:
(1,0)
(0,1)
(0.5,0.5)
Persistent miscoordination
Game Play
Weight / Time
A
B
A 0,0
1,1
B 1,1 0,0
Initial weights:
2
2
Payoff
1.4
1.4
Nash:
(1,0)
(0,1)
(0.5,0.5)
Persistent Miscoordination
Game Play
Weight / Time
A
B
A 0,0
1,1
B 1,1 0,0
Initial weights:
2
2
Payoff
2.4
2.4
Nash:
(1,0)
(0,1)
(0.5,0.5)
Summary on fictitious play
In case of convergence, the time average
of strategies forms a Nash Equilibrium
 The average payoff does not need to be
the one of a Nash (e.g. Miscoordination)
 Time average may not converge at all
(e.g. Shapley Game)

References

Fudenberg D., Levine D. K. (1998)
The Theory of Learning in Games
MIT Press
Nash Convergence of
General-Sum Games
Notation

2 Players:
  
  
 Strategies 
 and 

1   
1   
 Payoff
matricies
R=
r11 r12
c11 c12
r21 r22
C= c c
21
22
Objective Functions

Payoff Functions:
 Vr(,)=r11()+r22((1-)(1-))
+r12((1-))+r21((1-))
 Vc(,)=c11()+c22((1-)(1-))
+c12((1-))+c21((1-))
Hillclimbing Idea
With u=(r11+r22)-(r21+r12)

u’=(c11+c22)-(c21+c12)

Vr ( ,  )
 u  (r22  r12 )

Vc ( ,  )
 u '(c22  c12 )

Update Rule
Vr ( ,  )
 0, k 01 can
arbitrary strategies
 be
k 

Vc ( ,  )
 k 1   k  

Problem

infeasible point outside the unit square.
1
0
1
Solution:

Redefine the gradient to the projection of
the true gradient onto the boundary.
Let this denote the constrained dynamics!
1
0
1
lim
 (t ),  (t )
 0
Become functions
of time!
  
 t   0 u      (r22  r12 ) 

    






(
c

c
)
u
'
0





22
12


 
 t 
1. Case: U is invertible
The two possible qualitative forms of the unconstrained strategy pair:
2. Case: U is not invertible
Some examples of qualitative forms of the unconstrained strategy pair:
Convergence
If both players follow the IGA rule, then
both player’s average payoffs will
converge to the expected payoff of some
Nash equilibrium
If the strategy pair trajectory converges
at all, then it converges to a Nash pair.
Proposition
Both previous propositions also hold with
finite decreasing step size
References

Singh S., Kearns M., Yishay M. (2000)
Nash Convergence of Gradient Dynamics in
General-Sum Games
Proceedings of the Sixteenth Conference on
Uncertainty in Artificial Intelligence, Morgan
Kaufmann, pages 541-548
Dynamic computation of
Nash equilibria in TwoPlayer general-sum
games.
Notation

2 Players:
 p1 
 q1 
 
 
 Strategies    and   
p 
q 
 n
 n
 Payoff
R=
matricies
 r11  r1n 
  


rn1  rnn 
c11  c1n 
   
C= 

cn1  cnn 
Objective Functions

Payoff Functions:
 Row
 Col
Player: Vr ( p, q)  pT Rq
Player:
Vc ( p, q)  p T Cq
Observation!
Vr ( p, q)
is linear in each pi and qj
Let xi denote the pure strategy for action i.
This means:
If Vr ( xi , q)  Vr ( p, q) then increasing
the value of pi increases the payoff.
Hill climbing (again)
Multiplicative Update Rules
 
 
pipi
i
T


 pi (pt )i (Vt )r ( xRq, qi) p
Vr (Rq
p, q )
t t
 

qiqi
T i
T
 qi (qt )i (Vt )c ( p, xR )i Vpc ( Rq
p, q )
t t

Hill climbing (again)
System of Differential Equations (i=1..n)

pi
T


 pi (t ) Rq i  p Rq
t


qi
T
T
 qi (t ) p R i  p Rq
t


Fixed Points?
i either
pi pi (t )  0
T


 pi (t ) Rq i  p Rq
t or

Rq   p
i
T

Rq  0

When is a Fixpoint a Nash?
• Proposition:
Provided all pi(0) are neither
0 nor 1, then if (p,q) converges
to (p*,q*) then this is a
Nash Equilibrium.
Unit Square?
No Problem!
p i
pi=0 or pi=1 both set t to zero!
Convergence of the average of the
payoff

If the (p,q) trajectory and both player’s
payoffs converge in average, the average
payoff must be the payoff of some Nash
Equilibrium
2 Player 2 Action Case

Either the strategies converge immediately
to some pure strategy, or the difference
between the Kullback-Leibler distances of
(p,q) and some mixed Nash are constant.
p
1 p
*
KL( p, p )  p log( * )  (1  p) log(
)
*
p
1 p
KL( p, p* )  KL(q, q* )  const.
Trajectories of the difference
between the Kullback-Leibler
Distances
Nash
But…

… for games with more than 2 actions,
convergence is not guaranteed!
Counterexample: Shapley Game
```