Zero-determinant Strategies for Multi-player Multi

1
Zero-determinant Strategies for Multi-player
Multi-action Iterated Games
Xiaofan He, Student Member, IEEE Huaiyu Dai, Senior Member, IEEE and Peng Ning, Senior Member,
IEEE and Rudra Dutta, Member, IEEE
Abstract
Zero-determinant (ZD) strategies that allow a player to unilaterally control the linear combinations of its own
and other players’ expected rewards in iterated games have recently found wide applications. However, existing ZD
strategies mainly focus on some specific scenarios with restrictions on the number of players or actions a player
can take. Targeting wider applications and better performance, the ZD strategies along with corresponding existence
conditions for general multi-player multi-action iterated games are developed in this work, including existing ones as
special cases. In addition, an interesting fact that every player can have at most one master player (that can control
the expected reward of the given player) is revealed.
Keywords: Zero-determinant, multi-player, multi-action, game theory.
Copyright (c) 2015 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must
be obtained from the IEEE by sending a request to [email protected].
X. He ([email protected]) and H. Dai (Huaiyu [email protected]) are with the Department of Electrical and Computer Engineering, North
Carolina State University, NC, USA.
P. Ning ([email protected]) and R. Dutta ([email protected]) are with the Department of Computer Science, North Carolina State
University, NC, USA.
This work was supported in part by the National Science Foundation under Grants CNS-1016260, ECCS-1307949, and EARS-1444009.
2
I. I NTRODUCTION
The zero-determinant (ZD) strategy that enables a player to unilaterally set its own or another player’s expected
reward, or a weighted combination of these two, is developed in the seminal work [1] for two-player two-action
iterated games, and its extensions to the multi-player two-action and the two-player multi-action iterated games
have been studied in [2, 3] and [4], respectively. However, applications of these ZD strategies may be limited in
practice by the restrictions on the number of players or actions.
Considering this, the ZD strategy for general multi-player multi-action iterated games is developed in this work,
along with their existence conditions. Also, an important fact that every player can have at most one master player
(that can control the expected reward of the given player) is revealed in this work. The proposed ZD strategy is
expected to admit wider applications: Take the setting considered in [2] as an example where multiple users compete
for a shared spectrum band. The existing ZD strategy applies only when admissible actions of the users are restricted
to either transmitting with full power or keeping silent. Nonetheless, the ZD strategy proposed in this work can be
applied to the more general cases where multiple transmission power levels are allowed. Other potential applications
of the proposed ZD strategy include maintaining weighted social welfare in cooperative wireless communication
networks [5], stabilizing frequency in smart grid [6], and providing quality-of-service in femto-cell networks [4].
II. G ENERAL K × N I TERATED G AMES
A general K × N iterated game among K players where each player-k (1 ≤ k ≤ K ) has an action space Ak
consisting of N actions is considered. Background on basic iterated games can be found in [1]. In the following,
notations for the general K×N iterated game are introduced. Particularly, the tuple (ai1 , ..., aiK ) ∈ A , A1 ×...×AK
will be used to represent the case that player-k takes the ik -th (ik ∈ {1, ..., N }) action aik ∈ Ak , where A
is the set of all the N K possible action tuples. In addition, for ease of presentation, an index i ∈ I defined
as i = 1 +
PK
k=1 (ik
− 1)N K−k is assigned to each action-tuple (ai1 , ..., aiK ) ∈ A, with I , {1, ..., N K } the
corresponding action-tuple index set. In the iterated game, the action-tuple (a′i1 , ..., a′iK ) of the previous round is
defined as the current state si of the game, and the set of possible states is defined as S , A; the index i of a state
3
si ∈ S is given by that of the corresponding action-tuple (a′i1 , ..., a′iK ). At each round, players choose their actions
with probabilities depending on the current state and hence the game states form a Markov chain.1
The strategy matrix of player-k is denoted by pk|S|×N = [pki,j ] with |S| the cardinality of S and pki,j the probability
of taking action aj ∈ Ak at state si ∈ S . The reward vector of player-k is denoted by uk = [uk1 , ..., uk|S| ]T where
uki is the reward corresponding to players’ current actions (ai1 , ..., aiK ) with i ∈ I the index of (ai1 , ..., aiK ).
III. Z ERO - DETERMINANT S TRATEGIES
FOR
2 × 2 G AMES
When K = 2 and N = 2, the above general setting reduces to the 2 × 2 game considered in [1, 5, 6], where the
number of states |S| = 4. For a given strategy pair p1 and p2 of the two players, the transition matrix M of the
game state is given by

2
1
p1,1 p1,1


 1 2
p2,1 p2,1

M=

p1 p2
 3,1 3,1


 1 2
p4,1 p4,1

p11,1 p21,2 p11,2 p21,1 p11,2 p21,2 



p12,1 p22,2 p12,2 p22,1 p12,2 p22,2 

,

1
2
1
2
1
2 
p3,1 p3,2 p3,2 p3,1 p3,2 p3,2 



2
1
2
1
2
1
p4,1 p4,2 p4,2 p4,1 p4,2 p4,2
(1)
with the entry Mi,j at the i-th row and the j -th column representing the probability that the next state is sj given
current state si . The stationary distribution π of this Markov chain satisfies π T M = π T ,2 and the expected reward
ūk of player-k in the stationary state is given by
ūk = π T uk .
(2)
Define M̃ = M − I with I the identity matrix, it follows that
π T M̃ = 0,
1
(3)
To see that such a memory-one strategy (i.e., players only need to recall the actions played in the most recent round) is optimal, it is
shown in Appendix A that there is no advantage of recalling longer history records for general K × N iterated games.
2
As in [1, 2, 4–6], it is assumed in this work that M has a unique stationary distribution, which is true when all actions are played with
some probabilities greater than some arbitrarily small positive number ǫ.
4
and further by Cramer’s rule, we have3
Adj(M̃)M̃ = det(M̃)I = 0,
(4)
as M̃ has a zero eigenvalue implied by (3). By comparing (3) and (4), it can be noted that every row of Adj(M̃) is
proportional to π , and hence for any vector f = [f1 , ..., f4 ]T and strategy pair p1 and p2 , it can be shown through
some non-trivial derivations that [1]
π T f ∝ D(p1 , p2 , f )




p11,1 p21,2
p11,2 p21,1
−1 + p11,1 p21,1



  p1 p2
p12,2 p22,1
−1 + p12,1 p22,2
2,1 2,1


, det 


−1 + p13,2 p23,1
p13,1 p23,2
p13,1 p23,1




p14,2 p24,1
p14,1 p24,2
p14,1 p24,1

|
{z

,M′f



p11,1 p21,2
p11,2 p21,1
−1 + p11,1



−1 + p1 −1 + p1 p2
p12,2 p22,1
2,1
2,1 2,2

= det 



 p13,1
−1 + p13,2 p23,1
p13,1 p23,2




p14,1
p14,1 p24,2
p14,2 p24,1

|
{z
,M′′
f




f1 



f2 




f3 




f4 

}




f1 



f2 

,




f3 



f4 

}
(5)
where the last equality holds since determinant remains unchanged if the second column of M′f are added to the
first one. A key observation is that the first column of M′′f is solely under the control of player-1.
Consider, in particular, f = α0 1 + α1 u1 + α2 u2 , (with 1 the all-one vector and αk ∈ R, for k = 0, 1, 2), then
we have
3
α0 + α1 ū1 + α2 ū2 = π T α0 1 + α1 u1 + α2 u2 ∝ D(p1 , p2 , α0 1 + α1 u1 + α2 u2 ) = 0,
In (4), Adj(·) denotes the adjugate matrix which is the transpose of the cofactor matrix.
(6)
5
when p1 is such that the first column is proportional to the last column in the corresponding M′′f ; and such p1 is
termed the ZD strategy of player-1 [1]. This implies that player-1 can control the weighted reward α1 ū1 + α2 ū2
irrespective to player-2’s strategy (if such p1 exists); and similar analysis can be applied to player-2 as well.
As an example, the following result gives the ZD strategy of player-1 for controlling its own expected reward
(corresponding to α1 = 1 and α2 = 0) and the corresponding existence conditions [1, 2].
Theorem 1: If ∃ n, n′ ∈ {1, 2} such that u1n′ ,max ≤ u1n,min , where u1n′ ,max (u1n,min ) is the maximum (minimum)
reward of player-1 when he takes the n′ -th (n-th) action, then, irrespective to the strategy of player-2, any value
ũ1 from the interval [u1n′ ,max , u1n,min ] can be achieved by player-1; without loss of generality, assume n = 1, and
then the corresponding ZD strategies of player-1 are given by
u11
= 1 + 1 − 1 α0 ,
ũ
u12
1
p2,1 = 1 + 1 − 1 α0 ,
ũ
1
u
1
3
p3,1 = 1 − 1 α0 ,
ũ
u14
1
p4,1 = 1 − 1 α0 ,
ũ
p11,1
p11,2 = 1 − p11,1 ,
p12,2 = 1 − p12,1 ,
p13,2 = 1 − p13,1 ,
p14,2 = 1 − p14,1 ,
where α0 ∈ (0, min{ũ1 /(u1max − ũ1 ), ũ1 /(ũ1 − u1min )}) with u1max and u1min the maximum and the minimum
possible rewards of player-1, respectively.
IV. Z ERO - DETERMINANT S TRATEGIES
FOR
M ULTI - PLAYER M ULTI - ACTION G AMES
In this section, the ZD strategies for the general K × N iterated games described in Section II will be presented,
which include the ZD strategies in [1, 2, 4] as special cases. Also, the fact that each player can have at most one
master player will be shown.
A. ZD Strategy for General Iterated Games
Proposition 1 below provides the existence condition and the specific form of the ZD strategies (for player-1) in
general K × N iterated games.
6
Proposition 1: For a K -player N -action iterated game and a set of given coefficients {α1 , ..., αK }, if ∃ n ∈
{1, ..., N } such that
uαn,min
, min
i∈I|i1 =n
K
X
αk uki
≥
uαm6=n,max
, max
k=1
i∈I|i1 6=n
K
X
αk uki ,
(7)
k=1
where I|ik =n (I|ik 6=n ) denotes the set of action-tuples with player-k (not) taking the n-th action. Then,
(1) the action index n (for player-1) is unique;
(2) irrespective to the strategies of other players, player-1 can set the expected weighted social welfare
to be any value ũα ∈ [uαm6=n,max , uαn,min ]4 by taking any strategy that satisfies

!
K
k
X

α
u

k i


α0 , if i ∈ I|i1 =n
1+ 1−


ũα
k=1
p1si ,n =
!
K

k
X

α
u
k

i

α0 ,
otherwise,
1−


ũα
PK
k
k=1 αk ū
(8)
k=1
where
0 ≤ α0 ≤ min{ũα /(uαmax − ũα ), ũα /(ũα − uαmin )},
with uαmax , maxi∈I
PK
k
k=1 αk ui
and uαmin , mini∈I
weighted social welfares, respectively.5
PK
k
k=1 αk ui ,
(9)
representing the maximum and the minimum
Remark 1: Note that (8) does not specify the probabilities of taking actions {am } (m 6= n) at each state and
these probabilities can be arbitrary as long as
PN
1
e=1 psi ,e
= 1 for all states.
Remark 2 (Numerical result): Consider a scenario where a spectrum band is shared by K = 3 users each having
N = 3 different transmission powers Ak = {0, 0.1, 1}, 1 ≤ k ≤ 3. Our numerical results show that a user can
always find a ZD strategy by (8) to fix its average SINR to a value in the range [0, 0.25] (assuming unit channel
power gain and noise power).
Proof: Given existence, the uniqueness of the action index n can be proved by contradiction using (7).
4
Note that a player may be able to remove some actions from its action set (and then add duplicated actions back to keep the total number
α
of actions N unchanged) to minimize uα
m6=n,max , so as to obtain a larger feasible range for ũ .
5
In Proposition 1, it is assumed without loss of generality that all rewards are positive, since the desired ZD strategy is invariant to a
positive shift in reward functions.
7

p11,1 p21,1 ...pK
1,1
p11,1 p21,1 ...pK
1,2



 1 2
 p2,1 p2,1 ...pK
p12,1 p22,1 ...pK
2,1
2,2


M=
..
..

.
.



 1
1
2
K
p|S|,1 p2|S|,1 ...pK
|S|,1 p|S|,1 p|S|,1 ...p|S|,2

p11,1 p21,1 ...pK
1,1
−1 +



 p12,1 p22,1 ...pK
2,1

T
1
K

π f = D(p , ..., p , f ) = det 

..

.



 1
p|S|,1 p2|S|,1 ...pK
|S|,1

...
p11,1 p21,N ...pK
1,N
...
p12,1 p22,N ...pK
2,N
...
..
.

... 



... 

.

... 




...
... p1|S|,1 p2|S|,N ...pK
|S|,N
p11,1 p21,1 ...pK
1,2
...
p11,1 p21,N ...pK
1,N
− 1 + p12,1 p22,1 ...pK
2,2 ...
p12,1 p22,N ...pK
2,N
..
.



p11,1 p21,1 ...pK
 −1 + p11,1
1,2



 −1 + p12,1
−1 + p12,1 p22,1 ...pK
2,2



.
.

..
..




= det 
−1 + p1N K−1 ,1
p1N K−1 ,1 p2N K−1 ,1 ...pK

N K−1 ,2



 1
 pN K−1 +1,1 p1N K−1 +1,1 p2N K−1 +1,1 ...pK
N K−1 +1,2



..
..

.
.





p1|S|,1
p1|S|,1 p2|S|,1 ...pK
|S|,2

|
...
..
.
...
p1|S|,1 p2|S|,1 ...pK
|S|,2
(10)
... p1|S|,1 p2|S|,N ...pK
|S|,N
...
p11,1 p21,N ...pK
1,N
...
...
p12,1 p22,N ...pK
2,N
...
...
..
.
...
... −1 + p1N K−1 ,1 p2N K−1 ,N ...pK
N K−1 ,N ...
... p1N K−1 +1,1 p2N K−1 +1,N ...pK
N K−1 +1,N ...
...
..
.
...
...
p1|S|,1 p2|S|,N ...pK
|S|,N
...
{z
M′′′
f

f1 




... f2 



.. 

...
. 




... f|S|




f1




f2



..


.




.
fN K−1 




fN K−1 +1 



..


.





f|S|

}
(11)
As to the second part of the proposition, it can be first verified that (8) defines valid probability measures, i.e.,
p1si ,n ∈ [0, 1] for all i ∈ I , when (7) and (9) hold. To show that the chosen expected weighted social welfare ũα can
be achieved irrespective to other players’ actions, assume without loss of generality n = 1. Further assuming that
the other players take arbitrary strategies {pk } (for k = 2, ..., K ), the state transition matrix M of the corresponding
iterated game is given by (10).
Using similar arguments as in (3)–(5), it can be shown that, for any |S| × 1 vector f , (11) holds. It is worth
noting that the first column of the matrix M′′′
f in (11) is solely under player-1’s control.
8
When f = −ũα 1 +
PK
k=1 αk u
k
and p1 satisfies (8), it can be verified that the first column is proportional to the
last column (with ratio −ũα /α0 ) in M′′′
f , resulting in a zero determinant in (11). Therefore, it has
α
α
ũ = − −ũ +
K
X
αk ū
k
k=1
=−
T
α
π (−ũ 1 +
!
K
X
+
K
X
αk ūk
k=1
k
αk u )
k=1
|
{z
+
K
X
αk ūk ,
(12)
k=1
}
=0 due to zero determinant in (11)
where the second equality is due to the fact that π T · 1 = 1 and the definition of ūk given in (2).
Setting α1 = 1 and αk = 0 for k > 1, Proposition 1 gives the following corollary, which specifies how player-1
can set its own expected reward irrespective to other players’ strategies.
Corollary 1: For a K -player N -action iterated game, if ∃ n ∈ {1, ..., N } such that
u1n,min , min u1i ≥ u1m6=n,max , max u1i ,
i∈I|i1 =n
i∈I|i1 6=n
(13)
then (1) n is unique; (2) irrespective to the strategies of other players, any value ũ1 from the interval [u1m6=n,max , u1n,min ]
can be achieved by player-1 using any strategy that satisfies



 1 + 1 − u1i /ũ1 α0 , if s ∈ I|i1 =n
p1si ,n =



1 − u1i /ũ1 α0 , otherwise
(14)
where α0 ∈ (0, min{ũ1 /(u1max − ũ1 ), ũ1 /(ũ1 − u1min )}) with u1max and u1min denoting the maximum and the
minimum possible rewards of player-1, respectively.
Remark 3: Note that the ZD strategy presented in [2] generalizes Theorem 1 to the cases of K > 2 and is
included by Corollary 1 here as a special case of N = 2.
Similarly, player-1 can set the expected reward of player-k (k 6= 1) based on the following corollary.
Corollary 2: For a K -player N -action iterated game, if ∃ n ∈ {1, ..., N } such that
ukn,min , min uki ≥ ukm6=n,max , max uki ,
i∈I|i1 =n
i∈I|i1 6=n
(15)
then (1) n is unique; (2) irrespective to the strategies of other players (including player-k), any value ũk from the
9
interval [ukm6=n,max , ukn,min ] can be enforced by player-1 using any strategy that satisfies



 1 + 1 − uki /ũk α0 , if s ∈ I|i1 =n
p1si ,n =



1 − uk /ũk α , otherwise
(16)
0
i
where α0 ∈ (0, min{ũk /(ukmax − ũk ), ũk /(ũk − ukmin )}) with ukmax and ukmin denoting the maximum and the
minimum possible rewards of player-k, respectively.
Remark 4: Corollary 2 includes the ZD strategy presented in [4] as a special case of K = 2.
B. Uniqueness of Master Player
Seeing Corollary 1 and Corollary 2, a natural question is that what will happen when two players set different
values on the expected reward for the same player? Regarding this, it will be shown in the following that such
situation can never happen, except for the degenerated case.
Definition 1: For a given player, its master player is the one that has a valid ZD strategy to control the expected
reward of this given player. More specifically, player-k′ is the master player of player-k if and only if ∃ n′ ∈
{1, ..., N } such that
ukn′ ,min ,
min
i∈I|ik′ =n′
uki ≥ ukm6=n′ ,max ,
max
i∈I|ik′ 6=n′
uki .
(17)
Proposition 2: Each player k cannot have two different master players k′ and k′′ , except for the degenerated
case where equality holds in (17) for both k′ and k′′ (with possibly a different action index n′′ ) and both players
k′ and k′′ set the same expected reward ukn′ ,min for player-k.
Proof: Suppose by contradiction that player-k has two different master players k′ and k′′ in non-degenerated
cases. Then, ∃ n′ , n′′ ∈ {1, ..., N } such that
min uki ≥ max uki ,
i∈Iik′ =n′
min
i∈Iik′′ =n′′
i∈Iik′ 6=n′
uki ≥
max
i∈Iik′′ 6=n′′
uki ,
(18)
(19)
and at least one of (18) and (19) admits strict inequality. Without loss of generality, assume (19) admits strict
inequality.
10
Notice that, by (18), it has
min
uki =
max
uki =
i∈Iik′′ =n′′
i∈Ii
k
′′
′′ 6=n
min
uki ,
(20)
max
uki ,
(21)
i∈Iik′′ =n′′ ,ik′ 6=n′
i∈Ii
6=n′′ ,i ′ =n′
k′′
k
where Iik′′ =n′′ ,ik′ 6=n′ (Iik′′ 6=n′′ ,ik′ =n′ ) denotes the set of indices such that ik′′ = n′′ and ik′ 6= n′ (ik′′ 6= n′′ and
ik′ = n′ ) in the corresponding action-tuples. However, (19–21) imply
max uki ≥
i∈Ii
6=n
k′
=
min
i∈Ii
k′′
≥
′
i∈Ii
=n′′
max
i∈Ii
uki >
min
≥n′′ ,i ′ =n′
k′′
k
=n′′ ,i ′ 6=n′
k′′
k
uki ≥
max
i∈Ii
k′′
uki ≥
uki ≥
6 n′′
=
i∈Ii
i∈Ii
min
=n′′ ,i ′ 6=n′
k′′
k
max
6=n′′ ,i ′ =n′
k′′
k
uki
uki
min uki ,
i∈Ii
k′
(22)
=n′
which contradicts to (18).
V. C ONCLUSIONS
The ZD strategy for general multi-player multi-action iterated games is developed in this work, and the fact that
every player can have at most one master player is revealed. The proposed strategy can extend existing ZD strategy
based schemes to more general settings (e.g., more players and more actions) and may also find wider applications
in the future.
A PPENDIX A
N O A DVANTAGE
FOR
L ONG - MEMORY
Proof: Denote Hi as the recent history (of past actions) shared by players whose index k ≥ i, and accordingly
the history recorded by player-k is {H1 , ..., Hk }. To show that long-memory player has no advantage over shortmemory player in terms of expected reward, it is sufficient [1] to show that the expectation of the joint probability
P (ai1 , ..., aiK |H1 , ..., HK ) of taking actions (ai1 , ..., aiK ) is the same as that of the probability P (ai1 , ..., aiK |H1 )
that corresponds to the case that all players can only recall H1 , which can be shown by induction as follows
11
EH1 ,...,HK [P (a1 , ..., aK |H1 , ..., HK )]
=
X
P (a1 , ..., aK |H1 , ..., HK )P (H1 , ..., HK )
H1 ,...,HK
=
X
P (a1 |H1 ) · · · P (aK |H1 , ..., HK )P (H1 , ..., HK )
H1 ,...,HK
=
X
H1 ,...,HK−1
=
X

P (a1 |H1 ) · · · P (aK−1 |H1 , ..., HK−1 ) · 
X
HK

P (aK |H1 , ..., HK )P (HK |H1 , ..., HK−1 ) · P (H1 , ..., HK−1 )
P (a1 |H1 ) · · · P (aK−1 |H1 , ..., HK−1 ) · P (aK |H1 , ..., HK−1 )P (H1 , ..., HK−1 )
H1 ,...,HK−1
= EH1 ,...,HK−1 [P (a1 , ..., aK |H1 , ..., HK−1 )] .
(23)
R EFERENCES
[1] W. H. Press and F. J. Dyson. Iterated prisoners dilemma contains strategies that dominate any evolutionary opponent. Proc. Natl Acad.
Sci.,, 109(26):10409–10413, 2012.
[2] A. Daoud, G. Kesidis, and J. Liebeherr. Zero-determinant strategies: A game-theoretic approach for sharing licensed spectrum bands.
IEEE J. Sel. Areas Commun., 32(11):2297–2308, Nov. 2014.
[3] C. Hilbe, B. Wu, A. Traulsen, and M. A. Nowak. Cooperation and control in multiplayer social dilemmas. Proceedings of the National
Academy of Sciences, 111(46):16425–16430, 2014.
[4] H. Zhang, F. Li, D. Niyato, L. Song, T. Jiang, and Z. Han. Zero-determinant strategy for power control of small cell network. In Proc.
IEEE ICCS, Macau, China, Nov. 2014.
[5] H. Zhang, N. Dusit, L. Song, T. Jiang, and Z. Han. Zero-determinant strategy in cheating management of wireless cooperation. In Proc.
IEEE GLOBECOM, Austin, TX, Dec. 2014.
[6] A. K. Farraj, E. M. Hammad, A. Daoud, and D. Kundur. A game-theoretic control approach to mitigate cyber switching attacks in smart
grid systems. In Proc. IEEE SmartGridComm, Venice, Italy, Nov. 2014.