1 Zero-determinant Strategies for Multi-player Multi-action Iterated Games Xiaofan He, Student Member, IEEE Huaiyu Dai, Senior Member, IEEE and Peng Ning, Senior Member, IEEE and Rudra Dutta, Member, IEEE Abstract Zero-determinant (ZD) strategies that allow a player to unilaterally control the linear combinations of its own and other players’ expected rewards in iterated games have recently found wide applications. However, existing ZD strategies mainly focus on some specific scenarios with restrictions on the number of players or actions a player can take. Targeting wider applications and better performance, the ZD strategies along with corresponding existence conditions for general multi-player multi-action iterated games are developed in this work, including existing ones as special cases. In addition, an interesting fact that every player can have at most one master player (that can control the expected reward of the given player) is revealed. Keywords: Zero-determinant, multi-player, multi-action, game theory. Copyright (c) 2015 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to [email protected]. X. He ([email protected]) and H. Dai (Huaiyu [email protected]) are with the Department of Electrical and Computer Engineering, North Carolina State University, NC, USA. P. Ning ([email protected]) and R. Dutta ([email protected]) are with the Department of Computer Science, North Carolina State University, NC, USA. This work was supported in part by the National Science Foundation under Grants CNS-1016260, ECCS-1307949, and EARS-1444009. 2 I. I NTRODUCTION The zero-determinant (ZD) strategy that enables a player to unilaterally set its own or another player’s expected reward, or a weighted combination of these two, is developed in the seminal work [1] for two-player two-action iterated games, and its extensions to the multi-player two-action and the two-player multi-action iterated games have been studied in [2, 3] and [4], respectively. However, applications of these ZD strategies may be limited in practice by the restrictions on the number of players or actions. Considering this, the ZD strategy for general multi-player multi-action iterated games is developed in this work, along with their existence conditions. Also, an important fact that every player can have at most one master player (that can control the expected reward of the given player) is revealed in this work. The proposed ZD strategy is expected to admit wider applications: Take the setting considered in [2] as an example where multiple users compete for a shared spectrum band. The existing ZD strategy applies only when admissible actions of the users are restricted to either transmitting with full power or keeping silent. Nonetheless, the ZD strategy proposed in this work can be applied to the more general cases where multiple transmission power levels are allowed. Other potential applications of the proposed ZD strategy include maintaining weighted social welfare in cooperative wireless communication networks [5], stabilizing frequency in smart grid [6], and providing quality-of-service in femto-cell networks [4]. II. G ENERAL K × N I TERATED G AMES A general K × N iterated game among K players where each player-k (1 ≤ k ≤ K ) has an action space Ak consisting of N actions is considered. Background on basic iterated games can be found in [1]. In the following, notations for the general K×N iterated game are introduced. Particularly, the tuple (ai1 , ..., aiK ) ∈ A , A1 ×...×AK will be used to represent the case that player-k takes the ik -th (ik ∈ {1, ..., N }) action aik ∈ Ak , where A is the set of all the N K possible action tuples. In addition, for ease of presentation, an index i ∈ I defined as i = 1 + PK k=1 (ik − 1)N K−k is assigned to each action-tuple (ai1 , ..., aiK ) ∈ A, with I , {1, ..., N K } the corresponding action-tuple index set. In the iterated game, the action-tuple (a′i1 , ..., a′iK ) of the previous round is defined as the current state si of the game, and the set of possible states is defined as S , A; the index i of a state 3 si ∈ S is given by that of the corresponding action-tuple (a′i1 , ..., a′iK ). At each round, players choose their actions with probabilities depending on the current state and hence the game states form a Markov chain.1 The strategy matrix of player-k is denoted by pk|S|×N = [pki,j ] with |S| the cardinality of S and pki,j the probability of taking action aj ∈ Ak at state si ∈ S . The reward vector of player-k is denoted by uk = [uk1 , ..., uk|S| ]T where uki is the reward corresponding to players’ current actions (ai1 , ..., aiK ) with i ∈ I the index of (ai1 , ..., aiK ). III. Z ERO - DETERMINANT S TRATEGIES FOR 2 × 2 G AMES When K = 2 and N = 2, the above general setting reduces to the 2 × 2 game considered in [1, 5, 6], where the number of states |S| = 4. For a given strategy pair p1 and p2 of the two players, the transition matrix M of the game state is given by 2 1 p1,1 p1,1 1 2 p2,1 p2,1 M= p1 p2 3,1 3,1 1 2 p4,1 p4,1 p11,1 p21,2 p11,2 p21,1 p11,2 p21,2 p12,1 p22,2 p12,2 p22,1 p12,2 p22,2 , 1 2 1 2 1 2 p3,1 p3,2 p3,2 p3,1 p3,2 p3,2 2 1 2 1 2 1 p4,1 p4,2 p4,2 p4,1 p4,2 p4,2 (1) with the entry Mi,j at the i-th row and the j -th column representing the probability that the next state is sj given current state si . The stationary distribution π of this Markov chain satisfies π T M = π T ,2 and the expected reward ūk of player-k in the stationary state is given by ūk = π T uk . (2) Define M̃ = M − I with I the identity matrix, it follows that π T M̃ = 0, 1 (3) To see that such a memory-one strategy (i.e., players only need to recall the actions played in the most recent round) is optimal, it is shown in Appendix A that there is no advantage of recalling longer history records for general K × N iterated games. 2 As in [1, 2, 4–6], it is assumed in this work that M has a unique stationary distribution, which is true when all actions are played with some probabilities greater than some arbitrarily small positive number ǫ. 4 and further by Cramer’s rule, we have3 Adj(M̃)M̃ = det(M̃)I = 0, (4) as M̃ has a zero eigenvalue implied by (3). By comparing (3) and (4), it can be noted that every row of Adj(M̃) is proportional to π , and hence for any vector f = [f1 , ..., f4 ]T and strategy pair p1 and p2 , it can be shown through some non-trivial derivations that [1] π T f ∝ D(p1 , p2 , f ) p11,1 p21,2 p11,2 p21,1 −1 + p11,1 p21,1 p1 p2 p12,2 p22,1 −1 + p12,1 p22,2 2,1 2,1 , det −1 + p13,2 p23,1 p13,1 p23,2 p13,1 p23,1 p14,2 p24,1 p14,1 p24,2 p14,1 p24,1 | {z ,M′f p11,1 p21,2 p11,2 p21,1 −1 + p11,1 −1 + p1 −1 + p1 p2 p12,2 p22,1 2,1 2,1 2,2 = det p13,1 −1 + p13,2 p23,1 p13,1 p23,2 p14,1 p14,1 p24,2 p14,2 p24,1 | {z ,M′′ f f1 f2 f3 f4 } f1 f2 , f3 f4 } (5) where the last equality holds since determinant remains unchanged if the second column of M′f are added to the first one. A key observation is that the first column of M′′f is solely under the control of player-1. Consider, in particular, f = α0 1 + α1 u1 + α2 u2 , (with 1 the all-one vector and αk ∈ R, for k = 0, 1, 2), then we have 3 α0 + α1 ū1 + α2 ū2 = π T α0 1 + α1 u1 + α2 u2 ∝ D(p1 , p2 , α0 1 + α1 u1 + α2 u2 ) = 0, In (4), Adj(·) denotes the adjugate matrix which is the transpose of the cofactor matrix. (6) 5 when p1 is such that the first column is proportional to the last column in the corresponding M′′f ; and such p1 is termed the ZD strategy of player-1 [1]. This implies that player-1 can control the weighted reward α1 ū1 + α2 ū2 irrespective to player-2’s strategy (if such p1 exists); and similar analysis can be applied to player-2 as well. As an example, the following result gives the ZD strategy of player-1 for controlling its own expected reward (corresponding to α1 = 1 and α2 = 0) and the corresponding existence conditions [1, 2]. Theorem 1: If ∃ n, n′ ∈ {1, 2} such that u1n′ ,max ≤ u1n,min , where u1n′ ,max (u1n,min ) is the maximum (minimum) reward of player-1 when he takes the n′ -th (n-th) action, then, irrespective to the strategy of player-2, any value ũ1 from the interval [u1n′ ,max , u1n,min ] can be achieved by player-1; without loss of generality, assume n = 1, and then the corresponding ZD strategies of player-1 are given by u11 = 1 + 1 − 1 α0 , ũ u12 1 p2,1 = 1 + 1 − 1 α0 , ũ 1 u 1 3 p3,1 = 1 − 1 α0 , ũ u14 1 p4,1 = 1 − 1 α0 , ũ p11,1 p11,2 = 1 − p11,1 , p12,2 = 1 − p12,1 , p13,2 = 1 − p13,1 , p14,2 = 1 − p14,1 , where α0 ∈ (0, min{ũ1 /(u1max − ũ1 ), ũ1 /(ũ1 − u1min )}) with u1max and u1min the maximum and the minimum possible rewards of player-1, respectively. IV. Z ERO - DETERMINANT S TRATEGIES FOR M ULTI - PLAYER M ULTI - ACTION G AMES In this section, the ZD strategies for the general K × N iterated games described in Section II will be presented, which include the ZD strategies in [1, 2, 4] as special cases. Also, the fact that each player can have at most one master player will be shown. A. ZD Strategy for General Iterated Games Proposition 1 below provides the existence condition and the specific form of the ZD strategies (for player-1) in general K × N iterated games. 6 Proposition 1: For a K -player N -action iterated game and a set of given coefficients {α1 , ..., αK }, if ∃ n ∈ {1, ..., N } such that uαn,min , min i∈I|i1 =n K X αk uki ≥ uαm6=n,max , max k=1 i∈I|i1 6=n K X αk uki , (7) k=1 where I|ik =n (I|ik 6=n ) denotes the set of action-tuples with player-k (not) taking the n-th action. Then, (1) the action index n (for player-1) is unique; (2) irrespective to the strategies of other players, player-1 can set the expected weighted social welfare to be any value ũα ∈ [uαm6=n,max , uαn,min ]4 by taking any strategy that satisfies ! K k X α u k i α0 , if i ∈ I|i1 =n 1+ 1− ũα k=1 p1si ,n = ! K k X α u k i α0 , otherwise, 1− ũα PK k k=1 αk ū (8) k=1 where 0 ≤ α0 ≤ min{ũα /(uαmax − ũα ), ũα /(ũα − uαmin )}, with uαmax , maxi∈I PK k k=1 αk ui and uαmin , mini∈I weighted social welfares, respectively.5 PK k k=1 αk ui , (9) representing the maximum and the minimum Remark 1: Note that (8) does not specify the probabilities of taking actions {am } (m 6= n) at each state and these probabilities can be arbitrary as long as PN 1 e=1 psi ,e = 1 for all states. Remark 2 (Numerical result): Consider a scenario where a spectrum band is shared by K = 3 users each having N = 3 different transmission powers Ak = {0, 0.1, 1}, 1 ≤ k ≤ 3. Our numerical results show that a user can always find a ZD strategy by (8) to fix its average SINR to a value in the range [0, 0.25] (assuming unit channel power gain and noise power). Proof: Given existence, the uniqueness of the action index n can be proved by contradiction using (7). 4 Note that a player may be able to remove some actions from its action set (and then add duplicated actions back to keep the total number α of actions N unchanged) to minimize uα m6=n,max , so as to obtain a larger feasible range for ũ . 5 In Proposition 1, it is assumed without loss of generality that all rewards are positive, since the desired ZD strategy is invariant to a positive shift in reward functions. 7 p11,1 p21,1 ...pK 1,1 p11,1 p21,1 ...pK 1,2 1 2 p2,1 p2,1 ...pK p12,1 p22,1 ...pK 2,1 2,2 M= .. .. . . 1 1 2 K p|S|,1 p2|S|,1 ...pK |S|,1 p|S|,1 p|S|,1 ...p|S|,2 p11,1 p21,1 ...pK 1,1 −1 + p12,1 p22,1 ...pK 2,1 T 1 K π f = D(p , ..., p , f ) = det .. . 1 p|S|,1 p2|S|,1 ...pK |S|,1 ... p11,1 p21,N ...pK 1,N ... p12,1 p22,N ...pK 2,N ... .. . ... ... . ... ... ... p1|S|,1 p2|S|,N ...pK |S|,N p11,1 p21,1 ...pK 1,2 ... p11,1 p21,N ...pK 1,N − 1 + p12,1 p22,1 ...pK 2,2 ... p12,1 p22,N ...pK 2,N .. . p11,1 p21,1 ...pK −1 + p11,1 1,2 −1 + p12,1 −1 + p12,1 p22,1 ...pK 2,2 . . .. .. = det −1 + p1N K−1 ,1 p1N K−1 ,1 p2N K−1 ,1 ...pK N K−1 ,2 1 pN K−1 +1,1 p1N K−1 +1,1 p2N K−1 +1,1 ...pK N K−1 +1,2 .. .. . . p1|S|,1 p1|S|,1 p2|S|,1 ...pK |S|,2 | ... .. . ... p1|S|,1 p2|S|,1 ...pK |S|,2 (10) ... p1|S|,1 p2|S|,N ...pK |S|,N ... p11,1 p21,N ...pK 1,N ... ... p12,1 p22,N ...pK 2,N ... ... .. . ... ... −1 + p1N K−1 ,1 p2N K−1 ,N ...pK N K−1 ,N ... ... p1N K−1 +1,1 p2N K−1 +1,N ...pK N K−1 +1,N ... ... .. . ... ... p1|S|,1 p2|S|,N ...pK |S|,N ... {z M′′′ f f1 ... f2 .. ... . ... f|S| f1 f2 .. . . fN K−1 fN K−1 +1 .. . f|S| } (11) As to the second part of the proposition, it can be first verified that (8) defines valid probability measures, i.e., p1si ,n ∈ [0, 1] for all i ∈ I , when (7) and (9) hold. To show that the chosen expected weighted social welfare ũα can be achieved irrespective to other players’ actions, assume without loss of generality n = 1. Further assuming that the other players take arbitrary strategies {pk } (for k = 2, ..., K ), the state transition matrix M of the corresponding iterated game is given by (10). Using similar arguments as in (3)–(5), it can be shown that, for any |S| × 1 vector f , (11) holds. It is worth noting that the first column of the matrix M′′′ f in (11) is solely under player-1’s control. 8 When f = −ũα 1 + PK k=1 αk u k and p1 satisfies (8), it can be verified that the first column is proportional to the last column (with ratio −ũα /α0 ) in M′′′ f , resulting in a zero determinant in (11). Therefore, it has α α ũ = − −ũ + K X αk ū k k=1 =− T α π (−ũ 1 + ! K X + K X αk ūk k=1 k αk u ) k=1 | {z + K X αk ūk , (12) k=1 } =0 due to zero determinant in (11) where the second equality is due to the fact that π T · 1 = 1 and the definition of ūk given in (2). Setting α1 = 1 and αk = 0 for k > 1, Proposition 1 gives the following corollary, which specifies how player-1 can set its own expected reward irrespective to other players’ strategies. Corollary 1: For a K -player N -action iterated game, if ∃ n ∈ {1, ..., N } such that u1n,min , min u1i ≥ u1m6=n,max , max u1i , i∈I|i1 =n i∈I|i1 6=n (13) then (1) n is unique; (2) irrespective to the strategies of other players, any value ũ1 from the interval [u1m6=n,max , u1n,min ] can be achieved by player-1 using any strategy that satisfies 1 + 1 − u1i /ũ1 α0 , if s ∈ I|i1 =n p1si ,n = 1 − u1i /ũ1 α0 , otherwise (14) where α0 ∈ (0, min{ũ1 /(u1max − ũ1 ), ũ1 /(ũ1 − u1min )}) with u1max and u1min denoting the maximum and the minimum possible rewards of player-1, respectively. Remark 3: Note that the ZD strategy presented in [2] generalizes Theorem 1 to the cases of K > 2 and is included by Corollary 1 here as a special case of N = 2. Similarly, player-1 can set the expected reward of player-k (k 6= 1) based on the following corollary. Corollary 2: For a K -player N -action iterated game, if ∃ n ∈ {1, ..., N } such that ukn,min , min uki ≥ ukm6=n,max , max uki , i∈I|i1 =n i∈I|i1 6=n (15) then (1) n is unique; (2) irrespective to the strategies of other players (including player-k), any value ũk from the 9 interval [ukm6=n,max , ukn,min ] can be enforced by player-1 using any strategy that satisfies 1 + 1 − uki /ũk α0 , if s ∈ I|i1 =n p1si ,n = 1 − uk /ũk α , otherwise (16) 0 i where α0 ∈ (0, min{ũk /(ukmax − ũk ), ũk /(ũk − ukmin )}) with ukmax and ukmin denoting the maximum and the minimum possible rewards of player-k, respectively. Remark 4: Corollary 2 includes the ZD strategy presented in [4] as a special case of K = 2. B. Uniqueness of Master Player Seeing Corollary 1 and Corollary 2, a natural question is that what will happen when two players set different values on the expected reward for the same player? Regarding this, it will be shown in the following that such situation can never happen, except for the degenerated case. Definition 1: For a given player, its master player is the one that has a valid ZD strategy to control the expected reward of this given player. More specifically, player-k′ is the master player of player-k if and only if ∃ n′ ∈ {1, ..., N } such that ukn′ ,min , min i∈I|ik′ =n′ uki ≥ ukm6=n′ ,max , max i∈I|ik′ 6=n′ uki . (17) Proposition 2: Each player k cannot have two different master players k′ and k′′ , except for the degenerated case where equality holds in (17) for both k′ and k′′ (with possibly a different action index n′′ ) and both players k′ and k′′ set the same expected reward ukn′ ,min for player-k. Proof: Suppose by contradiction that player-k has two different master players k′ and k′′ in non-degenerated cases. Then, ∃ n′ , n′′ ∈ {1, ..., N } such that min uki ≥ max uki , i∈Iik′ =n′ min i∈Iik′′ =n′′ i∈Iik′ 6=n′ uki ≥ max i∈Iik′′ 6=n′′ uki , (18) (19) and at least one of (18) and (19) admits strict inequality. Without loss of generality, assume (19) admits strict inequality. 10 Notice that, by (18), it has min uki = max uki = i∈Iik′′ =n′′ i∈Ii k ′′ ′′ 6=n min uki , (20) max uki , (21) i∈Iik′′ =n′′ ,ik′ 6=n′ i∈Ii 6=n′′ ,i ′ =n′ k′′ k where Iik′′ =n′′ ,ik′ 6=n′ (Iik′′ 6=n′′ ,ik′ =n′ ) denotes the set of indices such that ik′′ = n′′ and ik′ 6= n′ (ik′′ 6= n′′ and ik′ = n′ ) in the corresponding action-tuples. However, (19–21) imply max uki ≥ i∈Ii 6=n k′ = min i∈Ii k′′ ≥ ′ i∈Ii =n′′ max i∈Ii uki > min ≥n′′ ,i ′ =n′ k′′ k =n′′ ,i ′ 6=n′ k′′ k uki ≥ max i∈Ii k′′ uki ≥ uki ≥ 6 n′′ = i∈Ii i∈Ii min =n′′ ,i ′ 6=n′ k′′ k max 6=n′′ ,i ′ =n′ k′′ k uki uki min uki , i∈Ii k′ (22) =n′ which contradicts to (18). V. C ONCLUSIONS The ZD strategy for general multi-player multi-action iterated games is developed in this work, and the fact that every player can have at most one master player is revealed. The proposed strategy can extend existing ZD strategy based schemes to more general settings (e.g., more players and more actions) and may also find wider applications in the future. A PPENDIX A N O A DVANTAGE FOR L ONG - MEMORY Proof: Denote Hi as the recent history (of past actions) shared by players whose index k ≥ i, and accordingly the history recorded by player-k is {H1 , ..., Hk }. To show that long-memory player has no advantage over shortmemory player in terms of expected reward, it is sufficient [1] to show that the expectation of the joint probability P (ai1 , ..., aiK |H1 , ..., HK ) of taking actions (ai1 , ..., aiK ) is the same as that of the probability P (ai1 , ..., aiK |H1 ) that corresponds to the case that all players can only recall H1 , which can be shown by induction as follows 11 EH1 ,...,HK [P (a1 , ..., aK |H1 , ..., HK )] = X P (a1 , ..., aK |H1 , ..., HK )P (H1 , ..., HK ) H1 ,...,HK = X P (a1 |H1 ) · · · P (aK |H1 , ..., HK )P (H1 , ..., HK ) H1 ,...,HK = X H1 ,...,HK−1 = X P (a1 |H1 ) · · · P (aK−1 |H1 , ..., HK−1 ) · X HK P (aK |H1 , ..., HK )P (HK |H1 , ..., HK−1 ) · P (H1 , ..., HK−1 ) P (a1 |H1 ) · · · P (aK−1 |H1 , ..., HK−1 ) · P (aK |H1 , ..., HK−1 )P (H1 , ..., HK−1 ) H1 ,...,HK−1 = EH1 ,...,HK−1 [P (a1 , ..., aK |H1 , ..., HK−1 )] . (23) R EFERENCES [1] W. H. Press and F. J. Dyson. Iterated prisoners dilemma contains strategies that dominate any evolutionary opponent. Proc. Natl Acad. Sci.,, 109(26):10409–10413, 2012. [2] A. Daoud, G. Kesidis, and J. Liebeherr. Zero-determinant strategies: A game-theoretic approach for sharing licensed spectrum bands. IEEE J. Sel. Areas Commun., 32(11):2297–2308, Nov. 2014. [3] C. Hilbe, B. Wu, A. Traulsen, and M. A. Nowak. Cooperation and control in multiplayer social dilemmas. Proceedings of the National Academy of Sciences, 111(46):16425–16430, 2014. [4] H. Zhang, F. Li, D. Niyato, L. Song, T. Jiang, and Z. Han. Zero-determinant strategy for power control of small cell network. In Proc. IEEE ICCS, Macau, China, Nov. 2014. [5] H. Zhang, N. Dusit, L. Song, T. Jiang, and Z. Han. Zero-determinant strategy in cheating management of wireless cooperation. In Proc. IEEE GLOBECOM, Austin, TX, Dec. 2014. [6] A. K. Farraj, E. M. Hammad, A. Daoud, and D. Kundur. A game-theoretic control approach to mitigate cyber switching attacks in smart grid systems. In Proc. IEEE SmartGridComm, Venice, Italy, Nov. 2014.
© Copyright 2026 Paperzz