Int J Game Theory (2000) 29:309±325
2000
9
99
9
Two-person repeated games with ®nite automata
Abraham Neyman1, Daijiro Okada2
1 Institute of Mathematics, The Hebrew University of Jerusalem, Givat Ram, Jerusalem 91904,
ISRAEL and SUNY at Stony Brook, Stony Brook, NY 11794-4384, USA
(email: [email protected])
2 Department of Economics, SUNY at Stony Brook, Stony Brook, NY 11794-4384, USA
(email: [email protected])
Received February 1997/revised version March 2000
Abstract. We study two-person repeated games in which a player with a restricted set of strategies plays against an unrestricted player. An exogenously
given bound on the complexity of strategies, which is measured by the size
of the smallest automata that implement them, gives rise to a restriction on
strategies available to a player.
We examine the asymptotic behavior of the set of equilibrium payo¨s as
the bound on the strategic complexity of the restricted player tends to in®nity,
but su½ciently slowly. Results from the study of zero sum case provide the
individually rational payo¨ levels.
JEL classi®cation: C73, C72.
Key words: repeated games, ®nite automata
1. Introduction
The objects of this study is two-person non-zero sum repeated games in which
there is a bound on the complexity of strategies for only one of the players.
Throughout the paper player 1 will be the restricted player. We employ automata to represent repeated game strategies. The complexity of a strategy is
then de®ned to be the smallest number of states of an automaton required to
implement it.
Speci®c models of repeated games studied are (1) The ®nitely repeated
game G n …m…n†† and (2) The l-discounted game Gl …m…l††. Here, m…† is the
bound on the number of states of automata available to player 1 and this is a
function of the number of repetitions, n, or the discount factor, l. We examine
the set of limit points of equilibrium payo¨s of G n …m…n†† (resp. Gl …m…l††) as
310
A. Neyman, D. Okada
m…n† ! y (n ! y) (resp. m…l† ! y (l ! 1)). The particular case considered in this paper is when m…† grows ``su½ciently slowly''1 Formally, we will
examine the cases when
lim
n!y
m…n† log m…n†
ˆ 0;
n
…1:1†
and
lim…1
l!1
l†m…l† log m…l† ˆ 0:
…1:2†
We will show that, under these conditions on m…†, the Hausdor¨ limit of the
set of equilibrium payo¨s, Limn!y E…m…n†† and Liml!1 E…m…l††, exist and
they coincide with the set of the feasible payo¨s above certain individually
rational levels. For player 1 this level will be his maxmin payo¨ in the oneshot game where max ranges over his pure actions and min ranges over player
2's pure actions, and for player 2 it will be her minmax of the one-shot game
where min ranges over player 1's pure actions and max ranges over her own
pure actions. The determination of these individually rational levels, under
the conditions (1.1) and (1.2), can be provided by our analysis of the zero sum
case, Neyman and Okada (1999). In this paper, however, we will explicitly
construct player 2's strategy that e¨ectively punishes the restricted player 1.
This will provide a simpler proof of a result originally proved in Neyman and
Okada (1999) using the concept of entropy. See also Neyman and Okada
(2000) for an alternative proof.
It will be seen that the equilibria that we construct are in pure strategies
and the equilibrium paths are cyclic. Our result for ®nitely repeated games
implies in particular that, in ®nitely repeated prisoner's dilemma, the friendly,
or nearly friendly, outcomes can be achieved in an equilibrium when there is a
bound on the strategic complexity on only one of the players. In addition,
Folk-Theorem type results like ours have an implication that in non-zero sum
games, being restricted in terms of strategic possibility is not necessarily detrimental even against a powerful unrestricted player. Only the punishment
will be severe for the restricted player.
Related literature includes Neyman (1999) and Papadimitriou and Yannakakis (1994) which contain several results on the asymptotic behavior of
the set of equilibrium payo¨s of two-person ®nitely repeated games when
there are bounds on the strategic complexity for both players. These results
encompass Neyman (1985)'s justi®cation of cooperation in ®nitely repeated
prisoners' dilemma. For example, the main theorem of Neyman (1999) states
that if the two bounds on the size of automata are subexponential as a function of minfthe number of repetitions; the larger boundg, then the asymptotic
folk theorem is obtained. More precisely, let G be a two-person game in strategic form. Denote by v i the minmax payo¨ for player i where min ranges
over the other player's mixed actions and max ranges over i's own pure ac1 For example, the condition (1.1) holds for all functions m…n† ˆ n a where 0 < a < 1, while it is
violated for m…n† ˆ n. In addition, the function m…n† ˆ n=log n, for which m…n†=n ! 0 …n ! y†
but m…n†=n a ! y …n ! y† for all 0 < a < 1, violates (1.1). The authors thank a referee for this
comment.
Two-person repeated games with ®nite automata
311
tions. Let G n …m1 …n†; m2 …n†† be the n-fold repetitions of G in which player i's
strategies are restricted to those implementable by automata of size at most
mi …n†, a function of n. If the sequence of triples …n; m1 …n†; m2 …n††y
nˆ1 satis®es
the conditions minfm1 …n†; m2 …n†g ! y …n ! y† and
lim
n!y
log…maxfm1 …n†; m2 …n†g†
ˆ 0;
minfn; m1 …n†; m2 …n†g
then the set of equilibrium payo¨ vectors of G n …m1 …n†; m2 …n†† converges to the
set of payo¨ vectors which are feasible and give player i at least v i . Zemel
(1989) contains results similar to Neyman (1985) but using modi®ed ®nite
automata which can send messages, in addition to those conveyed through the
actions taken, during the play.
Similar results have been obtained for other classes of repeated games.
Ben-Porath (1993) studies the undiscounted in®nitely repeated games with
N
N
; …ri †iˆ1
† be an N-person game in strategic
®nite automata. Let G ˆ ……Ai †iˆ1
form, and
vi ˆ
min
max Eq …r…a i ; b††
q A j0i D…A j † a i A A i
and
w i ˆ max
min
p A D…A i † b A j0i A i
Ep …r…a i ; b††
where D…X † denotes the set of probability distributions on a set X and Em denotes the expectation with respect to the probability m. Consider the in®nitely
repeated game in which player i has a complexity bound mi …k†, parameterized
by positive integer k, with m1 …k† U U mN …k† and m1 …k† ! y…k ! y†.
Denote this game by Gy …m1 …k†; . . . ; mN …k†† and the set of its equilibrium
payo¨s by E y …m1 …k†; . . . ; mN …k††. One of his results asserts that if
lim
k!y
log mN …k†
ˆ 0;
m1 …k†
then,
(i) the set of feasible payo¨s which give each player i at least v i is included in
lim inf k!y E y …m1 …k†; . . . ; mN …k†† and
(ii) lim supk!y E y …m1 …k†; . . . ; mN …k†† is included in the set of feasible payo¨s which give each player i at least w i .
Note that v i V w i . For two-person games, we have v i ˆ w i . Hence one can
conclude that limk!y E y …m1 …k†; m2 …k†† exists and it coincides with the set of
equilibrium payo¨s of the in®nitely repeated game without complexity bound
(the Folk Theorem). This result crucially depends on the study of the twoperson zero sum case which provides the individually rational levels v i and w i .
The exact asymptotics, i.e., the limit, if exists, of E y …m1 …k†; . . . ; mN …k†† for Nperson case is not known. See Section 4 of Neyman (1997). Lehrer (1988)
312
A. Neyman, D. Okada
contains a similar result for two-person games with bounded recall. Also see
Lehrer (1994) for N-person case with bounded recall.
The main contribution of this paper is a determination of individually
rational payo¨ levels together with the construction of equilibria which have
certain robustness properties and can be applied to a wider variety of conditions on the order of magnitude of the complexity bound m…†.
The next section introduces the model of repeated games and ®nite automata. In Section 3 we will construct player 2's strategy which will be used to
punish player 1 in equilibria constructed in the subsequent chapters. The results on the asymptotics of the set of equilibrium payo¨ vectors are presented
in Section 4 (the ®nitely repeated games) and Section 5 (the discounted games).
Section 6 concludes the paper.
2. Repeated games and automata
Let G ˆ …A; B; h; k† be a two-person game in strategic form where A and B are
®nite sets of actions, and, h : A B ! R and k : A B ! R are the payo¨
functions of player 1 and 2, respectively. We call G the stage game. Throughout the paper we will assume without loss of generality that all payo¨s are
nonnegative, i.e., h…a; b† V 0 and k…a; b† V 0 for all …a; b† A A B. Denote the
maxmin value of the stage game for player 1 in pure actions by h and the
minimax value for player 2 in pure actions by k , i.e.,
h ˆ max min h…a; b†
aAA bAB
and
k ˆ min max k…a; b†:
aAA bAB
Also set khk ˆ maxa; b jh…a; b†j and kkk ˆ maxa; b jk…a; b†j.
Given G ˆ …A; B; h; k† we next describe a new game in which G is played
repeatedly (with complete information and standard signaling).
For each positive integer n, let Sn (resp. Tn ) be the set of mappings from
…A B† n 1 to A (resp. to B) where …A B† 0 ˆ ffg. A pure strategy of player
1 (resp. player 2) is an element of S ˆ n Sn (resp. T ˆ n Tn ). Equivalently,
S (resp. T ) is the set of all mappings on the set of all ®nite histories
6nV1 …A B† n 1 to A (resp. B). A mixed strategy of player 1 (resp. player 2)
is a probability distribution on S (resp. T ). The sets of mixed strategies are
denoted by D…S† and D…T†.
Every pair of pure strategies …s; t† induces a play o…s; t† ˆ …ol …s; t††y
lˆ1 A
…A B†y where ol …s; t† is de®ned inductively as
for l ˆ 1
…s1 …f†; t1 …f††
ol …s; t† ˆ …al ; bl † ˆ
…sl …o1 ; . . . ; ol 1 †; tl …o1 ; . . . ; ol 1 †† for l > 1
Accordingly, every pair …s; t† of mixed strategies induces a random play
o…s; t† ˆ …ol …s; t††y
lˆ1 . We denote the corresponding probability distribution
on the set of plays …A B†y by Ps; t and the expectation with respect to Ps; t
by Es; t .
For each positive integer n we de®ne thePn-average payo¨ function of
n
h…ol …s; t††. Also, for each
player 1, hn : S T ! R, by hn …s; t† ˆ …1=n† lˆ1
l A ‰0; 1† we de®ne the l-discounted
payo¨
function
of player 1, hl : S T !
P
l 1
l
h…o
…s;
t††.
The
n-average and the lR by hl …s; t† ˆ …1 l† y
l
lˆ1
discounted payo¨ functions of player 2, kn and kl , are similarly de®ned. The
Two-person repeated games with ®nite automata
313
bilinear extensions of hn , hl , kn and kl to D…S† D…T†
denoted by the
Pare
n
h…al ; bl †e.
same symbols. Thus, for example, hn …s; t† ˆ Es; t d…1=n† lˆ1
In this paper we study two classes of repeated games di¨erentiated by their
payo¨ functions.
Finitely Repeated Game G n ˆ …S; T; hn ; kn †
The l-Discounted Game Gl ˆ …S; T; hl ; kl †
If two pure strategies of a player induce the same play against any pure
strategy of the other player, they are said to be equivalent. For example,
player 1's pure strategies s and s 0 are equivalent if ol …s; t† ˆ ol …s 0 ; t† for all
pure strategy t of player 2 and all stages l ˆ 1; 2; . . . . Extending this notion
to mixed strategies, we say that two strategies of a player are equivalent if,
against any strategy of the other player, they induce the same probability over
the plays of a repeated game.
Given the stage game G ˆ …A; B; h; k†, an automaton of player 1 is de®ned
by a four-tuple M ˆ hQ; q1 ; f ; gi. The ®rst component Q is a set of states,
and q1 A Q is an initial state. The third component is an action function,
f : Q ! A, and the last component is a transition function, g : Q B ! Q. By
the size of an automaton we mean the cardinality of the set of its states, jQj.
An automaton M plays a repeated game as follows. At each stage n it
takes an action prescribed by f for the current state, say qn , i.e., f …qn †; it is
set for q1 at the ®rst stage. Then it changes its state to qn‡1 speci®ed by g
as a function of the current state qn and player 2's action bn , that is, qn‡1 ˆ
g…qn ; bn †.
Every automaton M induces a pure strategy s for player 1 in a repeated
game in the following manner. First, for any sequence of player 2's actions
b1 ; . . . ; bn …n V 2†, de®ne an extension of the transition function inductively by
g…q; b1 ; . . . ; bn † ˆ g…g…q; b1 ; . . . ; bn 1 †; bn †:
Then for any history o ˆ ……a1 ; b1 †; . . . ; …an ; bn †† A …A B† n , set
s…o† ˆ f …g…q; b1 ; . . . ; bn ††
which is an action taken at stage n ‡ 1 …n V 1†. At the ®rst stage, s…f† ˆ
f …q1 †.
Also, for every pure strategy s A S in a repeated game, there is an automaton that induces an equivalent strategy. If s is equivalent to a pure strategy
induced by an automaton, we say that s is implementable by that automaton.
The size of the smallest automaton that implements a pure strategy serves
as a measure of complexity of that strategy. To be more precise, for a given
s ˆ …sn † A S, we say that a ®nite history o ˆ ……a1 ; b1 †; . . . ; …al ; bl †† is compatible with s if an ˆ sn ……a1 ; b1 †; . . . ; …an 1 ; bn 1 †† for every n ˆ 1; . . . ; l. Also, for
an arbitrary ®nite history o of length l, de®ne the induced strategy sjo ˆ
……sjo†n † by …sjo†n …o 0 † ˆ sl‡n …oo 0 † for each o 0 A …A B† n 1 where oo 0 is the
concatenation of o and o 0 . The number of distinct, or nonequivalent, strategies induced by s and ®nite histories compatible with s can be considered
as a measure of complexity of implementing s. Indeed, it can be shown that
314
A. Neyman, D. Okada
the size of the smallest automaton that implements s equals the number of
the equivalence classes of f…sjo†jo is a finite history compatible with sg. Kalai
and Stanford (1988) provides an analogous result for the full automata whose
transition depends on the player's own action as well as the actions of the
other players.
Henceforth, by the complexity of a pure strategy s, we mean the size of the
smallest automaton that implements s. For each positive integer m, we denote
by S…m† the subset of S consisting of those pure strategies of player 1 whose
complexity is at most m.
3. Individually rational payo¨ levels
We will present in this section a result which will be utilized in deriving much
of the subsequent results. The situation under consideration is the one in
which player 1 is restricted to a ®nite set of pure strategies. The nature of this
set is arbitrary. In particular, it may contain pure strategies which cannot be
implemented by any ®nite automata.
Theorem 3.1. For every ®nite subset S 0 of S there exists t^ A T such that for all
s A S0
(i)
hn …s; t^† U h ‡
khk log2 jS 0 j
n
for all n ˆ 1; 2; . . . ;
and
(ii) hl …s; t^† U h ‡ …1
l†khk log2 jS 0 j
for all l A ‰0; 1†:
Proof: For each ®nite history o ˆ …o1 ; . . . ; ol †, where oj ˆ …aj ; bj †, let S 0 …o†
be the set of strategies in S 0 that are compatible with o, i.e.,
S 0 …o† ˆ fs A S 0 j s…q† ˆ a1 ; and s…o1 ; . . . ; oj 1 † ˆ aj for all j ˆ 2; . . . ; lg:
For each a A A let S 0 …o; a† be the set of strategies in S 0 …o† that takes the
action a at the history o, i.e.,
S 0 …o; a† ˆ fs A S 0 …o† j s…o† ˆ ag:
Clearly, if a 0 a 0 , then S 0 …o; a† and S 0 …o; a 0 † are disjoint, and 6a A A S 0 …o; a†
ˆ S 0 …o†.
Let a…o† be an action of player 1 such that jS 0 …o; a…o††j V jS 0 …o; a†j for
all a A A. Notice that if a 0 a…o†, then jS 0 …o; a†j is at most one half of jS 0 …o†j:
otherwise, jS 0 …o; a…o††j ‡ jS 0 …o; a†j V 2jS 0 …o; a†j > jS 0 …o†j, a contradiction.
This implies that for every …a; b† A A B with a 0 a…o†, if o 0 ˆ …o1 ; . . . ; ol ;
…a; b††, then
jS 0 …o 0 †j U
jS 0 …o†j
:
2
De®ne t^ A T by t^…o† A argminb A B h…a…o†; b†. Take s A S 0 arbitrarily and
let …o1 ; o2 ; . . .† be the play generated by …s; t^†, oj ˆ …aj ; bj †. Denote o l ˆ
Two-person repeated games with ®nite automata
315
…o1 ; . . . ; ol †. Of course, s is compatible with o l for every l, i.e., s A S…o l †.
Therefore for all n,
Pn
I …al 0a…o l 1 ††
lˆ1
V jS…o n †j V 1
jS 0 j2
Pn
where I is the indicator function. This implies that lˆ1
I …al 0 a…ol 1 †† U
0
log2 jS j, that is, the number of stages at which player 1's action di¨ers from
a…o l † is at most log2 jS 0 j.
; y2 ; . . .† be a nonincreasing sequence of nonnegative
Now let y ˆ …y1P
Pyreal
numbers such that y
lˆ1 yl ˆ 1. De®ne hy : S T ! R by hy …s; t† ˆ
lˆ1 yl h…ol …s; t††. Take s A S 0 . Then, since
h…ol † U h I …al ˆ a…ol 1 †† ‡ khkI …al 0 a…ol 1 ††
for every l ˆ 1; 2; . . . ; we have
hy …s; t^† U
y
X
yl …h I …al ˆ a…ol 1 †† ‡ khkI …al 0 a…ol 1 †††
lˆ1
U h ‡ khk
y
X
yl I …al 0 a…ol 1 ††
lˆ1
U h ‡ khk
y
X
y1 I …al 0 a…ol 1 ††
lˆ1
U h ‡ y1 khk log2 jS 0 j:
(Recall our assumption h…a; b† V 0.) Note that if yl ˆ 1=n for l ˆ 1; . . . ; n
and yl ˆ 0 for l > n, then hy ˆ hn . Hence (i). Also, if yl ˆ …1 l†ll 1 , l ˆ
Q.E.D.
1; 2; . . . ; then hy ˆ hl . This proves (ii).
Remark 3.1: If s and s 0 in S 0 are equivalent, then for every ®nite history o and
every action a A A, either both s and s 0 are in S 0 …o; a† or neither is in S 0 …o; a†.
Therefore one can replace logjS 0 j in the statement of Theorem 3.1 by
logjS 0 =@j where S 0 =@ is the set of equivalence classes of S 0 .
Remark 3.2: Let S1 ; S2 ; . . . be a nondecreasing sequence of ®nite subsets of S.
If logjSn j=n ! 0 as n ! y, then Theorem 3.1 (i) implies that for every e > 0,
there is n0 such that for each n V n0 there is t A T for which maxs A Sn hn …s; t† U
h ‡ e. Similar result is obtained from Theorem 3.1 (ii) for l-discounted payo¨
by replacing the sequence S1 ; S2 ; . . . by a net …Sl j0 U l < 1† and the condition
logjSn j=n ! 0 as n ! y by …1 l† logjSl j ! 0 as l ! 1.
4. Finitely repeated game G n …m…n††
In this section we study the modi®ed version of the ®nitely repeated game,
G n …m…n†† ˆ …S…m…n††; T; hn ; kn †. The bound on the complexity of player 1's
316
A. Neyman, D. Okada
strategy, m…†, is a function of the number of repetitions n. Player 1 is allowed
to use a mixed strategy provided that its support lies in S…m…n††. He can also
use a behavioral strategy s : 6lV0 …A B†l ! D…A† as long as it is equivalent
to a mixed strategy with the support in S…m…n††.
A simple counting shows that the number of ®nite automata of size m is at
most mCm for some positive constant C. Thus the number of equivalence
classes of S…m† is also bounded by mCm . The next lemma follows from Theorem 3.1 (i), Remark 3.1 and Remark 3.2.
Lemma 4.1. Suppose that m…n† log m…n†=n ! 0 as n ! y. Then for every
e > 0, there is n0 such that for each n > n0 there is t A T such that
hn …s; t† U h ‡ e
for all s A S…m…n††:
Remark 4.1: As an immediate corollary of this lemma, we obtain the following result concerning the asymptotics of the value of two-person zero sum
repeated games with ®nite automata which was proved in Neyman and Okada
(1998a).2 Consider G ˆ …A; B; h; k† to be a two-person zero sum game, i.e.,
k ˆ h. Denote the value of G n …m…n†† by V n …m…n††.
Corollary 4.1. If m…n† log m…n†=n ! 0 as n ! y, then V n …m…n†† ! h as
n ! y.
Denote by E n …m…n†† the set of (Nash) equilibrium payo¨ vectors of
G …m…n††. The next theorem provides an asymptotics of E n …m…n†† when m…n†
grows su½ciently slowly. The convergence of sets is with respect to the Hausdor¨ topology in R 2 . To state the theorem formally we need some more notation. Let F be the convex hull of the set of payo¨s feasible in pure actions of
the stage game, that is, F ˆ Cof…h…a; b†; k…a; b†† j …a; b† A A Bg; and let
n
F~ ˆ f…x; y† A F j x V h ; y V k g:
The set F~ is nonempty. For example, let a A argmaxa A A ‰minb A B h…a; b†Š
and b A argmaxb A B k…a ; b†. Then it is easily seen that h…a ; b † V h and
k…a ; b † V k . Note that the point …h ; k † does not necessarily belong to
F and thus it may not belong to F~. For example, for the 2 2 stage game
L
R
T
0; 0
1; 2
B
2; 1
0; 0
we have …h ; k † ˆ …0; 1† and F ˆ Cof…0; 0†; …1; 2†; …2; 1†g. So …h ; k † B F .
…F~ ˆ Cof…12 ; 1†; …1; 2†; …2; 1†g† See Figure 1.
Theorem 4.1. If m…n† ! y, m…n† log m…n†=n ! 0 as n ! y, and if there is
…x; y† A F~ with x > h , then E n …m…n†† ! F~ as n ! y.
2 The previous proof utilized the notion of entropy.
Two-person repeated games with ®nite automata
317
Fig. 1. …h ; k † may not be in F
As demonstrated in the next example, the conclusion of the theorem fails if
the condition on F~ is not satis®ed.
Example (Neyman (1999)). Consider the 2 2 stage game given below.
L
R
T
1; 3
0; 4
B
1; 1
1; 0
Observe that h ˆ k ˆ 1 and F~ ˆ f…1; y† j 1 U y U 3g. For this game
E …m…n†† ˆ f…1; 1†g for every n regardless of m…n†. To see this ®rst note that
player 1 must receive 1 at every stage in any equilibrium path, and he can
guarantee 1 with an automaton of size 1 (play B at every stage).
Suppose that, in some equilibrium …s; t† of G n …m…n††, player 2 received
more than 1. Then …T; L† must be played with a positive probability at some
stage on the equilibrium path. Let n~ be the last stage at which …T; L† is played
with a positive probability, i.e.,
n
n~ ˆ maxfn 0 j 1 U n 0 U n; Ps; t ……an 0 ; bn 0 † ˆ …T; R†† > 0g:
De®ne t~ ˆ …~
tl † A D…T† as
8
< tl …o† for 1 U l U n~
t~l …o† ˆ R
for l ˆ n~
:
L
for l > n~
1
Then it is easily veri®ed that kn …s; ~t† > kn …s; t†, contradicting to the supposition that …s; t† is an equilibrium. r
We now turn to the proof of Theorem 4.1. Given a point z A R 2 and a
nonempty compact set Z H R 2 , de®ne d…z; Z† ˆ minz 0 A Z kz z 0 k. Since F~
and E n …m…n†† are nonempty compact subsets of F which is compact, the
318
A. Neyman, D. Okada
conclusion of the theorem, E n …m…n†† ! F~, is equivalent to
Lim E n …m…n†† ˆ Lim E n …m…n†† ˆ F~
n!y
n!y
where
Lim E n …m…n†† ˆ fzjEe > 0; En 0 ; bn V n 0 such that d…z; E n …m…n††† < eg;
n!y
and
Lim E n …m…n†† ˆ fzjEe > 0; bn0 such that En V n0 ; d…z; E n …m…n††† < eg:
n!y
We will establish the identity of the three sets through a pair of claims. Note
that the ®rst claim requires neither m…n† ! y …n ! y† nor the condition on
F~ present in the statement of Theorem 4.1.
Claim 4.1. If m…n† log m…n†=n ! 0 as n ! y, then Limn!y E n …m…n†† H F~.
Proof: Obviously, E n …m…n†† is included in the set of payo¨ vectors achieved
by mixed strategies which in turn is a subset of F. As the set F is closed,
Limn!y E n …m…n†† H F .
Take …x; y† A Limn!y E n …m…n††. First, the fact that player 1 can guarantee h at every stage using a pure action a A argmaxa A A ‰minb A B h…a; b†Š
shows that x V h . Next if G is zero sum so that h ˆ k, then
h ˆ max min k…a; b† ˆ
aAA bAB
min max k…a; b† ˆ
aAA bAB
k :
It follows from Lemma 4.1 that for every e > 0, there is n0 such that for each
n V n0 player 2 has a pure strategy t such that
kn …s; t† V k e
for all s A S…m…n††:
Therefore, for every n V n0 , player 2 must receive at least k librium of Gn …m…n††. This implies that y V k .
e in any equiQ.E.D.
Claim 4.2. If m…n† ! y, m…n† log m…n†=n ! 0 as n ! y, and if there is
…x; y† A F~ with x > h , then F~ H Lim n!y E n …m…n††.
Proof: First, we deal with the case in which there is …x; y† in F~ with x > h
and y > k . To show F~ H Lim n!y En …m…n†† it su½ces to show that, for
every d > 0, the set F~d ˆ f…x; y† A F~ j x > h ‡ d; y > k ‡ dg is contained in
Lim n!y En …m…n††.
Let K ˆ maxfkhk; kkkg. Since we have assumed that the payo¨s are
nonnegative, it follows that jr…a; b† r…a 0 ; b 0 †j U K for all r ˆ h; k and …a; b†;
…a 0 ; b 0 † A A B. Fix a d > 0 for which F~d 0 q and take …x; y† A F~d . Let e > 0
be su½ciently small so that e < minf1; K=4g, x > h ‡ 2e, and y > k ‡ 2e.
Let …ai ; bi † A A B, i ˆ 1; 2; 3, be such that …x; y† is a convex combination of …h…ai ; bi †; k…ai ; bi ††, i ˆ 1; 2; 3. Thus there are ai V 0, i ˆ 1; 2; 3, such
Two-person repeated games with ®nite automata
319
P3
that a1 ‡ a2 ‡ a3 ˆ 1 and …x; y† ˆ iˆ1
ai …h…ai ; bi †; k…ai ; bi ††. Assume that
k…a1 ; b1 † U k…a2 ; b2 † U k…a3 ; b3 † and, without loss of generality, a3 > 0. Let d
be a su½ciently large positive integer soP
that, by setting d1 ˆ ‰a1 dŠ, d2 ˆ ‰a2 dŠ,
3
di …h…ai ; bi †; k…ai ; bi ††, the followd3 ˆ d …d1 ‡ d2 †, and …x; y† ˆ …1=d† iˆ1
ing inequalities hold:
k…x; y†
e
…x; y†k < ;
2
…4:1†
and
k † > K:
d3 …y
…4:2†
Note that …x; y† converges to …x; y† as d tends to in®nity and thus (4.1) holds
for a su½ciently large d. Also, since a3 > 0 implies that d3 ! y as d ! y,
and since y > k , (4.2) holds for a su½ciently large d.
Let b4 A B be a best response of player 2 to the action a3 of player 1. De®ne
a sequence of action pairs of length d, x ˆ …x1 ; . . . ; xd †, by
8
< …a1 ; b1 † for j ˆ 1; . . . ; d1
xj ˆ …a2 ; b2 † for j ˆ d1 ‡ 1; . . . ; d1 ‡ d2
:
…a3 ; b3 † for j ˆ d1 ‡ d2 ‡ 1; . . . ; d.
(Recall that d1 ‡ d2 ‡ d3 ˆ d) De®ne a sequence of action pairs o ˆ
…o1 ; . . . ; on † A …A B† n as follows. Let q ˆ ‰n=dŠ. In the last d stages, o coincides with x up to the one stage before the end and then ®nishes with
…a3 ; b4 †:
…on
d‡1 ; . . . ; on †
From the stage n
…on
ˆ …x1 ; . . . ; xd 1 ; …a3 ; b4 ††;
qd ‡ 1 up to n
qd‡1 ; . . . ; on d †
Finally, in the ®rst n
…o1 ; . . . ; on
n
qd †
d, x is repeated q
ˆ ``…x1 ; . . . ; xd † repeated q
1 times:
1 times:''
qd stages, the tail part of x is played:
ˆ …x…q‡1†d
n‡1 ; . . . ; xd †:
Notice that …o1 ; . . . ; on 1 † is d-periodic. Clearly, for every p ˆ 1; . . . ;
1,
n
X
h…ol † V …n
p†x
dK
…4:3†
lˆp‡1
The assumption k…a1 ; b1 † U k…a2 ; b2 † U k…a3 ; b3 † and the choice of b4 imply
that for every p < n,
n
X
lˆp‡1
k…ol † V …n
p†y:
…4:4†
320
A. Neyman, D. Okada
De®ne a pair of pure strategies …~
s; t~† so that (i) they follow the path o as
long as the other player does so, and (ii) if player 2 deviated from o, then s~
takes a pure action a~ A argmina A A dmaxb A B k…a; b†e at every stage afterward,
while (iii) if player 1 deviated from o for the ®rst time at stage p, then t~ starts
playing a pure strategy t^ such that for every s A S…m…n†† and l ˆ 1; 2; . . . ;
hl …s; t^† U h ‡
khk logjS…m…n††j
:
l
Theorem 3.1 ensures the existence of such strategy.
The strategy s~ is implementable by an automaton of size d ‡ 1: d states for
playing the cycle phase of o and one for the punishment.3 So, if n is large
enough so that m…n† > d, then s~ A S…m…n††. Since the play induced by …~
s; t~† is
o, we have
s; t~†; kn …~
s; t~††
k…hn …~
…x; y†k <
dK e
<
n
2
for n >
2dK
:
e
It follows from (4.1), using the triangle inequality, that …hn …~
s; t~†; kn …~
s; t~†† is
within e of …x; y† for su½ciently large n.
Next we will show that no unilateral deviation from …~
s; t~† leads to a strict
improvement of the payo¨. We start with player 2. Take t A T. Assume that
the strategy t deviates from the play o at stage p. If p U n d3 , then the
inequalities (4.2) and (4.4) imply that
s; t~†
n…kn …~
kn …~
s; t†† V
V
while if n
K ‡ …n
K ‡ d3 …y
p†…y
k †
k † > 0;
d3 < p U n, recalling the choice of b4 ,
s; t~†
n…kn …~
kn …~
s; t†† V …n
p†k…a3 ; b3 † ‡ k…a3 ; b4 †
…k…a3 ; b4 † ‡ …n
ˆ …n
p†…k…a3 ; b3 †
p†k †
k † > 0:
Thus we conclude that player 2 cannot bene®t from a deviation from o at
any stage. Let us turn to player 1. Take s A S…m…n†† and suppose that …s; t~†
resulted in player 1's deviation from the path o. The fact that s A S…m…n††
implies that such deviation must occur in the ®rst m…n† repetitions of the cycle
o1 ; . . . ; od , and hence in the ®rst m…n†d stages of the repeated game. Thus
assume that the deviation occurred at stage p U m…n†d. Let …o10 ; . . . ; op0 † be
the play induced by …s; t~† up to stage p and set s 0 ˆ …sjo10 ; . . . ; op0 †. Then by
the construction of t~, we have, recalling that the payo¨s are assumed to be
nonnegative,
3 Although on 0 xd , player 1's action in on , a3 , is the same as his action in xd .
Two-person repeated games with ®nite automata
321
Fig. 2. F~ ˆ f…x; k †jh0 U x U h1 g, h U h0 < h1
hn …s; t~† U
U
pK ‡ 1
n
p
hn p …s 0 ; t^†
n
pK
khk logjS…m…n††j
‡ h ‡
n
n
which would be less than h ‡ e if, e.g.,
nV
2
maxfm…n†dK; khk logjS…m…n††jg:
e
…4:5†
Our assumption on the order of magnitude of m…n† guarantees (4.5) to hold
for all su½ciently large n. Since x > h ‡ 2e and hn …~
s; t~† is within e of x, we
s; t~†. We have thus shown that …~
s; t~† is an equilibrium of
have hn …s; t~† < hn …~
G n …m…n†† with a payo¨ vector within e of our target payo¨ vector …x; y† provided that n is large enough.
Next assume F~ ˆ f…x; k †jh0 U x U h1 g where h U h0 U h1 and h < h1 .
See Figure 2 for example. In this case there are two action pairs …a; b† and
…a 0 ; b 0 † such that k…a; b† ˆ k , h…a; b† U h0 and …h…a 0 ; b 0 †; k…a 0 ; b 0 †† ˆ …h1 ; k †.
Take …x; k † A F~ with x > h ‡ 2e where e > 0 is su½ciently small. Let d be
a su½ciently large positive integer and let x ˆ …x1 ; . . . ; xd † A …A B† d be such
Pd
that (i) xj ˆ …a; b† or …a 0 ; b 0 †, and (ii) j…1=d† jˆ1
h…xj † xj < e. Let n be a
su½ciently large positive integer and de®ne the path o ˆ …o1 ; . . . ; on † A
…A B† n by …o1 ; . . . ; on qd † ˆ ……a 0 ; b 0 †; . . . ; …a 0 ; b 0 †† and …on qd‡1 ; . . . ; on † ˆ
``x repeated q times'' where q ˆ ‰n=dŠ. Let s~ be player 1's pure strategy that
takes the action ol1 at stage l regardless of the past history. Let t~ be player 2's
pure strategy that follows the path o as long as player 1 does so, and if player
1 deviated from o at stage l for the ®rst time, it immediately reverts to t^
against S…m…n†† as in the previous case.
s; t~† is played and it is the
Since player 2 receives k at every stage when …~
highest payo¨ for her in the stage game, she has no incentive to deviate from
o at any stage. An argument similar to the ®rst case shows that player 1 cannot bene®t from the deviation from o, provided that n is su½ciently large.
322
A. Neyman, D. Okada
Thus …~
s; t~† is an equilibrium of G n …m…n†† with payo¨ within e of …x; k † for
su½ciently large n. This completes the proof.
Q.E.D.
Claims 4.1, 4.2 and the fact Lim n!y E n …m…n†† H Limn!y E n …m…n††
establish Theorem 4.1.
5. The l-discounted game G l (m(l))
In this section we will study a modi®ed version of the l-discounted game,
Gl …m…l†† ˆ …S…m…l††; T; hl ; kl †. We consider the bound on the complexity of
player 1's strategies to be a function of the discount factor l such that m…l† !
y as l ! 1. The next lemma is an analog of Lemma 4.1.
Lemma 5.1. Suppose that …1 l†m…l† log m…l† ! 0 as l ! 1. Then for every
e > 0, there is l0 such that for each l A ‰l0 ; 1† there is t A T such that
hl …s; t† U h ‡ e
for all s A S…m…l††:
Corollary 5.1. Let G be a two-person zero sum game and Vl …m…l†† be the
value of Gl …m…l††. If …1 l†m…l† log m…l† ! 0 as l ! 1, then Vl …m…l†† ! h
as l ! 1.
Let us denote by El …m…l†† the set of (Nash) equilibrium payo¨ vectors of
Gl …m…l††. The next theorem is an analogue of Theorem 4.1.
Theorem 5.1. If …1 l†m…l† log m…l† ! 0 as l ! 1 and if there is …x; y† A F~
with x > h or y > k , then El …m…l†† ! F~ as l ! 1.
Proof: De®ne the sets Liml!1 El …m…l†† and Lim l!1 El …m…l†† similarly to
Limn!y E n …m…n†† and Lim n!y E n …m…n††. An argument similar to the proof
of Claim 4.1 together with Lemma 5.1 shows that Liml!1 El …m…l†† H F~.
Below we will show that F~ H Lim l!1 El …m…l††.
First, assume that there is a point …x; y† in F~ such that x > h and y > k .
As in the proof of Claim 4.2, ®x d > 0 with F~d 0 q and take …x; y† A F~d . Let
e > 0 be su½ciently small so that x > h ‡ 4e and y > k ‡ 4e. Let d be a
su½ciently large positive integer and let x ˆ …x1 ; . . . ; xd † A …A B† d be a ®nite
sequence of action pairs such that
1 X
d
…h…xj †; k…xj ††
d jˆ1
e
…x; y† < :
2
…5:1†
De®ne a play o ˆ …o1 ; o2 ; . . .† by ol ˆ xj if l ˆ j (mod d ). That is, o consists of repetitions of the ®nite cycle x. For each positive integer p let
…xp …l†; yp …l†† ˆ …1
l†
y
X
lˆp
ll p …h…ol †; k…ol ††;
Two-person repeated games with ®nite automata
323
and set …x…l†; y…l††Pˆ …x1 …l†; y1 …l††. As …ol †y
lˆ1 is d-periodic, …xp …l†; yp …l††
d
…h…xj †; k…xj †† as l ! 1 for each p. This convergence is
converges to …1=d† jˆ1
uniform in p. So we can take l su½ciently close to 1 so that for every p ˆ
1; 2; . . . ;
e
d
1X
…5:2†
…h…xj †; k…xj †† < :
…xp …l†; yp …l††
2
d jˆ1
Now we describe the equilibrium strategies s~ A S and t~ A T. Player 1's
strategy s~ follows the play o as long as player 2 does so. If player 2 deviated
from o at some stage, then from the next stage on s~ takes a pure action a~ A
argmina A A dmaxb A B k…a; b†e. Player 2's strategy t~ also follows o as long as
player 1 does so. If player 1 deviated from o at some stage, then at the next
stage t~ starts playing the pure strategy t^ constructed in the proof of Theorem
3.1 against player 1's strategy set S…m…l††.
The strategy s~ is implementable by an automaton of size at most d ‡ 1.
So for l su½ciently close to 1 so that m…l† > d, we have s~ A S…m…l††. Note
s; t~†; kl …~
s; t~†† ˆ …x…l†; y…l††. Thus, by (5.1) and (5.2), the strategy
that …hl …~
pair …~
s; t~† yields a payo¨ vector within e of …x; y†.
~
Take s A S…m…l†† and let …ol0 †y
lˆ1 be the play induced by …s; t †. Then, either
ol ˆ ol0 for all l or there is the smallest p such that op 0 op0 . (Note that both
…ol † and …ol0 † are deterministic plays.) In the latter case, Lemma 5.1 implies
that
hl …s; t~† < …1
l†
p 1
X
l
l 1
h…ol † ‡ l
p 1
lˆ1
h…op0 †
‡
!
lp
1
l
…h ‡ e† :
It follows from (5.1), (5.2), and the assumption x > h ‡ 4e, that, for l su½ciently close to 1, …h ‡ e† xp‡1 …l† < 2e. Since l=…1 l† ! y as l ! 1,
we have K ‡ …l=…1 l††……h ‡ e† xp …l†† < e for l su½ciently close to 1
and hence
hl …s; t~†
hl …~
s; t~† ˆ hl …s; t~†
U …1
x…l†
p 1
K‡
l†l
U …1
l†l p 1 … e† < 0:
l
1
l
……h ‡ e†
xp‡1 …l††
If l is su½ciently close to 1 so that …1 l†dK U l d e, then player 2 would
have no incentive to deviate from o at any stage. Indeed, if player 2 deviated
from o for the ®rst time in the p-th cycle, then the gain within the p-th cycle
from the deviation is at most …1 l†l… p 1†d dK while the loss she incurs from
the punishment is at least l pd e.
Thus we have shown that, for all l su½ciently close to 1, …~
s; t~† is an equilibrium of Gl …m…l†† that yields a payo¨ vector within e of …x; y†.
Now suppose that F~ ˆ f…h ; y†jk0 U y U k1 g where k U k0 U k1 and
k < k1 . Then there are action pairs …a; b† and …a 0 ; b 0 † such that h…a; b† ˆ h ,
324
A. Neyman, D. Okada
k…a; b† U k0 and …h…a 0 ; b 0 †; k…a 0 ; b 0 †† ˆ …h ; k1 †. For a given payo¨ vector
…x; y† in F~ with y > k , de®ne x ˆ …x1 ; . . . ; xd † as in (5.1) except that xj ˆ
…a; b† or …a 0 ; b 0 †. Let o be the cyclic play with the cycle x. De®ne a strategy
pair …~
s; t~† as follows: s~ is the same as above, and t~ follows o regardless of the
history. Note that player 1 receives h at every stage on the play o and the
assumption on F~ implies that this is the highest payo¨ for him in the stage
game. Thus player 1 cannot bene®t by deviating from o. The same argument
as above shows that player 2 has no incentive to deviate from o provided that
l is su½ciently close to 1. The proof for the case F~ ˆ f…x; k †jh0 U x U h1 g
Q.E.D.
with h U h0 U h1 and h < h1 is similar and we omit it.
For the following 2 2 game
L
R
T
0; 1
1; 0
B
1; 0
0; 1
…h ; k † ˆ …0; 1† and hence F~ ˆ f…0; 1†g. It is clear, however, that player 1
must receive a strictly positive payo¨ in any equilibrium of the l-discounted
game. Thus one cannot dispense with the condition on F~ in the statement of
Theorem 5.1.
6. Concluding remarks
In the proof of Claim 4.2, we argued that player 1's deviation from equilibrium path, if any, must occur in a very early stage of the repeated game and so
there are enough stages after the deviation so that player 2's punishment is
e¨ective. For this we only needed the condition on the order of magnitude of
the complexity bound, m…n† ˆ o…n†, which is weaker than our original condition m…n†log m…n† ˆ o…n†. The latter condition is needed to determine the
individually rational levels, or rather, we know the individually rational levels
only under this condition on m…n†. Suppose that we obtained a result that, for
a particular sequence …m…n††y
nˆ1 with m…n† ˆ o…n†,
lim V n …m…n†† ˆ Val…G† ˆ min max h…a; b†
n!y
b A D…B† a A A
for every two-person zero sum game G ˆ …A; B; h†. (See Neyman (1997)
Conjecture 1 and 2.) Then an essentially the same proof as in Theorem 4.1
shows that, for such sequence …m…n††y
nˆ1 ,
n
E …m…n†† !
…x; y† A F x V min max h…a; b†; y V min max k…a; b†
b A D…B† a A A
a A D…A† b A B
as n ! y, provided that there is a feasible payo¨ vector in which player 1
receives strictly more than minb A D…B† maxa A A h…a; b†. Similar argument holds
for the discounted games.
Two-person repeated games with ®nite automata
325
References
Ben-Porath E (1993) Repeated games with ®nite automata. Journal of Economic Theory 59:17±32
Kalai E, Stanford W (1988) Finite rationality and interpersonal complexity in repeated games.
Econometrica 56:397±410
Lehrer E (1988) Repeated games with stationary bounded recall strategies. Journal of Economic
Theory 46:130±144
Lehrer E (1994) Finitely many players with bounded recall in in®nitely repeated games. Games
and Economic Behavior 7:390±405
Neyman A (1985) Bounded complexity justi®es cooperation in the ®nitely repeated prisoner's
dilemma. Economics Letters 19:227±229
Neyman A (1997) Cooperation, repetition, and automata. In Cooperation: Game-Theoretic Approaches, ed. by S. Hart, and A. Mas-Colell, vol. 155 of NATO ASI-Seies F, pp. 233±255.
Springer Verlag
Neyman A (1999) Finitely repeated games with ®nite automata. Mathematics of Operations Research 23:513±552
Neyman A, Okada D (1999) Strategic entropy and complexity in repeated games. Games and
Economic Behavior 29:191±223
Neyman A, Okada D (2000) Repeated games with bounded entropy. Games and Economic Behavior 30:228±247
Papadimitriou CH, Yannakakis M (1994) On complexity as bounded rationality: Extended abstract. In STOC 94, pp. 726±733, Montreal, Quebec, Canada
Zemel E (1989) Small talk and cooperation: A note on bounded rationality. Journal of Economic
Theory 49:1±9