Applying the Repeated Game Framework
to Multiparty Networked Applications
Mike Afergan
July 22, 2005
Joint work with Dave Clark, Rahul Sami and John Wroclawski
My Thesis
Repeated games can be an important and practical
tool for the design of networked applications.
Talk Overview




Fundamental Motivations
Background on Repeated Games
Example: Incentive-Based Routing
Research Overview and Concluding
Thoughts
3 of 59
Initial Assumptions
Networked applications are important
Incentives are a concern for a large
class of networked applications.




Routing
Peer-to-Peer
Network application developers need tools to build systems
robust to user incentives.
4 of 59
Properties Fundamental to
Networked Applications
Property #1: Multiple interacting self-interested parties


Direct communication or shared network
Motivates the use of game theory
Property #2: Interactions are repeated.


Causal relationship between one time period and the next
Examples:


ISPs in near identical BGP sessions
Users in similar interactions with similar users (e.g., web,
wireless, P2P)
Suggests that the repeated context should be
considered to use game theory effectively.
Repeated Games are Important


Repeated games are a well-studied area of
game theory.
The outcome of the repeated game can
significantly differ from the outcome of the
one-shot game.
 However, most relevant prior work considers
only the one-shot game.
This research is the first to consider repeated games
as a tool for networked applications.
A Practical Fit


Importantly, in each example we derive
practical results
These practical results stem from further
relationships between networked applications
and repeated games.
Networked
Repeated
Game Theory
Applications
7 of 59
Repeated Games are Practical
Property #3: Networked applications face
multiple constraints
 Example Constraints:



Need to realize system objectives
Cost, privacy, shared network
Impact of Constraints:


May not be able to realize a one-shot solution
Provides explanation for real-world phenomena
Repeated games work well with practical models of networked applications.
8 of 59
Repeated Games are Practical
Property #4: Actions in Networked Applications Are
Highly Parameterized
Parameter value is important
More interestingly, parameter granularity is
also important




In repeated games, the granularity of the action
qualitatively impacts the equilibrium
The freedom permitted can be a first order concern
9 of 59
Properties Fundamental to
Networked Applications
Repeated games are important and practical
Multiple
Parties
Repeated
Dynamics
Constraints
Parameterized
These four properties apply to a large
class of networked applications
Repeated games are an important and practical tool for
the design of networked applications.
Areas of Contribution
Exposition of Thesis



Introduce the concept of using repeated games
Demonstration of a fundamental relationship between
repeated games and networked applications
Present approaches and techniques
Application to Important Networked Problems
1. Inter-ISP Relationships with User-Directed Routing
(Chapter 3)
2. Design of Incentive-Based Routing Systems (Chapter 4)
3. Application-Layer Multicast Overlays (Chapter 5)
Later in this talk, I will present #2 in depth.
Talk Overview




Fundamental Motivations
Background on Repeated Games
Example: Incentive-Based Routing
Research Overview and Concluding
Thoughts
12 of 59
One-Shot Prisoner's Dilemma
P2
Cooperate
Defect
(5,5)
(0,9)
P1
Cooperate
Defect
(9,0)
(1,1)
Static Equilibrium
Outcome
In the one-shot game, (D,D) is the outcome
of the unique Nash Equilibrium.
Repeated Prisoner's Dilemma
Example Strategy:
1. Play C
2. If the other player defects, play D forever
P2
Cooperate
Defect
$$$
P1
Outcome of
the Repeated
Game
Cooperate
(5,5)
(0,9)
Defect
(9,0)
(1,1)
or
$+$+$
+ $+ $
+S
Key Takeaway: The equilibrium of the repeated game
may differ from the equilibrium of the corresponding
one-shot game.
Sample Analysis
P2
Cooperate
Defect
$$$
P1



(5,5)
(0,9)
Defect
(9,0)
(1,1)
$+$+$
+ $+ $ + S
Parameterized by discount factor ()


Cooperate
or
Patience Factor (infinite game)
Probability of game ending (finite game with unknown horizon)
Example: Strategy  is an equilibrium of the game iff:
(Playing  forever)  (One-time “defect”) + (Resulting payoffs)
“Play C forever. If other plays D, play D forever” is an equilibrium iff:


t 0
t 1
t
t
½
5


9


(
1
)



15 of 59
Repeated Equilibria Under
General Conditions


“Folk theorem” results show the feasibility of a
large set of potential outcome payoffs
Repeated equilibria feasible under a variety of
practical assumptions:

Imperfect Information
[Green-Porter ’84, Fudenberg-Levine-
Maskin ’94]


Players of different horizons [Fudenberg-Levine ’94]
Anonymous random matching [Ellison ’93]
In practice, this means many repeated outcomes are possible
under a broad class of restrictions.
Talk Overview



High Level Argument
Background on Repeated Games
Specific Example: Incentive-Based Routing






Problem Overview
The Problem of Repeated Dynamics
Finding Key Protocol Parameters
Generalizing the Results
Summary
Research Overview and Concluding Thoughts
17 of 59
The Context
Incentive-Based Interdomain Routing
A
PriceA
PriceB
s
PriceC


C
Routes as goods
Applied specifically and deployed incrementally
Well-motivated by:



t
Architecture Overview


B
Economic realities of today’s Internet
Increasingly prevalent technology (User-Directed Routing)
[A., Wroclawski ’04]
This talk does not defend such an architecture.
18 of 59
Protocol Design Question


We consider a single
competitive interchange
s
t
Our Question: How should one design a
protocol for conveying pricing information for routes?
Protocol Designer Does Control
Protocol Designer Does Not Control
Protocol period (time between
updates)
Number of networks
Unit of Measure (Mbps vs. MBps)
Network Cost
Width of protocol fields
(number of bits)
Strategies used by Networks
Our Analytical Framework:
Repeated Games
1. Routing is inherently a repeated process
2. The outcome of the repeated game can
differ qualitatively from that of the oneshot game
Our research is the first to consider routing
as a repeated game.
Our Contributions
Practical Conclusions
Although routing is repeated, important properties of prior models do
not hold in the repeated setting.
We find newfound importance for several parameters
1. The length of the protocol period
2. The granularity of the unit-of-measure
(e.g., Mbps, MBps, or Gbps)
3. The width of the price field
These provide practical insight for protocol designers.
It is possible to upper-bound prices using these parameters.
This helps designers (to the extent desired) control the uncertainty
presented by the repeated game.
21 of 59
Talk Overview



High Level Argument
Background on Repeated Games
Specific Example: Incentive-Based Routing






Problem Overview
The Problem of Repeated Dynamics
Finding Key Protocol Parameters
Generalizing the Results
Summary
Research Overview and Concluding Thoughts
22 of 59
Problem of Repeated Routing

An interconnect is


A repeated game
Between a small number of players (ISPs)
s
t
The repeated game may cause artificially higher prices

Standard pricing technique: Strategyproof Mechanisms




Truthtelling is at least as good as any other strategy
Benefits: Reduced strategizing and potential oscillation
Standard mechanism: Vickrey-Clark-Groves (VCG)
Feigenbaum, Papadimitriou, Sami, and Shenker (FPSS ’02) show
how to apply this to an Internet-like network efficiently
23 of 59
Applying VCG to a Network
[FPSS ’02]
A
1
s
B
1
1
t1
10
1
t2
10
Each node, i, on the Least Cost Path (LCP) paid:
pi = (LCP avoiding i) – LCP + ci
24 of 59
Applying VCG to a Network
[FPSS ’02]
A
1
s
B
1
1
t1
10
1
t2
10
Each node, i, on the Least Cost Path (LCP) paid:
pi = (LCP avoiding i) – (LCP) + ci
Example:
s -> t1: A is paid (10 + 1) – (1 + 1) + 1 = 10
s -> t2: B is paid (10 + 1) – (1 + 1) + 1 = 10
In the one-shot game, this is strategyproof.
The Repeated Version
A
1
s
B
1
1
t1
10
1
t2
10
In the repeated game A and B could both bid 20:
A is paid (10 + 20) – (1 + 20) + 20 = 29
B is paid (10 + 20) – (1 + 20) + 20 = 29
Conclusion #1: Although Internet routing is a repeated setting, the VCG
mechanism (and thus the FPSS implementation) is not strategyproof in the
repeated routing game.
26 of 59
Questions
1. What determines the equilibrium
price?
2.What can be done to control,
bound, or influence prices (if so
desirable)?
27 of 59
Talk Overview



High Level Argument
Background on Repeated Games
Specific Example: Incentive-Based Routing






Problem Overview
The Problem of Repeated Dynamics
Finding Key Protocol Parameters
Generalizing the Results
Summary
Research Overview and Concluding Thoughts
28 of 59
A Full Model of Routing
We:


Prove that particular parameters may significantly impact price
Formally analyze that impact (by looking at the derivatives)
given a model with:






Repeated interactions
Asynchronous interactions
Heterogeneous networks
Multi-hop paths and multiple destinations
Confluent (BGP-like) routing
Large class of strategies
This talk focuses on a simple model: Repeated Incentive
Routing Game (RIRG)


Intuition and analysis is similar for more general models
Will later briefly discuss generalizations (more details in thesis)
Repeated Incentive Routing Game (RIRG):
Topology
Direction of Traffic
Strategic Player
s

…
t
A particular interchange:





Single Source
Single Destination
Multiple homogenous networks offering connectivity
Networks compete for traffic on price (Bertrand competition)
Route is the market good
30 of 59
RIRG: Key Assumptions

Key Assumption #1: The game is played via a networked
protocol.



Key Assumption #2: The game is not infinite.



Protocol runs in a series of synchronized rounds (of length d)
There is a minimum bid granularity size (b).
Players only know length in expectation (D)
Note: D and d define :  = 1- d/D
Additional Assumptions that can be Relaxed





Traffic is fixed
Networks have fixed per unit cost
FPSS-like network
Networks have infinite capacity
Minimum bid becomes common knowledge
Traffic is splittable
31 of 59
RIRG: Play of the Game
In each round:
1.
All N players announce their bids
2.
Traffic is evenly split among the
provider(s) with the lowest price
3.
Provider is paid for the volume of traffic at
the price bid (1st price auction)
Key Decision: In each round, each network can either:
1. Try to be the low-price provider
2. Split the market with other firms at a higher price
Equilibrium Notion

The potential strategy space is quite large



An equilibrium notion refines the strategy space
Subgame perfect equilibrium (SPE) is natural and
standard for repeated games
A strategy  is subgame perfect if
i)  is a Nash equilibrium for the entire game
and
ii)  is a Nash equilibrium for each subgame.
33 of 59
Price Matching
For the purposes of this talk, I will focus on Price

Matching (PM) Strategies


Informally: “Bid the lowest price seen in the prior period”
Results generalize, for example:


“Match price and then raise later”
“Punish by doubling initial deviation”
Price Matching Strategy:
1.
2.
At t0, offer p*
t 1 

max
c
,
min
p

For all t>t0, pi =
j 

j

p* is the largest p such that PM is SPE
34 of 59
Defining Price Matching
Solving for p*
One Stage Deviation Principle (Abridged):
 is subgame perfect if and only if no player can
gain by deviating from  in a single stage and
conforming to  thereafter.


t
(1)    i  p, p    i  p  b, p       i  p  b, p  b 
t
t 1
t 1
(2)  i  p, p   1    i  p  b, p   i  p  b, p  b
p* is the maximum p such that
the inequality holds.
Term
Meaning
(pi, p-i)
Profit function

Period probability of
game ending (discount
factor)
Solving for Equilibrium
(2’)  i  p, p   1    i  p  b, p   i  p  b, p  b
(3)
Tp
T  p  b
 1   T  p  b   
N
N
(4) p  1   N  p  b    p  b
Theorem: In the RIRG, the unique
equilibrium price from Price
Variable
Matching is:
bN    N 
b
p
N
(1  N )(1   )

Meaning
Minimum bid size
Number of players
Period probability of
game ending (discount
factor)
Deriving Practical Intuition
Theorem: When playing Price Matching:
p
0
d
where d is the length of the protocol period.
Conclusion #2: A longer period may lead to lower prices
“A longer period may lead to lower prices”
Lowering price leads to:
Higher payoffs later
Big payoff now
$$$
or
$+$+$
+ $+ $
$
1sec
$
+S
$
Period of protocol
$
$
1 month
“A longer period may lead to lower prices”
Longer protocol period
Longer time before competitors react
More benefit to deviating
Lower prices
39 of 59
More Practical Intuition

Theorem: When playing PM:
p
0
b
where b is the minimum bid size.

“Minimum bid size” is not a protocol parameter.

But:
Unit-of-measure (Megabits, Megabytes, Terabits)

Width of price field (number of bits in protocol)
are protocol parameters

Conclusion #3: A wider price field and a more granular
unit of measure may reduce price.
Sensitivity to Parameters
Profit Margin vs Delta (N=2)
1
b=0.01
Profit Margin
0.8
b=0.05
b=0.1
0.6
0.4
0.2
0
0
0.2
0.4
Delta
0.6
0.8
1
Observations:
1. Sensitivity to delta is large, especially in the relevant range
2. Impact of b is qualitative, not just precision
Result Summary

As [Variable] Increases…
Prices
# of players
Decreases
Width of price field
Increases
Unit-of-Measure Granularity
Decreases
Protocol period
Decreases
Topology Stability
Increases
Example Takeaways:
1)
2)
Using Megabytes instead of Megabits can lead to lower
prices.
A system that runs faster may lead to higher prices.
A priori, some of these parameters seem benign or at most only
having impact as “rounding error.”
Constraining Prices

Sensitivity to parameters means:



This insight must be considered
They can help “solve the problem” of the repeated
dynamics (to the extent desirable)
Theorem: For all >0, there exists protocol
parameter settings such that pR  pS + ,
where:


pR is the equilibrium price in the repeated game
pS is the equilibrium price of the stage game.
43 of 59
Talk Overview



High Level Argument
Background on Repeated Games
Specific Example: Incentive-Based Routing




Problem Overview
The Problem of Repeated Dynamics
Finding Key Protocol Parameters
Generalizing the Results


Generalizing the Strategy Space
Generalizing the Game




Multiple Destinations and Confluent Flows
Heterogeneous Costs
Summary
Research Overview and Concluding Thoughts
44 of 59
Proportional Punishment (PP)
Strategies
Price Matching has two weaknesses:

1. Prices never rise
2. Punishment limited to matching price
Proportional Punishment





Strategies are SPE
Punishment is bound by some constant k
Class is very large (perhaps too large)
If ĥ is a one-stage deviation from h when
playing  at t0, then for PPk iff:
    k   p'
t
h
t
hˆ
t0
hˆ
45 of 59
Visualizing PPk
Price
Match then Raise
p
Price Matching
p’
Punish by Doubling
p-k(p-p’)
t0
Time
46 of 59
Analyzing PPk

Theorem 3: For any PPk, the maximal price
obtained by  is bound by
bN    k 
p 
(1  N )(1   )

Further, this bound is tight.

Other results follow similar to the simple Price
Matching case


Impact of b and 
Bounds on pR
47 of 59
A More General Model
Multiple Destinations and Confluent Flows
A
c
s
A wins traffic for t1
B wins traffic for t2
B
c

t1
t2
Multiple Destinations


Multiple goods, multiple markets
Provides for cooperation even with
confluent flows
48 of 59
A More General Model
Heterogeneous Networks
A
c
s


t2
Assume c > c’


A wins traffic for t1
B wins traffic for t2
B
c’

t1
Potential for a repeated equilibrium at p*(c’)
Requires that |c – c’| is sufficiently small
Equilibria may involve only a subset of the N players
Does not necessarily imply repeated equilibria


More general graph presents more options
A robust protocol must consider such conditions
49 of 59
Talk Overview



High Level Argument
Background on Repeated Games
Specific Example: Incentive-Based Routing






Problem Overview
The Problem of Repeated Dynamics
Finding Key Protocol Parameters
Generalizing the Results
Summary
Research Overview and Concluding Thoughts
50 of 59
Summary
The repeated setting is a vitally important
setting to consider.
2. Our analysis provides insight into the
importance of several protocol parameters
3. These parameters are:
1.


Under the control of the protocol designer
Unavoidable
Consideration of these parameters can help
build a robust system
5. Suggests that repeated game analysis can
be important and practical
4.
51 of 59
Talk Overview




Fundamental Motivations
Background on Repeated Games
Example: Incentive-Based Routing
Research Overview and Concluding
Thoughts
52 of 59
Benefits and Feasibility of Incentive
Based Routing (Chapter 3)
Business
Relationships


Traffic Policies
Traffic Patterns
Problem: User-directed routing (e.g., overlays) transforms
inter-domain routing into a meaningfully repeated game
Sample Contributions:


Exposition of the problem
Consideration of principles for why and how incentives (i.e., prices)
should be integrated to various routing architectures
53 of 59
Application-Layer Multicast
Overlays (Chapter 5)
……
……
Faithful Nodes create an efficient tree



Selfish Nodes able to alter the topology
Problem: Selfish users can degrade system performance
Contribution: A repeated model of cooperation
Contribution: Use model and simulation to descry practical
techniques and parameters that can aid in building more
robust systems
54 of 59
Meaningful Themes

For each problem considered:



The repeated dynamic plays a vital role in
defining the system equilibrium
Our model is the first to capture the
repeated dynamic
We are able to derive practical insight into
how to build more robust systems.
55 of 59
Exogenous Types vs
Endogenous Motivations

Some models use exogenous types:

Network type: business relationships (e.g,
[GaoRexford00])


Node type: cheater/not [Mathy et al ‘04],
generosity parameter [Feldman et al ‘04]
Repeated game models can capture these
factors in an endogenous fashion
56 of 59
Concluding Thoughts



The repeated dynamic must be considered in
modeling networked applications.
Repeated Games can provide practical results
Relevance of repeated games stems from
properties fundamental to networked
applications
57 of 59
Thank you for coming!
Questions?
Thesis (and slides) will be available at
http://www.mit.edu/~afergan/thesis/
58 of 59