Scaling Peer-to-Peer Games in
Low Bandwidth Environments
Jeff Pang
Frank Uyeda
Jacob R. Lorch
Carnegie Mellon
U.C. San Diego
Microsoft Research
1
P2P Games
Local View
Replica objects
Primary object
Internet
2
P2P Games
Local View
Replica objects
Primary object
Internet
3
P2P Games
Local View
Internet
Total Bandwidth (Mbps)
Replica objects
200
150
100
Primary object
50
0
0
100 200 300 400 500
Players
4
Internet
128 kbps
Total Bandwidth (Mbps)
P2P Games
200
150
Not enough bandwidth to send
100
to all peers at required rate
 choppy
and unsatisfying
50
game play
0
0
100 200 300 400 500
Players
5
Not Enough Bandwidth
P2P Quake on a LAN
P2P Quake on a Cable Modem
6
Our Solution: Donnybrook
• Contributions
– Provide insights that enable scaling
– Design techniques that codify these insights
– A prototype implementing these techniques
• User study demonstrates effectiveness
– Preserves fun in low bandwidth environments
– Increases scalability by an order of magnitude
7
Design Principles
• Player attention is bounded by a constant
Example: I focus on my current target
 Total attention scales linearly with # players
• Realism should not be sacrificed for accuracy
Example: “Teleporting” to correct locations is jarring
 Don’t always minimize error in because of updates
• Interaction must be timely and consistent
Example: If I shoot you, you should get hit immediately
 Prioritize interaction (i.e., inter-object writes)
8
Design Principles
• Player attention is bounded by a constant
– Technique: Focus Set
• Realism should not be sacrificed for accuracy
– Technique: Guidable AI
• Interaction must be timely and consistent
– Technique: Pairwise Rapid Agreement
9
Focus Set
• Each player divides other peers into two classes:
– Focus Set: players most focused on me
• Update every frame
– Best Effort: everyone else
• Update when bandwidth is available
•  bandwidth allocated to Focus Set, (1- ) to Best Effort
• Who goes in my Focus Set?
– Attention(i,j) = how much a player i is focused on player j
– Attention(i,j) sent from peer i to peer j periodically
– Peers with the highest Attention(i,j) go in peer j’s Focus Set
10
Focus Set
• Who goes in my Focus Set?
– Attention(i,j) = how much a player i is focused on player j
– Attention(i,j) sent from peer i to peer j periodically
– Peers with the highest Attention(i,j) go in peer j’s Focus Set
Attention(i,j) =
fproximity(d1)
j
+
i
faim(θ1)
+
finteraction-recency(t1)
θ1
θ2
d1
d2
11
Guidable AI
• Problem: Best effort peers receive infrequent updates
• Solution: Smooth state changes with AI
– Example: use existing path finding code to make replica move
update
update
update
12
Donnybrook in Action
P2P Quake on a Cable Modem
P2P Quake on a Cable Modem
with Donnybrook
13
Evaluation
• Questions
– How much does Donnybrook improve playability in
low bandwidth environments?
– How close is Donnybrook in a low bandwidth
environment to a LAN (“optimal”)?
• Methodology
– Modified Quake III to run on Donnybrook
– User study to evaluate user satisfaction
– Simulation to evaluate fairness (See paper)
14
Setup
30 bots
curr
curr
curr
… curr
donny donny donny
Internet
2 humans
curr
curr
~5 updates/sec
108 kbps
donny
LoBW
•
donny
curr
curr
… curr
Internet
108 kbps
curr
…
donny
LAN
curr
LoBW-Donny
•
•
Focus set size = ~4
Best effort rate = ~1 update/sec
curr
HiBW
•
20 updates/sec
15
Setup
15 min
User Study Procedure
– Before experiment, practice on HiBW
– Tell players two Quake III “servers” exist: A and B
– Start playing on A, can vote to switch to B
A
Switch!
This sucks
– When both players vote, game continues on B
B
– Can vote to switch back and forth
– Analog to how players choose game servers
(if good, stay, otherwise leave and try another)
Switch
back
OK
– Play new game on least-used version so they
can compare
5 min
16
Total Stay Time
1000
LoBW vs. LoBW-Donny
LoBW-Donny vs. HiBW
Seconds
800
600
400
200
0
LoBW
LoBWDonny
LoBWDonny
HiBW
Three additional metrics in the paper
How long does a pair play on each version?
support these conclusions.
17
Max # of players
Projected Scalability
1400
1200
With Donnybrook
1000
800
600
400
Without Donnybrook
200
0
0
128
256
384
512
640
768
896 1024 1152 1280 1408 1536
Upload bandwidth per peer (kbps)
18
Summary
• Donnybrook enables large-scale P2P games in
low bandwidth environments
• Three key design principles:
– Player attention is bounded by a constant
– Interaction must be timely and consistent
– Realism should not be sacrificed for accuracy
• Future work:
– Constant-size Focus Set doesn’t work if everyone is focused on
one player (e.g., the flag carrier in CTF)
– Use a multicast tree to deliver frequent updates in this scenario
19
Donnybrook: The Missing Slides
Expansion Pack
20
Meta-Conclusions
• We should do more games research
– Has the market-share and user base
– Numerous large scale distributed systems problems
• Bounded player attention applicable to other systems
– Prioritizing data collection in P2P sensor aggregation
– Varying fidelity in P2P video conferencing
• “End-to-end” evaluation should consider user experience
– Donnybrook’s replica consistency is not great
– Users don’t notice or don’t care
21
Other P2P Games Challenges
• Cheating
– Migration, Witnesses
• Bharambe [CMU thesis work]
– Trusted Computing / Hardware Sensors
• Johnson et al. [Intel]
• Fault Tolerance
– DHT self-healing / replica placement:
• Colyseus [NSDI 2006]
• SimMUD [INFOCOM 2004]
• Persistence
– General P2P storage replacements for DB
22
Why P2P Games?
www.mmogchart.com
• Massively multiplayer
games are popular
– Recent research:
• Colyseus [NSDI ‘06]
• SimMUD [INFOCOM ’04]
Subsciptions
12,500,000
10,000,000
7,500,000
5,000,000
2,500,000
0
1997 1999 2001 2003 2005
Year
• Centralized First Person
Shooter (FPS) games
scale poorly
– Quadratic Bandwidth
– CPU Limited
Bandwidth Used (kbps)
Colyseus [NSDI ’06]
CPU Bottleneck
23
Game Execution Model
• Game State:
– Collection of distinct objects (players, missiles, items, etc.)
• Game Execution:
– Each object has a Think function:
Think() { ReadPlayerInput(); DoActions(); ... }
– Execute each object once per frame:
Each 50ms do {
foreach object do {
object->Think();
}
}
24
Area of Interest Filtering
server s1
P1
Object Store
Colyseus [NSDI ’06]
P2
R3
Object Location
R4
Replica Management
DHT
Object Placement
P3 P4
server s2
25
Design Comparison
• Existing P2P game architectures
– Replicas can be as stale as the update interval
– Update rate per peer is fixed
• Donnybrook’s solution
– QoS: Vary update rate per peer, some get priority
– Guidable AI: Secondary replicas think for themselves
26
Pairwise Rapid Agreement
• Interaction: when player A modifies player B
(i.e. A performs a write on B)
• Goal: modification is consistent and applied quickly
• Insight: # interactions scales slowly
– Occur at human time scales  infrequent
– Involve only 2 players  unicast
• Solution: prioritize all inter-object writes
– Player A sends mod to Player B
– Player B checks preconditions locally
– Player B applies mod, sends result to A
• PRAs required in Quake III:
– Damage, Death, Item Pickup, Door Opening
27
Prediction
• Motivation: state snapshots get stale fast
– Example: players can traverse the entire diameter of a map in
several seconds in Quake III
– Goal: send prediction of state at time of next expected update
– Example: predict where a player will be at the next update
• Predicted Properties:
– Predict position: use in-game simulation to figure how where
physics brings player in next second
– Predict viewing angle: use view angles to estimate the target a
player is aiming at
– Predict Events: use number-of-shots-fired to estimate when a
player is “shooty,” etc.
28
Guidable AI: Convergence
• Problem: Best Effort peers receive very infrequent updates
• Solution: Smooth state changes with AI
– Position: use existing path finding code to make replica move smoothly
– Angle: have AI turn smoothly toward predicted targets
• Convergence
– Motivation: Players in focus should be represented more accurately,
but should not “warp” to actual position
– Solution: Converge to actual state when receiving frequent updates
• Focus on player B
 In player B’s Focus Set, get frequent updates
 Error(replica, actual) decreases with each update
• When Error() < , use player B’s update snapshots instead of AI
• Error(a,b) = distance(a.pos,b.pos)
29
Setup
• User Study Stats
– LoBW-Donny vs. LoBW: 12 trials
– LoBW-Donny vs. HiBW: 32 trials
– 98 total participants
How often did you play
FPS games in the past?
Every Day
62%
Every Week
25%
Less Often
13%
30
Departure Time
1000
Time until first vote
Time until second vote
Seconds
800
600
400
200
0
LoBW LoBW- HiBW
Donny
LoBW LoBW- HiBW
Donny
How long before a player wants to switch?
31
Fun Score
Score (1 to 10)
10
LoBW vs. LoBW-Donny
LoBW-Donny vs. HiBW
8
6
4
2
0
LoBW
LoBWDonny
LoBWDonny
HiBW
Survey: How fun was server A? Server B?
32
Preference
4%
LoBW
No Pref.
17%
LoBWDonny
96%
LoBW-Donny vs. LoBW
LoBWDonny
31%
52%
HiBW
LoBW-Donny vs. HiBW
Survey: Was A or B more Fun?
33
Avg. correlation
coefficient
Fairness
Random bots
All bots level 5
1
0.8
0.6
0.4
0.2
0
HiBW
LoBWDonny
LoBW
Rank
Scores
Rank
Scores
34
Donnybrook: The Elder Slides
Beta Test
35
P2P Games
Object Store
Internet
128 kbps
Bandwidth Used (kbps)
Colyseus [NSDI ’06]
• Not enough bandwidth to
send to all peers every frame
•  choppy and unsatisfying
game play
36
Outbound Capacity Problem
• Residential broadband is
asymmetric
• In many games, players
can see many others
• Insufficient upload
capacity to send updates
at required frequency

• Must send updates at
lower frequency
• Replicas can be as stale
as the update interval
128-512 kbps up
Cost per Peer in Quake III:
Update rate = 20 updates/sec
Update size = ~100 bytes

Bwidth per peer ≥ 16kbps

Supportable peers at 128kbps ≤ 8
37
Focus Set
• Who goes in my Focus Set?
– Attention(i,j) = how much a player i is focused on player j
– Attention(i,j) sent from peer i to peer j periodically
– Peers with the highest Attention(i,j) go in peer j’s Focus Set
t1
θ1
θ2
t2
d1
d2
38
Departure Time
Time until first vote
1000
Seconds
800
600
400
200
0
LoBW
LoBW-Donny
HiBW
How long before a player wants to switch?
39