Scalable Peer-to-Peer
Networked Virtual Environment
Master Thesis Oral Examination
Dept. of CSIE, Tamkang Univ.
Advisor: Dr. Chen Jui-Fa
Shun-Yun Hu
2005/01/07
1
Outline





Introduction
Voronoi-based Overlay Network (VON)
Simulation Results
Analysis
Conclusion
2
What is Networked Virtual
Environment (NVE)?

Virtual Reality + Internet

3D environment with people (avatar), objects, terrain,
agents

Military simulations (’80) 
Massively Multiplayer Online Games (mid-‘90)

Trends: larger scale, more realistic simulation
3
4
NVE: A Shared Space
5
Issues for Creating NVE






Consistency (events/states)
Responsiveness
Security
multiplayer
Scalability
Persistency
Reliability (Fault-tolerance)
massively multiplayer
6
The Scalability Problem



Many nodes on a 2D plane ( > 1,000)
Message exchange with those within Area of Interest (AOI)
How does each node receive the relevant messages?
Area of Interest
7
A simple solution (point-to-point)
Source: [Funkhouser95]
N * (N-1) connections ≈ O(N2)  Not scalable!
8
A better solution (client-server)
Source: [Funkhouser95]
Message filtering at server to reduce traffic
N connections = O(N)  server is bottleneck
9
Current solution (server-cluster)
Source: [Funkhouser95]
Still limited by servers. Expansive to deploy & maintain.
10
Scalability Analysis

Scalability constrains


Computing resource
Network resource
Non-scalable system
(CPU)
(Bandwidth)
vs.
Scalable system
Resource limit
x: number of entities
y: resource consumption at the limiting system component
11
What Next?

Strategies



Increase resource
Decrease consumption
Architectures




Point-to-point (LAN)
Client-server
Server-cluster
?
Peer-to-Peer


More servers
Message filtering
Scale
tens
hundreds
thousands
millions
10^1
10^2
10^3
10^6 …
12
What is Peer-to-Peer (P2P)?
[Stoica et al. 2003]
 Distributed systems without any centralized control
or hierarchical organization
 Runs software with equivalent functionality

Examples



File-sharing:
Napster, Gnutella, eDonkey
Distributed computing: SETI@Home (UC Berkeley)
VoIP:
Skype
13
Peer-to-Peer Overlay
A P2P overlay network
source: [Keller & Simon 2003]
14
Promise & Challenge of P2P

Promises



Growing resource, decentralized  Scalable
Commodity hardware
 Affordable
Challenges


Topology maintenance
 dynamic join/leave
Efficient content retrieval  no global knowledge
15
Issues for Creating P2P NVE
Consistency (events/states)
Responsiveness
Security
multiplayer
massively multiplayer

Scalability
Persistency
Reliability (Fault-tolerance)

Consistency (topology)





 P2P NVE
16
Related Works (1): SimMUD
[Knutsson et al. 2004] (Univ. of Pennsylvania)





Pastry + Scribe
Regions
Coordinators
(super-nodes)
Fixed-size region
Relay overhead
17
Related Works (2)
[Kawahara et al. 2004] (Univ. of Tokyo)





Fully-distributed
Nearest-neighbors
List exchange
High transmission
Overlay partition
18
Related Works (3): Solipsis
[Keller & Simon 2003] (France Telecomm R&D)





Links with AOI neighbor
Mutual cooperation
Inside convex hull
Potentially slow discovery
Inconsistent topology
19
Outline





Introduction
Voronoi-based Overlay Network (VON)
Simulation Results
Analysis
Conclusion
20
Design Goals

Observation:


for virtual environment applications, the contents we want
are messages from AOI neighbors
Content discovery is a neighbor discovery problem

Solve the Neighbor Discovery Problem in a fullydistributed, message-efficient manner.

Specific goals:


Scalable
Responsive
 Limit & minimize message traffics
 Direct connection with AOI neighbors
21
Voronoi Diagram


2D Plane partitioned into regions by sites, each
region contains all the points closest to its site
Can be used to find k-nearest neighbor easily
Neighbors
Region
Site
22
Design Concepts
Use Voronoi to solve the neighbor discovery problem




Identify enclosing and boundary neighbors
Each node constructs a Voronoi of its neighbors
Enclosing neighbors are minimally maintained
Mutual collaboration in neighbor discovery
Circle
Area of Interest (AOI)
White
self
Yellow
enclosing neighbor (E.N.)
L. Blue
boundary neighbor (B.N.)
Pink
E.N. & B.N.
Green
AOI neighbor
D. Blue unknown neighbor
23
Procedure (JOIN)
1) Joining node sends coordinates to any existing node
Join request is forwarded to acceptor
2) Acceptor sends back its own neighbor list
joining node connects with other nodes on the list
Joining node
Acceptor’s region
24
Procedure (MOVE)
1) Positions sent to all neighbors, mark messages to B.N.
B.N. checks for overlaps between mover’s AOI and its E.N.
2) Connect to new nodes upon notification by B.N.
Disconnect any non-overlapped neighbor
Boundary
neighbors
Non-overlapped
neighbors
New
neighbors
25
Procedure (LEAVE)
1) Simply disconnect
2) Others then update their Voronoi
new B.N. is discovered via existing B.N.
Leaving node
(also a B.N.)
New boundary neighbor
26
Dynamic AOI
Crowding within AOI can overload a particular node
It’s better if AOI-radius can be adjusted in real time
27
Adjustment Conditions

AOI-radius decrease


AOI-radius increase



Number of connections > maximum allowable connections
Maximum connections not exceeded
Current AOI-radius < preferred AOI-radius
Delay counter

To avoid fluctuations
28
Demonstration
Simulation video



General movements (20 nodes, 800x600 world)
Local vs. global view
Dynamic AOI adjustment
29
Outline





Introduction
Voronoi-based Overlay Network (VON)
Simulation Results
Analysis
Conclusion
31
Simulation Method

C++ implementation of Voronoi-based algorithm

World size: 1000 x 1000, AOI: 150
Trials from 10 – 250 nodes
Connection limit per node: 10
1000 time-steps



(~ 100 simulated seconds, assuming 10 updates/seconds)

Behavior model



Random movement:
Constant velocity:
Movement duration:
random direction
5 units/step
random (1 – 25 steps)
32
Consistency Metrics

Topology Consistency [Kawahara, 2004]
Number of observed AOI neighbors
Number of actual AOI neighbors

Drift Distance [Diot, 1999]
Distance between observed position and actual position
(average over all nodes)
33
Basic Model
Topology Consistency
Topology Consistency (%)
Topology Consistency Measurements
100
99
98
97
96
95
94
93
92
91
90
0
50
100
150
200
250
Number of Nodes
34
Basic Model
Scalability (1)
Transmission Size Per Node Per Second
6.0
5.0
Size (kb)
send (max)
4.0
send (avg)
recv (max)
3.0
recv (avg)
2.0
1.0
0.0
0
25
50
75
100
125 150
175
200
225
250
Number of Nodes
35
Basic Model
Scalability (2)
Average Neighbor Size Measurements
18
16
Neighbor Size
14
12
connected
10
AOI
8
6
4
2
0
0
50
100
150
200
250
Number of Nodes
36
Dynamic AOI Model
Effect of Node Increase on Average AOI-radius
160
Avg. AOI Radius
140
120
100
80
60
40
20
0
0
50
100
150
200
250
Number of Nodes
37
Dynamic AOI
Scalability (1)
Average Neighbor Size Measurements
10
9
Neighbor Size
8
7
connected
6
5
AOI
4
3
2
1
0
0
50
100
150
200
250
Number of Nodes
38
Dynamic AOI
Scalability (2)
Transmission Size Per Node Per Second
4.0
3.5
send (max)
Size (kb)
3.0
send (avg)
2.5
recv (max)
2.0
recv (avg)
1.5
1.0
0.5
0.0
0
25
50
75
100 125 150 175 200 225 250
Number of Nodes
39
Dynamic AOI
Scalability (3)
Comparison of Voronoi-based P2P and Client-Server
180
160
Size (kb)
140
send (avg)
120
recv (avg)
100
CS-send (avg)
80
CS-recv (avg)
60
40
20
0
0
25
50
75
100 125 150 175 200 225 250
Number of Nodes
40
Dynamic AOI
Topology Consistency (1)
Topology Consistency Measurements
Topology Consistency (%)
100
99
98
97
96
95
94
93
92
91
90
10
30
50
70
90
110 130 150 170 190 210 230 250
Number of Nodes
41
Dynamic AOI
Topology Consistency (2)
Drift Distance Measurements
100
Drift Distance
80
max
60
avg
40
20
0
0
50
100
150
200
250
Number of Nodes
42
Dynamic AOI
Reliability (1)
Change in Consistency of over 1000 time-steps (step 150 - 250)
Topology Consistency (%)
100
per step
90
node 1
node 91
80
70
150
160
170
180
190
200
210
220
230
240
250
time-step
43
Dynamic AOI
Reliability (2)
Average Number of Steps to Recover from Inconsistency
4
Steps
3
2
1
0
0
50
100
150
200
250
Number of Nodes
44
Outline





Introduction
Voronoi-based Overlay Network (VON)
Simulation Results
Analysis
Conclusion
45
Analysis of Design
Consistency (Topology)

Topology is fully connected & consistent  enclosing neighbors
Responsiveness

Lowest latency  direct connection, no relay
Scalability

Resource-growing & decentralized resource consumption
Reliability

Self-organizing for small number of node failures
46
P2P NVE Comparisons
SimMUD
Consistency
(topology)
Neighbor-list Solipsis
exchange
Supernode Neighbor listexchange
(partitioning)
Responsive- High
ness
overhead
High
overhead
VON
Neighbor
Neighbor
notify&query notify
(undiscovery) (consistent)
Medium
overhead
Low
overhead
Scalability
Relied on Fullysupernode distributed
Fullydistributed
Fullydistributed
Reliability
Long uptime
N/A
Selforganizing
N/A
47
Problems of Voronoi Approach

Message traffic


Circular round-up of nodes
Redundant message sending
(inherent to fully-distributed design)

Incomplete neighbor discovery


Can happen with inconsistent / incorrect neighbor list
Fast moving node
48
Outline





Introduction
Voronoi-based Overlay Network (VON)
Simulation Results
Analysis
Conclusion
49
Conclusion

NVE scalability is achievable with P2P architecture
and is a neighbor discovery problem

A promising solution: Voronoi-based P2P Overlay


Leverage knowledge of each peer to maintain topology
Properties



Scalable: fully-distributed, dynamic AOI
Efficient: low irrelevant messages, zero relay
Robust: consistent and self-organizing topology
50
Potential Applications

Online games
Relieve server from position updates in current MMOGs

Military
Enable large-scale, affordable military training simulation

3D Web
Provide multi-user interactivity to static 3D world

Scientific simulations
Distribute spatial simulation requiring frequent synchronization
51
Future Works

Short-term




Reliability measurements  latency, packet loss, node fail
Distributed event/state consistency
Recovery from overlay partition
Long-term



Persistency issue
(P2P-based database)
Security issue
(protection from malicious nodes)
3D content distribution (3D streaming on P2P)
Massive, persistent 3D environment sharable by all!
52
Acknowledgements











Dr. Jui-Fa Chen
(陳瑞發老師)
Dr. Wei-Chuan Lin
(林偉川老師)
Members of the Alpha Lab, TKU CS
Guan-Ming Liao
(廖冠名)
Dr. Chin-Kun Hu
(胡進錕老師)
LSCP, Institute of Physics, Academia Sinica
Joaquin Keller
Bart Whitebook
Jon Watte
(France Telecomm R&D, Solipsis)
(butterfly.net)
(there.com)
Dr. Wen-Bing Horng
Dr. Jiung-yao Huang
(洪文斌老師)
(黃俊堯老師)
53
Inconsistency caused by dAOI
54
Reliability (0-500 steps)
Change in Consistency of over 1000 time-steps (step 1 - 500)
Topology Consistency (%)
100
per step
90
node 1
node 91
80
70
0
100
200
300
400
500
time-step
55
Reliability (501-1000 steps)
Change in Consistency of over 1000 time-steps (step 501 - 1000)
Topology Consistency (%)
100
per step
90
node 1
node 91
80
70
500
600
700
time-step
800
900
1000
56