Opportunistic Multipath Forwarding
in Publish/Subscribe Systems
Reza Sherafat Kazemzadeh AND Hans-Arno Jacobsen
Middleware Systems Research Group
University of Toronto
MIDDLEWARE SYSTEMS
RESEARCH GROUP
MSRG.ORG
Agenda
• The content-based publish/subscribe model
– Characteristics and challenges
• Our approach
– Overlay neighborhoods
– Adaptive multipath forwarding
• Experimental evaluation results
Middleware 2012
Opportunistic Multipath Forwarding
2
Content-Based Pub/Sub Model
Publishers
P
PP
P
Publish
P
P
Pub/Sub
SS
S
S
S P
S
S
S
S
Subscribers
Middleware 2012
Opportunistic Multipath Forwarding
3
Content-Based Pub/Sub Model
Publishers
Many to many communication between a large number of
P
publishers/subscribers
P
Publish
Selective delivery based on subscription matching
• Unicast (one)
Pub/Sub
• Multicast (some)
• Broadcast (all)
P
Variations in traffic patterns makes it difficult to design an
optimal overlay network
S
S
Subscribers
Middleware 2012
Opportunistic Multipath Forwarding
4
Current Pub/Sub Overlays
Forwarding paths in the overlay are constructed in fixed end-to-end
manner (no/little path diversity)
P
✓
S
✗
D
E
✗
C
✗
B
✓
P
A
P
This results in a high number of “pure forwarding” brokers
System Yield =
Middleware 2012
#msgs delivered
= 1/5 = 20%
#msgs sent
Opportunistic Multipath Forwarding
5
Summary of Our Goals
• Construct a highly connected overlay mesh that provides
high path diversity between publishers and subscribers
• Avoid pure forwarders by allowing brokers to make finegrained forwarding decisions based on individual
publications and their matching sets
• Improve system yield, efficiency, scalability and delivery
delay
• Support dynamic adaptive routing based on live traffic
patters while avoiding high costs of full overlay
reconfiguration
Middleware 2012
Opportunistic Multipath Forwarding
6
Forwarding Strategies
p
• Fixed end-to-end (baseline)
A
*
B
C
*
D
*
Total msgs: 6
• Forwarding strategy 1
p
A
*
B
C
*
D
Total msgs: 5
*
• Forwarding strategy 2
Total msgs: 3
Middleware 2012
p
A
*
B
C
*
D
*
Opportunistic Multipath Forwarding
7
Our Approach in a Nutshell
Δ=3
Routing Tables
(Δ-neighborhoods
knowledge)
Δ=2
Δ=1
Links Management
(best links via a gain
function)
A
Pub. Forwarding
(Path Computations
for strategies)
Middleware 2012
S
Opportunistic Multipath Forwarding
8
Path Computation for
Forwarding Strategies
Strategy 1
p
p
A
Strategy 2
*
B
C
*
A
D
*
B
*
C
D
*
*
*
A
B
C
*
D
*
A
*
Middleware 2012
Opportunistic Multipath Forwarding
B
C
*
D
*
9
Experimental Evaluations
• We have implemented the algorithms and performed large-scale
experimental evaluations with up to 500 brokers
• Datasets
– Synthesized based on Zipf distribution
– Social networking traces from Facebook
• We measured performance of the system in terms of:
–
–
–
–
–
–
Overlay mesh connectivity
Delivery delay
Maximum system throughput
System yield
Publication propagation path length
Memory and CPU overhead
Middleware 2012
Opportunistic Multipath Forwarding
10
Network size:250
Delta=3
Network size:120
Delta=3
Number of Available paths
Path Diversity in Overlay Mesh
Path diversity:
10% of brokers w/ 1000 paths
Path diversity:
20% of brokers w/ 100 paths
Graph is based on a snapshot of the state of links in a running system
Middleware 2012
Opportunistic Multipath Forwarding
11
115% throughput
enhancement
100
CDF (%)
Strategy 2 80
60
40
S2,Fout=15
S1,Fout=15
S2,Fout=10
S1,Fout=10
S0
20
0
1
2
3
4
5
6
7
Path length
8
9
10
11
Fixed-end-to-end
Publication Hop Count
Experiment setup:
• 120 Brokers
• Publish rate is 1,800 msgs/sec and number of deliveries: 73,000 (in 5 min)
Middleware 2012
Opportunistic Multipath Forwarding
12
Conclusions
• Brokers build a highly connected overlay mesh and
make fine-grained forwarding decisions for each
publication in order to avoid pure forwarding neighbors
• We used the notion of overlay neighborhoods to
enable local traffic profiling and avoid high costs of
overlay reconfiguration
• Our approach enhances system’s efficiency and yield,
and ultimately improves its scalability and performance
Middleware 2012
Opportunistic Multipath Forwarding
13
Thanks
Questions!
MIDDLEWARE SYSTEMS
RESEARCH GROUP
MSRG.ORG
Middleware 2012
Opportunistic Multipath Forwarding
14
Pub/Sub Characteristics
• Many to many communication
between a large number of
publishers/subscribers
Client
• Selective delivery based on
subscription matching
– Unicast (one)
– multicast (some)
– broadcast (all)
Broker
• Traffic patterns depends on
workload application and may
change over time
Middleware 2012
Opportunistic Multipath Forwarding
15
Related Work: Overlay Reconfiguration
Broker overlay is “reconfigured” by addition
and removal of links between brokers
Advantages
– Forwarding path may be improved bringing some
publishers and subscribers closer together
Reconfigure
Disadvantages
– Some forwarding paths between publishers and
subscribers may indeed suffer
– Resulting overlay still relies on fixed end-to-end
paths
– Reconfiguration is costly and requires full or
partial re-propagation of subscriptions




[1] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks, FTDCS ‘03
[2] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic
MiddlewareIn:
2012
Opportunistic Multipath Forwarding
16
networks.
FTDCS. (2003)
Links Management
•
Overlay Network
Links types
– Primary links
– Soft links – shortcut links
– Candidate links – expeditionary links
•
Soft link
Candidate link
C
B
A
Traffic profiling
Soft link selection
– Profiling input
– Link state measured
– Broker load information exchanged
D
High rank
– In intervals of T sec, brokers count the
number of pubs sent over each links
– Gain function
gain(A,B,T) = pub_traffic_during_T * dist(A,B)
•
Primary link
Links ranking
high → low
A
Middleware 2012
B
Opportunistic Multipath Forwarding
C
D
E
F
G
E
F
…
17
Master v. Working Routing Data Structures
Master Routing Tables
• Overlay map captured by brokers’
Δ-neighborhoods are relatively
static
 Master overlay Map (MOM)
• Brokers link connectivity change
dynamically, brokers need an
efficient way to compute
forwarding paths over the changing
set of links
Working Routing Tables
 Working Overlay Map (WOM)
construct
• MOM is a concise representation
of the primary overlay that only
contains brokers with a direct link
Middleware 2012
Opportunistic Multipath Forwarding
18
Master v. Working Subscription Tables
Working Overlay Map
Between
Brokers
Sets
Beyond
Brokers
Sets
Master
Subscription
Table
Behind
Brokers
Sets
Working
Subscription
Table
• Similar to WOM, brokers adapt their subscription tables based
on the current set of available links:
Working Subscription Table (WST)
Middleware 2012
Opportunistic Multipath Forwarding
19
System Yield (measure of efficiency)
Delivered
publications
Strategy
Fixed end-to-end
73,000
(Sparse Workload)
Strategy 1
Strategy 2
Delivered
publications
Strategy
Baseline
284,000
(Dense Workload)
Strategy 1
Strategy 2
Middleware 2012
Number of pure
Pure Forwarders
System Yield
91,000
44%
42,000
63%
29,000
71%
Number of pure
Pure Forwarders
System Yield
195,000
59%
104,000
73%
69,000
80%
Opportunistic Multipath Forwarding
20
Publication Hop Count
Experiment setup:
•
120 Brokers
•
Sparse and dense workloads
•
Publish rate of 1,800 msgs/sec  Deliveries: 73,000 in 5 min
Middleware 2012
Opportunistic Multipath Forwarding
21
100
CDF (%)
80
60
40
S2,Fout=15
S1,Fout=15
S2,Fout=10
S1,Fout=10
S0
20
0
1
2
3
4
5
6
7
8
9
10
11
Path length
Publication Hop Count
Experiment setup:
•
120 Brokers
•
Sparse publication/subscription workload
•
Publish rate of 1,800 msgs/sec  Deliveries: 73,000 in 5 min
Middleware 2012
Opportunistic Multipath Forwarding
22
Sparse Matching Workload
Dense Matching Workload
100
100
80
80
CDF (%)
CDF (%)
Publication Hop Count
60
60
40
40
S2,Fout=15
S1,Fout=15
S2,Fout=10
S1,Fout=10
S0
20
0
1
2
3
4
5
6
7
8
9
10
11
S2,Fout=15
S1,Fout=15
S2,Fout=10
S1,Fout=10
S0
20
0
1
2
Path length
3
4
5
6
7
8
9
10
11
Path length
Multi-path forwarding is more effective in
sparse workloads
Middleware 2012
Opportunistic Multipath Forwarding
23
Increase (%)
180
Subscription covering set size
Predicate matching ops per pub
160
140
120
100
5
10
15
20
25
Broker Fanout
Impact of Broker Fanout on Subscription Covering
Experiment setup:
• 500 brokers
• Fanout of 5-25
Middleware 2012
Opportunistic Multipath Forwarding
24