How to Evaluate Exotic Wireless
Routing Protocols?
Dimitrios Koutsonikolas1,
Y. Charlie Hu1, Konstantina Papagiannaki2
1Purdue
University, 2Intel Research, Pittsburgh
1
Evolution of Wireless Routing
Protocols
• From the Ad Hoc Era to the Mesh Era
– New design goals
• High throughput vs. connectivity
– New “exotic” optimization techniques
– Cross – layer design
AODV
TORA
DSDV
DSR
1994
Performance
comparisons
1996 1997 1998
2000
Ad Hoc Networking Era
ExOR
ROMER SOAR
ETX ETT
MORE
MC2
COPE noCoCo
2003 2004 2005 2006
2007
Mesh Networking Era
2
In This Talk…
• Review the evolution of wireless protocol
design
– Reveal challenges to evaluation
methodology of new routing protocols
• Discuss current practices
– Weaknesses
• Suggest guidelines for fair and
meaningful evaluation
3
Ad Hoc Networking Era
• Primary challenge
– Deal with route breaks due to host mobility
• Layering principle
– Routing protocol discovers route
– 802.11 unicast transmits packets to next hop
• ACK/RETX, exponential backoff
• Evaluation
– PDR, control overhead, tradeoffs
– Low constant offered load
4
Mesh Networking Era
• Static routers
– Mobility not a concern
• Commercial applications
– Compete with other internet technologies
• New research focus
– High Throughput
5
Towards High Throughput
• Link-quality routing metrics
– Examples: ETX, ETT
– Still follow layering principle
• “Exotic” optimization techniques
– Examples: Opportunistic Routing, Network
Coding
– Abandon layering principle
6
Opportunistic Routing
• First demonstrated in ExOR [SIGCOMM ‘05]
• Packet broadcast at each hop, all neighbors
can receive it
• Neighbor closest to destination rebroadcasts
– Coordination required
A
0%
50%
S
A
B
C
D
S
50%
50%
B
0%
0%
C
D
Intra-Flow Network Coding
• First demonstrated in MORE [SIGCOMM ‘07]
• Routers randomly mix packets
• Benefits
– Remove need for coordination
– FEC-style reliability, no ACK/RETX
p1, p2
S
α*p1+ β*p2
p1, p2
A
Who forwards?
p1, p2
D
B
p1, p2
Coordination Required!
S
A
Both forward
B
γ*p1+ δ*p2
No Coordination!
D
Inter-Flow Network Coding
•
•
•
•
First demonstrated in COPE [SIGCOMM ‘06]
Routers mix packets from different flows
Increase network capacity!
Implied evaluation methodology
– Subject network to congestion
– Use network coding to eliminate congestion
Alice
1:p1
2:p2
4:p2
3:p1
Router
1:p1
Bob
Traditional Routing: 4 TX
2:p2
3:p1+p2 3:p1+p2
Alice
Router
Bob
Network Coding: 3 TX
9
Implications of 802.11 Broadcast
• 802.11 broadcast has no ACK/RETX, no
exponential backoff
– No reliability
– Nodes can send faster than in unicast
• Exotic techniques do not work well with TCP
– Batching
• Consequence
– Reliability and rate control are brought to routing
layer from lower or upper layers
10
Evolution of Protocol Stack
Application Layer
Application Layer
End-to-end Rate
Control
End-to-end Reliability
Packet Forwarding
Network Sublayer 3
End-to-end
Rate Control
End-to-end Reliability
Network Sublayer 2
Packet Forwarding
Network Sublayer 1
Network Coding
Hop-by-hop
Rate Control
Hop-by-hop Reliability
Transport Layer
Network Layer
Hop-by-hop Reliability
Medium Access
MAC Layer
Physical Layer
Traditional Network
Stack
MAC Layer
Medium Access
Physical Layer
New Network
Stack
11
Implications on Protocol
Evaluation
• Evaluation becomes a much subtler task
– Possible conflicts between new and old
mechanisms
• Inter-flow network coding vs. rate control
• Current state
– Diverse set of evaluation methodologies
– Lack of clear guidelines
12
Evaluation of Unreliable
Protocols
13
Practice 1: Making Both Protocols
Reliable
• Evaluation of ExOR, comparison with Srcr
– ExOR guarantees delivery of 90% of the file
– Srcr offers no guarantee
Problem
Removes spatial reuse from traditional routing
• Methodology
– Download a 1MB file
– Send 1.1MB with ExOR to compensate for loss
– Carry the whole file hop-by-hop with Srcr to avoid
collisions
14
Practice 2: No Rate Control –
Varying the Sending Rate
• Evaluation of COPE, comparison with Srcr
– COPE increases network
capacity
Problem
PDR drops quickly as network capacity is exceeded
• Methodology
– UDP traffic
– Vary offered load
– Exceed nominal
capacity (6Mbps)
15
Practice 3: A Protocol With Rate Control
Against a Protocol Without Rate Control
• Evaluation of SOAR, comparison with
Shortest Path (SP)
Problem
– SOAR applies rate control
Not clear what fraction of gain comes from
– SP has no rate
control
opportunistic
routing
and what from rate control
• Methodology
– Saturate the network
16
Evaluation of Reliable Protocols
17
Practice 5: A Reliable Against an
Unreliable Protocol
• Evaluation of MORE, comparison with
Srcr
– MORE offers FEC-style e2e reliability
Problem
– Srcr
offers
no reliability
•Srcr
suffers
losses
due to congestion
•Same amount of data sent by src, different amount
delivered to dst
• Methodology
– UDP sent at maximum possible rate
18
Practice 6: Running an Unreliable
Protocol Under TCP
• Evaluation of noCoCo, comparison with COPE
– noCoCo applies backpressure-based congestion
control/reliability
Problem
– COPE has no congestion control, weak reliability
TCP performs poorly in multihop wireless networks
Solution – Practice 7
Modify COPE to use noCoCo’s congestion
• Methodology control/reliability
– Run COPE under TCP
19
Use (or No Use) of Autorate
Adaptation
• Traditional routing uses 802.11 unicast
– Exploits autorate adaptation
• Exotic optimization techniques rely on 802.11
broadcast
– Operates on single Problem
rate
Methodology can be unfair to traditional routing
• Methodology
– Evaluation of most exotic protocols disables
autorate adaptation for traditional routing
• For “fair”comparison
20
Recommendations for more
consistent and meaningful
evaluation
21
The Importance of Rate Control I
Unreliable Protocols
• Traditional routing under
UDP has no rate control
– Packets dropped beyond
capacity
– Throughput reduction
• Exotic protocols w/o rate control
– Increase throughput, may increase capacity
– Packets still dropped beyond (new) capacity
• Exotic protocols w/ rate control
– Constant throughput beyond capacity
– No need to increase offered load beyond capacity
22
The Importance of Rate Control II
Reliable Protocols
• FEC-style reliability provides no rate
control
• PDR remains 100%, rate control still
needed
• Exceeding capacity may lead to
– Increased delays
– Unfairness among flows
• Related recommendation
– Evaluate with multiple flows
23
Isolating the Benefit from Exotic
Technique
• Evaluation should quantify the gain
from new exotic optimization technique
• Tricky part
– Adding an exotic technique may require old
techniques to move to the routing layer
• Recommendation
– Old techniques should also be incorporated
into traditional routing
24
Separating Rate Control from
End-to-end Reliability
• Running traditional routing under TCP
+ No modification to the protocol itself
– TCP performs poorly in multihop wireless networks
– TCP provides both rate control and reliability
• If new protocol has only one mechanism, overkill to run
old protocol under TCP
• Recommendation
– Incorporate reliability/rate control mechanism of
new protocol to old protocol
25
How to Incorporate Reliability To
Traditional Routing
• Case 1: reliability component disjoint to
exotic technique
– Example: ARQ component in noCoCo
– Method: add same component to traditional
routing
• Case 2: reliability component merged
with exotic technique
– Example: intra-flow NC in MORE
– Method: add FEC to traditional routing?
26
MAC Autorate Adaptation
• Exotic protocols should try to
incorporate autorate adaptation
– Not always feasible
• Recommendation
– Enable autorate adaptation for traditional
routing
– Show exotic protocol outperforms
traditional routing both with and without
autorate adaptation
27
Conclusions
• Inconsistencies in evaluating wireless mesh
routing protocols
• Fundamental reason
– No unified framework for understanding
interactions among
•
•
•
•
•
MAC
Congestion
Reliability
Interference
Network coding
• Real problem goes beyond how to evaluate
exotic protocols
28
Thank You!
29