Disruption Tolerant Networking:
Application of Approximate
Dynamic Programming
Dr. Donald Wunsch, [email protected]
Dr. Larry Pyeatt, [email protected]
Tae-hyung Kim, [email protected]
Applied Computational Intelligence Lab
University of Missouri, Rolla
Table of Contents
•
•
•
•
•
Goal: Optimizing multi-strategy selection
Approximate Dynamic Programming
Routing Policy vs. Routing Strategies
ADP in NS-2
Current Status
Goal: Optimizing multi-strategy selection
Packet delivery rate
• Select routing strategy using ADP
techniques.
• Provide near-optimal packet delivery rate
for any network situation.
Learning near-optimal
strategy selection
Network availability
An uninformed routing strategy
An informed routing strategy
e.g. gossiping
e.g. link-state
Approximate Dynamic Programming
Qt (st , at )  [rt   max Q(st 1 , at 1 )  Q(st , at )]
at 1
Environment
rt+1
st+1
Action
at
Reward
rt
Agent
State
st
Approximate Dynamic Programming
• State: Characterization of node and network
–
–
–
–
Node connectivity
Destination within transitive closure of AKB
Rate of local adjacency changes
Rate of link-state update
• Action: routing strategy to use for bundle
forwarding decision
• Reward: upon successful delivery (based on
ACKs)
Routing Policy vs. Routing Strategies
• Routing policy makes decisions regarding
– routing strategy
– control packet dissemination
• A node can have multiple routing strategies.
• Initially, two strategies will be incorporated.
– Uninformed strategy: First Contact
– Informed strategy: Link-State
ADP in NS-2 (1/3)
• Modular design with respect to
– routing strategies
– learning algorithm
DTN
Node
Local
port demux
Node entry
Routing
Agent
outgoing
links
Local
delivery
AKB
Manager
Routing
Policy
Routing
Strategy
Routing
Strategy
Routing
Strategy
ADP in NS-2 (2/3)
• Routing agent processes all incoming bundles.
– Data bundles for local delivery: local port demux.
– Other data bundles: routing strategy via the policy
– Control bundles: AKB manager.
DTN
Node
Local
port demux
Node entry
Routing
Agent
outgoing
links
Local
delivery
AKB
Manager
Routing
Policy
Routing
Strategy
Routing
Strategy
Routing
Strategy
ADP in NS-2 (3/3)
• AKB manager estimates the network conditions
– A control bundle is prepared when a link goes up or down.
• Routing policy module determines
– Dissemination of control bundles
– Routing strategy
DTN
Node
Local
port demux
Node entry
Routing
Agent
outgoing
links
Local
delivery
AKB
Manager
Routing
Policy
Routing
Strategy
Routing
Strategy
Routing
Strategy
Current Status
• Our existing RL library has been integrated into
NS-2.
• The routing agent and AKB manager are
completed and tested.
• The routing policy module is partially
constructed and being tested for dissemination
of control packets.