Automated Network Repair with Meta Provenance
Yang Wu*
Ang Chen*
Andreas Haeberlen*
Wenchao Zhou+
*
Y. Wu
University of Pennsylvania
+
Boon Thau Loo*
Georgetown University
HotNets-2015 (Nov 17, 2015)
1
Motivation: Automated repair
- Networks can have bugs
- How can we find and fix bugs quickly?
else if (switch == S1 && protocol == HTTP) then action = output:3.
else if (switch == S1 && protocol == HTTP) then action = output:5.
Copy-and-paste bug!!!
SDN
Controller
Why is the backup web
server not getting requests?
S1
S2
5
Y. Wu
Backup
Web Server
Off-loading
HTTP
S0
HTTP traffic
HotNets-2015 (Nov 17, 2015)
3
Main
Web Server
4
DNS
Server
2
Goal: Automated repair
-
Existing debuggers can identify problems
But finding effective fixes is still hard
Can we automate this?
Goal: “Fix it!” button for networks
NetSight (NSDI 2014):
requests dropped at S2
Treated as given!
SDN
Controller
Why to
is make
the backup
web
How
the backup
server
not getting
requests?
web server
get requests?
S1
S2
5
Y. Wu
Backup
Web Server
Off-loading
HTTP
S0
HotNets-2015 (Nov 17, 2015)
3
Main
Web Server
4
DNS
Server
3
Approach: Meta provenance
- Problem: Finding fixes is hard
- Idea: Provenance can pinpoint the root cause
- But previous provenance focus exclusively on data
- Key idea: Treating program as data
HTTP Packet received
at Main Web Server
meta
provenance
Matching Flow Entry
installed at S1
PacketIn
received at Controller
Y. Wu
HotNets-2015 (Nov 17, 2015)
Executed If Clause
in Controller Program
4
How could meta provenance be used?
Press
“Fix it!”
How to make the backup
web server get requests?
Meta provenance
is generated
No HTTP Packet
received at H2
Output:
Possible repairs
Fixed!
No Matching Flow Entry
installed at S2
No Executed If Clause
in Controller Program
No Specific Constant
in Controller Program
Change “switch == S1”
to “switch == S2”
…
Y. Wu
HotNets-2015 (Nov 17, 2015)
5
Goal: “Fix-it!” button for SDNs
Overview
Challenge: Cannot reason about program changes
Approach: Meta provenance
Mete provenance model
Solution
Generating repairs
Practical challenges
Case study
6
Meta provenance model
Provenance
Meta Provenance
Explains which inputs
contribute to the outputs
Explains which parts of the
program contribute to the outputs
Tuples: inputs, outputs
Meta tuples: inputs, outputs,
syntactic elements of the program
Program: how input derives output
Matching Flow Entry
installed at S1
FlowEntry(“S2”, “HTTP”, “Output-1”)
PacketIn
received at Controller
if (switch == S1 && protocol == HTTP)
then action = output:5
Meta program: how the
programming language operates
Matching Flow Entry
installed at S1
Constant(“S1”)
PacketIn
received at Controller
Executed If Clause
in Controller Program
if (operator == “==” && variable == constant)
then expression = true
Constant
in Controller Program
…
7
Meta provenance: Example
- Finds relevant syntactic elements
else if (switch == S1 && protocol == HTTP) then action = output:5.
else if (switch == S1 && protocol == HTTP) then action = output:3.
HTTP Packet received
at Main Web Server
SDN
Controller
Matching Flow Entry
installed at S1
1
S1
S0
3
Main
Web Server
Y. Wu
4
HTTP traffic
DNS
Server
Executed If Clause
in Program
Satisfied Condition
in Program
Operator ==
in Program
Variable
in Program
…
…
Constant
in Program
8
Goal: “Fix-it!” button for SDNs
Overview
Challenge: Cannot reason about program changes
Approach: Meta provenance
Mete provenance model
Solution
Generating repairs
Practical challenges
Case study
9
Generating repairs
- counter-factual reasoning generates targeted repairs
else if (switch == S1 && protocol == HTTP) then action = output:5.
else if (switch == S1 && protocol == HTTP) then action = output:3.
No HTTP Packet
received at H2
SDN
Controller
No Matching Flow Entry
installed at S2
No Executed If Clause
S1 Program
in Controller
S2
5
Off-loading
HTTP
S0
3
PacketIn
received at Controller
4
No Satisfied Condition
in Controller Program
Backup
Web Server
Y. Wu
Negative provenance
(SIGCOMM 2014)
Main
DNS
Web Server
Server
Operator ==
Variable
in Program
in Program
Change “switch == S1”
to “switch == S2”
Wrong Constant
in Program
10
Practical challenges: Infinite repairs
-
Problem: Meta-provenance tree is infinite
Idea: Programmers make certain errors more often
We can explore repairs in cost order, up to some cut-off
Other challenges are discussed in the paper
NEXIST[Tuple(Tab="FlowTable", Swi=3, Dip=10.0.0.15,
Dpt=80, Act=Output-1) @C, t1]
OR
(Enumerating
derived v.s. base)
OR
(Enumerating rules)
NEXIST[Head(Rul="f6", Tab="FlowTable", Swi=3,
Dip=10.0.0.15, Dpt=80, Act=Output-1) @C, t1]
...
(changing other rules)
NEXIST[Base(Tab="FlowTable", Swi=3,
Dip=10.0.0.15, Dpt=80, Act=Output-1) @C, t1]
...
(making new rule)
FIX: Make base tuple
OR
(Enumerating number of constraints)
OR
(Collective Enumeration: predicates of h3)
(with three constraints, i.e., changing constraint(s))
AND
(with JID==8538, i.e., with
materialized joins)
EXIST[Value(Rul="f6", JID=8538,
Arg=Swi', Val=3) @C, t1]
EXIST[Value(Rul="f6", JID=8538,
Arg=Dip', Val=10.0.0.15) @C, t1]
EXIST[Value(Rul="f6", JID=8538,
Arg=Dpt', Val=80) @C, t1]
EXIST[Value(Rul="f6", JID=*,
Arg=Act', Val=Output-1) @C, t1]
AND
AND
AND
AND
EXIST[Assignment(Rul="f6", Arg=Swi',
ID="Swi") @C]
EXIST[Expression(Rul="f6", JID=8538,
ID="Swi",Val=3) @C, t1]
EXIST[Assignment(Rul="f6", Arg=Dip',
ID="Dip") @C]
EXIST[Assignment(Rul="f6", Arg=Dpt',
ID="Dpt") @C]
EXIST[Expression(Rul="f6", JID=8538,
ID="Dip",Val=3) @C, t1]
EXIST[Value(Rul="f6", JID=8538,
Arg=Swi, Val=3) @C, t1]
EXIST[Assignment(Rul="f6", Arg=Act',
ID="Output-1") @C]
EXIST[HeadMeta(Rul="f6", Tab="FlowTable",
Args=Swi',Dip',Dpt',Act') @C]
EXIST[Value(Rul="f6", JID=8538,
Arg=Dpt, Val=80) @C, t1]
EXIST[Constraint(Rul="f6", JID=8538,
ID="Dpt == 80", Val=True) @C, t1]
NEXIST[Constraint(Rul="f6", JID=8538,
ID=?/*, Val=True) @C, t1]
AND
...
details
OR
EXIST[Expression(Rul="f6", JID=8538,
ID="Dip not in ...", Val=True) @C, t1]
EXIST[Constant(Rul="f6",
ID="Output-1",Val=Output-1) @C]
AND
EXIST[Expression(Rul="f6", JID=8538,
ID=Dip, Val=10.0.0.15) @C, t1]
EXIST[Expression(Rul="f6", JID=*,
ID="10.0.0.0/16", Val=10.0.0.0/16) @C, t1]
EXIST[Value(Rul="f6", JID=8538,
Arg=Dip, Val=10.0.0.15) @C, t1]
EXIST[Constant(Rul="f6",
ID="10.0.0.0/16",Val= 10.0.0.0/16) @C]
EXIST[Join(Rul="f6", JID=8538, Args=[Swi,Dip,Dpt],
Vals=[3,10.0.0.15,80]) @C, t1]
EXIST[IsConstraint(Rul="f6",
ID="Dip not in ...", ) @C, t1]
AND
NEXIST[Expression(Rul="f6", JID=8538,
ID="2", Val=3) @C, t1]
AND
EXIST[PredicateMeta(Rul="f6",Tab="PacketI
n", Args=[Swi,Dip,Dpt]) @C]
NEXIST[Constant(Rul="f6",
ID="2", Val=3) @C]
FIX: change constant value
"Swi == 2" => "Swi == 3"
...
(other predicates exist)
NEXIST[Constraint(Rul="f6", JID=8538,
ID="Swi == 2", Val=True) @C, t1]
EXIST[Operator(Rul="f6",
ID="not in", Opr="not in") @C]
...
details
NEXIST[Expression(Rul="f6", JID=8538,
ID="Swi == 2", Val=True) @C, t1]
EXIST[IsConstraint(Rul="f6",
ID="Swi == 2") @C, t1]
...
(other predicates exist)
EXIST[IsConstraint(Rul="f6",
ID="Dpt == 80", ) @C, t1]
AND
...
EXIST[Constraint(Rul="f6", JID=8538,
ID="Dip not in ...", Val=True) @C, t1]
EXIST[IsConstraint(Rul="f6",
ID="Swi == 2") @C, t1]
AND
(with JID!=8538, i.e., other
materialized/hypothetical joins)
...
todo
EXIST[Constraint(Rul="f6", JID=8538,
ID="Dpt == 80", Val=True) @C, t1]
NEXIST[IsConstraint ="f6",
ID=*/*) @C, t1]
AND
EXIST[IsConstraint(Rul="f6",
ID="Dip not in ...", ) @C, t1]
NEXIST[IsConstraint(Rul="f6",
ID="*/*") @C, t1]
...
details
NEXIST[ConstraintCount(Rul="f6",
N=2) @C, t1]
EXIST[IsConstraint(Rul="f6",
ID="Dpt == 80", ) @C, t1]
NEXIST[IsConstraint ="f6",
ID=*/*) @C, t1]
FIX: delete constraint
delete "Swi == 2"
...
(other operators)
NEXIST[Operator(Rul="f6", JID=8538,
ID="==", Val=">") @C]
NEXIST[Expression(Rul="f6", JID=8538,
ID="Swi", Val=2) @C, t1]
OR
...
todo
AND
EXIST[IsConstraint(Rul="f6",
ID="Dip not in ...", ) @C, t1]
AND
(with JID==8538 and two
specific constraints, i.e.,
with materialized joins)
...
todo
conflict with other
branches (solver)
AND
(with the ">" operator)
...
(change both subexpressions)
AND
(with non-existed JID, i.e.,
hypothetical joins)
...
todo
AND
OR
AND
EXIST[ConstraintCount(Rul="f6",
N=3) @C, t1]
NEXIST[Constraint(Rul="f6", JID=8538,
ID=*/*, Val=*) @C, t1]
AND
OR
(with the "==" operator)
EXIST[Predicate(Rul="f6", Tab="PacketIn",
Args=[Swi,Dip,Dpt], Vals=[3,10.0.0.15,80]) @C, t1]
EXIST[Tuple(Tab="PacketIn",
Vals=[3,10.0.0.15,80]) @C, t1]
EXIST[Constraint(Rul="f6", JID=8538, ID="Dip
not in ...", Val=True) @C, t1]
EXIST[Expression(Rul="f6", JID="*",
ID="Output-1",Val=Output-1) @C, t1]
EXIST[Expression(Rul="f6", JID=8538,
ID="Dpt",Val=3) @C, t1]
EXIST[Value(Rul="f6", JID=8538,
Arg=Dip, Val=10.0.0.15) @C, t1]
AND
(with JID!=8538, i.e., other
materialized joins)
OR
(Collective Enumeration: predicates of h3)
(with two constraints, i.e., deleting one constraint)
...
(other predicates exist)
FIX: change operator
"Swi == 2" => "Swi > 2"
OR
...
(composed from subexpressions)
NEXIST[Value(Rul="f6", JID=8538,
Arg=Swi, Val=2) @C, t1]
conflict with other
branches (solver)
OR
(composed from subexpressions)
...
(other ways to compose)
AND
NEXIST[Edge] * 3, @C
NEXIST[Operator(Rul="f6",
ID="-", Val="==") @C]
NEXIST[Expression(Rul="f6",
JID=8538, ID=..., Val=X) @C, t1]
NEXIST[Expression(Rul="f6",
JID=8538, ID=..., Val=Y) @C, t1]
OR
NEXIST[Value(Rul="f6", JID=8538,
Arg=Swi, Val=X) @C, t1]
OR
...
(other ways to compose)
NEXIST[Constant(Rul="f6",
ID=..., Val=Y) @C]
...
(other ways to compose)
FIX: add operator
"Swi == 2" => "Swi - 1 == 2"
Y. Wu
HotNets-2015 (Nov 17, 2015)
11
Case study
- We built an initial prototype of a debugger
- We defined a meta provenance model for NDLOG
- Our prototype fixed the copy-and-paste bug automatically
else if (switch == S1 && protocol == HTTP) then action = output:5.
else if (switch == S1 && protocol == HTTP) then action = output:3.
S2
SDN
Controller
How to make the backup
web server get requests?
S2
5
Y. Wu
Backup
Web Server
3
Off-loading
HTTP S0
Main
12 Server
Web
Summary
- Goal: Automated repairs for networks
Ideally, can we have an automatic “Fix it!” button?
- Approach: Meta Provenance
An extension of network provenance that treats program as data.
Uses counterfactual reasoning to generate targeted repairs.
- Several interesting challenges
Example: Provenance trees are infinite.
- We are currently building a complete debugger
based on meta provenance
Questions?
Y. Wu
HotNets-2015 (Nov 17, 2015)
13