A S URVEY OF I NCREMENTAL R EASONING A LGORITHMS
FOR DATALOG VARIANTS
Boris Motik
University of Oxford
December 8, 2016
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
0/20
Incremental Reasoning in Datalog
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
0/20
Incremental Reasoning in Datalog
DATALOG AND ITS A PPLICATION
Datalog: a DB query language / KR formalism based on if-then rules
E XAMPLE DATALOG P ROGRAM
hasParent(x, y ) ← hasMother (x, y)
hasAunt(x, z) ← hasParent(x, y ) ∧ hasSister (x, y )
Applications of datalog:
Semantic Web reasoning
RDFS and OWL 2 RL inference rules can be encoded into datalog
Supported in graphDB, Systap Big Data, Oracle DB, MarkLogic, RDFox, . . .
Data integration
Datalog rules (possibly with function symbols) can express source mappings
Enterprise data management
Datalog rules capture data dependencies and thus simplify maintenance
Supported in LogicBlox
Boris Motik
A Survey of Incremental Reasoning
1/20
Incremental Reasoning in Datalog
DATALOG R EASONING VIA M ATERIALISATION
Main task: compute entailments of program and a set of facts
E XAMPLE DATALOG P ROGRAM
hasParent(x, y ) ← hasMother (x, y)
hasAunt(x, z) ← hasParent(x, y ) ∧ hasSister (x, y )
E XAMPLE S ET OF FACTS
hasMother (Peter , Paula)
hasSister (Paula, Ann)
E XAMPLE E NTAILMENTS
hasParent(Peter , Paula)
hasAunt(Peter , Ann)
Materialisation: apply rules, store consequences, repeat while change
Main benefit: all consequences are precomputed during preprocessing
Main drawback: dealing with changes in the input
Boris Motik
A Survey of Incremental Reasoning
2/20
Incremental Reasoning in Datalog
I NCREMENTAL R EASONING
Inputs:
a set of explicit facts E
a datalog program P
an ‘old’ materialisation I of P on E
a set E − of facts to delete from E and a set E + of facts to add to E
Goal: compute the ‘new’ materialisation of P on (E \ E − ) ∪ E +
Boris Motik
A Survey of Incremental Reasoning
3/20
Incremental Reasoning in Datalog
I NCREMENTAL R EASONING
Inputs:
a set of explicit facts E
a datalog program P
an ‘old’ materialisation I of P on E
a set E − of facts to delete from E and a set E + of facts to add to E
Goal: compute the ‘new’ materialisation of P on (E \ E − ) ∪ E +
Do this efficiently ⇒ without recomputing most of I
Boris Motik
A Survey of Incremental Reasoning
3/20
Incremental Reasoning in Datalog
I NCREMENTAL R EASONING
Inputs:
a set of explicit facts E
a datalog program P
an ‘old’ materialisation I of P on E
a set E − of facts to delete from E and a set E + of facts to add to E
Goal: compute the ‘new’ materialisation of P on (E \ E − ) ∪ E +
Do this efficiently ⇒ without recomputing most of I
Note: we delete only explicit facts
Users cannot delete inferred facts
⇒ That is a subject for belief revision
Boris Motik
A Survey of Incremental Reasoning
3/20
Incremental Reasoning in Datalog
K EY I SSUES
Addition usually not a problem: just use the seminaı̈ve algorithm
Deleting facts with multiple derivations is a source of difficulty
E XAMPLE P ROGRAM
A(y ) ← A(x) ∧ R(x, y)
E XAMPLE DATA & C ONSEQUENCES
Explicit Data
A(a), R(a, c), R(a, d), A(b), R(b, c)
Consequences
A(c), A(d)
Delete A(a)
Boris Motik
A Survey of Incremental Reasoning
4/20
Incremental Reasoning in Datalog
K EY I SSUES
Addition usually not a problem: just use the seminaı̈ve algorithm
Deleting facts with multiple derivations is a source of difficulty
E XAMPLE P ROGRAM
A(y ) ← A(x) ∧ R(x, y)
E XAMPLE DATA & C ONSEQUENCES
Explicit Data
A(a), R(a, c), R(a, d), A(b), R(b, c)
Consequences
A(c), A(d)
Delete A(a)
A(c) ‘survives’
A(b) and R(b, c) still derive A(c)
Boris Motik
A Survey of Incremental Reasoning
4/20
Incremental Reasoning in Datalog
K EY I SSUES
Addition usually not a problem: just use the seminaı̈ve algorithm
Deleting facts with multiple derivations is a source of difficulty
E XAMPLE P ROGRAM
A(y ) ← A(x) ∧ R(x, y)
E XAMPLE DATA & C ONSEQUENCES
Explicit Data
A(a), R(a, c), R(a, d), A(b), R(b, c)
Consequences
A(c), A(d)
Delete A(a)
A(c) ‘survives’
A(b) and R(b, c) still derive A(c)
A(d) ‘dies’
There is no alternative derivation of A(d)
Boris Motik
A Survey of Incremental Reasoning
4/20
Incremental Reasoning in Datalog
D ESIDERATA FOR AND L IMITS OF I NCREMENTAL R EASONING
Rematerialisation (i.e., restarting from scratch) is a possible solution
Ideally, incremental reasoning should be faster than rematerialisation
Unrealistic requirement: if E − = E, rematerialisation requires no work
⇒ An incremental algorithm will necessarily do more work
Next best goal for worst-case performance:
Undo precisely rule instances that do not hold any more, and apply precisely new
rule instances
⇒ Still not attainable in most cases
Weak worst-case performance:
Do not perform more work than in the ‘old’ and the ‘new’ materialisation combined
Overall goal: more efficient than rematerialisation on small E − and E +
Boris Motik
A Survey of Incremental Reasoning
5/20
Counting-Based Algorithms
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
5/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
1
1
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
1
1
2
Increment the counter after each derivation
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
1
1
2
1
Increment the counter after each derivation
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
1
1
2
1
Increment the counter after each derivation
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
1
1
2
1
Increment the counter after each derivation
1
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
1
1
2
1
Increment the counter after each derivation
Delete A(a):
Decrease its counter
1
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
0
1
2
1
Increment the counter after each derivation
Delete A(a):
Decrease its counter
The counter of A(a) reaches zero, so propagate deletion
1
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
0
1
1
1
Increment the counter after each derivation
Delete A(a):
Decrease its counter
The counter of A(a) reaches zero, so propagate deletion
1
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
0
1
1
1
Increment the counter after each derivation
Delete A(a):
Decrease its counter
The counter of A(a) reaches zero, so propagate deletion
Delete B(a):
1
Decrease its counter
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
0
0
1
1
Increment the counter after each derivation
Delete A(a):
Decrease its counter
The counter of A(a) reaches zero, so propagate deletion
Delete B(a):
1
Decrease its counter
The counter of B(a) reaches zero, so propagate deletion
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
0
0
0
1
Increment the counter after each derivation
Delete A(a):
Decrease its counter
The counter of A(a) reaches zero, so propagate deletion
Delete B(a):
1
Decrease its counter
The counter of B(a) reaches zero, so propagate deletion
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
0
0
0
0
Increment the counter after each derivation
Delete A(a):
Decrease its counter
The counter of A(a) reaches zero, so propagate deletion
Delete B(a):
1
Decrease its counter
The counter of B(a) reaches zero, so propagate deletion
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
0
0
0
0
Increment the counter after each derivation
Delete A(a):
Decrease its counter
The counter of A(a) reaches zero, so propagate deletion
Delete B(a):
0
Decrease its counter
The counter of B(a) reaches zero, so propagate deletion
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
BASIC C OUNTING (N ONRECURSIVE VARIANT )
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
Associate with each fact a counter initialised to zero
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
0
0
0
0
Increment the counter after each derivation
Delete A(a):
Decrease its counter
The counter of A(a) reaches zero, so propagate deletion
Delete B(a):
0
Decrease its counter
The counter of B(a) reaches zero, so propagate deletion
Worst-case optimal for nonrecursive rules
Boris Motik
A Survey of Incremental Reasoning
6/20
Counting-Based Algorithms
P ROBLEMS OF C OUNTING AND R ECURSION
E XAMPLE
B(x) ← A(x),
A(a)
B(a)
A(x) ← B(x)
1
1
Boris Motik
A Survey of Incremental Reasoning
7/20
Counting-Based Algorithms
P ROBLEMS OF C OUNTING AND R ECURSION
E XAMPLE
B(x) ← A(x),
A(a)
B(a)
A(x) ← B(x)
1
2
Boris Motik
A Survey of Incremental Reasoning
7/20
Counting-Based Algorithms
P ROBLEMS OF C OUNTING AND R ECURSION
E XAMPLE
B(x) ← A(x),
A(a)
B(a)
A(x) ← B(x)
2
2
Boris Motik
A Survey of Incremental Reasoning
7/20
Counting-Based Algorithms
P ROBLEMS OF C OUNTING AND R ECURSION
E XAMPLE
B(x) ← A(x),
A(a)
B(a)
A(x) ← B(x)
2
2
Boris Motik
A Survey of Incremental Reasoning
7/20
Counting-Based Algorithms
P ROBLEMS OF C OUNTING AND R ECURSION
E XAMPLE
B(x) ← A(x),
A(a)
B(a)
A(x) ← B(x)
1
2
Boris Motik
A Survey of Incremental Reasoning
7/20
Counting-Based Algorithms
P ROBLEMS OF C OUNTING AND R ECURSION
E XAMPLE
B(x) ← A(x),
A(a)
B(a)
A(x) ← B(x)
1
2
Boris Motik
A Survey of Incremental Reasoning
7/20
Counting-Based Algorithms
P ROBLEMS OF C OUNTING AND R ECURSION
E XAMPLE
B(x) ← A(x),
A(a)
B(a)
A(x) ← B(x)
1
1
Boris Motik
A Survey of Incremental Reasoning
7/20
Counting-Based Algorithms
P ROBLEMS OF C OUNTING AND R ECURSION
E XAMPLE
B(x) ← A(x),
A(a)
B(a)
1
1
A(x) ← B(x)
Cyclic dependencies cause problems
Similar to why garbage collection by reference counting
does not
Boris Motik
A Survey of Incremental Reasoning
7/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
1
1
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
1
1
1
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
1
1
2
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
1
1
2
1
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
...
1
1
2
1
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
1
1
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
2
1
1
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
C0 (x) ← B(x)
1
1
2
1
1
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
1
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
C0 (x) ← B(x)
0
0
2
1
1
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Delete A(a) and B(a) by undoing derivations
1
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
C0 (x) ← B(x)
0
0
0
1
1
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Delete A(a) and B(a) by undoing derivations
1
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
C0 (x) ← B(x)
0
0
0
1
0
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Delete A(a) and B(a) by undoing derivations
1
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
C0 (x) ← B(x)
0
0
0
1
0
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Delete A(a) and B(a) by undoing derivations
0
Boris Motik
A Survey of Incremental Reasoning
8/20
Counting-Based Algorithms
C OUNTING AND R ECURSION
E XAMPLE
C0 (x) ← A(x)
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
C0 (x) ← B(x)
0
0
0
0
0
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Associate with each fact an array of counters, one
per iteration
Delete A(a) and B(a) by undoing derivations
0
Boris Motik
A Survey of Incremental Reasoning
8/20
The Delete/Rederive (DRed) Algorithm
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
8/20
The Delete/Rederive (DRed) Algorithm
T HE DR ED A LGORITHM AT A G LANCE
Delete/Rederive (DRed): state of the art incremental maintenance algorithm
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
A(a)
B(a)
Boris Motik
A Survey of Incremental Reasoning
9/20
The Delete/Rederive (DRed) Algorithm
T HE DR ED A LGORITHM AT A G LANCE
Delete/Rederive (DRed): state of the art incremental maintenance algorithm
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Materialise initial facts
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
Boris Motik
A Survey of Incremental Reasoning
9/20
The Delete/Rederive (DRed) Algorithm
T HE DR ED A LGORITHM AT A G LANCE
Delete/Rederive (DRed): state of the art incremental maintenance algorithm
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Materialise initial facts
Delete A(a) using DRed:
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
Boris Motik
A Survey of Incremental Reasoning
9/20
The Delete/Rederive (DRed) Algorithm
T HE DR ED A LGORITHM AT A G LANCE
Delete/Rederive (DRed): state of the art incremental maintenance algorithm
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Materialise initial facts
Delete A(a) using DRed:
1
Delete all facts with a derivation from A(a)
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
C0D (x) ← AD (x)
C0D (x) ← B D (x)
D (x) for 1 ≤ i ≤ n
CiD (x) ← Ci−1
C0D (x) ← CnD (x)
Boris Motik
A Survey of Incremental Reasoning
9/20
The Delete/Rederive (DRed) Algorithm
T HE DR ED A LGORITHM AT A G LANCE
Delete/Rederive (DRed): state of the art incremental maintenance algorithm
E XAMPLE
C0 (x) ← A(x)
C0 (x) ← B(x)
Ci (x) ← Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← Cn (x)
Materialise initial facts
Delete A(a) using DRed:
1
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
Delete all facts with a derivation from A(a)
C0D (x) ← AD (x)
C0D (x) ← B D (x)
D (x) for 1 ≤ i ≤ n
CiD (x) ← Ci−1
C0D (x) ← CnD (x)
2
Rederive facts that have an alternative derivation
C0 (x) ← C0D (x) ∧ A(x)
C0 (x) ← C0D (x) ∧ B(x)
Ci (x) ← CiD (x) ∧ Ci−1 (x) for 1 ≤ i ≤ n
C0 (x) ← C0D (x) ∧ Cn (x)
Boris Motik
A Survey of Incremental Reasoning
9/20
The Forward/Backward/Forward (FBF) Algorithm
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
9/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
Delete A(a) using FBF:
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
?
Delete A(a) using FBF:
1
Is A(a) derivable in any other way?
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
×
Delete A(a) using FBF:
1
2
Is A(a) derivable in any other way?
No ⇒ delete
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
×
Delete A(a) using FBF:
1
?
2
3
Is A(a) derivable in any other way?
No ⇒ delete
As in DRed, identify C0 (a) as derivable from A(a)
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
×
?
?
Delete A(a) using FBF:
1
2
3
4
Is A(a) derivable in any other way?
No ⇒ delete
As in DRed, identify C0 (a) as derivable from A(a)
Apply the rules to C0 (a) ‘backwards’ ⇒ by C0 (x) ← B(x), we get B(a)
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
×
X
?
Delete A(a) using FBF:
1
2
3
4
5
Is A(a) derivable in any other way?
No ⇒ delete
As in DRed, identify C0 (a) as derivable from A(a)
Apply the rules to C0 (a) ‘backwards’ ⇒ by C0 (x) ← B(x), we get B(a)
B(a) is explicit so it is derivable
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
×
X
X
Delete A(a) using FBF:
1
2
3
4
5
6
Is A(a) derivable in any other way?
No ⇒ delete
As in DRed, identify C0 (a) as derivable from A(a)
Apply the rules to C0 (a) ‘backwards’ ⇒ by C0 (x) ← B(x), we get B(a)
B(a) is explicit so it is derivable
So C0 (a) is derivable too
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
The Forward/Backward/Forward (FBF) Algorithm
T HE F ORWARD /BACKWARD /F ORWARD A LGORITHM
Facts often have many derivations, so many facts get deleted in the first step
The Forward/Backward/Forward algorithm looks for alternatives immediately
A(a)
B(a)
C0 (a)
C1 (a)
...
Cn (a)
×
X
X
Delete A(a) using FBF:
1
2
3
4
5
6
7
Is A(a) derivable in any other way?
No ⇒ delete
As in DRed, identify C0 (a) as derivable from A(a)
Apply the rules to C0 (a) ‘backwards’ ⇒ by C0 (x) ← B(x), we get B(a)
B(a) is explicit so it is derivable
So C0 (a) is derivable too
Stop propagation and terminate
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Incremental Update of Datalog Materialisation: the
Backward/Forward Algorithm. AAAI 2015
Boris Motik
A Survey of Incremental Reasoning
10/20
Comparing the Three Algorithms
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
10/20
Comparing the Three Algorithms
C OMPARISON : R ECURSIVE C OUNTING
E XAMPLE P ROGRAM + I NITIAL DATA
A(y) ← A(x) ∧ R(x, y ),
A(a0 ), R(a0 , a1 ), R(a1 , a2 ), . . . , R(an−1 , an )
After adding A(ai )
Initial state
A(a0 )
A(a1 )
...
A(ai )
A(ai+1 )
...
A(an )
1
A(a0 )
A(a1 )
...
A(ai )
A(ai+1 )
...
A(an )
1
1
1
1
1
1
1
1
1
1
Even if nothing changes, we can redo initial work twice (if i = 2)
Not worst-case optimal
The amount of repeated work gets smaller as i approaches n
⇒ Intuition: long multiple inference chains can be problematical
Can be made weakly worst-case optimal
Boris Motik
A Survey of Incremental Reasoning
11/20
Comparing the Three Algorithms
C OMPARISON : DR ED AND FBF
E XAMPLE P ROGRAM + I NITIAL DATA
A(y ) ← A(x) ∧ R(x, y),
A(a0 ), A(ai ), R(a0 , a1 ), R(a1 , a2 ), . . . , R(an−1 , an )
Update task: delete A(ai )
FBF reproves A(aj ) with 1 ≤ j ≤ i
DRed deletes and then reproves
A(aj ) with j ≥ i
More efficient if j is close to 0
More efficient if j is close to n
⇒ Suitable when deletion is at the
‘front’ of inference chains
⇒ Suitable when deletion is at the
‘end’ of inference chains
⇒ Works well if facts can be
reproved easily
Can be made weakly worst-case
optimal
Not weakly worst-case optimal:
‘old’ rule instances examined once
more in backward chaining
Boris Motik
A Survey of Incremental Reasoning
12/20
Extending Datalog: Stratified Negation and Equality
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
12/20
Extending Datalog: Stratified Negation and Equality
DATALOG WITH S TRATIFIED N EGATION
E XAMPLE P ROGRAM + I NITIAL DATA
B(x) ← U(x) ∧ not A(x),
C(x) ← U(x) ∧ not B(x),
D(x) ← U(x) ∧ not C(x),
U(a)
After adding A(a)
Initial state
B(a)
D(a)
A(a)
C(a)
Deletion and insertion are not independent:
Insertion into negated atoms is deletion and vice versa
⇒ Must apply deletion and insertion per stratum
Considerable source of complexity for DRed and FBF:
Cannot update materialisation in the end
For each stratum, both ‘old’ and ‘new’ materialisation must be kept
Ensuring nonrepetition is not straightforward
Boris Motik
A Survey of Incremental Reasoning
13/20
Extending Datalog: Stratified Negation and Equality
DATALOG WITH E QUALITY (I)
E XAMPLE P ROGRAM + I NITIAL DATA
y1 ≈ y2 ← R(x, y1 ) ∧ R(x, y2 ),
C(x) ← A(x) ∧ B(x),
Materialisation
R(c, a), R(c, b), A(a), B(b)
Eliminate redundancy by keeping
representatives only:
R(c, a)
R(c, b)
a≈b
A(a)
A(b)
B(a)
B(b)
C(a)
C(b)
Representative of a is a
Representative of b is a
Representative of c is c
R(c, a)
a≈a
A(a)
B(a)
C(a)
Boris Motik
A Survey of Incremental Reasoning
14/20
Extending Datalog: Stratified Negation and Equality
DATALOG WITH E QUALITY (II)
E XAMPLE P ROGRAM + I NITIAL DATA
y1 ≈ y2 ← R(x, y1 ) ∧ R(x, y2 ),
C(x) ← A(x) ∧ B(x),
Initial representation
R(c, a), R(c, b), A(a), B(b)
Representation after deleting
R(c, b)
R(c, a)
a≈a
A(a)
B(a)
C(a)
R(c, a)
A(a)
B(b)
⇒ Deletion may require addition — B(b) in this case
In fact, the representation after deletion can have more facts!
Boris Motik
A Survey of Incremental Reasoning
15/20
Extending Datalog: Stratified Negation and Equality
DATALOG WITH E QUALITY (III)
DRed algorithm not very efficient due to equality replacement rules:
A(x) ← A(y) ∧ x ≈ y, B(x) ← B(y) ∧ x ≈ y, . . .
Deleting a ≈ b results in deleting all facts containing a or b
Equality rules are highly recursive — the problematical case for DRed
⇒ Often overdeletes most of the materialisation
FBF is much more efficient:
Eager reproving particularly effective with equality
Can efficiently handle small updates, particularly if equality does not change
B. Motik, Y. Nenov, R. Piro, and I. Horrocks: Combining Rewriting and Incremental
Materialisation Maintenance for Datalog Programs with Equality. IJCAI 2015
Boris Motik
A Survey of Incremental Reasoning
16/20
Other Approaches: FOIESs & Truth Maintenance Systems
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
16/20
Other Approaches: FOIESs & Truth Maintenance Systems
F IRST-O RDER I NCREMENTAL E VALUATION S YSTEMS (FOIES S )
For single-tuple changes, update the materialisation using a finite number of
first-order queries
I.e., compute updates using SQL!
E XAMPLE P ROGRAM
R(x, z) ← R(x, y) ∧ R(y, z)
R
a
R
b
R
R
Boris Motik
c
d
R
e
R
f
A Survey of Incremental Reasoning
17/20
Other Approaches: FOIESs & Truth Maintenance Systems
F IRST-O RDER I NCREMENTAL E VALUATION S YSTEMS (FOIES S )
For single-tuple changes, update the materialisation using a finite number of
first-order queries
I.e., compute updates using SQL!
E XAMPLE P ROGRAM
R(x, z) ← R(x, y) ∧ R(y, z)
R
a
R
b
R
R
c
Rnew
d
R
e
R
f
Add Rnew (c, d)
Boris Motik
A Survey of Incremental Reasoning
17/20
Other Approaches: FOIESs & Truth Maintenance Systems
F IRST-O RDER I NCREMENTAL E VALUATION S YSTEMS (FOIES S )
For single-tuple changes, update the materialisation using a finite number of
first-order queries
I.e., compute updates using SQL!
E XAMPLE P ROGRAM
R(x, z) ← R(x, y) ∧ R(y, z)
R
R
R
a
R
b
R
c
R
Rnew
d
R
e
R
f
Add Rnew (c, d)
Evaluate R(x, y ) ∧ Rnew (y, z) and add R(x, z)
Boris Motik
A Survey of Incremental Reasoning
17/20
Other Approaches: FOIESs & Truth Maintenance Systems
F IRST-O RDER I NCREMENTAL E VALUATION S YSTEMS (FOIES S )
For single-tuple changes, update the materialisation using a finite number of
first-order queries
I.e., compute updates using SQL!
E XAMPLE P ROGRAM
R(x, z) ← R(x, y) ∧ R(y, z)
R
a
R
b
R
R
R
R
c
R
Rnew
d
R
R
e
R
f
Add Rnew (c, d)
Evaluate R(x, y ) ∧ Rnew (y, z) and add R(x, z)
Evaluate Rnew (x, y ) ∧ R(y, z) and add R(x, z)
Boris Motik
A Survey of Incremental Reasoning
17/20
Other Approaches: FOIESs & Truth Maintenance Systems
F IRST-O RDER I NCREMENTAL E VALUATION S YSTEMS (FOIES S )
For single-tuple changes, update the materialisation using a finite number of
first-order queries
I.e., compute updates using SQL!
E XAMPLE P ROGRAM
R(x, z) ← R(x, y) ∧ R(y, z)
R
a
R
b
R
R
R
R
c
R
R
Rnew
d
R
R
e
R
f
R
R
R
Add Rnew (c, d)
Evaluate R(x, y ) ∧ Rnew (y, z) and add R(x, z)
Evaluate Rnew (x, y ) ∧ R(y, z) and add R(x, z)
Evaluate R(x, y ) ∧ Rnew (y, z) ∧ R(z, w) and add R(x, w)
Boris Motik
A Survey of Incremental Reasoning
17/20
Other Approaches: FOIESs & Truth Maintenance Systems
C APABILITIES OF FOIES S
Possible only in limited situations:
Adding (but not deleting) tuples on chain programs
Deleting tuples for transitivity over acyclic data
No FOIES exists for transitivity and cyclic graphs
Capabilities increase if we can maintain additional auxiliary relations
Still cannot handle transitivity over cyclic graphs for deletion
⇒ Neat idea, but not really suitable for general use
Boris Motik
A Survey of Incremental Reasoning
18/20
Other Approaches: FOIESs & Truth Maintenance Systems
T RUTH M AINTENANCE S YSTEMS (TMS S )
Justification: a set of facts and a rule that prove some fact
Rule A(x) ← B(x) ∧ C(x) and facts B(a) and C(a) justify A(a)
Can be seen as a rule instance A(a) ← B(a) ∧ C(a)
Main idea: store with each fact a set of justifications
Add a justification each time a fact is derived
Delete a justification each time a rule instance goes away
Delete a fact when its set of justifications drops to zero
Main problem: storing justifications incurs significant overhead
Particularly on highly recursive programs
⇒ Unlikely to be efficient enough for large knowledge bases
Boris Motik
A Survey of Incremental Reasoning
19/20
Conclusion
TABLE OF C ONTENTS
1
I NCREMENTAL R EASONING IN DATALOG
2
C OUNTING -BASED A LGORITHMS
3
T HE D ELETE /R EDERIVE (DR ED ) A LGORITHM
4
T HE F ORWARD /BACKWARD /F ORWARD (FBF) A LGORITHM
5
C OMPARING THE T HREE A LGORITHMS
6
E XTENDING DATALOG : S TRATIFIED N EGATION AND E QUALITY
7
OTHER A PPROACHES : FOIES S & T RUTH M AINTENANCE S YSTEMS
8
C ONCLUSION
Boris Motik
A Survey of Incremental Reasoning
19/20
Conclusion
C ONCLUSION
Three algorithms most suitable for practice:
Counting (with a nontrivial extension to recursion)
DRed
FBF
No algorithm is worst-case optimal
Can in fact be twice as inefficient as rematerialisation
Counting and DRed are optimal in a weak sense
They consider at most all inferences from the ‘old’ and the ‘new’ derivation
FBF is not weakly optimal due to backward chaining
Counting has a fixed overhead that shows up on small updates
FBF seems to be particularly suited to small updates
Boris Motik
A Survey of Incremental Reasoning
20/20