Model Checking with
Proofs and Counterexamples
Anvesh Komuravelli
Carnegie Mellon University
Joint work with Arie Gurfinkel, Sagar Chaki, and Edmund Clarke
Spacer
© Anvesh Komuravelli
Safety of Programs
Safe + Proof
Program P
+ Safety
Assertions
Automatic verification
for
assertion failures
0. x := 0;
1. while (x < n) {
// x ≥ 0
1. x := x + 1;
}
3. x := -x;
4. assert (x ≤ 0)
Unsafe +
Counterexample
Loop
Invariant
Spacer
© Anvesh Komuravelli
2
Safety of Programs
While-programs
(unbounded variables,
no procedure calls,
no memory operations)
Safe + Proof
Program P
+ Safety
Assertions
Automatic verification
for
assertion failures
0. x := 0;
1. while (x < n) {
2. x := x + 1;
}
3. x := -x;
4. assert (x ≤ -1)
Unsafe +
Counterexample
Counterexample Trace:
pc=0. x=0, n=0
pc=1. else branch
pc=3. x=0
pc=4. error
Spacer
© Anvesh Komuravelli
3
Algorithms for Safety
1. Safety is undecidable!
• Reduction from Halting Problem to safety of a 2-counter machine
2. Existing algorithms use heuristics for verifying many programs in practice
3. Two broad classes of model checking algorithms:
A. Generalize feasible and safe behaviors (Proof-Based)
B. Eliminate infeasible and unsafe behaviors (Counterexample-based)
This talk: Improve (A) based on ideas from (B)
Spacer
© Anvesh Komuravelli
4
Background
on Proof-Based algorithms
Spacer
© Anvesh Komuravelli
5
Programs as Transition Systems
Transition System
Variables
Init condition
should never hold
Error condition
Transition relation
encodes how data and control
change after every instruction
“
0. x := 0;
1. while (x < n) {
2. x := x + 1;
}
3. x := -x;
4. assert (x ≤ 0)
“
Spacer
© Anvesh Komuravelli
6
SAT-Based Model Checking
Transition System
Counterexample of length 0?
Counterexample of length 1?
SAT?
SAT?
…
1. Boolean SAT is NP-complete, but we have efficient solvers today
2. SAT modulo theories (SMT) for handling arithmetic, etc.
3. Eg:
is unsatisfiable for integers x, y
Bounded Model Checking, Clarke et al., TACAS 1999
Spacer
© Anvesh Komuravelli
7
SAT-Based Model Checking
Transition System
Are initial states safe?
Counterexample of length 0?
Counterexample of length 1?
SAT?
SAT?
…
Are 1 step-reachable
states safe?
Keep track of the
reachable states!
When do we stop?
No upper bound on the length of a counterexample!
Even for finite-state systems, the upper bound can be huge!
Bounded Model Checking, Clarke et al., TACAS 1999
Spacer
© Anvesh Komuravelli
8
Keep Track of the Reachable States
Usually
Hopeless!
reach(P)
err(x)
Initial
States
States
reachable in
≤1 steps
Spacer
States
reachable in
≤2 steps
© Anvesh Komuravelli
9
Reachable states can be diverging!
reachable states
at (pc=1)
0. x := 0;
1. while (x < n) {
2. x := x + 1;
}
3. x := -x;
4. assert (x ≤ 0)
n is a
symbolic
input
converged!
(diverging)
…
Spacer
© Anvesh Komuravelli
10
Generalize the reachable states!
err(x)
Generalize
[1] McMillan, Interpolation and SAT-Based Model Checking, CAV 2003
[2] McMillan, Lazy Abstraction with Interpolants, CAV 2006
Spacer
(Heuristics using
Craig Interpolation[1,2])
© Anvesh Komuravelli
11
Generalize the reachable states!
reach(P)
err(x)
Spacer
© Anvesh Komuravelli
12
Proofs and Invariants
reachable states
at (pc=1)
0. x := 0;
1. while (x < n) {
2. x := x + 1;
}
3. x := -x;
4. assert (x ≤ 0)
x ≥ 0 is a
loop invariant!
The actual set of
reachable states
is stronger:
0≤x≤n
Proof of
Safety
Proof of
“Bounded” Safety
(diverging)
…
Spacer
© Anvesh Komuravelli
13
Many heuristics for generalizations!
One possible
generalization
err(x)
Another
possible
generalization
• No unique generalization!
• Today’s best algorithms for hardware verification are SAT-based
• Several competitive algorithms exist for software
Spacer
© Anvesh Komuravelli
14
The Problem
Generalizations are not always sufficient
Spacer
© Anvesh Komuravelli
15
Generalizations can suffer from local view
x = y = z = w = 0;
while (*) {
// loop invariant:
// (x ≥ 4 => y ≥ 100) && (z ≤ 10w)
if (*) {x++; y += 100;}
else if (*)
if (x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {
y = −y;
}
t = 1;
w += t; z += 10t;
}
assert (!(x ≥ 4 && y ≤ 2))
State-of-the-art Tool Z3
cannot verify in an hour
Proofs of
Bounded Safety
never connect
z and w
Source: Automatically Refining Abstract Interpretations, Gulavani, Chakraborty, Nori and Rajamani, TACAS ‘08.
Spacer
© Anvesh Komuravelli
16
Abstractions for better generalizations!
x = y = z = w = 0;
while (*) {
// loop invariant:
// (x ≥ 4 => y ≥ 100) && (z ≤ 10w)
if (*) {x++; y += 100;}
else if (*)
if (x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {
y = −y;
}
t = 1;
w += t; z += 10t;
}
assert (!(x ≥ 4 && y ≤ 2))
State-of-the-art Tool Z3
cannot verify in an hour
t = *;
Verifies the abstraction
in < 1 sec.
Abstractions only add
behaviors
Source: Automatically Refining Abstract Interpretations, Gulavani, Chakraborty, Nori and Rajamani, TACAS ‘08.
Spacer
© Anvesh Komuravelli
17
How to obtain helpful abstractions automatically?
1. An abstraction of the program can dramatically improve
generalizations!
2. How to obtain helpful abstractions?
3. How to efficiently and automatically maintain
abstractions?
Spacer
© Anvesh Komuravelli
18
Abstractions are great, but not always!
Reachable
states of an
abstraction
Spurious
counterexample
err(x)
reach(P)
The second
class of
algorithms
CounterExampleGuided Abstraction
Refinement
(CEGAR)[1]
[1] Clarke et al., Counterexample-Guided Abstraction Refinement, CAV 2000.
Spacer
© Anvesh Komuravelli
19
Our algorithm Spacer
Spacer
© Anvesh Komuravelli
20
Spacer
(Software Proof-based Abstraction with CounterExample-based Refinement)
Program
Abstract
No
Fix a Bound
Proof-Based Abstraction
Invariants?
Safety Proof
Check
Safety
Refine
CEGAR
No
Feasible?
Counterexample
Yes
Yes
Spacer
© Anvesh Komuravelli
21
Spacer
(Software Proof-based Abstraction with CounterExample-based Refinement)
Program
Abstract
No
Fix a Bound
Proof-Based Abstraction
Invariants?
Safety Proof
Check
Safety
Refine
CEGAR
No
Proofs from
Abstractions
Feasible?
Counterexample
Yes
Yes
Spacer
© Anvesh Komuravelli
22
Spacer
(Software Proof-based Abstraction with CounterExample-based Refinement)
Program
Refinement using
Spurious
Abstract
Counterexamples
No
Fix a Bound
Proof-Based Abstraction
Invariants?
Safety Proof
Check
Safety
Refine
CEGAR
No
Feasible?
Counterexample
Yes
Yes
Spacer
© Anvesh Komuravelli
23
Spacer
(Software Proof-based Abstraction with CounterExample-based Refinement)
Program
Abstract
No
Fix a Proof-Based
Bound
Abstraction
Proof-Based Abstraction
Invariants?
Safety Proof
Check
Safety
Refine
CEGAR
No
Feasible?
Counterexample
Yes
Yes
Spacer
© Anvesh Komuravelli
24
Spacer
(Software Proof-based Abstraction with CounterExample-based Refinement)
Program
Abstract
No
Fix a Bound
Proof-Based Abstraction
Invariants?
Safety Proof
Check
Safety
Refine
CEGAR
No
Feasible?
Counterexample
Yes
Yes
Spacer
© Anvesh Komuravelli
25
Spacer at a high level
Abstraction
reach(A1)
reach(P)
err(x)
Initial
States
of P
Generalization/Proof
Initial
States
of A1
Proof-Based
Abstraction
Spacer
© Anvesh Komuravelli
26
Spacer at a high level
reach(A1)
reach(P)
err(x)
Spurious
counterexample
Reachable states
of A1 in ≤1 steps
Spacer
© Anvesh Komuravelli
27
Spacer at a high level
Refine A1 to A2
eliminating the
spurious
counterexample
reach(A2)
reach(P)
err(x)
Reachable states
of A2 in ≤1 steps
Generalization/
Proof
Reachable states
of P in ≤1 steps
Spacer
© Anvesh Komuravelli
28
Spacer at a high level
Fresh abstraction,
to avoid bias
reach(A3)
reach(P)
err(x)
Proof-Based
Abstraction
Reachable states
of A3 in ≤1 steps
Reachable states
of P in ≤1 steps
Spacer
© Anvesh Komuravelli
29
Key Ideas of Spacer
1. Abstractions help obtain (hopefully) more general proofs
2. First integration of Proof-Based Abstraction with
SAT/SMT-Based Model Checking
3. Orthogonal to heuristics for Interpolation/Generalization
Implementation and Experimental Evidence on C Programs
Spacer
© Anvesh Komuravelli
30
Abstractions add a new dimension
SAT-Based Model Checking
…
A
b
s
t
r
a
c
t
…
…
…
Spacer
© Anvesh Komuravelli
31
SAT-Based Model Checking with Abstractions
Under-approximations
need not be
monotonic
…
A
b
s
t
r
a
c
t
…
…
…
Spacer
© Anvesh Komuravelli
32
SAT-Based Model Checking with Abstractions
Under-approximations
…
…
A
b
s
t
r
a
c
t
…
non-trivial
abstraction
…
Spacer
© Anvesh Komuravelli
33
Spacer
(Software Proof-based Abstraction with CounterExample-based Refinement)
Program
Abstract
No
Fix a Bound
Proof-Based Abstraction
Invariants?
Safety Proof
Check
Safety
Refine
CEGAR
No
Feasible?
Counterexample
Yes
Yes
Spacer
© Anvesh Komuravelli
34
Spacer on Example
Spacer
© Anvesh Komuravelli
35
Spacer on Example
x = y = z = w = 0;
c = 0;
Add Counters
while (*) {
// (y > 10w) => (z < 100x), z ≤ 100x,
// x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1
assume (c < 2);
if (*) {x++; y += 100;}
else if (* && x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {y = −y;}
else assume (false);
w++; z += 10;
c += 1;
Loop
Invariants
}
assert (!(x ≥ 4 && y ≤ 2));
Bound  Solve
Spacer
© Anvesh Komuravelli
36
Spacer on Example
Treat as conjectured
unbounded invariants.
(as in Houdini[1]).
x = y = z = w = 0;
c = 0;
Specific to
under-approx.
Extract
Unbounded
Invariants
while (*) {
// (y > 10w) => (z < 100x), z ≤ 100x,
// x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1
assume (c < 2);
if (*) {x++; y += 100;}
else if (* && x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {y = −y;}
else assume (false);
w++; z += 10;
c += 1;
Preserved!
Depend on
counter
}
assert (!(x ≥ 4 && y ≤ 2));
[1] Houdini, an annotation
assistant for ESC/Java,
C. Flanagan and K.R.M. Leino, 2001
Bound  Solve  Unbounded?
Spacer
© Anvesh Komuravelli
37
Spacer on Example
x = y = z = w = 0;
c = 0;
Invariants are
too weak!
while (*) {
// (y > 10w) => (z < 100x), z ≤ 100x,
// x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1
if (*) {x++; y += 100;}
else if (* && x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {y = −y;}
else assume (false);
w++; z += 10;
c += 1;
✖
}
assert (!(x ≥ 4 && y ≤ 2));
Bound  Solve  Unbounded? NO  Abstract
Spacer
© Anvesh Komuravelli
38
Spacer on Example
Strengthen
with
Invariants
x = y = z = w = 0;
c = 0;
assume (y > 10w => z < 100x, z ≤ 100x);
while (*) {
// (y > 10w) => (z < 100x), z ≤ 100x,
// x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1
assume (c < 2);
if (*) {x++; y += 100;}
else if (* && x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {y = −y;}
else assume (false);
w++; z += 10;
c += 1;
assume (y > 10w => z < 100x, z ≤ 100x);
}
Redundant
assert (!(x ≥ 4 && y ≤ 2));
Bound  Solve  Unbounded? NO  Abstract
Spacer
© Anvesh Komuravelli
39
Spacer on Example
x = y = z = w = 0;
c = 0;
assume (y > 10w => z < 100x, z ≤ 100x);
while (*) {
// (y > 10w) => (z < 100x), z ≤ 100x,
// x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1
assume (c < 2);
if (*) {x++; y = *;}
else if (* && x ≥ 4) {x++; y = *;}
else if (y > 10w && z ≥ 100x) {y = *;}
else assume (false);
w = *; z = *;
c += 1;
assume (y > 10w => z < 100x, z ≤ 100x);
}
Proof-Based
Abstraction
assert (!(x ≥ 4 && y ≤ 2));
Bound  Solve  Unbounded? NO  Abstract
Spacer
© Anvesh Komuravelli
40
Spacer on Example
x = y = z = w = 0;
c = 0;
assume (y > 10w => z < 100x, z ≤ 100x);
while (*) {
Counterexample!
assume (c < 4);
if (*) {x++; y = *;}
else if (* && x ≥ 4) {x++; y = *;}
else if (y > 10w && z ≥ 100x) {y = *;}
else assume (false);
w = *; z = *;
c += 1;
assume (y > 10w => z < 100x, z ≤ 100x);
Increment x to 4
Choose y arbitrarily
Concrete control
path is infeasible
Concretize
}
assert (!(x ≥ 4 && y ≤ 2));
Bound  Solve  Feasible? NO  Refine
Spacer
© Anvesh Komuravelli
41
Spacer on Example
x = y = z = w = 0;
c = 0;
assume (y > 10w => z < 100x, z ≤ 100x);
while (*) {
assume (c < 4);
if (*) {x++; y += 100;}
else if (* && x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {y = −y;}
else assume (false);
w = *; z = *;
c += 1;
assume (y > 10w => z < 100x, z ≤ 100x);
CEGAR
}
assert (!(x ≥ 4 && y ≤ 2));
Bound
 Solve  Feasible? NO  Refine
Spacer
© Anvesh Komuravelli
42
Spacer on Example
Invariants
x = y = z = w = 0;
c = 0;
assume (y > 10w => z < 100x, z ≤ 100x);
while (*) {
// (y > 10w) => (z < 100x), z ≤ 100x
// y > 0, (x > 0) => (y ≥ 100)
assume (c < 4);
if (*) {x++; y += 100;}
else if (* && x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {y = −y;}
else assume (false);
w = *; z = *;
c += 1;
assume (y > 10w => z < 100x, z ≤ 100x);
}
assert (!(x ≥ 4 && y ≤ 2));
Bound  Solve  Unbounded? YES
Spacer
© Anvesh Komuravelli
43
Implementation Details
Spacer
© Anvesh Komuravelli
44
Three Key Components
Focus
1. Extracting Unbounded Invariants
2. Proof-Based Abstraction
3. Counterexample Analysis and Refinement
How can we efficiently use today’s SAT/SMT solvers?
Spacer
© Anvesh Komuravelli
45
Extracting Unbounded Invariants
An invariant for the transition system
is a formula that holds for the initial states and after every transition
(initial)
φ is an invariant iff
(transition)
Given: A set L of conjectures for invariants, each including “initial states”.
Goal: A maximal I
L s.t.
Spacer
© Anvesh Komuravelli
46
Extracting Unbounded Invariants
Given: A set L of conjectures for invariants, each including “initial states”.
Goal: A maximal I
L s.t.
SAT?
unsat
sat, making
true
Re-start from
scratch!
…
SAT?
until fixed point
Spacer
© Anvesh Komuravelli
47
Extracting Unbounded Invariants
Given: A set L of conjectures for invariants, each including “initial states”.
L s.t.
Introduce
assumption
variables
unsat
sat, making
true
…
Goal: A maximal I
Spacer
© Anvesh Komuravelli
48
Proof-Based Abstraction
A proof of “bounded” safety for
is a formula that holds of the initial states and after every transition
up to the given bound, and excludes error states.
Proof of
Bounded Safety
(initial)
(transition)
(safety)
Not all of is
necessary!
Spacer
© Anvesh Komuravelli
49
Proof-Based Abstraction
unsat
unsat
UNSAT
core
unsat
Iteratively
minimize
Spacer
© Anvesh Komuravelli
50
What have we seen so far?
1. Generalizing reachable states can be hard!
2. Abstractions can really help!
3. Algorithm Spacer that combines abstraction refinement
with SAT-based model checking
4. How Spacer can be efficiently automated
Spacer
© Anvesh Komuravelli
51
Tool and
Experimental Evaluation
Spacer
© Anvesh Komuravelli
52
Spacer Tool
C Program
Existing Front-end based on LLVM
(Horn-SMT) Logical Encoding
Proof-Based
Abstraction,
CEGAR, etc.
Spacer Backend
(using Z3’s framework)
Theories handled:
Linear Arithmetic
(Rationals and Integers),
Bitvectors
Spacer
© Anvesh Komuravelli
53
Spacer Tool
Program
UnderApproximate
Abstract
No
Proof-Based Abstraction
Feasible?
Safety Proof
Check
Safety
Refine
CEGAR
No
Feasible?
Counterexample
Yes
Yes
SMT-Based
Model Checker in Z3
Spacer
© Anvesh Komuravelli
54
The hard example mentioned in the beginning
State-of-the-art Tool Z3
cannot verify in an hour
x = y = z = w = 0;
while (*) {
if (*) {x++; y += 100;}
else if (*)
if (x ≥ 4) {x++; y++;}
else if (y > 10w && z ≥ 100x) {
y = −y;
}
t = 1;
w += t; z += 10t;
}
assert (!(x ≥ 4 && y ≤ 2))
Spacer
automatically
verifies in under a
minute!
Source: Automatically Refining Abstract Interpretations, Gulavani, Chakraborty, Nori and Rajamani, TACAS ‘08.
Spacer
© Anvesh Komuravelli
55
Results on SV-COMP’13 Benchmarks
900
Advantage!
With abstraction (secs)
800
700
600
Time-out
500
Mem-out
400
300
200
100
0
0
100 200 300 400 500 600 700 800 900
No abstraction (secs)
Spacer
© Anvesh Komuravelli
56
Summary
Spacer
© Anvesh Komuravelli
57
Conclusion
 Focused Proofs: Abstractions guide Interpolation towards certain generalizations
 Combine Proof-Based Abstraction and Counterexample-Guided Refinement
 General framework independent of the underlying model checker
 Works in practice!
Future Directions
 Verification in presence of assumptions
 Different kinds of bounding/abstraction
 Synthesizing ghost code to help verification
Spacer
© Anvesh Komuravelli
For more details, read our CAV’13 paper!
Questions?
Spacer
© Anvesh Komuravelli
59
Counterexample Analysis and Refinement
Concrete:
Abstract:
An “abstract counterexample” is a finite length path consistent with error states
SAT
SAT ?
Feasibility
Check
Spacer
© Anvesh Komuravelli
60