Model Checking
Doron A. Peled
University of Warwick
Coventry, UK
[email protected]
Modelling and specification for
verification and validation


How to specify what the software is
supposed to do?
How to model it in a way that allows us
to check it?
Sequential systems.




Perform some computational task.
Have some initial condition, e.g.,
0in A[i] integer.
Have some final assertion, e.g.,
0in-1 A[i]A[i+1].
(What is the problem with this spec?)
Are supposed to terminate.
Concurrent Systems
Involve several computation agents.
Termination may indicate an abnormal
event (interrupt, strike).
May exploit diverse computational power.
May involve remote components.
May interact with users (Reactive).
May involve hardware components
(Embedded).
Problems in modeling systems


Representing concurrency:
- Allow one transition at a time, or
- Allow coinciding transitions.
Granularity of transitions.



Assignments and checks?
Application of methods?
Global (all the system) or local (one
thread at a time) states.
Modeling.
The states based model.




V={v0,v1,v2, …} - set of variables.
p(v0, v1, …, vn) - a parametrized assertion, e.g.,
v0=v1+v2 /\ v3>v4.
A state is an assignment of values to the program
variables. For example:
s=<v0=1,v2=3,v3=7,…,v18=2>
For predicate (first order assertion) p:
p (s ) is p under the assignment s.
Example: p is x>y /\ y>z. s=<x=4, y=3, z=5>.
Then we have 4>3 /\ 3>5, which is false.
State space


The state space of a program is the set
of all possible states for it.
For example, if V={a, b, c} and the
variables are over the naturals, then the
state space includes:
<a=0,b=0,c=0>,<a=1,b=0,c=0>,
<a=1,b=1,c=0>,<a=932,b=5609,c=6658>…
Atomic Transitions



Each atomic transition represents a
small piece of code such that no smaller
piece of code is observable.
Is a:=a+1 atomic?
In some systems, e.g., when a is a
register and the transition is executed
using an inc command.
Non atomicity





Execute the
following when x=0
in two concurrent
processes:
P1:a=a+1
P2:a=a+1
Result: a=2.
Is this always the
case?
Consider the actual
translation:
P1:load R1,a
inc R1
store R1,a
P2:load R2,a
inc R2
store R2,a
 a may be also 1.

Scenario
P1:load R1,a
inc R1
store R1,a
a=0
R1=0
R2=0
P2:load R2,a
R1=1
R2=1
inc R2
a=1
store R2,a
a=1
Representing transitions

Each transition has two parts:



The enabling condition: a predicate.
The transformation: a multiple assignment.
For example:
a>b  (c,d):=(d,c)
This transition can be executed in states
where a>b. The result of executing it is
switching the value of c with d.
Initial condition



A predicate I.
The program can
start from states s
such that I (s )
holds.
For example:
I (s )=a>b /\ b>c.
A transition system



A (finite) set of variables V over some
domain.
A set of states S.
A (finite) set of transitions T, each
transition et has



an enabling condition e, and
a transformation t.
An initial condition I.
Example





V={a, b, c, d, e}.
S: all assignments of natural numbers
for variables in V.
T={c>0(c,e):=(c-1,e+1),
d>0(d,e):=(d-1,e+1)}
I : c=a /\ d=b /\ e=0
What does this transition system do?
The interleaving model



An execution is a finite or infinite
sequence of states s0, s1, s2, …
The initial state satisfies the initial
condition, I.e., I (s0).
Moving from one state si to si+1 is by
executing a transition et:


E (si ), I.e., si satisfies e.
si+1 is obtained by applying t to si.
Example:
T={c>0(c,e):=(c-1,e+1),

s0=<a=2, b=1, c=2, d=1, e=0>

s1=<a=2, b=1, c=1, d=1, e=1>

s2=<a=2, b=1, c=1, d=0, e=2>

s3=<a=2, b=1 ,c=0, d=0, e=3>
d>0(d,e):=(d-1,e+1)}
I: c=a /\ d=b /\ e=0
The transitions
L0:While True do
NC0:wait(Turn=0);
CR0:Turn=1
endwhile ||
L1:While True do
NC1:wait(Turn=1);
CR1:Turn=0
endwhile
T0:PC0=L0PC0:=NC0
T1:PC0=NC0/\Turn=0
PC0:=CR0
T2:PC0=CR0
(PC0,Turn):=(L0,1)
T3:PC1=L1PC1=NC1
T4:PC1=NC1/\Turn=1
PC1:=CR1
T5:PC1=CR1
(PC1,Turn):=(L1,0)
Initially: PC0=L0/\PC1=L1
The state graph:Successor
relation between states.
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
Some observations


Executions : the set of maximal paths
(finite or terminating in a node where
nothing is enabled).
Nondeterministic choice : when more
than a single transition is enabled at a
given state. We have a nondeterministic
choice when at least one node at the
state graph has more than one
successor.
Always ¬(PC0=CR0/\PC1=CR1)
(Mutual exclusion)
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
Always if Turn=0 the at
some point Turn=1
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
Always if Turn=0 the at
some point Turn=1
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
Interleaving semantics:
Execute one transition at a time.
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=1
L0,NC1
Turn=0
NC0,NC1
Turn=1
L0,CR1
Turn=0
CR0,NC1
Need to check the property
for every possible interleaving!
Interleaving semantics
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
Turn=0
NC0,NC1
CR0,NC1
Turn=1
L0,NC1
Turn=1
L0,CR1
Turn=0
L0,L1
Turn=0
L0,NC1
Busy waiting
L0:While True do
NC0:wait(Turn=0);
CR0:Turn=1
endwhile ||
L1:While True do
NC1:wait(Turn=1);
CR1:Turn=0
endwhile
T0:PC0=L0PC0:=NC0
T1:PC0=NC0/\Turn=0PC0:=CR0
T1’:PC0=NC0/\Turn=1PC0:=NC0
T2:PC0=CR0(PC0,Turn):=(L0,1)
T3:PC1==L1PC1=NC1
T4:PC1=NC1/\Turn=1PC1:=CR1
T4’:PC1=NC1/\Turn=0PC1:=N1
T5:PC1=CR1(PC1,Turn):=(L1,0)
Initially: PC0=L0/\PC1=L1
Always when Turn=0 then
sometimes Turn=1
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
Now it does not hold!
(Red subgraph generates a counterexample execution.)
How can we check the model?



The model is a graph.
The specification should refer the the
graph representation.
Apply graph theory algorithms.
What properties can we check?



Invariants: a property that need to
hold in each state.
Deadlock detection: can we reach a
state where the program is blocked?
Dead code: does the program have
parts that are never executed.
How to perform the checking?



Apply a search strategy (Depth first
search, Breadth first search).
Check states/transitions during the
search.
If property does not hold, report
counter example!
If it is so good, why learn deductive
verification methods?

Model checking works only for finite
state systems. Would not work with



Unconstrained integers.
Unbounded message queues.
General data structures:




queues
trees
stacks
parametric algorithms and systems.
The state space explosion

Need to represent the state space of a
program in the computer memory.


Each state can be as big as the entire
memory!
Many states:


Each integer variable has 2^32 possibilities.
Two such variables have 2^64 possibilities.
In concurrent protocols, the number of states
usually grows exponentially with the number of
processes.
If it is so constrained, is it of any use?




Many protocols are finite state.
Many programs or procedure are finite state
in nature. Can use abstraction techniques.
Sometimes it is possible to decompose a
program, and prove part of it by model
checking and part by theorem proving.
Many techniques to reduce the state space
explosion.
Depth First Search
Program DFS
For each s such that
Init(s)
dfs(s)
end DFS
Procedure dfs(s)
for each s’ such that
R(s,s’) do
If new(s’) then
dfs(s’)
end dfs.
How can we check properties with DFS?



Invariants: check that all reachable states
satisfy the invariant property. If not, show
a path from an initial state to a bad state.
Deadlocks: check whether a state where no
process can continue is reached.
Dead code: as you progress with the DFS,
mark all the transitions that are executed at
least once.
¬(PC0=CR0/\PC1=CR1) is
an invariant!
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
Want to do more!




Want to check more properties.
Want to have a unique algorithm to
deal with all kinds of properties.
This is done by writing specification in
more complicated formalisms.
We will see that in the next lecture.
[](Turn=0 --> <>Turn=1)
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
init
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
Turn=0
L0,L1
init
Turn=0
L0,L1
Turn=1
L0,L1
Turn=1
L0,L1
•Add an additional initial node.
•Propositions are attached to incoming nodes.
•All nodes are accepting.
Correctness condition

We want to find a correctness condition
for a model to satisfy a specification.
Language of a model: L(Model)
Language of a specification: L(Spec).

We need: L(Model)  L(Spec).


Correctness
Sequences satisfying Spec
Program executions
All sequences
How to prove correctness?



Show that L(Model)  L(Spec).
Equivalently:
______
Show that L(Model)  L(Spec) = Ø.
Also: can obtain L(Spec) by translating
from LTL!
What do we need to know?



How to intersect two automata?
How to complement an automaton?
How to translate from LTL to an
automaton?
Specification Formalisms
Properties of formalisms




Formal. Unique interpretation.
Intuitive. Simple to understand (visual).
Succinct. Spec. of reasonable size.
Effective.




Check that there are no contradictions.
Check that the spec. is implementable.
Check that the implementation satisfies spec.
Expressive.
May be used to generate initial code.
Specifying the implementation or its properties?

A transition system





A (finite) set of variables V.
A set of states S.
A (finite) set of transitions T, each transition e==>t
has
 an enabling condition e and a transformation t.
An initial condition I.
Denote by R(s, s’) the fact that s’ is a successor of s.
The interleaving model




An execution is a finite or infinite sequence of states s0, s1,
s2, …
The initial state satisfies the initial condition, I.e., I (s0).
Moving from one state si to si+1 is by executing a transition
e==>t:
 e(si), I.e., si satisfies e.
 si+1 is obtained by applying t to si.
Lets assume all sequences are infinite by extending finite
ones by “stuttering” the last state.
Temporal logic




Dynamic, speaks about several “worlds”
and the relation between them.
Our “worlds” are the states in an
execution.
There is a linear relation between them,
each two sequences in our execution
are ordered.
Interpretation: over an execution,
later over all executions.
LTL: Syntax
 ::= () | ¬ | /\  \/ U
 |O  | p
“box”, “always”, “forever”
“diamond”, “eventually”, “sometimes”
O “nexttime”
U“until”
Propositions p, q, r, … Each represents some
state property (x>y+1, z=t, at-CR, etc.)
Semantics over suffixes of execution









O
U








Combinations



[]<>p “p will happen infinitely often”
<>[]p “p will happen from some point
forever”.
([]<>p) --> ([]<>q) “If p happens
infinitely often, then q also happens
infinitely often”.
Some relations:


[](a/\b)=([]a)/\([]b)
But <>(a/\b)(<>a)/\(<>b)
b


a
<>(a\/b)=(<>a)\/(<>b)
But [](a\/b)([]a)\/([]b)
b
a
b
b
a
b
a
b
a
What about

([]<>A)/\([]<>B)=[]<>(A/\B)? No, just <--

([]<>A)\/([]<>B)=[]<>(A\/B)? Yes!!!

(<>[]A)/\(<>[]B)=<>[](A/\B)? Yes!!!

(<>[]A)\/(<>[]B)=<>[](A\/B)? No, just -->
Can discard some operators


Instead of <>p, write true U p.
Instead of []p, we can write ¬<>¬p,
or ¬(true U ¬p).
Because []p=¬¬[]p.
¬[]p means it is not true that p holds
forever, or at some point ¬p holds or
<>¬p.
Formal semantic definition








Let  be a sequence s0 s1 s2 …
Let i be a suffix of : si si+1 si+2 … (0 = )
i |= p, where p a proposition, if si|=p.
i |= /\ if i |=  and i |= .
i |= \/ if i |=  or i |= .
i |= <> if for some ji, j |= .
i |= [] if for each ji, j |= .
i |= U  if for some ji, j|=.
and for each ik<j, k |=.
Then we interpret:



s|=p as in propositional logic.
|= is interpreted over a sequence, as
in previous slide.
P|= holds if |= for every sequence
 of P.
Spring Example
release
s1
s2
pull
s3
release
extended
r0 = s1 s2 s1 s2 s1 s2 s1 …
r1 = s1 s2 s3 s3 s3 s3 s3 …
r2 = s1 s2 s1 s2 s3 s3 s3 …
…
extended
malfunction
LTL satisfaction by a single
sequence
r2 = s1 s2 s1 s2 s3 s3 s3 …
release
s1
pull
s2
release
extended
r2 |= extended ??
r2 |= O extended ??
r2 |= O O extended ??
r2 |= <> extended ??
r2 |= [] extended ??
s3
extended
malfunction
r2 |= <>[] extended ??
r2 |= ¬ <>[] extended ??
r2 |= (¬extended) U malfunction ??
r2 |= [](¬extended->O extended) ??
LTL satisfaction by a system
release
s1
pull
s2
release
extended
P |= extended ??
P |= O extended ??
P |= O O extended ??
P |= <> extended ??
P|= [] extended ??
s3
extended
malfunction
P |= <>[] extended ??
P |= ¬ <>[] extended ??
P |= (¬extended) U malfunction ??
P |= [](¬extended->O extended) ??
The state space
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=0
CR0,NC1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=1
NC0,CR1
[] ¬(PC0=CR0/\PC1=CR1)
(Mutual exclusion)
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=1
L0,L1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=0
CR0,NC1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=1
NC0,CR1
[](Turn=0 --> <>Turn=1)
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=1
L0,L1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=0
CR0,NC1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=1
NC0,CR1
Interleaving semantics:
Execute one transition at a time.
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=1
L0,NC1
Turn=0
NC0,NC1
Turn=0
CR0,NC1
Turn=1
L0,CR1
Need to check the property
for every possible interleaving!
More specifications



[](PC0=NC0  <> PC0=CR0)
[](PC0=NC0 U Turn=0)
Try at home:
- The processes alternate in entering
their critical sections.
- Each process enters its critical section
infinitely often.
Proof system







¬<>p<-->[]¬p
[](pq)([]p[]q)
[]p(p/\O[]p)
O¬p<-->¬Op
[](pOp)(p[]p)
(pUq)<-->(q\/(p/\O(pUq)))
(pUq)<>q


+ propositional logic
axiomatization.
+ axiom:
p
[]p
Traffic light example
Green --> Yellow --> Red --> Green
Always has exactly one light:
[](¬(gr/\ye)/\¬(ye/\re)/\¬(re/\gr)/\(gr\/ye\/re))
Correct change of color:
[]((grUye)\/(yeUre)\/(reUgr))
Another kind of traffic light
Green-->Yellow-->Red-->Yellow-->Green
First attempt:
[](((gr\/re) U ye)\/(ye U (gr\/re)))
Correct specification:
[]( (gr-->(gr U (ye /\ ( ye U re ))))
/\(re-->(re U (ye /\ ( ye U gr ))))
/\(ye-->(ye U (gr \/ re))))
Needed only when we
can start with yellow
Properties of sequential
programs






init-when the program starts and
satisfies the initial condition.
finish-when the program terminates and
nothing is enabled.
Partial correctness: init/\[](finish)
Termination: init/\<>finish
Total correctness: init/\<>(finish/\ )
Invariant: init/\[]
Automata over finite words





A=<S, S, , I, F>
S (finite) the alphabet, S (finite) the states.
: S x S x S is the transition relation
I : S are the starting states
F: S are the accepting states.
a
S0
a
b
S1
b
The transition relation




(S0,
(S0,
(S1,
(S1,
a, S0)
b, S1)
a, S0)
b, S1)
a
S0
a
b
S1
b
A run over a word




A word over S, e.g., abaab.
A sequence of states, e.g. S0 S0 S1 S0 S0 S1.
Starts with an initial state.
Accepting if ends at accepting state.
a
S0
a
b
S1
b
The language of an automaton




The words that are accepted by the
automaton.
Includes aabbba, abbbba.
Does not include abab, abbb.
What is the language?
a
S0
a
b
S1
b
Nondeterministic automaton


Transitions: (S0,a,S0), (S0,b,S0),
(S0,a,S1),(S1,a,S1).
What is the language of this
automaton?
S0
A,b
a
S1
a
Equivalent deterministic automaton
S0
a,b
a
S1
a
S1
a
a
S0
b
b
Automata over infinite words



Similar definition.
Runs on infinite words over S.
Accepts when an accepting state occurs
infinitely often in a run.
a
S0
a
b
S1
b
Automata over infinite words



Consider the word a b a b a b a b …
There is a run S0 S0 S1 S0 S1 S0 S1 …
This run in accepting, since S0 appears
infinitely many times.
A
S0
A
B
S1
B
Other runs



For the word b b b b b … the run is
S0 S1 S1 S1 S1… and is not accepting.
For the word a a a b b b b b …, the
run is S0 S0 S0 S0 S1 S1 S1 S1 …
What is the run for a b a b b a b b b …?
a
S0
a
b
S1
b
Nondeterministic automaton


What is the language of this
automaton?
What is the LTL specification if
b -- PC0=CR0, a=¬b?
S0
a,b
a
S1
a
•Can you find a deterministic automaton with same language?
•Can you prove there is no such deterministic automaton?
Specification using Automata



Let each letter correspond to some propositional
property.
Example:
a -- P0 enters critical section,
b -- P0 does not enter section.
[]<>PC0=CR0
a
S0
a
b
S1
b
Mutual Exclusion




A -- PC0=CR0/\PC1=CR1
B -- ¬(PC0=CR0/\PC1=CR1)
C -- TRUE
[]¬(PC0=CR0/\PC1=CR1)
S0
b
a
S1
c
L0:While True do
NC0:wait(Turn=0);
CR0:Turn=1
endwhile ||
L1:While True do
NC1:wait(Turn=1);
CR1:Turn=0
endwhile
T0:PC0=L0==>PC0=NC0
T1:PC0=NC0/\Turn=0==>
PC0:=CR0
T2:PC0=CR0==>
(PC0,Turn):=(L0,1)
T3:PC1==L1==>PC1=NC1
T4:PC1=NC1/\Turn=1==>
PC1:=CR1
T5:PC1=CR1==>
(PC1,Turn):=(L1,0)
Initially: PC0=L0/\PC1=L1
The state space
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
[]¬(PC0=CR0/\PC1=CR1)
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
[](Turn=0 --> <>Turn=1)
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
Correctness condition

We want to find a correctness condition
for a model to satisfy a specification.
Language of a model: L(Model)
Language of a specification: L(Spec).

We need: L(Model)  L(Spec).


Correctness
Sequences satisfying Spec
Program executions
All sequences
Incorrectness
Counter
examples
Sequences satisfying Spec
Program executions
All sequences
Intersecting M1=(S1,S,T1,I1,A1)
and M2=(S2,S,T2,I2,S2)





Run the two automata in parallel.
Each state is a pair of states: S1 x S2
Initial states are pairs of initials: I1 x I2
Acceptance depends on first
component: A1 x S2
Conforms with transition relation:
(x1,y1)-a->(x2,y2) when
x1-a->x2 and y1-a->y2.
Example
(all states of second
automaton accepting!)
a
s0
a
b,c
s1
b,c
t1
c
a
t0
b
States: (s0,t0), (s0,t1), (s1,t0), (s1,t1).
Accepting: (s0,t0), (s0,t1). Initial: (s0,t0).
a
a
s0
b,c
b,c
t1
c
a
t0
b
a
s0,t1
Also state (s0,t1) is
unreachable but we
will keep it for the
time being!
s1
c
s0,t0
b
s1,t0
a
s1,t1
b
c
More complicated when A2S2
a
s0
a
b,c
s1
b,c
a
s0,t1
a
t0
b
t1
c
c
s0,t0
b
a
s1,t1
c
Should we have acceptance when both components
accepting? I.e., {(s0,t1)}?
No, consider (ba)
It should be accepted, but never passes that state.
More complicated when A2S2
a
s0
A
b,c
s1
b,c
a
s0,t1
a
t0
b
t1
c
c
s0,t0
b
a
s1,t1
c
Should we have acceptance when at least one components
is accepting? I.e., {(s0,t0),(s0,t1),(s1,t1)}?
No, consider b c
It should not be accepted, but here will loop through
(s1,t1)
Intersection - general case
Also: propositions on nodes
A/\¬B
¬A
q0
q2
q1
q3
A/\¬B
q0,q3
¬A/\B
q0 and q2 give false.
A\/¬B
¬A/\B
q1,q2
¬A/\¬B
q1,q3
Version 0: to catch q0
Version 1: to catch q2
Version 0
A/\¬B
q0,q3
¬A/\B
q1,q2
Move when see accepting of left (q0)
A/\¬B
q0,q3
Move when see accepting of right (q2)
¬A/\B
q1,q2
Version 1
¬A/\¬B
q1,q3
¬A/\¬B
q1,q3
Make an accepting state in one of the
version according to a component
accepting state
Version 0
A/\¬B
q0,q3,0
A/\¬B
q0,q3,1
¬A/\B
q1,q2,0
¬A/\B
q1,q2 ,1
Version 1
¬A/\¬B
q1,q3,0
¬A/\¬B
q1,q3 ,1
How to check for emptiness?
a
s0,t1
c
s0,t0
b
a
s1,t1
b
c
Emptiness...
Need to check if there exists an accepting
run (passes through an accepting state
infinitely often).
Finding accepting runs
If there is an accepting run, then at least
one accepting state repeats on it
forever.
This state appears on a cycle. So, find a
reachable accepting state on a cycle.
Equivalently...

A strongly connected component: a set
of nodes where each node is reachable
by a path from each other node. Find a
reachable strongly connected
component with an accepting node.
How to complement?



Complementation is hard!
Can ask for the negated property (the
sequences that should never occur).
Can translate from LTL formula  to
automaton A, and complement A. But:
can translate ¬ into an automaton
directly!
Model Checking under Fairness
Express the fairness as a property φ.
To prove a property ψ under fairness,
model check φψ.
Counter
example
Fair (φ)
Bad (¬ψ)
Program
Model Checking under Fairness
Specialize model checking. For weak
process fairness: search for a
reachable strongly connected
component, where for each process
P either

it contains on occurrence of a
transition from P, or

it contains a state where P is
disabled.
Dekker’s
algorithm
P1::while true do
begin
non-critical section 1
c1:=0;
while c2=0 do
begin
if turn=2 then
begin
c1:=1;
wait until turn=1;
c1:=0;
end
end
critical section 1
c1:=1;
turn:=2
end.
boolean c1 initially 1;
boolean c2 initially 1;
integer (1..2) turn initially 1;
P2::while true do
begin
non-critical section 2
c2:=0;
while c1=0 do
begin
if turn=1 then
begin
c2:=1;
wait until turn=2;
c2:=0;
end
end
critical section 2
c2:=1;
turn:=1
end.
Dekker’s
algorithm
boolean c1 initially 1;
boolean c2 initially 1;
integer (1..2) turn initially 1;
P2::while true do
P1::while true do
begin
begin
non-critical section 2
non-critical section 1
c2:=0;
c1:=0;
c1=c2=0,
while c1=0 do
while c2=0 do
begin
turn=1
begin
if turn=1 then
if turn=2 then
begin
begin
c2:=1;
c1:=1;
wait until turn=2;
wait until turn=1;
c2:=0;
c1:=0;
end
end
end
end
critical section 2
critical section 1
c2:=1;
c1:=1;
turn:=1
turn:=2
end.
end.
Dekker’s
algorithm
boolean c1 initially 1;
boolean c2 initially 1;
integer (1..2) turn initially 1;
P2::while true do
P1::while true do
begin
begin
non-critical section 2
non-critical section 1
c2:=0;
c1:=0;
c1=c2=0,
while c1=0 do
while c2=0 do
begin
turn=1
begin
if turn=1 then
if turn=2 then
begin
begin
c2:=1;
c1:=1;
wait until turn=2;
wait until turn=1;
c2:=0;
c1:=0;
end
end
end
end
critical section 2
critical section 1
c2:=1;
c1:=1;
turn:=1
turn:=2
end.
end.
Dekker’s
algorithm
P1 waits for P2 to set c2 to 1 again.
Since turn=1 (priority for P1), P2 is
ready to do that. But never gets the
chance, since P1 is constantly active
checking c2 in its while loop.
P2::while true do
P1::while true do
begin
begin
non-critical section 2
non-critical section 1
c2:=0;
c1:=0;
c1=c2=0,
while c1=0 do
while c2=0 do
begin
turn=1
begin
if turn=1 then
if turn=2 then
begin
begin
c2:=1;
c1:=1;
wait until turn=2;
wait until turn=1;
c2:=0;
c1:=0;
end
end
end
end
critical section 2
critical section 1
c2:=1;
c1:=1;
turn:=1
turn:=2
end.
end.
What went wrong?




The execution is unfair to P2. It
is not allowed a chance to
execute.
Such an execution is due to the
interleaving model (just picking
an enabled transition.
If it did, it would continue and
set c2 to 0, which would allow
P1 to progress.
Fairness = excluding some of
the executions in the
interleaving model, which do
not correspond to actual
behavior of the system.
while c1=0 do
begin
if turn=1 then
begin
c2:=1;
wait until turn=2;
c2:=0;
end
end
Recall:
The interleaving model



An execution is a finite or infinite sequence of states s0, s1, s2,
…
The initial state satisfies the initial condition, I.e., I (s0).
Moving from one state si to si+1 is by executing a transition
et:
 e(si), I.e., si satisfies e.
 si+1 is obtained by applying t to si.
Now: consider only “fair” executions. Fairness constrains
sequences that are considered to be executions.
Sequences
Executions
Fair
executions
Some fairness definitions

Weak transition fairness:
It cannot happen that a transition is enabled indefinitely,
but is never executed.

Weak process fairness:
It cannot happen that a process is enabled indefinitely,
but non of its transitions is ever executed

Strong transition fairness:
If a transition is infinitely often enabled, it will get
executed.

Strong process fairness:
If at least one transition of a process is infinitely often
enabled, a transition of this process will be executed.
How to use fairness
constraints?



Assume in the negative that some
property (e.g., termination) does not
hold, because of some transitions or
processes prevented from execution.
Show that such executions are
impossible, as they contradict fairness
assumption.
We sometimes do not know in reality
which fairness constraint is guaranteed.
Example
Initially: x=0; y=0;
P2: do
:: y==0 
if
In order for the loop to
:: true
terminate we need P1 to
:: x==1  y=1
execute the assignment.
fi
But P1 may never execute,
od
since P2 is in a loop
P1::x=1
executing true.
Consequently, x==1 never
holds, and y is never
assigned a 1.
pc1=l0-->(pc1,x):=(l1,1) /* x=1 */
pc2=r0/\y=0-->pc2=r1
pc2=r1pc2=r0
/* y==0*/
/* true */
pc2=r1/\x=1(pc2,y):=(r0,1)
/* x==1 --> y:=1 */
Weak transition fairness
Initially: x=0; y=0;
P2: do
P1::x=1
:: y==0 
Under weak transition
if
fairness, P1 would assign
:: true
1 to x, but this does not
:: x==1  y=1
guarantee that 1 is
fi
assigned to y and thus the
od
P2 loop will terminates,
since the transition for
checking x==1 is not
continuously enabled
(program counter not
always there).
Weak process fairness only
guarantees P2 to execute, but it
can still choose to do the true.
Strong process fairness:
same.
Strong transition fairness
Initially: x=0; y=0;
P1::x=1
P2: do
Under strong transition
:: y==0 
fairness, P1 would assign
if
1 to x. If the execution was
:: true
infinite, the transition
:: x==1  y=1
checking x==1 was
infinitely often enabled.
fi
Hence it would be
od
eventually selected. Then
assigning y=1, the main
loop is not enabled
anymore.
Some fairness definitions
exec  is executed.

/\ T (<>[]en  []<>exec).
Equivalently: /\aT ¬<>[](en /\¬exec)
execPi some transition
of Pi is executed.
en  is enabled.

Weak process fairness:
/\Pi (<>[]enPi  []<>execPi )
enPi some transition of
process Pi is enabled.

enPi = \/ T en

execPi = \/ T exec
Weak transition fairness:
Strong transition fairness:
/\ T ([]<>en  []<>exec )
Strong process fairness:
/\Pi ([]<>enPi  []<>execPi )
“Weaker fairness condition”




A is weaker than B if BA.
Consider the executions L(A)
and L(B).
Then L(B)  L(A).
An execution is
strong {process,transition} fair
implies that it is also weak
{process,transition} fair.
There are fewer strong
{process,transition} fair
executions.
Strong
transition
fair execs
Strong
process
fair execs
Weak
process
fair execs
Weak
transition
fair execs
Model Checking under fairness



Instead of verifying that the program
satisfies , verify it satisfies fair 
Problem: may be inefficient. Also
fairness formula may involves special
arrangement for specifying what exec
means.
May specialize model checking
algorithm instead.
Model Checking under Fairness
Specialize model checking. For weak process
fairness: search for a reachable strongly
connected component, where for each
process P either

it contains on occurrence of a transition
from P, or

it contains a state where P is disabled.

Weak transition fairness: similar.

Strong fairness: much more difficult
algorithm.
Abstractions
Problems with software
analysis



Many possible outcomes and
interactions.
Not manageable by an algorithm
(undecideable, complex).
Requires a lot of practice and ingenuity
(e.g., finding invariants).
More problems



Testing methods fail to cover potential errors.
Deductive verification techniques require
 too much time,
 mathematical expertise,
 ingenuity.
Model checking requires a lot of time/space and may
introduce modeling errors.
How to alleviate the
complexity?




Abstraction
Compositionality
Partial Order Reduction
Symmetry
Abstraction



Represent the program using a smaller
model.
Pay attention to preserving the checked
properties.
Do not affect the flow of control.
Main idea

Use smaller data objects.
x:= f(m)
y:=g(n)
if x*y>0 then …
else …
x, y never used again.
How to abstract?


Assign values {-1, 0, 1} to x and y.
Based on the following connection:
sgn(x) = 1 if x>0,
0 if x=0, and
-1 if x<0.
sgn(x)*sgn(y)=sgn(x*y).
Abstraction mapping



S - states, I - initial states. L(s) - labeling.
R(S,S) - transition relation.
h(s) maps s into its abstract image.
Full model
--h-->
Abstract model
I(s)
-->
I(h(s))
R(s, t)
-->
R(h(s),h(t))
L(h(s))=L(s)
go
Traffic light
example
stop
stop
go
go
stop
stop
stop
What do we preserve?
Every execution of the
full model can be
simulated by an
execution of the reduced
one.
go
go
Every LTL property that
holds in the reduced
model hold in the full
one.
But there can be
properties holding for
the original model but
not the abstract one.
stop
stop
stop
Preserved:
[](go->O stop)
go
go
Not preserved:
[]<>go
stop
stop
stop
Counterexamples
need to be
checked.
Symmetry



A permutation is a one-one and onto function
p:A->A.
For example, 1->3, 2->4, 3->1, 4->5, 5->2.
One can combine permutations, e.g.,
p1: 1->3, 2->1, 3->2
p2: 1->2, 2->1, 3->3
p1@p2: 1->3, 2->2, 3->1
A set of permutations with @ is called a
symmetry group.
Using symmetry in analysis


Want to find some symmetry group such
that for each permutation p in it,
R(s,t) if and only if R(p(s), p(t))
and L(p(s))=L(s).
Let K(s) be all the states that can be
permuted to s. This is a set of states such
that each one can be permuted to the other.
init
Turn=1
L0,L1
Turn=0
L0,L1
Turn=0
L0,NC1
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=1
L0,NC1
Turn=0
CR0,L1
Turn=1
L0,CR1
Turn=1
NC0,L1
Turn=1
NC0,NC1
Turn=0
Turn=1
CR0,NC1
NC0,CR1
init
Turn=0
L0,L1
Turn=0
L0,NC1
The quotient model
Turn=0
NC0,L1
Turn=0
NC0,NC1
Turn=0
CR0,L1
Turn=0
CR0,NC1
What is preserved in the following buffer
abstraction? What is not preserved?
e
empty
q
q
quasi
q
full
f
Translating from logic to automata
Comment: the language of LTL is a
proper subset of the language of Buchi
Automata. For example,
one cannot express in LTL the
following automaton [Wolper]:
a
a ,¬a
Why translating?


Want to write the specification in some
logic.
Want model-checking tools to be able
to check the specification automatically.
Preprocessing





Convert into normal form, where negation
only applies to propositional variables.
¬[] becomes <>¬.
¬<> becomes [] ¬.
What about ¬ ( U )?
Define operator V such that
¬ (  U ) = (¬) R (¬),
¬ (  R ) = (¬) U (¬).
Semantics of pR q
(the Release operator, dual to Until)
¬p
¬p
¬p
¬p
¬p
¬p
¬p
¬p
¬p
q
q
q
q
q
q
q
q
q
¬p
¬p
¬p
¬p
p
q
q
q
q
q


Replace ¬T by F, and ¬F by T.
Replace ¬ ( \/ ) by (¬) /\ (¬) and
¬ ( /\ ) by (¬) \/ (¬)
Eliminate implications, <>, []



Replace  ->  by (¬ ) \/ .
Replace <> by (T U ).
Replace [] by (F R ).
Example




Translate ( []<>P )  ( []<>Q )
Eliminate implication ¬( []<>P ) \/ (
[]<>Q )
Eliminate [], <>:
¬( F R ( T U P ) ) \/ ( F R ( T U Q ) )
Push negation inwards:
(T U (F R ¬P ) ) \/ ( F R ( T U Q ) )
The data structure
Incoming
New
Old
Name
Next
The main idea


 U  =  \/ (  /\ O (  U  ) )
 R  =  /\ (  \/ O (  R  ) )
This separates the formulas to two parts:
one holds in the current state, and the
other in the next state.
How to translate?


Take one formula from “New” and add it
to “Old”.
According to the formula, either


Split the current node into two, or
Evolve the node into a new version.
Splitting
Copy incoming
edges, update
other field.
Incoming
New
Old
Next
Incoming
New
Old
Next
Incoming
New
Old
Next
Evolving
Copy incoming
edges, update
other field.
Incoming
New
Old
Next
Incoming
New
Old
Next
Possible cases:

 U  , split:
Add  to New, add  U  to Next.
 Add  to New.
Because  U  =  \/ (  /\ O ( U  )).


 R  , split:
Add  to New.
 Add  to New,  R  to Next.
Because  R  =  /\ (  \/ O ( R  )).

More cases:

 \/ , split:



 /\ , evolve:


Add  to New.
Add  to New.
Add  to New.
O , evolve:

Add  to Next.
How to start?
init
Incoming
New
aU(bUc)
Old
Next
init
Incoming
aU(bUc)
init
init
Incoming
a
aU(bUc)
aU(bUc)
Incoming
bUc
aU(bUc)
init
Incoming
bUc
aU(bUc)
init
init
Incoming
b
aU(bUc)
(bUc)
Incoming
c
aU(bUc)
When to stop splitting?


When “New” is empty.
Then compare against a list of existing nodes
“Nodes”:


If such a with same “Old”, “Next” exists,
just add the incoming edges of the new version
to the old one.
Otherwise, add the node to “Nodes”. Generate a
successor with “New” set to “Next” of father.
init
Incoming
a,aU(bUc)
Creating a
successor node.
aU(bUc)
Incoming
aU(bUc)
How to obtain the automaton?
X
There is an edge from
node X to Y labeled
with propositions P
(negated or non
negated), if X is in the
incoming list of Y, and
Y has propositions P
in field “Old”.
Initial nodes are those
marked as init.
Incoming
New
Old
a, b, ¬c
Next
Node Y
The resulted nodes.
a, aU(bUc)
b, bUc, aU(bUc)
b, bUc
c, bUc
c, bUc, aU(bUc)
a, aU(bUc)
b, bUc, aU(bUc)
b, bUc
c, bUc
c, bUc, aU(bUc)
Initial nodes:
All nodes with incoming edge from “init”.
Acceptance conditions



Use “generalized Buchi automata”, where
there are several acceptance sets F1, F2, …,
Fn, and each accepted infinite sequence must
include at least one state from each set
infinitely often.
Each set corresponds to a subformula of form
U . Guarantees that it is never the case
that U  holds forever, without .
Translate to Buchi acceptance condition (will
be shown later).
Accepting w.r.t. bUc
(green)
a, aU(bUc)
b, bUc, aU(bUc)
b, bUc
c, bUc
All nodes with c, or without bUc.
c, bUc, aU(bUc)
Acceptance w.r.t. aU(bUc)
(blue)
a, aU(bUc)
b, bUc, aU(bUc)
b, bUc
c, bUc
All nodes with bUc or without aU(bUc).
c, bUc, aU(bUc)
Translating Multiple Buchi into
Buchi automata




For acceptance sets F1, F2, …, Fn, generate n
copies of the automaton.
Move from the ith copy to the i+1th copy
when passing a state of Fi.
In the first copy, the states of F1 are
accepting.
Same intuition as the unrestricted intersection:
Need to see each accepting set infinitely often,
and it does not matter that we miss some of
them (there must be an infinite supply
anyway…).
Conclusions






Model checking: automatic verification of
finite state systems.
Modeling: affects how we can view and check
the system’s properties.
Specification formalisms, e.g., LTL or Buchi
automata
Apply graph algorithms (or BDD, or BMD).
Methods to alleviate state space explosion.
Translate LTL into Buchi automata.
Algorithmic Testing and
Black Box Checking
Why testing?






Reduce design/programming errors.
Can be done during development,
before
production/marketing.
Practical, simple to do.
Check the real thing, not a model.
Scales up reasonably.
Being state of the practice for decades.
Part 1: Testing of
black box finite state
machine
Wants to know:
Know:
In what state we started?
Transition relation
In what state we are?
Size or bound on size
Transition relation
Conformance
Satisfaction of a
temporal property
Finite
automata
(Mealy machines)
S - finite set of states. (size n)
S– set of inputs.
(size d)
O – set of outputs, for each transition.
(s0  S - initial state).
  S  S S - transition relation.
  S  SO – output on edge.
Why deterministic machines?





Otherwise no amount of experiments would
guarantee anything.
If dependent on some parameter (e.g.,
temperature), we can determinize, by taking
parameter as additional input.
We still can model concurrent system. It
means just that the transitions are
deterministic.
All kinds of equivalences are unified into
language equivalence.
Also: connected machine (otherwise we may
Determinism
a/1
b/1
a/1
When the black box is nondeterministic, we
might never test some choices.
Preliminaries: separating
sequences
b/1
s1
a/0
a/0
b/1
s2
b/0
s3
a/0
Start with one block containing
all states {s1, s2, s3}.
A: separate to blocks of states
with different output.
b/1
s1
a/0
a/0
b/1
s2
b/0
s3
a/0
Two sets, separated using
the string b {s1, s3}, {s2}.
Repeat B: Separate blocks based on
moving to different blocks.
b/1
s1
a/0
a/0
b/1
s2
b/0
s3
a/0
Separate first block using b to three singleton blocks.
Separating sequences: b, bb.
Max rounds: n-1, sequences: n-1, length: n-1.
For each pair of states there is a separating sequence.
Want to know the state of the
machine (at end). Homing sequence.
Depending on output, would know in what
state we are.
Algorithm: Put all the states in one block (initially
we do not know what is the state).
Then repeatedly partitions blocks of states, as long
as they are not singletons, as follows:
 Take a non singleton block, append a
distinguishing sequence  that separates at least
two states.
 Update all blocks to the states after executing .
Max length: (n-1)2
(Lower bound: n(n-1)/2.)
Example (homing sequence)
b/1
s1
a/0
a/0
b/1
s2
b/0
s3
{s1, s2, s3}
1
0
{s1, s2} {s3}
1
a/0
1
b
1
0
b
{s1} {s2} {s3}
On input b and output 1, still don’t know if
was in s1 or s3, i.e., if currently in s2 or s1.
So separate these cases with another b.
Synchronizing sequence



One sequence takes the machine to the
same final state, regardless of the initial
state or the outputs.
Not every machine has a synchronizing
sequence.
Can be checked whether exists and can
be found in polynomial time.
State identification:




Want to know in which state the
system has started (was reset).
Can be a preset distinguishing
sequence (fixed), or a tree
(adaptive).
May not exist (PSPACE complete
to check if preset exists,
polynomial for adaptive).
Best known algorithm:
exponential length for preset,
polynomial for adaptive [LY].
Sometimes cannot identify
initial state
a/1
s1
b/1
s2
a/1
a/1
b/0
b/1
s3
Start with a:
in case of being in
s1 or s3 we’ll move
to s1 and cannot
distinguish.
Start with b:
In case of being in
s1 or s2 we’ll move
to s2 and cannot
distinguish.
The kind of experiment we do affects what we can
distinguish. Much like the Heisenberg principle in Physics.
Conformance testing





Unknown deterministic finite state system B.
Known: n states and alphabet S.
An abstract model C of B. C satisfies all the properties
we want from B. C has m states.
Check conformance of B and C.
Another version: only a bound n on the number of
states l is known.

Check conformance with a
given state machine
b/1
a/1
?
=



s1
b/1
a/1
a/1
b/0
s2
s3
Black box machine has no more states than specification machine
(errors are mistakes in outputs, mistargeted edges).
Specification machine is reduced, connected, deterministic.
Machine resets reliably to a single initial state (or use homing
sequence).
Conformance testing [Ch,V]
a/1
b/1
a/1

b/1
b/1
a/1
a/1
b/1

a/1
b/1
Cannot distinguish if reduced or not.
Conformance testing (cont.)
b
a
a
a
a
b
a
b
a
b
a
b
Need: bound on number of states of B.
Preparation:
Construct a spanning tree
a/1
s1
b/1
s2
a/1
a/1
b/0
b/1
s3
a/1
s3
s1
b/1
s2
How the algorithm works?
a/1
s3
s1
b/1
Reset or
homing
Reset or
homing
According to the spanning tree,
force a sequence of inputs
to go to each state.
1.
From each state, perform
the distinguishing
sequences.
2.
From each state, make a
single transition, check
output, and use
distinguishing sequences to
check that in correct target
state.
s2
Distinguishing sequences
Comments
1.
2.
3.
4.
Checking the different distinguishing sequences
(m-1 of them) means each time resetting and
returning to the state under experiment.
A reset can be performed to a distinguished
state through a homing sequence. Then we
can perform a sequence that brings us to the
distinguished initial state.
Since there are no more than m states, and
according to the experiment, no less than m
states, there are m states exactly.
Isomorphism between the transition relation is
found, hence from minimality the two automata
recognize the same languages.
Combination lock automaton


Assume accepting states.
Accepts only words with a specific suffix
(cdab in the example).
s1
c
s2
d
s3
a
Any other input
s4
b
s5
When only a bound on size of
black box is known…

Black box can “pretend” to behave as a
specification automaton for a long time,
then upon using the right combination,
make a mistake.
a/1
a/1
s1
a/1
b/1
s2
b/1
b/1
Pretends to be S1
a/1
s3
b/0
Pretends to be S3
Conformance testing algorithm
[VC]




Reset or
homing
Reset or
homing
The worst that can happen is a
combination lock automaton that
behaves differently only in the last
s1
state. The length of it is the
a/1
b/1
difference between the size n of the
black box and the specification m.
Reach every state on the spanning
s3
s2
tree and check every word of length
n-m+1 or less. Check that after the
combination we are at the state we
Words of length n-m+1
are supposed to be, using the
distinguishing sequences.
No need to check transitions: already
included in above check.
Complexity: m2 n dn-m+1
Distinguishing sequences
Probabilistic complexity: Polynomial.
Model Checking





Finite state description of a system B.
LTL formula . Translate  into an automaton P.
Check whether L(B)  L(P)=.
If so, S satisfies . Otherwise, the intersection includes a
counterexample.
Repeat for different properties.



Buchi automata (-automata)
S - finite set of states. (B has l  n states)
S0  S - initial states. (P has m states)
S - finite alphabet.
(contains p letters)
 S  S  S - transition relation.
F  S - accepting states.
Accepting run: passes a state in F infinitely often.
System automata: F=S, deterministic, one initial state.
Property automaton: not necessarily deterministic.
Example: check a
a
a
<>a
a, a
Example: check <>a
a
a
<>a
a
a
Example: check  <>a
a, a
a
<>a
a
Use automatic translation algorithms, e.g.,
[Gerth,Peled,Vardi,Wolper 95]
System
a
c
b
Every element in the product is a counter
example for the checked property.
s1
a
c
s2
q1
b
a
s3
Acceptance is
determined by
automaton P.
s1,q1
s1,q2
a
<>a
a
q2
a s ,q
2 1
b
a
c
s3,q2
a
Model Checking / Testing






Given Finite state
system B.
Transition relation of B
known.
Property represent by
automaton P.
Check if L(B)  L(P)=.
Graph theory or BDD
techniques.
Complexity: polynomial.





Unknown Finite state
system B.
Alphabet and number of
states of B or upper
bound known.
Specification given as
an abstract system C.
Check if B C.
Complexity: polynomial
if number states known.
Exponential otherwise.
Black box checking [PVY]




Property represent by
automaton P.
Check if L(B)  L(P)=.
Graph theory
techniques.



Unknown Finite state
system B.
Alphabet and Upper
bound on Number of
states of B known.
Complexity:
exponential.
Experiments
reset
a
b
c
a
c
b
try c
try b
a
c
b
a
c
b
a
c
b
a
c
b
fail
Simpler problem: deadlock?


Nondeterministic algorithm:
guess a path of length  n from the initial
state to a deadlock state.
Linear time, logarithmic space.
Deterministic algorithm:
systematically try paths of length n, one
after the other (and use reset), until
deadlock is reached.
Exponential time, linear space.
Deadlock complexity




Nondeterministic algorithm:
Linear time, logarithmic space.
Deterministic algorithm:
Exponential (p n-1) time, linear space.
Lower bound: Exponential time (use
combination lock automata).
How does this conform with what we
know about complexity theory?
Modeling black box checking



Cannot model using Turing machines:
not all the information about B is
given. Only certain experiments are
allowed.
We learn the model as we make the
experiments.
Can use the model of games of
incomplete information.
Games of incomplete
information







Two players: $player, player (here, deterministic).
Finitely many configurations C. Including:
Initial Ci , Winning : W+ and W- .
An equivalence relation @ on C (the $player cannot
distinguish between equivalent states).
Labels L on moves (try a, reset, success, fail).
The $player has the moves labeled the same from
configurations that are equivalent.
Deterministic strategy for the $player: will lead to a
configuration in W+  W-. Cannot distinguish between
equivalent configurations.
Nondeterministic strategy: Can distinguish between
equivalent configurations..
Modeling BBC as games



Each configuration contains an
automaton and its current state (and
more).
Moves of the $player are labeled with
try a, reset... Moves of the -player
with
success, fail.
c1 @ c2 when the automata in c1 and c2
would respond in the same way to the
A naive strategy for BBC





Learn first the structure of the black box.
Then apply the intersection.
Enumerate automata with n states (without
repeating isomorphic automata).
For a current automata and new automata,
construct a distinguishing sequence. Only one
of them survives.
Complexity: O((n+1)p (n+1)/n!)
On-the-fly strategy





Systematically (as in the deadlock
case), find two sequences v1 and v2
of length <=m n.
Applying v1 to P brings us to a state
t that is accepting.
Applying v2 to P brings us back to t.
Apply v1 v2 n to B. If this succeeds,
there is a cycle in the intersection
labeled with v2, with t as the P
(accepting) component.
Complexity: O(n2p2mnm).
v1
v2
Learning an automaton




Use Angluin’s algorithm for learning an
automaton.
The learning algorithm queries whether
some strings are in the automaton B.
It can also conjecture an automaton Mi
and asks for a counterexample.
It then generates an automaton with
more states Mi+1 and so forth.
A strategy based on learning





Start the learning algorithm.
Queries are just experiments to B.
For a conjectured automaton Mi ,
check if Mi  P = 
If so, we check conformance of Mi with B
([VC] algorithm).
If nonempty, it contains some v1 v2 . We
test B with v1 v2n. If this succeeds: error,
otherwise, this is a counterexample for Mi .
Complexity



l - actual size of B.
n - an upper bound of size of B.
d - size of alphabet.
Lower bound: reachability is similar to
deadlock.
 O(l 3 d l + l 2mn) if there is an error.
 O(l 3 d l + l 2 n dn-l+1+ l 2mn) if there is no
error.
If n is not known, check while time allows.
 Probabilistic complexity: polynomial.

Some experiments




Basic system written in SML (by Alex
Groce, CMU).
Experiment with black box using Unix
I/O.
Allows model-free model checking of C
code with inter-process communication.
Compiling tested code in SML with BBC
program as one process.
Conclusions




Black Box Checking: automatic
verification of unspecified system.
A hard problem, exponential in number
of states, but polynomial on average.
Implemented, tested.
Another use: when the model is given,
but is not exact.
The rest is not part of this
lecture
Part 2: Software testing
Testing is not about showing that there are
no errors in the program.
 Testing cannot show that the program
performs its intended goal correctly.
So, what is software testing?
Testing is the process of executing the program
in order to find errors.
A successful test is one that finds an error.

Some software testing stages







Unit testing – the lowest level, testing
some procedures.
Integration testing – different pieces of code.
System testing – testing a system as a whole.
Acceptance testing – performed by the
customer.
Regression testing – performed after updates.
Stress testing – checking the code under
extreme conditions.
Mutation testing – testing the quality of the test
suite.
Some drawbacks of testing




There are never sufficiently many test
cases.
Testing does not find all the errors.
Testing is not trivial and requires
considerable time and effort.
Testing is still a largely informal task.
Black-Box (data-driven, inputoutput) testing
The testing is not based on the structure
of the program (which is unknown).
In order to ensure correctness, every
possible input needs to be tested - this
is impossible!
The goal: to maximize the number of
errors found.
testing
Is based on the internal structure of the
program.
There are several alternative criterions for
checking “enough” paths in the
program.
Even checking all paths (highly
impractical) does not guarantee finding
all errors (e.g., missing paths!)
Some testing principles






A programmer should not test his/her own
program.
One should test not only that the program does
what it is supposed to do, but that it does not do
what it is not supposed to.
The goal of testing is to find errors, not to show
that the program is errorless.
No amount of testing can guarantee error-free
program.
Parts of programs where a lot of errors have
already been found are a good place to look for
more errors.
The goal is not to humiliate the programmer!
Inspections and Walkthroughs





Manual testing methods.
Done by a team of people.
Performed at a meeting
(brainstorming).
Takes 90-120 minutes.
Can find 30%-70% of errors.
Code Inspection





Team of 3-5 people.
One is the moderator. He
distributes materials and
records the errors.
The programmer
explains the program
line by line.
Questions are raised.
The program is analyzed
w.r.t. a checklist of
errors.
Checklist for
inspections
Data declaration
All variables declared?
Default values
understood?
Arrays and strings
initialized?
Variables with similar
names?
Correct initialization?

Control flow
Each loop terminates?
DO/END statements
match?

Input/output
OPEN statements
correct?
Format specification
correct?
End-of-file case handled?

Walkthrough




Team of 3-5 people.
Moderator, as
before.
Secretary, records
errors.
Tester, play the role
of a computer on
some test suits on
paper and board.
Selection of test cases
(for white-box testing)
The main problem is to select a good coverage
criterion. Some options are:





Cover all paths of the program.
Execute every statement at least once.
Each decision has a true or false value at least
once.
Each condition is taking each truth value at least
once.
Check all possible combinations of conditions in
each decision.
Cover all the paths of the program
Infeasible.
Consider the flow diagram
on the left.
It corresponds to a loop.
The loop body has 5 paths.
If the loops executes 20
times there are 5^20
different paths!
May also be unbounded!
How to cover the executions?
IF (A>1)&(B=0) THEN X=X/A;
END;
IF (A=2)|(X>1) THEN X=X+1;
END;



Choose values for A,B,X.
Value of X may change, depending on A,B.
What do we want to cover? Paths? Statements?
Conditions?
Statement coverage
Execute every statement at least
once
By choosing
IF (A>1)&(B=0)
THEN X=X/A;
A=2,B=0,X=3
END;
each statement will be
IF (A=2)|(X>1)
chosen.
THEN
X=X+1;
The case where the
END;
tests fail is not
checked!
Now x=1.5
Decision coverage
Each decision has a true and false
outcome at least once.
Can be achieved using
IF (A>1)&(B=0)
THEN X=X/A;
 A=3,B=0,X=3
END;
 A=2,B=1,X=1
Problem: Does not test IF (A=2)|(X>1)
THEN X=X+1;
individual conditions.
END;
E.g., when X>1 is
erroneous in second
decision.
Decision coverage
A=3,B=0,X=3
Now x=1
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
Decision coverage
A=2,B=1,X=1

The case where
A1 and the
case where x>1
where not
checked!
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
Condition coverage
Each condition has a true and false
value at least once.
For example:


A=1,B=0,X=3
A=2,B=1,X=0
lets each condition be
true and false once.
Problem:covers only the
path where the first
test fails and the
second succeeds.
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
Condition coverage
A=1,B=0,X=3

IF (A>1) & (B=0)
THEN X=X/A;
END;
IF (A=2) | (X>1)
THEN X=X+1;
END;
Condition coverage
A=2,B=1,X=0

Did not check the
first THEN part at
all!!!
Can use
condition+decisio
n coverage.
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
Multiple Condition Coverage
Test all combinations of all
conditions in each test.








A>1,B=0
A>1,B≠0
A1,B=0
A1,B≠0
A=2,X>1
A=2,X1
A≠2,X>1
A≠2,X1
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
A smaller number of cases:
A=2,B=0,X=4
 A=2,B=1,X=1
 A=1,B=0,X=2
 A=1,B=1,X=1
Note the X=4 in the first
case: it is due to the fact
that X changes before
being used!

IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
Further optimization: not all combinations.
For C /\ D, check (C, D), (C, D), (C, D).
For C \/ D, check (C, D), (C, D), (C, D).
Preliminary:
Relativizing assertions
(B) : x1= y1 * x2 + y2 /\ y2 >= 0
Relativize B) w.r.t. the assignment
becomes B) [Y\g(X,Y)]
I.e.( B) expressed w.r.t. variables at A.)
(B)A = x1=0 * x2 + x1 /\ x1>=0
Think about two sets of variables,
before={x, y, z, …} after={x’,y’,z’…}.
Rewrite (B) using after, and the
assignment as a relation between the
set of variables. Then eliminate after.
Here:
x1’=y1’ * x2’ + y2’ /\ y2’>=0 /\
x1=x1’ /\ x2=x2’ /\ y1’=0 /\ y2’=x1
now eliminate x1’, x2’, y1’, y2’.
A
(y1,y2)=(0,x1)
Y=g(X,Y)
B
A
(y1,y2)=(0,x1)
B
Verification conditions: tests
T
C)  B)= t(X,Y) /\
C
C)
D)  B)=t(X,Y) /\
D)
B
F
t(X,Y)
D
B
T
C
B)= D) /\ y2x2
y2>=x2
F
D
How to find values for coverage?
•Put true at end of path.
•Propagate path backwards.
•On assignment, relativize
expression.
•On “yes” edge of decision,
add decision as conjunction.
•On “no” edge, add negation
of decision as conjunction.
•Can be more specific when
calculating condition with
multiple condition coverage.
A>1 & B=0
no
yes
X=X/A
A=2 | X>1
no
yes
true
X=X+1
true
How to find values for coverage?
(A≠2 /\ X/A>1) /\ (A>1 & B=0)
A>1 & B=0
Need to find a satisfying
assignment:
no
yes
X=X/A
A=3, X=6, B=0
Can also calculate path
condition forwards.
A≠2 /\ X/A>1
A≠2 /\ X>1
A=2 | X>1
no
yes
true
X=X+1
true
How to cover a flow chart?





Cover all nodes, e.g., using search strategies:
DFS, BFS.
Cover all paths (usually impractical).
Cover each adjacent sequence of N nodes.
Probabilistic testing. Using random number
generator simulation. Based on typical use.
Chinese Postman: minimize edge traversal
Find minimal number of times time to travel each
edge using linear programming or dataflow
algorithms.
Duplicate edges and find an Euler path.
Test cases based on data-flow
analysis



Partition the program
into pieces of code with
a single entry/exit point.
For each piece find which
variables are
set/used/tested.
Various covering criteria:
 from each set to each
use/test
 From each set to
some use/test.
X:=3
t>y
x>y
z:=z+x
Test case design for black box
testing



Equivalence partition
Boundary value analysis
Cause-effect graphs
Equivalence partition


Goals:
 Find a small number of test cases.
 Cover as much possibilities as you can.
Try to group together inputs for which the program is
likely to behave the same.
Specification
condition
Valid equivalence
class
Invalid equivalence
class
Example: A legal variable



Begins with A-Z
Contains [A-Z0-9]
Has 1-6 characters.
Valid equivalence
class
Invalid equivalence
class
Starting char
Starts A-Z 1
Starts other 2
Chars
[A-Z0-9]
Has others
Length
1-6 chars 5
Specification
condition
3
4
0 chars, >6 chars
6
7
Equivalence partition (cont.)


Add a new test case until all valid equivalence classes
have been covered. A test case can cover multiple
such classes.
Add a new test case until all invalid equivalence class
have been covered. Each test case can cover only one
such class.
Specification
condition
Valid equivalence
class
Invalid equivalence
class
Example



AB36P
(1,3,5)
1XY12
(2)
A17#%X (4)


(6)
VERYLONG (7)
Valid equivalence
class
Invalid equivalence
class
Starting char
Starts A-Z 1
Starts other 2
Chars
[A-Z0-9]
Has others
Length
1-6 chars 5
Specification
condition
3
4
0 chars, >6 chars
6
7
Boundary value analysis

In every element class, select values
that are closed to the boundary.


If input is within range -1.0 to +1.0, select
values -1.001, -1.0, -0.999, 0.999, 1.0,
1.001.
If needs to read N data elements, check
with
N-1, N, N+1. Also, check with
N=0.
Test case generation based on
LTL specification
LTLAut
Flow
Compiler chart
Model
Checker
Transitions
Path
Path condition
calculation
First order
instantiator
Test
monitoring
Goals




Verification of software.
Compositional verification. Only a unit of
code.
Parametrized verification.
Generating test cases.
A path found with some truth assignment satisfying
the path condition. In deterministic code, this
assignment guarantees to derive the execution of
the path.
In nondeterministic code, this is one of the
possibilities.
Divide and Conquer




Intersect property automaton with the
flow chart, regardless of the statements and
program variables expressions.
Add assertions from the property automaton
to further restrict the path condition.
Calculate path conditions for sequences found
in the intersection.
Calculate path conditions on-the-fly.
Backtrack when condition is false.
Thus, advantage to forward calculation of
path conditions (incrementally).
Spec:
¬at l2U (at l2/\ ¬at l2/\(¬at l2U at l2))
¬at l2
l2:x:=x+z
X
l3:x<t
at l2
l2:x:=x+z
=
l3:x<t
l1:…
¬at l2
l2:x:=x+z
at l2
Spec:
¬at l2U (at l2/\ xy /\
(¬at l2/\(¬at l2U at l2 /\ x2y )))
xy
¬at l2
l2:x:=x+z
X
at l2/\
xy
l3:x<t
l2:x:=x+z
=
l1:…
¬at l2
at l2/\
x2y
l3:x<t
x2y
l2:x:=x+z
l0
Example: GCD
l1:x:=a
l2:y:=b
l3:z:=x rem y
l4:x:=y
l5:y:=z
no
l6:z=0?
yes
l7
l0
Example: GCD
l1:x:=a
l2:y:=b
Oops…with an
error (l4 and l5
were switched).
l3:z:=x rem y
l4:y:=z
l5:x:=y
no
l6:z=0?
yes
l7
Why use Temporal specification



Temporal specification for sequential
software?
Deadlock? Liveness? – No!
Captures the tester’s intuition about the
location of an error:
“I think a problem may occur when the
program runs through the main while loop
twice, then the if condition holds, while t>17.”
l0
Example: GCD
l1:x:=a
l2:y:=b
a>0/\b>0/\at l0 /\at l7
l3:z:=x rem y
at l0/\
a>0/\
b>0
l4:y:=z
l5:x:=y
no
at l7
l6:z=0?
yes
l7
l0
Example: GCD
l1:x:=a
l2:y:=b
a>0/\b>0/\at l0/\at l7
l3:z:=x rem y
Path 1: l0l1l2l3l4l5l6l7
a>0/\b>0/\a rem b=0
Path 2: l0l1l2l3l4l5l6l3l4l5l6l7
a>0/\b>0/\a rem b0
l4:y:=z
l5:x:=y
no
l6:z=0?
yes
l7
Potential explosion
Bad point: potential explosion
Good point: may be chopped on-the-fly
l0
Drivers and Stubs




l1:x:=a
l2:y:=b
Driver: represents the program
or procedure that called our
l3:z’=x rem y
checked unit.
/\x’=x/\y’=x
Stub: represents a procedure
called by our checked unit.
l4:y:=z
In our approach: replace both of
them with a formula
representing the effect the
l5:x:=y
missing code has on the program
variables.
no
Integrate the driver and stub
l6:z=0?
specification into the calculation
of the path condition.
yes
l7
Conclusions



Black box testing: Know transition relation,
or bound on number of states, want to find
initial
state, structure, conformance, temporal
property.
Software testing:
Unit testing, code inspection, coverage, test
case generation.
Model checking and testing have a lot in
common.