QuickCheck@Neilfest
(or what I learned from Neil about software testing)
John Hughes
Chalmers University/Quviq AB
• Company founded May 2006
– Tools for testing software
– Profitable in the first year!
• Customers include…
Soon
TBA
QuickCheck
• Tests software automatically against a formal
specification
prop_reverse() ->
?FORALL({Xs,Ys},
{list(int()),list(int())},
reverse(Xs++Ys) ==
reverse(Xs)++reverse(Ys)).
Testing…
19> eqc:quickcheck(ex:prop_reverse()).
.........Failed! After 10 tests.
{[2],[3,2]}
Shrinking....(4 times)
{[0],[1]}
false
QuickCheck
• Tests software automatically against a formal
specification
prop_reverse() ->
?FORALL({Xs,Ys},
{list(int()),list(int())},
reverse(Xs++Ys) ==
reverse(Xs)++reverse(Ys)).
QuickCheck in Brief
Specification
Test
outcome
Test case
Generate
Execute
Simplify
Minimal
counterexample
e.g. {[0],[1]}
A More Realistic Test Case
• A test case for the Erlang process registry
Spawn a new process
test_register() ->
(set up test data)
Pid = spawn(),
register(name,Pid),
Register it—a sidePid2 = whereis(name),
effect
assert(Pid==Pid2),
unregister(name).
Inspect the results
Restore the state,
ready for next test
Did the test pass?
Industrial Test Cases
• A sequence of
– Calls to an API under test
– Checks on the results
• Not much like {[0],[1]}…
• How can we convert this kind of test case into
a logical property?
QuickCheck in Industry
Test case:
a program
Specification
Generate
Test
outcome
Execute
Simplify
Minimal
program
provoking a
fault
Programs as Data Objects
• Why?
– So failing test cases can be printed
– So failing test cases can be repeated
– So failing test cases can be simplified
Can we mix generation and execution?
• Randomly generate and perform each call
– Save a test case as a list of functions and actual
parameters?
– NO!!!
test_register() ->
Pid = spawn(),
register(name,Pid),…
• When we repeat the test, Pid has a different
value
– We must use a symbolic variable!
Test Case Language
• Do we need conditionals?
X = foo(…),
case p(X) of
true -> baz(X);
false -> bar(X)
end
X = foo(…),
assert not(p(X)),
bar(X)
• Tests should be deterministic
– Check the right branch, don’t test
• Users need simplicity—straight-line code is
simple, and suffices
Generating Test Cases
• How do we know
– Which commands are valid at each point?
– What test data is available at each point?
– What results are expected?
• Track a test case state
– Check preconditions before generating each
command
– Store available data in the state
– Check postconditions wrt test case state
Process Registry State
type state() = record
pids::list(pid()),
regs::list({atom(),pid()})
end.
Available
process ids
Currently
registered
processes
Process Registry State
Available
process ids
type state() = record
pids::list(symbolic(pid())),
regs::list({atom(),symbolic(pid())})
end.
Currently
registered
processes
• Two-level programs!
– Static = test case generation time
– Dynamic = test execution time
Two-level State Machines
• Preconditions
– Checked during generation, purely static
• Postconditions
– Checked during execution, purely dynamic
• Next state function
– Used at both times, two-level
• Command generator
– Used during generation, two-level
Process Registry
• spawn()
– Adds a (dynamic) pid to the state
• unregister(Name)
– Pre: Name must be registered
– Removes Name from the state
• register(Name,Pid)
– Adds {Name,Pid} to the state
– Post: exception if Name or Pid already registered
A Failing Test Case
[{set,{var,2},{call,…,spawn,[]}},
{set,{var,3},{call,…,register,[a,{var,2}]}},
{set,{var,6},{call,…,register,[b,{var,2}]}},
{set,{var,8},{call,…,spawn,[]}},
{set,{var,9},{call,…,register,[b,{var,8}]}}]
V2=spawn(),
register(a,V2),
register(b,V2),
V8=spawn(),
register(b,V8)
Code
Spec
Inconsistency!
A Buggy Spec
• spawn()
– Adds a (dynamic) pid to the state
• unregister(Name)
– Pre: must be registered
– Removes Name from the state
• register(Name,Pid)
…provided there
is no exception!
– Adds {Name,Pid} to the state
– Post: exception if Name or Pid already registered
”The Trick”
Added to the state
Purely dynamic
[{set,{var,2},{call,…,spawn,[]}},
…]
• What if we know part of the structure?
– Check the known part in the postcondition
– Add the known part statically to the state
• ”The Trick” is as useful as ever!
Test Case
Generator
=
Randomized
Generating
Extension
Two-level languages are helping to find thorny
bugs in industrial systems!
Best Bug!
• In Ericsson’s Media Proxy (Multimedia IPtelephony product, SIP)
Add
Add
Sub
Add
Sub
Add
Sub
• A serious bug
• Obtained from a simple specification, by
simplifying 160-command sequence!