Property-Based Test Generation
Li Tan, Oleg Sokolsky, and Insup Lee
University of Pennsylvania
The Overview of Our Approach
Behavior specification
Feature conflict
detection
LTL formulae
F1, …, Fn
Model Checker
Quasi linear (Lassoshape) proof structure
Test trace
generator
LTL formulae
F1, …, Fn
Temporal Property
Translator
Criteria not
being covered
Specification Model
(CHARON)
Coverage
Criteria
System Modeling
Informal
System Specification
Traces {r1,…,rn}
Simulation-based
randomized
test generator
Traces
Testing result
Environ. Modeling
Hardware
Specification/limitation
Interface Definition
Test Harness
Implementation
Test result
Temporal (feature) Spec.
+
Model (optional)
Model-checking based
Test generator
Test suite
(Finite set of finite traces)
Goal: Using model-checking technique to make test generation more efficient, flexible,
and centered on the system-specific properties (features).
Step I. Preparing specifications
Properties (feature specification) as linear temporal logic formula
(optional) System specification system as CHARON (for hybrid systems) and EFSM
(for discrete systems)


Step II. Test generation using model checkers.
(For hybrid systems) Simulation-based test generation with the
assistance of predicate abstraction reachability analysis.

(For Discrete system)



(Option A) Using the proof structures of evidence-ready model checkers.
(Option B) Reducing the test generation for LTL formula to safety check
(For temporal specification only) Functional test.



Generating non-trivial test traces for temporal specification (feature
specification)
Detecting conflicting in temporal specification.
Step III: Realizing test harness.
From Property and Model to Test suite: Property-based test generation
I. From infinite length to finite: Synthesizing test suites for 9LTL property
A infinite Lasso-shaped test suite can be checked adequately by finite steps if the implementation
is bounded.
Turn=1, :c1, : c2
Turn=1, c1, : c2
Turn=2, c1, : c2
+
Turn=2, c1, c2
Turn=1, c1, c2
Turn=1, c1, c2
Estimating the number of relevant implementation states using slicing
A quasi-linear proof skeleton
A finite test suite
Test Generation using Model Checkers
Option A: Modifying model checkers and retaining proofs.
Option B: Using the idea of reducing LTL model checking to reachability
analysis [A. Biere etc], but enhancing the observer to retain proof
SMV model
+
Linear Temporal
Logic Specification
SMV model
+
Extended
Observer Model
Repetition
information
Extracted
Proof
Generated
test trace
II. From infinite numbers of traces to finite: selecting interesting traces
System properties are required to be held on all the paths, we will select only nontrivial paths, whose
characteristics are caught by ELTL formula systematically deriving from the properties.
LTL f => ELTL formulae a2e(f)={f(f’ ! (f’))|f’ Á f}
F= G(req -> F(cancelÇresponse))
F Æ (: G(req ! F(cancelÇ false)))
= F Æ F(req Æ G(: cancel))
Test the case that no cancel follows a request
(hence a reponse must be placed)
F Æ (: G(req ! F(false Ç response)))
= F Æ F(req Æ G(: response))
F Æ (: G(true ! F(cancel Ç response)))
= F Æ FG(: (cancel Ç response))
Test the case that no cancel or reponses occurs
after time t, (hence should not a request occur).
F Æ (: G(req ! false))
= F Æ F(req)
Test the case that there is request
Test the case that no response follows a request
(hence a cancel must be placed)
From only Property to Test suite: Functional test generation
So, what if only behavior (feature) specification is available ……
LTL formulae
F
Nontrivial ELTL formulae
Derived from F
Y=a2e(F)
f0 2 F Æ 2
f1 2 F Æ 2
Y
…….
Buchi automaton B1
Buchi automaton B0
Check nonemptiness
A trace satisfies f0
Y
fn 2 F Æ 2
Y
Buchi automaton Bn
Check nonemptiness
A trace satisfies f1
A trace satisfies fn
Testing Hybrid system: simulation-based test generator with predicate-abstract
reachability analysis
System
Modeling
CHARON
(Model)
Coverage
Criteria
No
Bad set
Reachability
Checker
Flatten hybrid
model
Predicate set
NO w/ more predicates
Simulation
/refinement
Yes w/ Trace
YES
Test Suite
Concretize
Implementation
An implementation of simulation-based test generator
a. CHARON simulator with test generator
b. Progress report of test generator
c. Visual display of generated test traces.
Realizing Test Harness
Charon model
for CARA
Charon model
For patient
Closed Charon Model for CARA
Test trace
Simulation-based
test generator
Coverage criteriae
Variable back_EMF
Value
Time
60.0
0.001
70.0
0.002
……
Test harness as
I/O Interface
Test Result
CARA simulator
/model-generated code
Standalone executable program
Conclusion
1.
Applying model-checking technique to traditional domain of test
generation is appealing.
1.
Test generation is centralized on system-specific properties
2.
State-of-art model checkers may be adapted as general purpose
test generator (and think properties as programs ).
3.
Techniques in model checking may help find interesting test
traces and provide new angle to view and think test generation.
Property-based test generation requires integrated efforts.
2.
1.
Test generation  witness generation.
1.
2.
2.
Proof is necessary to generate partial test suite and perform optimization.
Proof is also needed to extend the notion of “testable” properties.
Model-based code generation may help build test harness.