Model-Checking in the Early
Lifecycle
Zoë Stephenson
1
Acknowledgements
• The work presented here was conducted under the MOSAIC project, project
number TP/3/DSM/6/I/15780, a collaboration between the University of York,
the University of Sheffield, Rolls-Royce plc., Goodrich Engine Control Systems
and Jaguar Cars Limited.
• This project is co-funded by the Technology Strategy Board's Collaborative
Research and Development programme, following an open competition.
• The Technology Strategy Board is an executive body established by the
Government to drive innovation. It promotes and invests in research,
development and the exploitation of science, technology and new ideas for the
benefit of business --- increasing sustainable economic growth in the UK and
improving quality of life.
2
Motivation
V&V activities
roughly 50%
Management
8%
System Integration
17%
Other
Softw are
3%
System
Specification
25%
Hardw are
Softw are
Integration
1%
Softw are
Integration Test
7%
Low Level
Softw are Test
17%
Softw are
Static Analysis
1%
Softw are
Implementation
10%
Softw are
Design
3%
Review s and
Inspections
8%
Proprietary industrial data, 2000
3
Motivation
• Reducing V&V effort should be very effective in
reducing project cost
– Especially by addressing issues earlier in the lifecycle
• We use model-checking to address V&V issues
• Questions from industrial parties:
– Is it applicable to our systems?
– Does it scale?
– How much effort does it save?
4
Outline
• Early lifecycle characteristics
• Eliciting individual properties
• Demonstrating applicability
5
santi/MOCpages.com
the LEGO Company
Early vs. Late
the MathWorks
6
LTSA
Early vs. Late
Early Lifecycle
Late lifecycle
• Few stable requirements
• Complete requirements
• Some experience of
previous systems
• Full system available to
evaluate
• Assessment targeted at
technical risks – novelty
and concurrent design
• Assessment targeted at
properties of complete
system
7
Property Elicitation
• Early lifecycle issues are:
– Individual aspects
– Not necessarily captured in requirements
• Where do they come from?
• Which ones are important?
• How do we know we’ve found all the important
ones?
• How should we write them down?
8
Property Source
• Current draft
requirements
• Sketches
• Documentation and
standards
• Existing systems
• Domain experts
• Technical risk
assessment
• Project priorities
Properties that the system
ought to satisfy
9
Technology-Independence
• System needs to satisfy properties regardless of
design, implementation, V&V technologies
– Different properties will be amenable to different types
of V&V technology
– Properties will need to be translated into various
different forms
• At the point of elicitation:
– Those forms are not guaranteed to be known
– Those participating in elicitation may not be
comfortable with those forms
10
Explicit Elicitation Process
Bounds elicitation
and helps to focus
discussion
Requirements
scoping
Scope description
Other sources
elicitation
Natural-language
properties
Model
formulation
Properties
Should ideally be concise
and precise
11
Elicitation Process
• Facilitated discussion
– Can use a separate scribe and chair
– Our experiments combined these roles
• Series of prompts:
– stimulate discussion
– explore different kinds of behaviour
– start with general issues and lead on to specific
issues
• e.g. from “expectations” to specific invariants
12
Typical Prompt Areas
• Boundary properties
– “is that all that the system needs to do?”
• Completeness
– “Did we cover every value/every state?”
• Dynamic completeness
– “…every value/state change?”
• Feasibility
– “…every high-risk area?”
• Interaction
– “…every sequence of interactions?”
• Expectation
– “Does it behave reasonably?”
13
Pilot Studies
• Fixed-priority scheduler implementation
– 23 properties elicited
– Around 15 checkable by model-checking
– Model-checking study ongoing
• New implementation of surface friction estimator
– 12 properties, 6 checkable
– Small model-checking study conducted
14
Observations
• Elicited properties not always easily checkable in a
model-checker
– “Detection time plus adjustment time to correct for the change in
surface friction ought to be within 100ms.”
• Elicited properties not always easily checkable in early
lifecycle
– “The error in estimation for lateral friction is approximately the
same across the whole friction range”
• Skills of a property elicitation specialist different to those
of a domain expert
– “Strategic pedantry”
15
Studies Performed
spin
Simulink Design Verifier
Fixed-priority
scheduler

Active roll control


Throttle resolver input
validation



Surface friction
estimator
16
Throttle Resolver Study
• Based on three problem reports
Function
Requirement
Problem Statement
extract
Underlying Error
Test Case
Test Case
Action Taken
17
Study Aims
• Given latest system specification, error
definition, requirement and test case:
– Produce a “damaged” version of the system that
contains the error
– Express the requirement as a property to check
– Model-check the latest system and the damaged
system against the property
• Current system should satisfy property
• Damaged system should violate property
• Generated counterexample should be similar to
test case
18
Requirements
Study Setup for spin
Latest Specification
Requirement
Requirement
Requirement
Requirement
CSS
197432.3
The rate of change threshold of dvTPAMovDet deg/s shall be
Unchanged
lowered to dvTPASSDet deg/s when one valid signal exceeds
dvTPAMovDet deg/s; it shall return to dvTPAMovDet deg/s when
all valid signals are moving at a rate of less than dvTPASSDet
deg/s.
CSS Following throttle movement there shall be a confirmation period Unchanged
197- of dvTPAMovConf seconds before the movement detection
44- become false.
2.3
CSS The crosscheck tolerance between the TRA signals and selected Unchanged
197- TPA shall be set to dvTRATPAXCTol deg.
452.3
CSS A crosscheck failure between a TRA signal and selected TPA
Unchanged
197- shall be confirmed during dvTRATPAXCConf seconds. During
46- the confirmation time, the last valid selected TRAV shall be held.
2.3
endeec: do
::
sync_eec?_;
atomic {
Hysterysis is applied to
movement detection to
prevent intermittent
detection.
by hand
To allow signals to settle
after transient.
Inhibit short duration
faults from latching fault.
Requirement
assert
Underlying Error
damage
Test Case
/* account for bus fault */
/* Detect movement */
mtpafault1 = (tpafault1 || busfault);
mtpafault2 = (tpafault2 || busfault);
/* BEGIN TRACE 24 CSS-197-149-2.3 */
mtpafault3 = (tpafault3 || busfault);
/* movTPA1 = | otpa1_u - tpa1_u | > tpamovt */
if
ntpafault = mtpafault1 + mtpafault2 + mtpafault3;
:: otpa1_u == tpa1_u -> movTPA1 = 0
:: otpa1_u - tpa1_u - 1 >= tpamovt -> movTPA1 = 1
/* Detect movement */
/* BEGIN TRACE 24 CSS-197-149-2.3 */
/* movTPA1 = | otpa1_u - tpa1_u | > tpamovt */
if
:: otpa1_u == tpa1_u -> movTPA1 = 0
:: otpa1_u - tpa1_u - 1 >= tpamovt -> movTPA1 = 1
check
assert
/* account for bus fault */
mtpafault1 = (tpafault1 || busfault);
endeec: do
mtpafault2 = (tpafault2 || busfault);
::
mtpafault3 = (tpafault3 || busfault);
sync_eec?_;
atomic {
ntpafault = mtpafault1 + mtpafault2 + mtpafault3;
compare
visualise
20
Counterexample
Results
• All properties satisfied by current model
• All properties give counterexamples in damaged
models
• All counterexamples comparable with actual
test-cases used
21
Effort Profile
hours
57
60
50
40
30
23
15
15
12
12
11
C
AR
n
io
at
is
A
al
su
TR
n
Vi
tio
is a
al
C
su
Vi
AR
n
io
at
C
lid
A
Va
AR
TR
n
n
tio
io
at
ra
a
A
lid
ep
Va
TR
pr
n
n
io
io
at
at
ar
lid
ep
Va
pr
n
io
at
C
lid
AR
Va
n
io
at
ic
A
rif
TR
Ve
n
io
at
ic
rif
C
Ve
AR
g
lin
el
A
od
R
M
T
g
C
lin
el
AR
n
od
io
M
at
ris
A
ilia
TR
m
C
n
io
Fa
AR
at
ch
ris
oa
ilia
pr
m
A
ap
Fa
TR
ic
at
ch
oa
em
pr
st
ap
Sy
ic
at
em
st
y
d
Sy
ar
r
ho
li b
et
al
m
rv
is
te
ys
In
al
an
al
rv
te
In
22
4
3
8
6
7
6
7
10
16
19
20
0
Simulink Design Verifier
• Extension to “Verification and
Validation” toolbox
• Integrates Prover Technology
“Prover plug-in”
• Adds a new library
– Contains blocks that you use
to specify assumptions and
properties
• Adds a new Tools sub-menu
–
–
–
–
Check model compatibility
Generate tests (coverage)
Prove properties
Options
23
Simulink Model
Study Setup for SDV
Latest Specification
extract
Requirement
assert
assert
damage
Test Case
compare
check
Underlying Error
Counterexample
25
Results
• All properties satisfied by current model
• All properties give counterexamples in damaged
models
• All counterexamples comparable with actual
test-cases used
26
Effort Profile
Problem Report
Setup time
Max checking
time
2388 (wrong flags)
1 hour
2s
2836 (reheal)
2 hours
5s
3307 (reselection)
2 days
14s
27
Observations
spin:
• Manual translation
• No model extraction
required
• Test cases can be
complex
• Test cases in terms of
Promela, not original
specification
SDV:
• Automatic translation
• Model extraction required
• Test cases relatively
simple
• Test cases in terms of
original specification
28
Results from Other Studies
• Active roll control
– Examined for mode conflict
– Much easier in spin than SDV
• Surface friction estimation
– Very difficult to check due to arithmetic involved
• Fixed-priority scheduler
– Challenging to model preemption accurately in
Promela
– Discovered one interrupt-masking issue that we fed
back into development cycle
29
Summary
• Used property elicitation to determine most
relevant properties in the early lifecycle
• Investigated model-checking tools using casestudies from industrial parties:
– Applicable to some parts of examined systems
– Submodel extraction needed to address scalability
– Effort profile results to be fed into cost-benefit
analysis task next year
30