New Development Techniques:
New Challenges for Verification and Validation
Mats Heimdahl
Critical Systems Research Group
Department of Computer Science and Engineering
University of Minnesota
4-192 EE/CS; 200 Union Street SE
Minneapolis, MN 55455
1
Domain of Concern
http://www.cs.umn.edu/crisys
2
How we Develop Software
Concept
Formation
System Test
http://www.cs.umn.edu/crisys
Requirements
Specification
System
Test
Integration
Test
Design
Integration
Unit Test
Implementation
Analysis
Object Code
3
Validation and Verification
Concept
Formation
http://www.cs.umn.edu/crisys
Requirements
Specification
System
Design
Verification:
Are we building the
Integration
thing right?
Implementation
Validation:
Are we building the right thing?
4
Model-Based Development
Properties
http://www.cs.umn.edu/crisys
Visualization
Analysis
Testing
Specification
Model
Code
5
Prototyping
Model-Based Development Tools
• Commercial Products

http://www.cs.umn.edu/crisys





Esterel Studio and
SCADE Studio from
Esterel Technologies
SpecTRM from
Safeware Engineering
Rhapsody from I-Logix
Simulink and Stateflow
from Mathworks Inc.
Rose Real-Time from
Rational
Etc. Etc.
6
Research Tools (many):
RSML-e and Nimbus
Simulations of
environment
http://www.cs.umn.edu/crisys
RSML-e Formal
Models
(~20 running
concurrently)
7
How we Will Develop Software
Concept
Formation
Requirements
http://www.cs.umn.edu/crisys
Analysi
s
Properties
System
Specification/Model
Syste
m
Test
Integration
Test
Integration
Specification
Test
Implementation
8
FGS/FMS Mode Logic
RSML-e and Nimbus
Simulations of
environment
http://www.cs.umn.edu/crisys
RSML-e Formal
Models
(~20 running
concurrently)
9
Sample RSML-e Specification
http://www.cs.umn.edu/crisys
10
Capture Requirements as Shalls
http://www.cs.umn.edu/crisys
11
Translated All the Shalls into
SMV Properties
http://www.cs.umn.edu/crisys
12
Early Validation of Requirements
Using Model-Checking (NuSMV)
http://www.cs.umn.edu/crisys
•
•
•
Prove Over 300+ Properties in Less Than an Hour
Found Several Errors in Our Models Using Model-Checking
Substantially Revised the Shalls to Correct Errors
13
Early Validation of Requirements
Using Theorem Proving (PVS)
http://www.cs.umn.edu/crisys
•
•
•
Proved Several Hundred Properties Using PVS
More Time Consuming than Model-Checking
Use When Model-Checking Won’t Work
14
Model-Based Development Examples
http://www.cs.umn.edu/crisys
Company
Product
Tools
Specified & Autocoded
Benefits Claimed
Airbus
A340
SCADE
With Code
Generator
 20X Reduction in Errors
 Reduced Time to Market
Eurocopter
EC-155/135
Autopilot
GE &
Lockheed
Martin
Schneider
Electric
FADEDC Engine
Controls
SCADE
With Code
Generator
ADI Beacon





US
Spaceware
PSA
CSEE
Transport
Honeywell
Commercial
Aviation
Systems
70% Fly-by-wire Controls
70% Automatic Flight Controls
50% Display Computer
40% Warning & Maint Computer
90 % of Autopilot
 Not Stated
Nuclear Power
Plant Safety
Control
DCX Rocket
SCADE
With Code
Generator
MATRIXx
 200,000 SLOC Auto Generated
from 1,200 Design Views
Electrical
Management
System
Subway
Signaling System
SCADE
With Code
Generator
SCADE
With Code
Generator
MATLAB
Simulink
 50% SLOC Auto Generated
Primus Epic
Flight Control
System
 Not Stated
 50% Reduction in Cycle Time




Reduction in Errors
50% Reduction in Cycle Time
Decreased Cost
8X Reduction in Errors while
Complexity Increased 4x




50-75% Reduction in Cost
Reduced Schedule & Risk
60% Reduction in Cycle Time
5X Reduction in Errors
 80,000 C SLOC Auto Generated
 Improved Productivity from
20 to 300 SLOC/day
 60% Automatic Flight Controls
 5X Increase in Productivity
 No Coding Errors
 Received FAA Certification
15
A Simplified Development Model
Requirements and
Specification
http://www.cs.umn.edu/crisys
System Test
Code
Unit Test
Time
16
Ongoing Research
RSML-e, SCR, SpecTRM,
Statecharts, Esterel, SCADE,
Simulink, Etc. Etc.
CMU, SRI, Stanford, UC
Berkley, VERIMAG,
NASA, Etc., Etc.
Minnesota, Pennsylvania,
George Mason, NRL,
NASA Ames, Etc.
RSML-e, SCR, SpecTRM,
Statecharts, Esterel, SCADE,
Simulink, Etc. Etc. –UML
Properties
http://www.cs.umn.edu/crisys
Analysis
Testing
Prototyping
Visualization
Specification
Model
RSML-e, SCR, SpecTRM,
Statecharts, Esterel, SCADE,
Simulink, Etc. Etc. –UML
Code
Proof carrying code,
Provably correct compilers,
Test for correctness
17
Problems…
Tested
enough?
Trust the
results?
Can we trust execution
environment?
Can we trust execution
environment?
Properties
http://www.cs.umn.edu/crisys
Analysis
Testing
Prototyping
Visualization
Are the languages
usable—syntax and
semantics?
Can they play nice
together?
Specification
Model
Code
Can we really
trust the code?
18
Benefits of Modeling
Fewer
“Bugs”
http://www.cs.umn.edu/crisys
Savings
Time
19
Code Generation
Fewer
“Bugs”
http://www.cs.umn.edu/crisys
Coding effort
greatly reduced
Savings
Time
20
Qualified Code Generation
(theory)
http://www.cs.umn.edu/crisys
Unit testing
moved here.
Unit testing
eliminated for
generated code
Time
Savings
21
Code Generation Concerns
Concept
Formation
Is our model “right”?
Requirements
Properties
http://www.cs.umn.edu/crisys
Can we trust the execution
environment?
Can we trust our analysis
tools?
Can we trust our
properties?
System
Specification/Model
Integration
Can we
trust the
code
generator?
Implementation
22
“Correct” Code Generation
•
Provably correct compilers

•
http://www.cs.umn.edu/crisys
•
Proof carrying code

Generate
Specification/Model
Total correctness required
Base all specification testing on
the generated code

•
Very hard (and often not
convincing)
Loose the benefits of working
at the specification level
Output
Specification
Based Tests
Generate test suites from
specification


Implementation
Compare specification behavior
with generated code to better
trust your specification testing
Unit testing is now not
eliminated, but completely
automated
23
Output
Specification Testing
• Certify the execution • When have we tested
environment

Too costly and
probably impossible
enough?

http://www.cs.umn.edu/crisys
• Specification based

testing

Specification coverage
criteria

Any discrepancy and
either



the code generator is
wrong, or
the execution
environment is wrong,
or
the target platform is
faulty
What is adequate
coverage?
Criteria for
measurement are not
good for generation
– Technically covering
the specification, but
with useless tests

Do we reveal faults

Tradeoff between the
impossible and the
inadequate
24
Proof Techniques
(theory)
Reduced testing
since properties
proved correct in
specification stage
http://www.cs.umn.edu/crisys
Proofs
performed here
Time
Savings
25
Verification Trust
We need
properties
(requirements)!!!
Concept
Formation
Often lost in
the modeling
“frenzy”
Requirements
http://www.cs.umn.edu/crisys
Specification/Model
Properties
How System
do we
trust our
proofs?
Integration
Implementation
Proof validity in
production
environment?
26
Proof Techniques
•
Certify analysis tools

•
http://www.cs.umn.edu/crisys

Technically feasible, but is
the redundancy
“trustworthy”??
Cost…
Theorem Prover
Translation
RSML-e
Translation
State Exploration
Must keep analysis cost
under control
Generate test suites from
specification

Model Checker
Translation
Automation is key

•
Too costly and probably
impossible
Use redundant proof paths

•
Trusted
Translators
?
Low cost since it is already
done for the code generator
Many languages
Trusted and
many
analysis
Translators?
techniques
27
Proof Techniques
(worst case)
Added burden
that cannot be
leveraged later
http://www.cs.umn.edu/crisys
Most analysis is not
easy, nor cheap!
Savings
Time
28
Regression Verification
Large
Evolving
Model
100s, if not
1000s, of
properties
http://www.cs.umn.edu/crisys
Iterated
Weekly?
Daily?
Hourly?
Analysis Result
• Abstraction cost amortized
• Impact of change on abstraction
• Approximate techniques in day-to-day activities
29
Can We Achieve the Goal?
Redundant proof
process (PVS, SMV,
Prover, SAL,…)
http://www.cs.umn.edu/crisys
Specification testing
Test case generation
Verifiable
code
generator
?
Yes!
Abbreviated system
testing augmented
with generated tests
?
?
?
Automated unit testing (to
MC/DC?)—to check code
generator and specification
execution environment
?
Time
Savings
30
Perfection is Not Necessary
≥
Missed Faults
http://www.cs.umn.edu/crisys
• We only need to be better than what we are
now…

How do we demonstrate this?

Empirical studies are of great importance
31
Education of Regulatory Agencies
• Regulatory agencies are very conservative
And rightly so…
 Avionics software is very good

http://www.cs.umn.edu/crisys
• We need to understand regulatory and
•
industry concerns to get our techniques into
practice
We need to have convincing evidence that
our techniques work and are effective
32
New Challenges for V&V
•
Validate models


http://www.cs.umn.edu/crisys
•

Validate tools

•

We will rely a lot on tools for model
validation, can we trust them?
Creative use of testing necessary
Verify and Validate generated code


•
The models must satisfy the “real” requirements
Validate the properties used in analysis
Model testing crucial to success

Can we trust that the translation was correct?
Test automation crucial
Includes output to analysis tools
Adapt to the various modeling notations


Models will not come in one language
Translation between notations and tools
33
Discussion
http://www.cs.umn.edu/crisys
34