V&V&S
Verification, Validation and
Synthesis:
doing away with defects
SDS
Foil no 1
Quality and quality assurance
methods
Quality:
process quality = meeting the specification.
system quality = playing the role required by the environment.
Methods:
1. Constructive methods that aim to generate the right results in the
first place
=> The SDL methodology, program generation.
2. Corrective methods that aim to detect errors and make corrections.
=> Verification and validation.
SDS
Foil no 2
Verification and Validation
Barry Boehm (Boehm, 1981):
Verification: To establish the truth of correspondence between a
software product and its specification (from the Latin veritas, “truth”).
Validation: To establish the fitness or worth of a software product for its
operational mission (from the Latin valere, “to be worth”).
Verification: Are we building the product right?
Validation: are we building the right product?”
SDS
Foil no 3
Verification and Validation in TIMe
verify
domain
verify
specification
Developers
Application design
Framework design
Architecture design
implementation
Validate
Market
and
users
SDS
Foil no 4
needs
verify
instance
config.
needs
needs
validate
system
Techniques
Constructive
methods
Defects and errors
Corrective
methods (V&V)
Results
Synthesize
-design
-implement.
Requirements
Design
Reviews
analysis
Implementation
Testing
SDS
Foil no 5
Analysi
s
• Syntax analysis: by editor
• Static semantic analysis: by analyzer
• Dynamic semantic analysis: by simulation and
reachability analysis (modelchecking)
• Verification: by manual inspection and checking formal
relationships between models
• Validation: by manual inspection, by testing and formal
compatibility checks
SDS
Foil no 6
Static signal check 1
• Derive valid input and output signal sets from process graph
• Check signal sets against signal lists on each connection:
P2in = L2 + L3 + L5 P2out = L1 + L4 + L6
• Check signal destinations:
P2 out-to-p1 = L1 P2 out-to-p3 = L4 P2 out-to-p3 = L6
SDS
Foil no 7
Static signal check 2
• Valid input and output signal
sets against gate signal lists
once for each type
• Signal lists against each
other for each connection
• Signal destinations
SDS
Foil no 8
Static signal check 3
Who talks
together here?
• Find logical signal routes by
signal list partitioning
conserving flows
• Signal sets against signal lists
• Signal destinations
SDS
Foil no 9
Dynamic analysis
Liveness = something good will eventually happen (system specific)
safety = nothing bad will ever happen (system independent)
1. Reachability analysis
Full (up to 10**5 states)
Controlled partial (up to 10**8 states)
Random simulation (larger)
2. Structural reasoning
Invariants
Logical inference on equations, rules, axioms
Model checking
3. Execution
Simulation
Testing
SDS
Foil no 10
Reachability analysis
• Iteratively generate all reachable system states
• Analyse each state for
• deadlocks;
• incompleteness;
• other errors.
• Analyse the generated graph
• partial deadlock;
• liveness;
• etc.
SDS
Foil no 11
Are there any problems here?
[Aa]
A
[Gb,Sa,Qa]
Foil no 12
[Rb,Eb]
[Ga, Sb,Qb]
[Ra,Ea]
SDS
B
[Ab]
A
B
1
1
Ra
Sa
Rb
Sb
Sb
Ga
Sa
Gb
3
8
3
8
Gb
Qa
Ga
Qb
Aa
1
Ab
1
5
5
Ea
Eb
Qb
Qa
1
1
Step 1: Make transition charts
SDS
Foil no 13
Step 2: Generate reachability graph
Find all possible
behaviours
considering that a
signal transfer
takes two steps:
• send
• consume
SDS
Foil no 14
Findings
The errors:
• Unspecified receptions and deadlocks may occur
The reason:
• mixed initiatives not properly handled
• Graph “explodes” because independent behaviours are interleaved
and signals are queued
• Many paths lead to the same error
Is there a simpler way??
SDS
Foil no 15
Working directly in SDL
[Aa]
A
[Gb,Sa,Qa]
Mixed initiative state = a state
with inputs from two
independent sources (i.e.
association roles)
SDS
Foil no 16
mixed
initiative
state
invisible
from B
[Rb,Eb]
[Ga, Sb,Qb]
[Ra,Ea]
Association role behaviour =
the visible behaviour of one
object when observed from
another (associated) object.
B
[Ab]
A
B
1
1
Ra
Sa
Rb
Sb
Sb
Ga
Sa
Gb
3
8
3
8
Gb
Qa
Ga
Qb
Aa
1
Ab
1
5
5
Ea
Eb
Qb
Qa
1
1
Role behaviour and input consistent roles
Deriving a role behaviour:
• Replace invisible inputs by “none” (or
just make a mental note)
• Remove invisible outputs (or just ignore
them)
The result is a process graph with only
visible inputs and outputs. (or just a
mental view on the original graph)
Input consistency:
• An invisible transition is a transition with
a none input.
• An invisible transition is input consistent
iff the next-state explicitly accepts all
the visible inputs of the (present) state.
• The role is input consistent iff every
invisible transition in the role is input
consistent.
SDS
Foil no 17
A
1
invisible transitions
1 and 3 are not
input consistent
because 3 does
not accept Sa
none
Sa
Sb
Ga
3
8
Gb
Qa
1
5
5 and 1 are
input consistent
because 1 accepts
more than 5
none
Qb
1
How to ensure input consistency ?
When checking - for each
role:
• Find invisible transitions
• Check that every invisible
transition is input consistent
• Consider transitions from
mixed initiative states first.
When designing:
• for each state, in particular
mixed initiative states,
check for each role that all
transitions are input
consistent.
The intuition behind this is that the
associated process will not immediately
observe an invisible transition and therefore
the same visible outputs
should be accepted in both states
SDS
Foil no 18
Mixed initiative state
A
invisible transition
1 and 3 are not
input consistent
1
none
Sa
Sb
Ga
3
8
Gb
Qa
1
5
none
5 and 1 are
input consistent
Qb
1
S-Rule: Input consistent process
Design SDL processes such that all their role behaviours,
i.e. the behaviour visible to processes in their
environment, are input consistent.
In particular check the vicinity of mixed initiative states
SDS
Foil no 19
SDL and MSC
• The SDL behaviour shall “contain” all the MSC behaviours
in addition the SDL behaviour may contain:
- cases not treated by the MSCs
- behaviour on other interfaces
- behaviour in other services
SDL
structure
SDS
Foil no 20
SDL (object)
behaviour tree
MSC (instance)
behaviour tree
MSCs
“contains”
Verifying SDL
• generate all possible traces defined by the MSCs
• check that all traces may execute in the global behaviour tree for the
corresponding SDL (using the Validator)
SDL
SDS
Foil no 21
SDL behaviour tree
MSC behaviour tree
MSCs
Synthesizing behaviour 1
msc User_accepted
User
AC System
Idle
process ACSystem
Code
OK
Card out
Unlock
Idle
Door unlocked
msc User_rejected
User
AC System
Idle
Code
Code
CardOut
CardOut
OK
NOK
Unlock
Code
Card out
NOK
Idle
SDS
Foil no 22
Door unlocked
Idle
How to correct this?
...by combining and branching
Idle
Idle
Code
Code
Code
CardOut
CardOut
OK
NOK
SDS
Foil no 23
Validate
CardOut
OK
NOK
CardOut
NOK
Unlock
Unlock
Door unlocked
OK
Idle
Door unlocked
Idle
Synthesizing behaviour 2
MSC EnterZone
User
Process Type UserServer
UserServer
Authenticator
Authorizer
Idle
Card [cardid]
Enter PIN
Givepin [PIN]
DCL Cardid, PIN, zone integer;
Card (Cardid)
ValidatePIN
[cardid,PIN]
Enter PIN
NOK
exc
Illegal PIN
wait for PIN
OK
ValidateRights
[cardid,zone]
GivePIN (PIN)
NOK
exc
ValidatePIN
(cardid, PIN)
AccessDenied
OK
AccessGranted
SDS
Foil no 24
Validate PIN
Synthesizing behaviour 3
MSC EnterZone
User
Process Type UserServer
UserServer
Authenticator
Authorizer
Validate PIN
Card [cardid]
OK
NOK
Enter PIN
Givepin [PIN]
ValidatePIN
[cardid,PIN]
ValidateRights
(cardid, zone)
IllegalPIN
NOK
exc
Illegal PIN
ValidateRights
Idle
OK
ValidateRights
[cardid,zone]
OK
NOK
NOK
exc
AccessDenied
OK
AccessGranted
AccessDenied
AccessGranted
Idle
SDS
Foil no 25
Idle
Synthesizing behaviour 4
General approach:
• Split each MSC into instances, and make corresponding SDL fragments.
• Combine all the SDL fragments for a given process into a more
complete behaviour
• Remove non-determinism by combining and adding decisions or states
• Add unspecified behaviour details
• Add exceptions
• Consider all possibilities in each state: ensure that all roles are input
consistent!
SDS
Foil no 26
Specification and design
Specification
•••
Decompose
Verify MSC
Synthesize
Design
•••
Validate
interface
Verify SDL
SDS
Foil no 27
Validating interfaces
• Static signal checks: are all output signals received?
• Aligning MSC: do both sides align with the same MSCs?
• Comparing role behaviours manually: are required roles contained in
provided roles?
• Simulating with SDT: will the interface work for all simulated cases?
• Reachability analysis with SDT: will the validator find errors using state space
exploration?
structure
Type
SDS
Foil no 28
Type
Verify
context
verifying MSC decomposition
structure
aligning
Verifying SDL behaviour
behaviour
SDS
Foil no 29
Aspects:
• verifying MSC decomposition
• verifying SDL behaviour:
• object behaviour
• composition behaviour
Verifying SDL behaviour
Principle:
• every trace defined by the MSCs
shall be possible in the SDL
MSC and testing
MSC
system
Abstract test suite
MSC
application
infrastructure
support sw
hw
MSC
SDS
Foil no 30
system
under
test
Executable test suite
Tester
verdict
Testing what?
Application (SDL).
Application support:
•
Protocols.
•
Input-output modules.
General support:
•
Routing.
•
Operating system.
•
SDL runtime support.
•
Error handling.
•
Test and debug facilities.
Performance (non-functional):
SDS
Foil no 31
•
Real time response, delays.
•
Traffic handling capacity.
•
Load control.