The State of Hybrid Model-Based Testing
Michiel van Osch
21-06-2005
Index
• Introduction
• Testing a Water Tank Controller
• Hybrid Model-Based Testing
– Conformance Relation
– Test-case generation
• Work in Progress
• Questions
2
Model-Based Testing
?“stop traffic”
?“stop traffic”
!“red”
Discrete
Model
!“red”
Test-tool
Discrete
System
pass
3
Hybrid Systems
Desired
Temperature: 25o
Coffee with
Milk and Sugar
Room
Temperature
“1”, “2”, “1”
“brake”
Hydrolic
pressure
4
Hybrid Model-based Testing
SP > ST
?Set_setpoint(sp)
?Set_setpoint(sp)
Heater ON
Heater OFF
SP £ ST
Lens
temperature
Lens
temperature
!Print(“warning”)
!”open
Cooling
Valve”
Test-tool
Hybrid-Model
!”open
Cooling
Valve”
Hybrid System
pass
5
Challenges
• Theory:
– Develop Conformance Concepts
– Develop Test-case Generation Algorithms
– Prove Correctness and Applicability of Theory
• Practice:
– Implement Tooling
– Exactly Model Relevant Aspects
– Connect Models, Tools and Systems
6
Index
• Introduction
• Testing a Water Tank Controller
• Hybrid Model-Based Testing
– Conformance Relation
– Test-case generation
• Work in Progress
• Questions
7
The water Tank Controller
set point
Tank Controller
messages
Water
Tank
Temperature
Tank Controller
• Discrete input: Set point
• Discrete output: communication
• Continuous input: Temperature
(from sensor)
• Continuous output: Heat
(from Heater)
Heater
Sensor
8
Specification (1)
Set_setpoint(sp)
Tank
Controller
Print(“Warning”)
ST
Tank Controller
• Discrete input: Set_setpoint(sp)
• Discrete output: Print(“warning”)
• Continuous input: ST
• Continuous output: H
H
Hybrid I/O
automaton
9
Specification (2)
SP > ST
?Set_setpoint(sp)
?Set_setpoint(sp)
Heater ON
Heater OFF
SP £ ST
!Print(“warning”)
ST
H
ST
H
input
Heater OFF
output
input
output
Heater ON
10
Test Setup
Set_Setpoint(sp)
Set_setpoint(sp)
ST
Set-point
Temp. Sens.
Tank
Controller
Print(“Warning”)
Print(..)
ST
Specification
H
Tank Controller
Message
H
Heat
Test-tool
Temperature
Sensor
Heater
Implementation
11
Testing...
Set_setpoint(sp)
Tank
Controller
Tank Controller
Print(“Warning”)
ST
H
Test-tool
Temperature
Sensor
Heater
Specification
SP > ST
?Set_setpoint(sp)
Implementation
?Set_setpoint(sp)
Heater ON
Heater OFF
SP £ ST
!Print(“warning”)
ST
Temp. Sens.
H
Heat
Specification
Implementation
12
Testing…
Set_setpoint(SP)
Set-point
Set_setpoint(sp)
Tank
Controller
Tank Controller
Print(“Warning”)
ST
H
Test-tool
Temperature
Sensor
Heater
Specification
SP > ST
?Set_setpoint(sp)
Implementation
?Set_setpoint(sp)
Heater ON
Heater OFF
SP £ ST
!Print(“warning”)
ST
Temp. Sens.
H
Heat
Specification
Implementation
13
Testing…
Set_setpoint(sp)
Tank
Controller
Tank Controller
Print(“Warning”)
ST
H
Test-tool
Temperature
Sensor
Heater
Specification
SP > ST
?Set_setpoint(sp)
Implementation
?Set_setpoint(sp)
Heater ON
Heater OFF
SP £ ST
!Print(“warning”)
ST
Temp. Sens.
H
Heat
Specification
Implementation
14
Testing
fail
Set_setpoint(sp)
Tank
Controller
Tank Controller
Print(“Warning”)
Print(“warning”)
ST
H
“error”
Test-tool
Temperature
Sensor
Heater
Specification
SP > ST
?Set_setpoint(sp)
Implementation
?Set_setpoint(sp)
Heater ON
Heater OFF
SP £ ST
!Print(“warning”)
ST
Temp. Sens.
H
Heat
Specification
Implementation
15
Testing…
fail
Set_setpoint(sp)
Tank
Controller
Tank Controller
Print(“Warning”)
ST
H
Test-tool
Temperature
Sensor
Heater
Specification
SP > ST
?Set_setpoint(sp)
Implementation
?Set_setpoint(sp)
Heater ON
Heater OFF
SP £ ST
!Print(“warning”)
ST
Temp. Sens.
H
Heat
Specification
Implementation
16
Index
• Introduction
• Testing a Water Tank Controller
• Hybrid Model-Based Testing
– Conformance Relation
– Test-case generation
• Work in Progress
• Questions
17
Conformance
A hybrid implementation is input-output conform a hybrid
specification if for all observable traces of the
specification:
The set of possible output actions and trajectories,
after the execution of a trace on the implementation,
is a subset of
the set of possible output actions or trajectories,
after the execution of a trace on the specification.
18
Some Decisions
• Defined on Hybrid I/O Automata
– Input Action Enabled
– Input Trajectory Enabled
– Non-zeno
• State Based: possible output actions and trajectories
in a certain state
• Input and output trajectories are treated together
• There is no quiescence observation (as of yet)
19
Conformance relation
We Formally Define:
• A hybrid trace  = 0 a1 1 a2 … as a sequence of discrete actions
and trajectories of continuous variables (alternating)
• A after : the set of states in which A can be after executing trace

• out(q): the set of output actions or trajectories possible in state q
• Out(Q): the set of output actions or trajectories possible in the set
of states Q
Then implementation HIOA i is hioco specification HIOA s if:
i hioco s if traces(s): out(i after )  out(s after )
20
Examples
!a
!a
!c
Correct!
!b
!b
specification
q
specification
implementation
Correct!
q
implementation
21
Examples
!a
!a
!c
Incorrect!
!b
!b
specification
q
specification
implementation
Incorrect!
q
implementation
22
Test-Case generation &
Execution
•
•
•
Input actions: Select and stimulate
Output actions: Observe and validate
Trajectories:
1. Select a time up to which to apply input
2. Select an input trajectory to use for stimulation
3. Stimulate and observe output variables :
– Until the end of the trajectory, without observing a
mismatch between observed and specified output
trajectories, in this case continue
– Until an output action is observed, in this case validate the
output action and if correct, then continue
– Until a mismatch between observed and specified output
variables occurs
•
As long as verdict “fail” is given, the verdict “pass”
can be given
23
Test-case generation
fail
?Set_setpoint(SP)
{SP > ST}
pass
!Print(“warning”)
pass
fail
!Print(“error”)
fail
24
Index
• Introduction
• Testing a Water Tank Controller
• Hybrid Model-Based Testing
– Conformance Relation
– Test-case generation
• Work in Progress
• Questions
25
Work in Progress (1)
• Does a continuous variable always have a value?
“1”,”2”
Coffee
temperature
When no coffee comes
out of the machine, temperature
Is not continuously 0!
• State Changes During a Trajectory
q
q’
While applying an input trajectory
and observing the output,
States change
26
Work in Progress (2)
• Quiescence
– Is quiescence a property on the discrete actions of the system
or also on continuous behavior of the system?
• Notation
What does this mean?
27
Current & Future Work
• Further Develop these Concepts (Formalization,
Theorems, Proofs)
• Make Concepts Workable in Practice
– Sampling of continuous behavior
– Selecting Realistic Input Trajectories
– Deal with Uncountable Many Possible Input and Output
Trajectories and States
– Deal with Inaccuracy of Observed Output
• Implement Proof of Concept
28
Questions
29