TorX
Automated Model Based Testing
with Formal Methods
©
Jan Tretmans
Radboud University
Nijmegen (NL)
together with:
University of Twente
Enschede (NL)
[email protected]
[email protected]
Jan Tretmans Radboud University Nijmegen
1
TorX : Automated Model Based Testing
with Formal Methods
Contents
 Model based testing
 Formal, model based testing with transition systems
 Transition systems testing and ioco
 A Tool for transition systems testing
 TorX
 Current and future research
 What does it mean for ARTIST2
©
Jan Tretmans Radboud University Nijmegen
2
Automated
Automated Model
Model Based
Based Testing
test
TTCN
test
generation
cases
tool
model
IUT
conf
model
IUT conf model
  sound

exhaustive
test
test
execution
tool
tool
IUT
IUT passes tests
©
Jan Tretmans Radboud University Nijmegen
pass fail
3
Model Based Testing
 Testing with respect to a (formal) model / specification
 SDL, CSP, Lotos, Promela, UML, state diagrams, Spec#, . . . .
 Precise, formal definition of correctness
 good and unambiguous basis for testing
 Formal validation of tests
 Algorithmic derivation of tests
 tools for automatic test generation
 Maintenance of models / specifications, not of test suites
 regression testing
©
Jan Tretmans Radboud University Nijmegen
4
Model Based Testing
with Transition Systems
dertest
: LTS
generation

(TTS)
tool

IUT conf
i ioco
model
s
exhaustive
  sound
test
test
execution
t ||
tooli
tool
smodel
 LTS
i
IUT
confs
ioco
model
i IUT
 IOTS
IUT
i || passes
der(s) tests
 pass
©
Jan Tretmans Radboud University Nijmegen
pass fail
5
Formal Testing with Transition Systems
Test hypothesis :
s  LTS
der : LTS 
(TTS)
ioco
Proof soundness and exhaustiveness:
Ts  TTS
iIUT
IUT IOTS
IMPS
passes
exec : :
TESTS
IOTS 
IMPS

TTS 
{pass,fail}
(OBS)
©
IUTIMP . iIUT IOTS .
tTTS . IUT passes t
 iIUT passes t
Jan Tretmans Radboud University Nijmegen
iIOTS .
( tder(s) . i passes t )
 i ioco s
pass / fail
6
Implementation Relation
ioco
Correctness expressed by implementation relation ioco:
i ioco s =def   Straces (s) : out (i after )  out (s after )
p 
©
p
=  !x  LU {} . p !x
Straces ( s )
= {   (L{})* | s
p after 
= { p’ | p
out ( P )
= { !x  LU | p !x
Jan Tretmans Radboud University Nijmegen


}
p’ }
, pP }  {  | p 
p, pP }
7
Test Generation Algorithm
Algorithm
To generate a test case from transition system specification s0
compute T(S), with S a set of states, and initially S = s0 after  ;
For T(S), apply the following recursively, non-deterministically:
1
end test case
pass
2
supply input
!a
T( S after ?a   )
©
Jan Tretmans Radboud University Nijmegen
3
observe output
forbidden outputs
?y
allowed outputs
?x

fail fail
T ( S after !x )
allowed outputs or :
!x  out ( S )
forbidden outputs or : !y  out ( S )
8
Validity of Test Generation
For every test t generated with algorithm we have:
 Soundness :
t will never fail with correct implementation
i ioco s
implies
i passes t
 Exhaustiveness :
each incorrect implementation can be detected
with a generated test t
i ioco s
©
Jan Tretmans Radboud University Nijmegen
implies
 t : i fails t
9
A Tool for Transition Systems
Testing: TorX
 On-the-fly test generation and test execution
 Implementation relation: ioco
 Mainly applicable to reactive systems / state based systems;
 specification languages: LOTOS, Promela, FSP, Automata
user:
manual
automatic
next
input
specification
check
output
offer
input
TorX
observe
output
IUT
pass
fail
inconclusive
©
Jan Tretmans Radboud University Nijmegen
10
TorX Tool Architecture
spec.
explorer
specification
specification
text
©
primer
states
transitions
Jan Tretmans Radboud University Nijmegen
TorX
driver
abstract
actions
adapter
abstract
actions
IUT
IUT
concrete
actions
11
TorX
©
Jan Tretmans Radboud University Nijmegen
12
TorX Case Studies
 Conference Protocol
 EasyLink TV-VCR protocol
 Cell Broadcast Centre component
 ‘’Rekeningrijden’’ Payment Box protocol
 V5.1 Access Network protocol
academic
Philips
LogicaCMG
Interpay
Lucent
LogicaCMG
 Easy Mail Melder
academic
 FTP Client
LogicaCMG
 “Oosterschelde” storm surge barrier-control
ASML/Tangram
 DO/DG dose control
 Laser interface
©
Jan Tretmans Radboud University Nijmegen
ASML/Tangram
13
What has been Achieved ……

Sound and precise formal basis for model based testing
 ioco test theory
 proved test derivation algorithm

Test tool TorX
 prototype tool for model-based formal testing
 “is at least as good as conventional testing”
 supports test generation and test execution
 more, longer, and provably correct test cases

©
Applied successfully to different cases studies
Jan Tretmans Radboud University Nijmegen
14
Testing Transition Systems: Extensions
Status
model
with data
and time
and hybrid
and action
refinement
test case
?coin1
?coin2
? money  n: int 
! money
?coin3
?
[ n  35 ] ->
[ n  50 ] ->
? button1
? button2
:= 00
c
Vt:=:=00 Vcc :=
d Vct </10
dt =
3
! button2
d Vcc</15
dt =
2
[Vt = 15 ] ->
! tea
[[V
c c =510
] ->] ->
! coffee
? coffee
? tea
pass
©
Jan Tretmans Radboud University Nijmegen

fai
l
fai
l
15
Current and Future Research
Twente & Radboud
 Testing real-time aspects
 multi-channel real-time
 Testing complicated data structures
 transformational- + transition system based testing
 Action refinement
 when an abstract action is implemented as sequence of actions
 What is a good test suite
 test selection and test coverage
 Test adapter and test interface
 generic test environment
 Compositionality and integration testing
 differences diminish
©
Jan Tretmans Radboud University Nijmegen
16
Current and Future Research
Twente & Radboud
 Hybrid testing
 when continuous variables occur
 Compositionality and integration testing
 differences diminish
 Testing stochastic and probabilistic properties
 Multi-disciplinary
 system testing
 Relations between model checking, testing, static analysis,
theorem proving, etc.
 differences diminish
. . . . .
©
Jan Tretmans Radboud University Nijmegen
17
Some Dutch Testing Projects
 Côte de Resyste
(1998 - 2002)
- Conformance Testing of Reactive Systems: TorX
Philips
Lucent
TU Eindhoven
Uni. of Twente
 Atomyste
(LogicaCMG)
(Interpay)
(KPN)
- ATOm splitting in eMbedded sYStem TEsting
Uni. of Twente
Radboud Uni. Nijmegen
 Stress - Systematic Testing of Real-time Embedded Systems
 Testing real-time properties
Uni. of Twente
 Testing data-intensive systems
Radboud Uni. Nijmegen
 Tangram
- Model Based Testing and Diagnosis
 Testing ASML Wafer Stepper machines - application oriented
ASML, ESI, TUD, TUE, UT, RU, S&T, TNO
©
Jan Tretmans Radboud University Nijmegen
18
ARTIST2 Activities
Quantitative Testing & Verification
1. Theory for testing embedded systems
a.
b.
c.
d.
real-time aspects
data aspects
extended conformance testing theories integrating a. and b.
test action refinement
a.
b.
real-time schedulability analysis
optimal control synthesis
a.
b.
model checking algorithms for CTMC, MDP
integration of performance analysis into verification
a.
b.
data structures for real-time and stochastic modelling and analysis
test interfaces and test adapters
a.
b.
c.
collection of case studies
comparison
identification of links to industrial tools
2. Verification and scheduling
3. Verification of stochastic systems
4. Tool-oriented research
5. Application of testing and verification tools in industrial settings
©
Jan Tretmans Radboud University Nijmegen
19
ARTIST2 Activities
Quantitative Testing & Verification
1. Theory for testing embedded systems
a.
b.
c.
d.
2.
Verification and scheduling
a.
b.
3.
real-time schedulability analysis
optimal control synthesis
Verification of stochastic systems
a.
b.
4.
real-time aspects
data aspects
extended conformance testing theories integrating a. and b.
test action refinement
model checking algorithms for CTMC, MDP
integration of performance analysis into verification
Tool-oriented research
a.
b.
data structures for real-time and stochastic modelling and analysis
test interfaces and test adapters
a.
b.
c.
collection of case studies
comparison
identification of links to industrial tools
5. Application of testing and verification tools in industrial settings
©
Jan Tretmans Radboud University Nijmegen
20