Integration of Functional and
Timed Testing of
Real Time and Concurrent
Systems
V. Kuliamin, A. Petrenko, N.!Pakoulin,
I.!Bourdonov, A.!Kossatchev
[email protected]
ISP RAS, Russia
Classic testing scheme
s1
stimulus
Test System
r1
s2
r2
reaction
s3
r3
Test =
Test sequence
s4
r4
s5
r5
Target System
Concurrency and real-time issues
stimulus1
stimulus
reaction1
reaction3
reaction





stimulus2
Target System
Multiple access points
Simultaneous inputs
Multiple reactions
Spontaneous reactions
Temporal constraints
reaction2
reaction
stimulus
reaction
Δt <= T0
Outline
Theoretical background
 Practical considerations
 Case studies

Proposed model
IOSM – Input/Output State Machine
?b
 States
S
!z
 Stimuli
X
?a
 Reactions R
!y
!x
 Transitions T
 φ : T → S×(X ∪ R ∪ {i})× S
Multiple
reactions
Internal
transition
Spontaneous
reaction
Empty stimulus
X = I ∪ {e}
Empty stimulus e models the knowledge of
system that it has no inputs
!x
!y
?a
?e
Asynchronous State Machine –
IOSM with empty stimulus
Behavior function
!x
!y
!x
?a
?e
?a
!y
B : X* → P (R*)
B : X* → P (R*)
ai ↦ { xi }
ai ↦ { yjxk…xlyn : k+…+l = i }
I.Burdonov, A.Kossatchev, V.Kuliamin. Asynchronous State Machines: Classification
and Applications to Testing. Proceedings of ISP RAS, Moscow, 2003. In Russian
Modeling parallel inputs
?a
?b
?{a,a}
?{a,a,b}
?{b,b}
?{a,b}
How to model responses
to all possible multistimuli?
?{a,b,b}
Plain concurrency : multistimulus is
equivalent to some sequence of the stimulus
contained
Outline
Theoretical background
 Practical considerations
 UniTesK
test development method
 Case
studies

 Modifications
for testing concurrent systems
UniTesK test construction
Behavior Model
Testing Model
✕
Target System
✕

Behavior Correctness
Checking
Test Input Construction
UniTesK test architecture
Test sequence construction
Test
Engine
Test
Test
sequence
Action
construction
Iterator
Test
Scenario
Specification

✕
Oracle
Mediator
Target
system
Example of specifications
specification int deposit ( int x )
{
pre
{
return x >= 0;
}
post
{
return balance == @balance + x;
}
}
Example of scenario
scenario class AccountTest
{
AcountSpecification model;
AbstractState getState()
{ return new IntState( model.balance ); }
scenario deposit()
{
iterate(int i = 0; i < 5; i++;) model.deposit(i);
}
scenario withdraw()
{
iterate(int i = 0; i < 5; i++;) model.withdraw(i);
}
}
Test sequence construction
Test
Engine
Test Action
Iterator
Modifications for testing concurrency

Specifications
 Asynchronous
reactions
 Temporal constraints

Testing
 Providing
parallel inputs
 Collecting reactions
 Checking the behavior as a whole
Specifying asynchronous reactions
specification reaction UDPPacket UDPResponse ( )
{
pre { return !ModelUDPPackets.isEmpty ( ); }
post
{
return
@ModelUDPPackets.clone().contains(UDPResponse)
&& !ModelUDPPackets.contains(UDPResponse);
}
}
Specifying temporal constraints
specification reaction UDPPacket UDPResponse ( )
{
pre { return !ModelUDPPackets.isEmpty ( ); }
post
{
return
@ModelUDPPackets.clone().contains(UDPResponse)
&& !ModelUDPPackets.contains(UDPResponse)
&& time –
@ModelUDPPackets.clone().time(UDPResponse)
<= 50*numberOfNodes;
}
}
Providing parallel inputs
s14
s24
s33
s13
s23
s12
s22
s11
s21
s32
Target
System
s31
Multisequence is used instead of sequence
of stimuli
Collecting asynchronous reactions
11
r11
Target
System
r21
21
31
r12
r22
12
r23
22
32
23
r31
r32
r33
33
Time
Reactions form a partially ordered set
Constructing model action series
Stimuli
Reactions
Plain concurrency axiom






✕

Modified test architecture
Test sequence construction
Test
Engine
Test Action
Iterator
✕
Synchronization
Manager
Serializer
Oracle
Mediator
Target
system

Reaction
Collector
Outline
Theoretical background
 Practical considerations
 Case studies

Case studies



Nortel Networks SOS – 1994-1996
 about 30 serious bugs found
MSR IPv6
– 2001-2002
 several serious bugs found
MS Mobile IPv6
– 2002-2003
References






http://www.ispras.ru/groups/rv/rv.html
I. Burdonov, A. Kossatchev, V. Kuliamin. Asynchronous State Machines:
Classification and Applications to Testing. Proceedings of ISP RAS,
Moscow, 2003. In Russian
V. Kuliamin, A. Petrenko, I. Bourdonov, and A. Kossatchev. UniTesK Test
Suite Architecture. Proceedings of FME 2002. LNCS 2391, pp.!77-88,
Springer-Verlag, 2002.
A. Petrenko. Specification-based Testing: Towards Practice. Proceedings of
PSI’01, LNCS, Springer-Verlag. v. 2244, pp. 287-300
V. Kuliamin, A. Petrenko, I. Bourdonov, A. Demakov, A. Jarov, A.
Kossatchev, S.!Zelenov. Java Specification Extension for Automated Test
Development. Proceedings of PSI'01. LNCS, Springer-Verlag. v. 2244, pp.
301-307.
A. Petrenko, I. Bourdonov, A. Kossatchev, and V. Kuliamin. Experiences in
using testing tools and technology in real-life applications. Proceedings of
SETT'01, India, Pune, 2001.
Thank you!