1
Accuracy Analysis of
Long-Run Average Performance Metrics
B.D. Theelen , J.P.M. Voeten
and Y. Pribadi
Information and Communication Systems Group, Faculty of Electrical Engineering
Eindhoven Embedded Systems Institute
Eindhoven University of Technology, P.O. Box 513, 5600 MB Eindhoven, The Netherlands
E-mail: [email protected]
Abstract— Before implementing a system with hardware
and software components, performance modelling is often
used for evaluating design alternatives. The formal modelling language Parallel Object-Oriented Specification Language (POOSL) has proven to be very useful for analysing
performance metrics of design alternatives for real-life industrial systems. Based on its mathematically defined semantics, POOSL enables to analytically compute performance metrics by analysing the Markov chain that is implicitly defined by a POOSL model.
Models of real-life industrial systems are however often
too complex to be analysed exhaustively. Performance evaluation is therefore based on simulation, enabling the estimation of performance metrics. However, simulation results
only have a proper meaning if their accuracy is known. In
general, longer simulation yields more accurate results. To
analyse the accuracy of simulation results and automatically
terminate the simulation after accurate results have been obtained, confidence intervals can be used.
Based on the properties of the Markov chains implicitly
defined by POOSL models, this paper presents a technique
of regenerative cycles for analysing the accuracy of longrun sample average estimation. Furthermore, an algebra
of confidence intervals is defined to enable accuracy analysis of long-run time averages as well as long-run sample
and time variances. Finally, library classes are introduced
for POOSL, which ease accuracy analysis for the mentioned
performance metric types.
Keywords—Markov Chains, Performance Estimation, Reward Structures, Confidence Intervals.
I. I NTRODUCTION
To manage complexity and to shorten design time, industry is forced to deploy system-level specification and
design approaches that enable analysing the performance
of design alternatives. Such system-level specification and
design methods define frameworks for developing a performance model, which describes the system in the earThis research is supported by PROGRESS, the embedded systems
research program of the Dutch organization for Scientific Research
NWO, the Dutch Ministry of Economic Affairs and the Technology
Foundation STW.
liest phases of the design. This performance model allows to evaluate whether the conceptual solutions for realising the system will satisfy the performance requirements before actually implementing the system with hardware and software components. To ensure that a performance model is adequate for answering the performance
questions, system-level specification and design methods
should utilise well-defined modelling languages. An example of a system-level modelling language for complex real-time hardware/software systems is the Parallel
Object-Oriented Specification Language (POOSL), which
was introduced in [12], [5] and [10]. The POOSL language
has proven to be very useful for describing real-life industrial systems and for analysing their performance. For instance in [13], POOSL has been applied for analysing the
performance of design alternatives for an industrial Internet core router.
The POOSL language is equipped with a mathematically defined semantics, which is based on a two-phase execution model [9]. The state of a model can either change
by asynchronously executing actions (taking no time) or
by letting the time pass synchronously. The semantics of
the non-real-time part of POOSL is given in [12], whereas
the formalisation of the real-time extension is described in
[4] and [5]. For the purpose of performance evaluation, the
semantics also supports expressing probabilistic behaviour
[17]. Founded on the semantics, a POOSL model defines a
unique timed probabilistic labelled transition system [17],
which is obtained after resolving the non-determinism in
the POOSL model by an external scheduler [15]. This labelled transition system can be transformed into a discretetime Markov chain [17]. Evaluating the performance of
design alternatives with POOSL is performed by analysing
this Markov chain.
In case the number of states of the Markov chain is not
too large, the result for a performance metric can be computed analytically by means of equilibrium analysis [17],
[11]. However, real-life industrial systems often result in
Markov chains with too many states to be analysed ex-
2
haustively [13]. Performance evaluation of such systems
is therefore based on simulation, which enables to estimate
the performance results. It is however unclear how long a
simulation should run before the performance results are
sufficiently accurate with respect to the actual performance
figures. Generally, more accurate results are obtained for
longer simulation runs. To analyse the accuracy of simulation results and automatically terminate the simulation after obtaining accurate results, the technique of confidence
intervals [16] can be applied.
A prerequisite for applying the classical technique of
confidence intervals is that the subsequent samples for a
specific performance metric are independent and identically distributed (i.i.d.). In general, this property can not
be justified. This paper derives a technique of regenerative
cycles for analysing the accuracy of performance metrics
that can be expressed as a long-run average of sample values, where the subsequent samples do not satisfy the i.i.d.
property. The classical technique of confidence intervals
is furthermore only applicable for long-run sample averages. To enable analysing performance metrics that are
expressed as a long-run time average or as a long-run sample or time variance, this paper defines an algebra of confidence intervals. The algebra is used to construct POOSL
library classes that ease accuracy analysis of the mentioned
performance metric types.
The remainder of this paper is organised as follows. The
next section gives a brief overview of the Markov Chains
that are implicitly defined by POOSL models. Section III
presents the technique of regenerative cycles for analysing
the accuracy of long-run average performance metrics. In
section IV, an algebra of confidence intervals is defined to
enable discussing the accuracy analysis of long-run time
averages as well as long-run sample and time variances
in section V. Section VI introduces the resulting POOSL
library classes. Conclusions are given in section VII.
II. M ARKOV C HAINS
FOR
"!
# $
&%
'( #( $ %
%
4
)
* +, .-0/ 1 23
+
65 7 )
) $8 9 : ;6< $ =
&%
> , -@?
III. E STIMATION
POOSL
Each POOSL model, together with an external scheduler, defines a discrete-time Markov chain
,
represents the state of the
where each random variable
model at discrete time-epoch . The (countable) set of
all states is denoted with . The sequence of states that
is visited during execution of the model is subject to the
(stochastic) matrix of transition probabilities. A finite
sequence of states
or an infinite sequence of states
is called a trace if, for
, respectively each
, the probability
each
of changing from state
to
, denoted as
, is
larger than 0.
A state of a Markov chain may be repeatedly visited at
different time-epochs. If a state is visited more than once
with probability 1, the state is called recurrent. Any nonrecurrent state is transient. Now, let
denote the cycle time through a recurrent state . If the expected cycle time through , denoted with
, is finite,
is
named positive. Markov chains which have a positive state
that can be reached from any other state with probability
1, are called ergodic [14].
Ergodic Markov chains have the property that the set
of states can be split into a set of positive states and
a set of transient states. All infinite traces of an ergodic
Markov chain visit transient states only finitely many times
with probability 1, whereas all positive states are visited
infinitely many times with probability 1. Furthermore, a
unique equilibrium distribution
can be defined, such that the equilibrium
probability
of a state is equal to the fraction of timeepochs that the Markov chain is in state . As a result,
it can be derived that the equilibrium probability of any
positive state
equals
[14], whereas the
equilibrium probability of a transient state is equal to 0.
Performance evaluation with POOSL is based on
analysing reward information that is contained in the states
of the Markov chain. Each time a certain state is visited,
a reward as specified by a reward function
is
obtained. Many performance metrics can be expressed as
long-run averages of such rewards. In a POOSL model,
(intermediate) performance results are stored in variables.
These variables are updated every time a new reward is obtained for the considered performance metric. A Markov
chain, together with some set of reward functions, is called
a reward structure. A POOSL model defines such a reward structure. This reward structure is defined implicitly
as tools for executing POOSL models [6], [1], [10] do not
generate it explicitly.
OF
L ONG - RUN S AMPLE AVERAGE
Many performance metrics are defined as a long-run
sample average of rewards (or combinations thereof, see
section V). An example is the average latency of transferring a packet through a system. For a reward function
and an ergodic Markov chain
, the long-run
12
sample average of rewards is defined as
A
>
! Actually, the long-run sample average of rewards is defined by the
$ H I E H J I K L H I E J , where M is
limiting behaviour of the random variable BDCE F G
B F G L E J by the POOSL
a (conditional) reward function that is implicitlyCE defined
model indicating the condition when the variable which stores the cumulative
O result for N is updated, see also [11].
With the limiting behaviour of a random variable, we mean almost
sure convergence [7].
3
A > (1)
Based on the ergodic theorem for Markov chains [2],
[14], the exact value for can be computed analytically
[17], [11]. To perform such computation, it is necessary
to generate the Markov chain explicitly and determine the
equilibrium distribution. Because of the large number of
states, such exhaustive analysis can often not be completed
within reasonable time for real-life industrial systems. Performance evaluation of such systems is therefore based on
estimating using simulation.
With simulation, a finite trace
of
the Markov chain is generated. Now, can be estimated
by the point-estimator
A
A
"
A
> (2)
As a result of the simulation, a point-estimation A
4 > is obtained. To analyse the accuracy of this
performance result, the technique of confidence intervals
can be applied. In this paper, we define a confidence interval as a stochastic interval for which the prob is larger than or equal to a cerability that
tain confidence level . Before a simulation run is started,
the desired confidence level of can be specified for every performance metric. Based on the realisation (which we also call a
of the confidence interval confidence interval) that is generated during simulation,
it is possible to check the accuracy of the estimated performance result. This accuracy is expressed as an upper
bound for the relative error of the estimated mean with
respect to the exact performance result , given by
/ 2
A*/ 2
/ 2
/ 2
A
A
O ! O! A A ! O A
1 66 if 1
if
"
"
(3)
otherwise
A. Confidence Intervals for i.i.d. Rewards
> > > A
4 > According to the central limiting theorem [2], it can be
stated that, in case the rewards
are assumed to be i.i.d. random variables with mean and
is
variance , the random variable A
A /
2
bility that is in the stochastic interval ()$ " *%)$ " is close to . We now know that an interval realisation
+#$ " ,%-$ " will contain with a probability close
is deterto . As a result, the confidence interval $
$
mined by ." and /%#" .
To calculate the confidence interval in this way, the variance for the rewards
must be
known. The variance , given by
A
/A
A
2
A
/ 2
A > > > > A is in general not known. With simulation,
mated by the point-estimator
> A can be esti-
(4)
By the Slutsky-theorem [7], the random variable
converges again to a standard normal distribution.
A second application of the Slutksy-theorem enables replacing the in formula 4 by the point-estimator of for can be calmula 2. Hence, the confidence interval culated during simulation.
For Markov chains, the assumption that the rewards
are i.i.d. random variables can
not be justified. In fact, the sequence of obtained rewards
depends on the sequence of visited states, which depends
on the transition probabilities .
A
/ 2
> > > "
B. Regenerative Cycles
When the relative error becomes sufficiently small during simulation, e.g., by checking that it goes below a certain limit, the simulation can be terminated automatically.
A
O
asymptotically normally distributed with mean and
variance . Consequently, the random variable con
verges to a standard normal distribution [3]. Now, a constant can be
determined such that the probability that
is close to . Rewriting this expres &%'$ " . Hence, the probasion yields !#$ " As discussed in section II, any infinite trace of an ergodic Markov chain visits all positive states infinitely
many times with probability 1. Assuming that the finite trace generated during simulation incorporates visits
to some positive state , a more general technique for
calculating the confidence interval can be developed. To this purpose, we introduce some additional random variables.
For
, let the random variables
denote the
0
1
length of the
cycle through
and let the random variables 2
indicate the cumulative reward that is obtained
0 1 cycle through according to reward function
in the
%
/ 2
:=
$
&%
&%
:=
# $
4
C
u
m
u
R
S
l a t i v
e w
a r d
e
S
s
C
L
y
e n
g
c l e
t h
s
( 1
Y
s
S
)
S
r
L
( 1
S
( 2
Y
r
)
)
r
( 2
S
( 3
Y
r
S
L
r
S
S
)
S
r
)
L
( 3
r
S
>.
)
r
and ( 4
S
)
r
.
:=
# $
are i.i.d., see [2].
The expected length of a cycle through , which can
be estimated by the point-estimator
, is equal
. On the other hand, the expected cumuto
lative reward per cycle through , which can be estimated
2 , equals 2
by the point-estimator
[2], where is defined according to formula 1. The latter
result can be understood by realising that 2
is equal to
times the expected performance result .
Now, we additionally define for
, the rewards
2 . Because the random variables
'( # $ < $
4 &# %
'( $ < $
A
$
#( $
A
:=
:=
:=
:=
A
#
#
$ := $
$
$
: =
and 2 $ are both i.i.d., the rewards
$ are i.i.d. as well.
: =
The mean of the rewards $ is equal to < $ 8A < $ 1 ,
whereas the variance is denoted with .
By applying the central limiting theorem, the ran is asymptotically normally disdom variable
tributed with mean 0 and variance . Hence, the ran
4
! "! < H $ J
dom variable O
converges to a standard nor
mal distribution. Similar to the previous section, a constant can be determined such that the probability that
! <H : : " ! $ J : is close to . By substitut =
=
=
ing $ 2 $ A # $ , we obtain (after rewriting) that
! "! < H $ J
A the probability that ! ; <H $ ! " ; <H 6
!
"
!
J
J
$
$
! ! 6! < H H $ J % $ ! " H is close to . As a result, A
"! ; < $ J
6! ; < $ J
can be estimated by the new point-estimator
4 2 : =
4 # : $ =
$
<H $ J A
variance
, defined as
:=
:=
2 $ A # $ (5)
is however not known. With simulation,
mated by the point-estimator
2
:=
$
4 & % : =
$
:=
$
2
A
! "! ; <H $ J
2
probability . Hence, the confidence interval / is determined by &
A $ ! " 6! ; <H $ J and A/% $ ! " 6! ; <H $ J .
interval in this way,
In order to calculate the confidence
: =
the variance for the rewards $ must be known. The
! "! ;
Figure 1 illustrates these definitions with respect to a
stochastic trace that starts in state .
Since the future probabilistic behaviour of a Markov
chain only depends on the present state, the behaviour exhibited when starting from a recurrent state is independent
of the behaviour that was exhibited when starting from that
state at some previous time-epoch (the behaviour is ’regenerated’). Consequently, the random variables
are i.i.d.
:=
and also the random variables 2 $
A
With simulation, a new point-estimation is obtained
when applying 5. Consequently, the concrete interval
$ " % $ " will contain with
/A
L
r
S
r
)
Fig. 1. Definition of the random variables
( 4
Y
r
:=
:=
$ A # $
can be esti-
(6)
A
in which can again be replaced by the point-estimator of
formula 5 based on the Slutsky-theorem.
To apply the point-estimators of formula 5 and 6, it is
required that 2
and
are random variables
for
which the second moment exists. Furthermore, must hold for the central limiting theorem to apply.
The application of formulas 5 and 6 is often referred to
as the technique of regenerative cycles. A practical problem of applying this technique during simulation is however the automatic recognition of visiting the positive state
for the first time. It is furthermore expensive to automatically check whether
is visited again after each state
transition. To avoid these problems, a constant length for
the cycle of revisiting state
is often used. This is called
the batch-means technique, which is therefore a special
case of the technique of regenerative cycles.
:= $ #
:=
$
%
:=
$
1
%
&%
IV. A N ALGEBRA
OF
C ONFIDENCE I NTERVALS
In order to investigate the accuracy of performance metrics other then long-run sample averages, an algebra of
confidence intervals is introduced. To this purpose, we
consider random variables that assume values of the ex ,
tended real numbers , also denoted with
with the extended rules of arithmetic from [7]. Now, the
definition of confidence intervals is as follows.
Definition 1: Let
and let . The
stochastic interval with and is called a confidence interval for if . As indicated in section III, is named a
confidence level.
?
?
A.*?
/ 2
/ 2 ?
* A
2
*
A *
5
On the set of confidence intervals defined by definition
1, we define the operations addition, negation, substraction, square, multiplication, reciprocal and division.
/ 2
2
A 2 / / 2
A 2
/ 2
/
/ confidence intervals as follows. Let be a confidence interval for and let
be a confidence inter confidence interval
val for . Then
is a %
for ,
. The confidence interval
is obtained by
with the negation of
adding .
A. Addition
Before defining the addition operation on confidence intervals, we first prove the following lemma.
Lemma 1: Let
be two properties. Then % .
Proof: Let and denote the complements of and
respectively . Then,
( ( (
( %
% A
/
( A%3
*
A
2
/ % ( %
( A
( A * / % 2 A % %
/
2 3* / 2 The negation operation on confidence intervals is defined by the following theorem.
Theorem 2: Let be a confidence interval for
. Then is a confidence interval for .
Proof: The proof is similar to the proof of theorem 1.
/
/2 2
A
C. Substraction
Based on the negation and addition operations on confidence intervals, we define the substraction operation on
1
11
and 1
Proof: The proof is similar to the proof of theorem 1
and involves distinguishing three cases.
E. Multiplication
The multiplication operation on confidence intervals is
defined by the following theorem.
be a confidence interval for
Theorem 4: Let and let
be a confidence interval for . Then
is a % confidence interval for
, where
/ 2
2
A
/ 2
/ 1 and if 1 if 1 and if 1 if if 1 1 and if 1 A if if if 1
1 and 1
1
1 and 1
1 1 1 and 1
1
B. Negation
A
/ % % 2
2 3* / 2 ( A * / % (
% 3*
By applying lemma 1, this yields
( A%
if if if A
/ 2
2
/ 1 /
A
Taking the square of a confidence interval is defined by
the following theorem.
be a confidence interval for
Theorem 3: Let . Then
is a confidence interval for , where
Now, the addition operation on confidence intervals is
defined by the following theorem.
be a confidence interval for
Theorem 1: Let and let
be a confidence interval for . Then
%
%
is a % confidence interval for
/% .
Proof: For the probability that %
is in the interval
%
%
we obtain
A
D. Square
% ( 2
/
/ 2 2 1
1 and 1
1
1 and 1
and
Proof: The proof involves distinguishing nine cases
and is similar to the proof of theorem 1.
F. Reciprocal
Taking the reciprocal of a confidence interval is defined
by the following theorem.
Theorem 5: Let be a confidence interval for
. Then
is a confidence interval for ,
A 1
/ 2 2
/ 6
if if if if if if 1
1
1
11
1
> A
and 1
and and and 1
1
1
A
G. Division
Based on the reciprocal and multiplication operations on
confidence intervals, we define the division operation on
be a conconfidence intervals as follows. Let fidence interval for and let
be a confidence
confidence
interval for . Then
is a %
interval for . The confidence interval
is obtained
with the reciprocal of
.
by multiplying 1
/ 2
/ / 2 /2 2
/
2
/ 2
V. C OMPLEX P ERFORMANCE M ETRICS
Complex performance metrics consist of a combination
of several long-run averages. Estimating such a complex
performance metric can be performed by adding, subtracting, multiplying and dividing the results that are obtained
by separately estimating the involved long-run averages.
Furthermore, a confidence interval can be derived for the
complex performance metric based on the algebra of confidence intervals. With that, the accuracy of the result for
the complex performance metric can be analysed as well.
A. Long-Run Sample Variances
Next to the long-run sample average of rewards, performance metrics may be expressed as a long-run sample variance of rewards. An example is the variance in
the latency or jitter of transferring a packet through a system. For a reward function and an ergodic Markov chain
, the long-run sample variance of rewards is
defined as
( ( Notice that the left term of this expression is a pointestimator for the long-run sample average of the square of
reward . As such, a confidence interval for this term can
be determined by applying the technique of regenerative
cycles. When a confidence interval for is available, a
confidence interval for the combined result can be derived
by applying the square and substraction operations on confidence intervals defined in section IV. With the resulting
confidence interval, the accuracy of the estimated long-run
sample variance can be analysed.
>
Proof: The proof is similar to the proof of theorem 1
and involves distinguishing six cases.
A
>
> A B. Long-Run Time Averages
Often performance metrics are defined as a long-run
time average of rewards. An example of such a performance metric is the time average occupancy level of a
queue. For a reward function and an ergodic Markov
chain
, the long-run time average of rewards
is defined as
>
6 > (7)
> 4 > 4 By multiplying both the numerator and denominator
with , we see that this quotient consists of two pointestimators for long-run sample averages. In case a point-
r ( X
i 1
)
r ( X
A
> A 4 > 4 where
represent the time difference between the time
of occurrence of
and of
as illustrated in
figure 2. With simulation, can be estimated by the pointestimator
r ( X
where is defined according to formula 1. With simulation, can be estimated using the point-estimator
a r d
O !
O
!
O!
e w
Rewriting this expression yields
R
where
D
t
i - 1
D
t
i
i
i + 1
)
r ( X
)
D
t
i +
i +
2
)
T
1
Fig. 2. Definition of the time difference
.
i m
e
7
estimation is available for both the numerator and the denominator, the accuracy of the estimated long-run time average can be determined by applying the division operation
on confidence intervals defined in section IV.
C. Long-Run Time Variances
Performance metrics may also be expressed as a longrun time variance of rewards. An example is the time variance in the occupancy level of a queue. For a reward function and an ergodic Markov chain
, the
long-run time variance of rewards is defined as
( >
4 > 4 where is defined by formula 7 and
according to figure 2. With simulation, can be estimated by the pointestimator
4 > 4 If both the nominator and the denominator are multiplied
by , this expression can be rewritten to
4 4 > Notice that the left term of this expression is a pointestimator for the long-run time average of the square of
reward . If a confidence interval for is available, a confidence interval for the combined result can be derived based
on the square and substraction operations defined in section IV. As such, the accuracy of the estimated long-run
time variance can be analysed.
>
VI. POOSL C LASSES
FOR
ACCURACY A NALYSIS
To ease the accuracy analysis of long-run sample and
time averages as well as long-run sample and time variances with POOSL, we introduced several library classes
for POOSL. First, a class for representing the extended real
numbers is defined. This class additionally enables to perform calculations with extended real numbers according to
the extended rules of arithmetic from [7].
Using the class of extended real numbers, a class for
representing confidence intervals is introduced. In addition, this class enables to apply the operations on confidence intervals defined in section IV. Moreover, a method
is included for testing the accuracy of the point-estimation
that is generated during simulation. This is performed in
accordance with formula 3. Based on initialising an instance of the class with a desired confidence level and a
desired upper bound for the relative error, a simulation can
be terminated automatically when the realised relative error goes below the upper bound.
To perform accuracy analysis of the different types of
long-run average performance metrics discussed in this
paper, classes for observing these performance metrics
are introduced. For validation purposes, these classes
also incorporate a method for logging intermediate pointestimations and the accompanying confidence intervals.
For observing long-run sample averages, a class is introduced which implements the technique of regenerative
cycles based on the point-estimator of formula 5. After
every regenerative cycle, an instance of the confidence interval class is generated based on the result obtained with
the point-estimator of formula 6. The classes for observing
long-run time averages and long-run sample and time variances apply the algebra of confidence intervals of section
IV for the combination of long-run averages according to
their respective formulas in section V.
Because of using the technique of regenerative cycles,
the user is required to explicitly indicate the so-called recurrence condition of starting a new cycle. As such, the
user is able to analyse the performance metric on a fixed
batch-length
(e.g., by using the condition that after
0 1 base
every
obtained reward for a queue occupancy level,
a new cycle starts) or on a variable cycle-length base (e.g.,
by using the condition that the inspected queue is empty).
Explicitly indicating the recurrence condition makes it unnecessary to actually generate the Markov chain defined by
a POOSL model. This is because automatically recognising the first visit to that state and automatically checking
whether this state is visited again after each transition can
now be based on the recurrence condition.
111
VII. C ONCLUSIONS
In this paper, we presented how the properties of the
Markov chains defined by POOSL models enable to evaluate long-run average performance metrics. Although these
Markov chains allow computing such performance metrics
analytically, performance evaluation of real-life industrial
systems is often based on simulation.
To analyse the accuracy of simulation results for longrun sample average performance metrics, a technique of
regenerative cycles is presented. This technique enables to
generate confidence intervals if the rewards for a long-run
sample average are not independent and identically distributed, which is the case for reward structures based on
Markov chains. With the resulting confidence intervals, a
simulation can be automatically terminated when the simulation results are sufficiently accurate.
Next to long-run sample averages, performance metrics
may be expressed as a long-run time average or as a long-
8
run sample or time variance. These performance metrics
are defined as a combination of long-run sample averages.
To determine confidence intervals for simulation results of
such complex long-run average performance metrics, an
algebra of confidence intervals is defined. Consequently,
the accuracy of simulation results for these performance
metrics can be analysed as well.
To ease accuracy analysis with POOSL, library classes
are introduced that implement the results of this paper.
These classes allow to observe performance results during
simulation and automatically terminate a simulation when
accurate simulation results are obtained.
R EFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
L.J. van Bokhoven, J.P.M. Voeten and M.C.W. Geilen. Software Synthesis for System Level Design Using Process Execution
Trees. In: B. Werner, ed., Proceedings of EUROMICRO’99, pp.
463–467. Los Alamitos, California (U.S.A.): IEEE, 1999.
K.L. Chung. Markov Chains with Stationary Transition Probabilities. Second Edition. Berlin (Germany): Springer-Verlag, 1967.
J.L. Devore. Probability and Statistics for Engineering and the
Sciences. Monterey, California (U.S.A.): Brooks, 1987.
M.C.W. Geilen. Real-Time Concepts for Software/Hardware Engineering. Graduation Report. Eindhoven University of Technology, Eindhoven (Netherlands), 1996.
M.C.W. Geilen and J.P.M. Voeten. Real-time Concepts for a Formal Specification Language for Software/Hardware Systems. In:
J.P. Veen, Ed., Proceedings of ProRISC’97, pp. 185–192. Utrecht
(Netherlands): STW, Technology Foundation, 1997.
M.C.W. Geilen and J.P.M. Voeten. Object-Oriented Modelling
and Specification using SHE. In: R.C. Backhouse and J.C.M.
Baeten, Eds., Proceedings of VFM’99, pp. 16–24. Eindhoven
(The Netherlands): Eindhoven University of Technology, 1999.
A.F. Karr. Probability. New York (U.S.A.): Springer-Verlag, 1993
R. Milner. Communication and Concurrency. Englewood Cliffs,
New Yersey (U.S.A.): Prentice-Hall, 1989.
X. Nicollin and J. Sifakis. An Overview and Synthesis on Timed
Process Algebras. In: K. Larsen and A. Skou, Eds., Proceedings of CAV’91, pp. 376–398. Berlin (Germany): Springer-Verlag,
1991.
www.ics.ele.tue.nl/ lvbokhov/poosl
Y. Pribadi, J.P.M. Voeten and B.D. Theelen. Reducing Markov
Chains for Performance Evaluation. In: Proceedings of
PROGRESS’01. Utrecht (The Netherlands): STW Technology
Foundation, 2001.
P.H.A. van der Putten and J.P.M. Voeten. Specification of Reactive
Hardware/Software Systems. Ph.D. thesis. Eindhoven (Netherlands): Eindhoven University of Technology, 1997.
B.D. Theelen, J.P.M. Voeten, L.J. van Bokhoven, G.G. de Jong,
A.M.M. Niemegeers, P.H.A. van der Putten, M.P.J. Stevens,
J.C.M. Baeten. System-level Modeling and Performance Analysis. In: J.P. Veen, Ed., Proceedings of PROGRESS’00, pp. 141147. Utrecht (The Netherlands): STW Technology Foundation,
2000.
H.C. Tijms. Stochastic Models; An Algorithmic Approach. Chichester (England): John Wiley & Sons, 1994.
M. Vardi. Automatic Verification of Probabilistic Concurrent
Finite-State Programs. In: Proceedings of FOCS’85, pp. 327–
338. IEEE, 1985.
[16] J.P.M. Voeten, P.H.A. van der Putten, M.C.W. Geilen and M.P.J.
Stevens. Towards System Level Performance Modelling. In: J.P.
Veen, Ed., Proceedings of ProRISC’98, pp 593-597. Utrecht (The
Netherlands): STW Technology Foundation, 1998.
[17] J.P.M. Voeten. Performance Evaluation with Temporal Rewards.
To be published in: Journal of Performance Evaluation.