Extension of PRISM by Synthesis of
Optimal Timeouts in Fixed-delay CTMC
L’uboš Korenčiak, Vojtěch Řehák, and Adrian Farmadin
Faculty of Informatics, Masaryk University, Brno, Czech Republic
{korenciak, rehak, xfarmad}@fi.muni.cz
Abstract. We present a practically appealing extension of the probabilistic model
checker PRISM rendering it to handle fixed-delay continuous-time Markov chains
(fdCTMCs) with rewards, the equivalent formalism to the deterministic and stochastic Petri nets (DSPNs). fdCTMCs allow transitions with fixed-delays (or timeouts)
on top of the traditional transitions with exponential rates. Our extension supports
an evaluation of expected reward until reaching a given set of target states. The main
contribution is that, considering the fixed-delays as parameters, we implemented a
synthesis algorithm that computes the epsilon-optimal values of the fixed-delays
minimizing the expected reward. We provide a performance evaluation of the
synthesis on practical examples.
1
Introduction
PRISM is an efficient tool for probabilistic model-checking of stochastic systems such
as Markov decision processes (MDPs), discrete-time Markov chains (DTMCs), or
continuous-time Markov chains (CTMCs). The PRISM community frequently raises
requests to incorporate the possibility to express delays with deterministic durations in
a CTMC1 . The standard PRISM recommendation is to approximate the deterministic
durations using a phase-type technique [10] and thus obtaining a CTMC. This works for
some models, however there are models for which such approximation can cause either a
large error or a state space explosion (see, e.g. [7,2]). However, there is a formalism called
fixed-delay CTMCs (fdCTMCs) [4,7,1] that is the requested extension of CTMCs by
fixed-delay (fd) events, modeling the deterministic transitions or timeouts. Recent result
[1] came up with new synthesis algorithms working directly on fdCTMCs (rather than
approximating them with CTMCs). Here we provide the first attempt to experimental
evaluation of such synthesis algorithms and show that they are practically applicable. In
the following running example we demonstrate the fdCTMC semantics as well as the
parameters and objectives of the synthesis.
Example 1. The figure bellow depicts fdCTMC of a slightly modified model of dynamic
power management of a Fujitsu disk drive taken from the PRISM case studies2 [12]. The
disk has three modes idle, busy, and sleep. In the idle and sleep modes the disk receives
1
2
http://www.prismmodelchecker.org/manual/FrequentlyAskedQuestions/
PRISMModelling#det_delay
http://www.prismmodelchecker.org/casestudies/power_ctmc3.php
requests, in the busy mode it also serves them. The disk is equipped with a bounded
buffer, where it stores requests when they arrive. The requests arrive with an exponential
inter-arrival time of rate 1.39 and increase the current size of the buffer. The requests are
served in an exponential time of/with rate 12.5, what decreases the buffer size. Note that
restricting the model to the idle and busy modes only, we obtain a CTMC model of an
M/M/1/n queue.
Moreover, the disk can move from the idle mode to the sleep mode where it saves
energy. Switching of the disk to the sleep mode is driven by timeout. This is modeled by
an fd event f1 moving the state from (idle, 0) to (sleep, 0) when the disk is steadily idle
for a specified amount of time (e.g. 1 second). The disk is woken up by another timeout
modeled by an fd event f2 , which is active in all sleep states. After staying in the sleep
mode for, e.g. 2 seconds, f2 changes the state according to the dashed arrows.
1.39
1.39
busy, 1
idle, 0
12.5
f1
f2
sleep, 0
sleep, 1
1.39
···
12.5
12.5
f2
1.39
1.39
busy, 2
12.5
f2
1.39
sleep, 2
1.39
busy, n
f2
1.39
···
1.39
1.39
sleep, n
Additionally, every state is given a rate cost that specifies an amount of energy consumed per each second spent there. Optionally, an impulse cost can be specified, e.g., say
that the change from (idle, 0) to (sleep, 0) consumes 0.006 energy units instantaneously.
Now, one might be interested in how much energy in average is consumed before emptying the buffer, i.e. to compute the expected energy consumed until reaching target that is
a new successor of (busy, 1) instead of the initial state (idle, 0). But, being a developer
of the disk, can we set better timeouts for f1 and f2 ? Hence, we consider timeouts as
parameters and synthesize them in order to minimize the expected amount of consumed
energy.
Our contribution is as follows. 1. We provide an extension of the PRISM language and
of the internal data structures to support specification of fdCTMC with impulse and
rate costs (or equivalently rewards). Hence, our version of PRISM is now ready for
other experiments with fdCTMC algorithms including the possibility to support modelchecking options as for CTMCs and DTMCs. 2. We added an evaluation of expected
reward until reaching a given set of target states. 3. We analyzed the synthesis algorithm
from [1], derived exact formulas and implemented the algorithm. 4. Additionally, we
accelerated the implementation by few structural changes, that significantly improved the
running time and the space requirements of the synthesis implementation. 5. We provide
a performance evaluation proving that current implementation is practically applicable to
a complex model from the PRISM case-study.
Related work There are many papers that contain models with fd events suitable for
synthesis such as deterministic durations in train control systems [14], time of server
rejuvenation [3], timeouts in power management systems [12], etc. Some of the models
already contain specified impulse or rate costs.
In [13] authors compute the optimal value of webserver timeout using impulse and
rate costs. The implementation can dynamically change the optimal value of timeout
based on the current inter-arrival times of requests. It works on the exact fdCTMC model
and cannot be easily applied to the more general fdCTMC models our implementation
can handle.
The formalism of deterministic and stochastic Petri nets (DSPNs) is equivalent to
fdCTMCs. DSPNs have been extensively studied and many useful results are directly
applicable to fdCTMCs. To the best of our knowledge the synthesis of fd events has not
been studied for DSPNs. The most useful tools for DSPNs are ORIS [6] and TimeNET
[15].
There was also an option to implement the synthesis algorithm as an extension of
ORIS. However, PRISM is much more used in practice and contains solution methods
for MDPs, that we needed for our implementation. Thus, we decided to implement the
synthesis into PRISM, even thought we had to extend the PRISM language and data
structures. Therefore, the ORIS and TimeNET algorithms can be now reimplemented for
fdCTMCs in PRISM easily, exploiting its efficient symbolic structures and algorithms
for CTMCs or MDPs.
In the rest of the paper we first formally define the fdCTMC and explain the extension
of PRISM language. Then we discuss the implemented algorithms and the performance
results.
2
Preliminaries
We use N0 , R≥0 , and R>0 to denote the set of all non-negative integers, non-negative
real numbers, and positive real numbers, respectively. Furthermore, for a countable set
A, we denote by D(A) the set of discrete probability distributions over A, i.e. functions
P
µ : A → R≥0 such that a∈A µ(a) = 1.
Definition 1. A fixed-delay CTMC (fdCTMC) C is a tuple (S , Q, F, A, N, d, sin ) where
–
–
–
–
–
S is a finite set of states,
Q : S × S → R≥0 is a rate matrix,
F is a finite set of fixed-delay (fd) events,
A : S → 2F assigns to each state s a set of active fd events in s,
N : S × F → D(S ) is the successor function, i.e. assigns a probability distribution
specifying the successor state to each state and fd event that is active there,
– d : F → R>0 is a delay vector that assigns a positive delay to each fd event,
– sin ∈ S is an initial state.
Note that fdCTMC C with empty set of fd events is a CTMC. The fdCTMC formalism
can be understood as a stochastic event-driven system, i.e. the amount of time spent in
each state and the probability of moving to the next state is driven by the occurrence of
events. In addition to the fd events of F, there is an exponential event E that is active in
P
all states s where s0 ∈S Q(s, s0 ) > 0. During an execution of an fdCTMC all active events
keep one timer, that holds the remaining time until the event occurs. The execution starts
in the state sin . The timer of each fd event f in A(sin ) is set to d( f ). The timer of the
exponential event is set randomly according to the exponential distribution with a rate
P
0
3
s0 ∈S Q(sin , s ). The event e with least timer value t occurs and causes change of state.
In case e is an fd event, the next state is chosen randomly according to the distribution
N(sin , e), otherwise e is an exponential event and the probability of choosing s as a next
P
state is Q(sin , s)/ s0 ∈S Q(sin , s0 ). In the next state s, the timers of all newly active fd
events (i.e. A(s) \ A(sin )), the occurred event e, and the exponential event are set in the
same way as above. Observe that the timers of the remaining active fd events decreased
by time t spent in the previous state. The execution then proceeds in the same manner.
We illustrate the definition on the fdCTMC model from Example 1. The execution
starts in (idle, 0). The events f1 and E are active and their timers are set to 1 and e.g.
1.18, respectively. Hence, after 1 second f1 occurs and changes the state to (sleep, 0)
with probability 1. The timers of newly active event f2 and E are set to 2 and e.g. 1.5,
respectively. Now, E occurs and changes the state to (sleep, 1). Here f2 is still active
and thus its timer holds the original value subtracted by the time spent in (sleep, 0), i.e.
2 − 1.5 = 0.5. The timer of the exponential event is set, etc.
A run of the fdCTMC is an infinite sequence (s0 , e0 , t0 )(s1 , e1 , t1 ) . . . where s0 = sin
and for each i ∈ N0 it holds that si ∈ S is the i-th visited state, ei ∈ {E} ∪ F is the event
that occurred in si , and ti ∈ R≥0 is the time spent in si . For the formal definition of the
semantics of fdCTMC and the probability space on runs see [8].
Total reward before reaching a target To allow formalization of performance properties
we enrich the model in a standard way with rewards or costs (see, e.g. [11]). For an
fdCTMC C with a state space S we additionally define a set of target states T , reward rates
R, and impulse rewards I. Formally, the target state T is a subset of S \ sin , R : S → R≥0
assigns a reward rate to every state, and I : S × ({E} ∪ F) × S → R≥0 assigns an impulse
reward to every change of state. Now the reward assigned to a run (s0 , e0 , t0 )(s1 , e1 , t1 ) . . .
P
is the reward accumulated before reaching a state of T , i.e. n−1
i=0 (ti · R(si ) + I(si , ei , si+1 ))
where n > 0 is the minimal index such that sn ∈ T . We set the reward to infinity whenever
there is no such n. The reward of a run can be viewed as a random variable, say CostC,T,R,I .
By EC,T,R,I (or simply EC ) we denote the expected value of CostC,T,R,I .
Synthesis Given a delay vector d0 , let (parametric) fdCTMC C(d0 ) be the fdCTMC C
where the delay vector is changed to d0 . Our aim is to find a delay vector d such that the
expected reward EC(d) is minimal. Formally, given an error bound ε > 0 the synthesis
algorithm computes delay vector d, such that EC(d) ≤ Val [C] + ε, where Val [C] denotes
the optimal reward inf d0 EC(d0 ) .
3
PRISM Language and User Interface Extension
Each fdCTMC model file must begin with the keyword fdctmc. For the purpose of our
synthesis and expected reward implementation, the set of target states has to be specified
by label "target", e.g.
label "target" = s=2;
3
For the sake of simplicity, when multiple events X = {e1 , . . . , en } occur simultaneously, the
successor is determined by the minimal element of X according to some fixed total order on F.
The exponential event (the matrix Q) is specified the same way as in CTMC models of
PRISM. The fd events are local to a module and must be declared immediately after the
module name. E.g. the fdelay f = 1.0 defines the fd event f with delay of a double
value 1.0. For an fd event f we specify its set of active states (i.e. A−1 ( f )) and transition
kernel (i.e. N(·, f )) by PRISM commands where the identifier f is in the arrow. E.g.
[L] s=1 --f-> 0.3:(s’=0) + 0.7:(s’=2)
specifies that the fd event f is active in all states where s=1 and whenever it occurs, the
next state is derived from the original one by changing variable s to 0 with probability
0.3 and to 2 with probability 0.7. The probabilities in each command have to sum to
one. Observe that fd event commands are similar to DTMC commands in PRISM. The
synchronization labels are used only to impose impulse rewards as for CTMC, e.g.
rewards
[L] true : 1.0;
endrewards
The rate rewards are specified the same way as for CTMC in PRISM. The PRISM
source code for the fdCTMC of Example 1 is in Appendix B. The implementation details
containing the fdCTMC structure are provided in Appendix B as well.
Users can run the implemented algorithms from both the graphical and the
command-line interfaces of PRISM. The expected reward and synthesis implementations are available in menu Model -> Compute -> Exp. reachability reward
and Model -> Compute -> FD synthesis, respectively or using the command-line
option -expreachreward and -fdsynthesis, respectively. The error bound ε is specified in Options -> Options -> Termination epsilon or in the command-line
option -epsilon.
4
Implementation Issues
Implementation of the expected reward computation was a straightforward application of
existing PRISM methods. For the synthesis we implemented the unbounded optimization
algorithm from [1]. The algorithm is based on discretization, i.e. we provide discretization
bounds and restrict the uncountable space of delay vectors into a finite space. Instead of
an exhaustive search through the finite space, we use the idea of [1] and transform the
parametric (discretized) fdCTMC into an MDP where actions correspond to the choices
of fd event delays. Now, the minimal solution of the MDP yields the optimal delay vector.
The discretization bounds consist of the discretization step δ, the upper bound on
fd event delay d and the precision κ for computation of action parameters. They are
computed for each fd event separately from the error bound ε, the number of states, the
minimal transition probability, and other fdCTMC model attributes. For more detail see
Appendix A. Note that in every fdCTMC model, the delays for all fd events have to
be specified. Applying these delays, we compute the corresponding expected reward
Val which is used as an upper bound for the optimal reward. Then Val is employed
when computing the discretization bounds. The lower the Val is, the faster the synthesis
implementation performs. Thus it is worth to think of good delays of fd events when
specifying the model.
Given the discretization bounds one has to compute the transition probabilities
and expected accumulated reward for each action in the MDP corresponding to the
discretized delay of fd event. This can be done using the transient analysis of subordinated
CTMCs [9].
Prototype implementation In the first implementation we used straightforward approach
to call built-in methods of PRISM to compute the required quantities for each discretized
fd event delay separately. This is reasonable since the built-in methods are correctly
and efficiently programmed for all PRISM engines and methods of computing transient
analysis. However, we experienced that most of the time was spent computing the
transient analysis rather than solving the created MDP, e.g. 520 seconds out of 540
seconds of total time4 . One of the reasons is that in each iteration a small portion
of memory is allocated and freed by built-in PRISM methods. Since there is a large
number of actions, the amount of reallocated memory was slowing down the computation.
Thus we decided to reimplement the computation of transient probabilities the applying
principles of dynamic programming.
Iterative computation of transient analysis The transient probabilities can be very
efficiently approximated up to an arbitrary small error using the uniformization technique.
The problem is that we have to compute the transient probabilities for each value of a very
large set {i · δ | i ∈ N0 and 0 < i ≤ d/δ} and allow only fixed error κ for each computation.
The transient probability vector π(δ) of a CTMC C at time δ can be computed using
uniformization by
J
X
(λ · δ) j −λ·δ
π(δ) =
1 sin · P j ·
·e ,
(1)
j!
j=0
where 1 sin is the initial vector of C, λ is the uniformization rate of C, and P is the transition
kernel of the uniformized C. The choice of number J influences the error of the formula.
It is easy to compute the value of J such that the error is sufficiently small.
However, for time i · δ we can use the previously computed transient probabilities as
π(i · δ) =
J
X
j=0
π((i − 1) · δ) · P j ·
(λ · δ) j −λ·δ
·e .
j!
(2)
It is again easy to compute J such that the overall allowed error is not exceeded. Instead
of performing naïve computation for each number in {i · δ | i ∈ N0 and 0 < i ≤ d/δ} with
according number of steps J1 , . . . , Jd/δ to cause error bounded by κ in each computation,
we compute the transient probabilities iteratively with sufficiently large J to cause small
error in all computations. For example, if we have δ = 0.1, d/δ = 1000, rate λ = 1.0
and κ = 0.01 using the naïve method we have to do J1 + · · · + Jd/δ = 66, 265 steps and
using the iterative method J · d/δ = 3, 000 steps. This is significant difference since a
vector matrix multiplication is performed in each step. Thus we hard-programmed the
iterative computation of transient probabilities and accumulated rewards in CTMC what
caused a dramatic speedup thanks to the smaller number of arithmetic operations and
better memory management.
4
Computed for the rejuv model and the error bound 0.001, see Section 5.
Precomputation Careful reader may have noticed that (2) can be further simplified to
π(i · δ) = π((i − 1) · δ) · e−λ·δ ·
J
X
Pj ·
j=0
(λ · δ) j
.
j!
(3)
P
Hence, the matrix e−λ·δ · Jj=0 P j · (λ · δ) j / j! can be easily precomputed beforehand and
used for computation of each π(i · δ) to increase the savings even more. However, this is
not true. J is small and the matrix P is sparse for the most reasonable models and error
P
bounds. But e−λ·δ · Jj=0 P j · (λ · δ) j / j! is not sparse for almost each error bound, P, and
λ, what is known as "fill-in" phenomenon. Thus using (2) is typically more efficient than
using (3). Similar observations were discussed in [5].
Implementing the synthesis algorithm of [1], we inherited the following restrictions
on the input fdCTMC models. There is at most one concurrently active fd event in each
state, i.e. ∀s ∈ S : |A(s)| ≤ 1. For each fd event there is at most one state where its timer
is set. Every state has a positive rate reward, i.e. ∀s ∈ S : R(s) > 0. Moreover, we add
that all fd events have positive impulse rewards, i.e. ∀ f ∈ F ∧ s, s0 ∈ S : N(s, f )(s0 ) >
0 =⇒ I(s, f, s0 ) > 0. For the expected reward implementation only the first two
restrictions are valid.
5
Experimental Results
We tested the performance of our synthesis implementation on the model from Example 1
for various sizes of the queue (2, 4, 6, and 8) and the rejuvenation model provided in
Appendix B. The considered error bounds are 0.005, 0.0025, 0.0016, 0.00125, and 0.001.
The following table shows the expected rewards and the computation times for a given
error bound. As the expected rewards are very similar for different error bounds, we show
their longest common prefix, instead of listing five similar long numbers.
Model
rejuv
dpm2
dpm4
dpm6
dpm8
ε : 0.005
1/ε : 200
5.87
58.22
156.02
259.76
616.47
CPU time [s]
0.0025
0.0016
400
600
12.09
14.71
121.15
195.61
354.35
509.19
532.47 2705.45
3142.44 6362.79
0.00125
800
21.60
234.58
2197.10
3026.77
22507.55
Longest
0.00100 common prefix
1000 of exp. rewards
23.84 0.94431314832
248.52 0.336634754
2652.05 0.337592724
5124.10 0.337583980
27406.62 0.337537611
It is worth mentioning that the longest computation (dpm8 for error 0.001) took only
1 hour and 30 minutes of real clock time thanks to the native parallelism of Java (the table
shows the sum for all threads). Note that the computed values of the expected reward
are of a much better precision than required. This indicates that there might even be
a space for improvements of the synthesis algorithm. Our experiments show that the
implementation retains the theoretical complexity bounds saying that the computation
time is exponential to the number of states and polynomial to 1/ε.
The computations were run on platform HP DL980 G7 with 8 64-bit processors Intel
Xeon X7560 2.26GHz (together 64 cores) and 448 GiB DDR3 RAM, but only 304GB
was provided to Java. The time was measured by the Linux command time.
6
Conclusions and Future Work
In this paper, we incorporated the fdCTMC models into PRISM and implemented
the expected reward computation and the synthesis algorithm. The tool is available
on http://www.fi.muni.cz/~xrehak/fdPRISM/. We have used the explicit state
PRISM engine. Based on the promising results, it is reasonable to (re)implement the
synthesis and other model checking algorithms for fdCTMCs in the more efficient PRISM
engines. Moreover, new effort can be put to reduce the number of current restrictions on
the fdCTMC models. For instance the method of stochastic state classes [6] implemented
in ORIS may be applied for computation of transient analysis instead of uniformization.
Acknowledgments We thank Vojtěch Forejt and David Parker for fruitful discussions.
References
1. T. Brázdil, L’. Korenčiak, J. Krčál, P. Novotný, and V. Řehák. Optimizing performance of
continuous-time stochastic systems using timeout synthesis. In QEST, volume 9259 of LNCS,
pages 141–159. Springer, 2015.
2. M. Fackrell. Fitting with matrix-exponential distributions. Stochastic models, 21(2-3):377–400,
2005.
3. R. German. Performance Analysis of Communication Systems with Non-Markovian Stochastic
Petri Nets. Wiley, 2000.
4. C. Guet, A. Gupta, T. Henzinger, M. Mateescu, and A. Sezgin. Delayed continuous-time
Markov chains for genetic regulatory circuits. In CAV, pages 294–309. Springer, 2012.
5. S. Haddad, L. Mokdad, and P. Moreaux. A new approach to the evaluation of non markovian
stochastic petri nets. In ICATPN, volume 4024 of LNCS, pages 221–240. Springer, 2006.
6. A. Horváth, M. Paolieri, L. Ridi, and E. Vicario. Transient analysis of non-markovian models
using stochastic state classes. Perform. Eval., 69(7-8):315–335, 2012.
7. L’. Korenčiak, J. Krčál, and V. Řehák. Dealing with zero density using piecewise phase-type
approximation. In EPEW, volume 8721 of LNCS, pages 119–134. Springer, 2014.
8. J. Krčál. Formal Analysis of Discrete-Event Systems with Hard RealTime Bounds. PhD thesis,
Faculty of Informatics, Masaryk University, Brno, 2014.
9. Ch. Lindemann. An improved numerical algorithm for calculating steady-state solutions of
deterministic and stochastic petri net models. Perform. Eval., 18(1):79–95, 1993.
10. M.F. Neuts. Matrix-Geometric Solutions in Stochastic Models: An Algorithmic Approach. The
Johns Hopkins University Press, Baltimore, 1981.
11. M.L. Puterman. Markov Decision Processes. Wiley, 1994.
12. Q. Qiu, Q. Wu, and Pedram M. Stochastic modeling of a power-managed system: construction
and optimization. In Proc. ISLPED, pages 194–199. ACM Press, 1999.
13. W. Xie, H. Sun, Y. Cao, and K. S. Trivedi. Optimal webserver session timeout settings for
web users. In Computer Measurement Group Conferenceries, pages 799–820, 2002.
14. A. Zimmermann. Applied restart estimation of general reward measures. In RESIM, pages
196–204, 2006.
15. A. Zimmermann. Modeling and evaluation of stochastic petri nets with timenet 4.1. In ICST,
pages 54–63. IEEE, 2012.
A
Discretization Bounds
Using the full version of [1] we derived the exact formulas of the discretization bounds
for each fd event:
d = max
δ=
κ=
n
Val
e · | ln(α/2)| o
;
,
λ · minP
minP|S f d | · minR
α
,
D1
ε · δ · minR
2 · |S 0 | · (1 + Val)
,
where
α = min
n
ε
;
Bound[#] · (1 + Val) · |S 0 |
D1 = max{2 · λ ; 1 · (λ + 1) · maxR},
o
1
,
0
2 · Bound[#] · |S |
– Bound[#] is an upper bound on expected number of steps to reach target from any
state in the created MDP, i.e.
Bound[#] =
–
–
–
–
B
Val
,
minimal expected one-step reward in the created MDP
Val is the upper bound on the expected reward,
S 0 is the state space of the created MDP,
λ is the uniformization rate, and
|S f d |, minP, maxR, and minR is the number of states, the minimal branching probability, the maximal reward, and the minimal reward in the subordinated CTMC for
the given fd event, respectively.
fdCTMC in PRISM Language
Extension of PRISM data structures The FDCTMCSimple class extends the CTMCSimple
class by a vector of objects of type FDEvent and few methods to work with them (the
methods are explained in the corresponding interface FDCTMC that is an extension of
interface CTMC). The FDEvent is basically an extension of DTMCSimple class by one
double attribute that keeps delay of the fd event and one String attribute that keeps
the label of the fd event. The transition kernel is kept in the inherited attributes from the
DTMCSimple class.
Fig. 1. The graphical user interface of PRISM with the source code of the rejuvenation model [3].
Fig. 2. The source code of the fdCTMC from Example 1 in the PRISM language.