Communications in Statistics - Stochastic Models, 16(5), 2000
OCCUPATION TIMES IN MARKOV PROCESSES
Bruno Sericola
IRISA - INRIA
Campus de Beaulieu
35042 Rennes Cedex, France
ABSTRACT
In a homogeneous finite-state Markov process, we consider the occupation
times, that is, the times spent by the process in given subsets of the state
space during a finite interval of time. We first derive the distribution of the
occupation time of one subset and then we generalize that result to the joint
distribution of occupation times of different subsets of the state space by the
use of order statistics from the uniform distribution. Next, we consider the
distribution of weighted sums of occupation times. We obtain forward and
backward equations describing the behavior of these weighted sums and we
show how these lead to simple expressions for that distribution.
1
INTRODUCTION
Let X = {Xu , u ≥ 0} be a homogeneous Markov process with finite state space
S. The occupation time of a subset U ⊂ S over [0, t) is defined as the random
variable
Z
Wt =
0
t
1{Xu ∈U } du,
where 1{c} = 1, if condition c holds and 0 otherwise. That random variable
has drawn much attention as it is also known as the interval availability in
reliability and dependability theory. In [2] an expression for the distribution
of Wt was obtained using uniform order statistics on [0, t). From a computational point of view that expression is very interesting; various methods were
developed to compute it even in the case of denumerable state spaces (see [2],
[10], [8], [9] and the references therein).
1
We first recall how the joint distribution of the pair (Wt , Xt ) was obtained
in [2] by using the forward and backward equations associated with the uniformized Markov chain of the process X. We then generalize that result to
the joint distribution of Wt1 , . . . , Wtm , Xt , where Wti is the occupation time of
a subset Bi over the interval [0, t). Finally, we consider a weighted sum of
occupation times, that is the random variable Yt defined by
Z
Yt =
0
t
ρ(Xu )du,
where for each i ∈ S, ρ(i) is a nonnegative constant. The quantity Yt arises in the performability analysis in reliability and dependability theory (see
[3], [5] and the references therein). Here we derive backward and forward
equations describing the behavior of the joint distribution of (Yt , Xt ). These
partial differential equations are then solved. We show that they lead to simple
expressions for the joint distribution of (Yt , Xt ).
The remainder of the paper is organized as follows. In the next section, we
consider the joint distribution of uniform order statistics and the joint conditional distribution of the jumps in a Poisson process and we recall how they
are related. In Section 3, we consider the case m = 1, to obtain the distribution of occupation time for a discrete time Markov chain. That distribution,
combined with the results of Section 2, leads to a simple expression for the
joint distribution of the pair (Wt , Xt ). In Section 4, the results of Section 3 are
generalized to the case m > 1. Finally, Section 5 deals with the distribution
of the pair (Yt , Xt ).
2
2 ORDER STATISTICS
2.1
The Uniform Distribution
We consider the order statistics of uniform random variables on [0, t). Formally,
let X1 , X2 , . . ., Xn be n i.i.d. uniform random variables on [0, t). If the random
variables X1 , X2 , . . ., Xn are rearranged in ascending order of magnitude and
written as
X(1) ≤ X(2) ≤ · · · ≤ X(n) ,
we call X(i) the ith order statistic, i = 1, 2, . . . , n.
Let Fr (x) be the distribution of X(r) . We know that (see for instance [1]),
for x ∈ (0, t),
Fr (x) = P{X(r) ≤ x} =
n
X
i=r
n ! x i x n−i
1−
.
t
t
i
(1)
In [1] it is shown that the joint density of X(l1 ) , X(l1 +l2 ) , . . . , X(l1 +l2 +···+lk ) is
given, for 1 ≤ k ≤ n, 1 ≤ l1 + l2 + · · · + lk ≤ n (li ≥ 1), and x1 ≤ x2 ≤ · · · ≤ xk
(xi ∈ (0, t)), by
gl1 ,l2 ,...,lk (x1 , x2 , . . . , xk ) =
k 1
n!
t
x1 l1 −1 x2 − x1 l2 −1
xk − xk−1 lk −1
xk n−(l1 +l2 ···+lk )
···
1−
t
t
t
t
.
(l1 − 1)!(l2 − 1)! · · · (lk − 1)!(n − (l1 + l2 · · · + lk ))!
Furthermore, if for fixed 1 ≤ k ≤ n and 1 ≤ l1 +l2 +· · ·+lk ≤ n (li ≥ 1), we
define Y1 = X(l1 ) and Yi = X(l1 +l2 +···+li ) − X(l1 +l2 +···+li−1 ) for i = 2, . . . , k, then
the joint density function of Y1 , Y2 , . . . , Yk is given, for 0 < s1 + s2 + · · ·+ sk < t
(si ∈ (0, t)), by
hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk ) = gl1 ,l2 ,...,lk (s1 , s1 + s2 , . . . , s1 + · · · + sk ),
that is,
3
hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk ) =
k 1
n!
t
s1
t
l1 −1 n−(l1 +l2 ···+lk )
s1 + s2 + · · · + sk
s2 l2 −1
sk lk −1
···
1−
t
t
t
(l1 − 1)!(l2 − 1)! · · · (lk − 1)!(n − (l1 + l2 · · · + lk ))!
.
In particular, for k = n, we get the joint density function of the spacings
Y1 = X(1) , Y2 = X(2) −X(1) , . . . , Yn = X(n) −X(n−1) , denoted by h(x1 , x2 , . . . , xn )
by setting l1 = l2 = · · · = ln = 1 in the preceding expression, that is

n!



h(x1 , x2 , . . . , xn ) =
tn


 0
if
n
X
i=1
xi ≤ t
otherwise.
By writing Yn+1 = t − X(n) , this also determines the (degenerate) joint density
of Y1 , Y2 , . . . , Yn , Yn+1 on the set
xi ≥ 0
(i = 1, . . . , n, n + 1)
n+1
X
i=1
xi = t.
The joint distribution Hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk ) of Yl1 , Yl2 , . . . , Ylk is given in
the following lemma.
Lemma 2.1 For 1 ≤ k ≤ n, 1 ≤ l1 + l2 + · · · + lk ≤ n (li ≥ 1), and 0 <
s1 + s2 + · · · + sk < t (si ∈ (0, t)), we have that
Hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk ) =
s1
X n! t
i1 s2
t
i2
i1 ≥ l1 , i2 ≥ l2 , . . . , ik ≥ lk ,
s1 + s2 + · · · + sk
sk ik
···
1−
t
t
i1 !i2 ! · · · ik !(n − (i1 + i2 · · · + ik ))!
n−(i1 +i2 ···+ik )
.
i1 + i2 + · · · + ik ≤ n
Proof. It suffices to show that
∂ k Hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk )
= hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk ).
∂s1 ∂s2 · · · ∂sk
To simplify notation, we define
t;s1 ,s2 ,...,sk
θn;i
=
1 ,i2 ,...,ik
s1
t
i1 s2
t
i2
sk ik
s1 + s2 + · · · + sk
1−
t
t
i1 !i2 ! · · · ik !(n − (i1 + i2 · · · + ik ))!
···
4
n−(i1 +i2 ···+ik )
.
We obtain
∂Hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk )
n!
=
∂sk
t
X
t;s1 ,s2 ,...,sk
θn;i
1 ,i2 ,...,ik−1 ,ik −1
i1 ≥ l1 , i2 ≥ l2 , . . . , ik ≥ lk ,
i1 + i2 + · · · + ik ≤ n
−
n!
t
X
t;s1 ,s2 ,...,sk
θn−1;i
1 ,i2 ,...,ik
i1 ≥ l1 , i2 ≥ l2 , . . . , ik ≥ lk ,
i1 + i2 + · · · + ik ≤ n − 1
=
n!
t
X
t;s1 ,s2 ,...,sk
θn−1;i
1 ,i2 ,...,ik
i1 ≥ l1 , i2 ≥ l2 , . . . , ik−1 ≥ lk−1 , ik ≥ lk − 1,
i1 + i2 + · · · + ik ≤ n − 1
−
n!
t
X
t;s1 ,s2 ,...,sk
θn−1;i
1 ,i2 ,...,ik
i1 ≥ l1 , i2 ≥ l2 , . . . , ik ≥ lk ,
i1 + i2 + · · · + ik ≤ n − 1
=
n!
t
X
t;s1 ,s2 ,...,sk
θn−1;i
,
1 ,i2 ,...,ik−1 ,lk −1
i1 ≥ l1 , i2 ≥ l2 , . . . , ik−1 ≥ lk−1 ,
i1 + i2 + · · · + ik−1 ≤ n − 1
where the second equality is obtained by the change of variable ik −→ ik +1. By
successively iterating the same argument with respect to variables sk−1 , . . . , s2 ,
we obtain
X t;s ,s ,...,s
n! n−k+1
∂ k−1 Hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk )
1 2
k
= k−1
θn−k+1;i
,
1 ,l2 −1,...,lk−1 −1,lk −1
∂sk ∂sk−1 · · · ∂s2
t
i1 =l1
and finally,
∂ k Hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk )
∂sk · · · ∂s2 ∂s1
=
X t;s ,s ,...,s
X t;s ,s ,...,s
n! n−k+1
n! n−k
1 2
k
θ
−
θ 1 2 k
tk i1 =l1 n−k+1;i1 −1,l2 −1,...,lk −1 tk i1 =l1 n−k;i1 ,l2 −1,...,lk −1
=
X
X t;s ,s ,...,s
n! n−k
n! n−k
t;s1 ,s2 ,...,sk
θ
−
θ 1 2 k
n−k;i1 ,l2 −1,...,lk −1
k
k
t i1 =l1 −1
t i1 =l1 n−k;i1 ,l2 −1,...,lk −1
n! t;s1 ,s2 ,...,sk
θ
tk n−k;l1 −1,l2 −1,...,lk −1
= hl1 ,l2 ,...,lk (s1 , s2 , . . . , sk ),
=
5
where the second equality is obtained by the change of variable i1 −→ i1 + 1.
2.2
The Poisson Process
Let {Nt , t ∈ R} be a Poisson process of rate λ and T0 , T0 + T1 , . . ., T0 + T1 +
· · ·+Tn−1 , be the first n instants of jumps of {Nt } in [0, t). It is well-known, see
[4], that the density of the conditional distribution of T0 , T1 , . . . , Tn−1 , given
{Nt = n} is


n!



f (x0 , x1 , . . . , xn−1 ) =
tn



 0
if
n−1
X
i=0
xi ≤ t
otherwise.
That is also, as seen in the previous subsection, the joint density of the order
statistics from the uniform distribution on (0, t).
If we write Tn = t − (T0 + T1 + · · · + Tn−1 ), this also determines the (degenerate) joint density function of T0 , T1 , . . . , Tn−1 , Tn on the set
xi ≥ 0
(i = 0, . . . , n − 1, n)
n
X
i=0
xi = t.
The symmetric role of the variables x0 , x1 , . . . , xn−1 , xn shows that the random
variables T0 , T1 , . . . , Tn−1 are exchangeable. It follows from relation (1) that,
for 1 ≤ l ≤ n, {i1 , · · · , il } ⊂ {0, 1, . . . , n}, and s ∈ (0, t), we have
P{Ti1 + · · · + Til ≤ s | Nt = n} = P{T0 + · · · + Tl−1 ≤ s | Nt = n}
= P{X(l) ≤ s}
 
n
n s k X
s n−k
 
=
1−
.
t
t
k
k=l
(2)
More generally, let k be an integer such that 1 ≤ k ≤ n and let l1 , l2 , . . . , lk
be integers such that 1 ≤ l1 + l2 + · · · + lk ≤ n (li ≥ 1). For any subset
{i1 , i2 , . . . , il1 +l2 +···+lk } of {0, 1, . . . , n − 1, n}, the vectors


l1
X
j=1
Tij ,
l1X
+l2
j=l1 +1
Tij , . . . ,
6
l1 +l2X
+···+lk
j=l1 +l2 +···+lk−1 +1

Tij 

and

lX
1 −1
j=0
Tj ,
l1 +l
2 −1
X
j=l1
Tj , . . . ,
l1 +l2 +···+l
X k −1
j=l1 +l2 +···+lk−1

Tj 
have the same conditional distribution given that Nt = n. By lemma 2.1, for
0 < s1 + s2 + · · · + sk < t (si ∈ (0, t)), we thus get
P{
l1
X
j=1
Tj ≤ s1 ,
l1X
+l2
j=l1 +1
s1
X n! t
Tj ≤ s2 , . . . ,
i1 i1 ≥ l1 , i2 ≥ l2 , . . . , ik ≥ lk ,
s2
t
i2
l1 +l2X
+···+lk
j=l1 +l2 +···+lk−1 +1
Tj ≤ sk | Nt = n} =
sk ik
s1 + s2 + · · · + sk
···
1−
t
t
i1 !i2 ! · · · ik !(n − (i1 + i2 · · · + ik ))!
n−(i1 +i2 ···+ik )
.
i1 + i2 + · · · + ik ≤ n
(3)
3 DISTRIBUTION OF OCCUPATION TIMES
Let X = {Xu , u ≥ 0} be a homogeneous Markov process with finite state
space S. The process X is characterized by its infinitesimal generator A and
its initial probability distribution α. We denote by Z = {Zn , n ≥ 0} the
uniformized Markov chain [7] associated to the Markov process X, with the
same initial distribution α. Its transition probability matrix P is related to
the matrix A by the relation P = I + A/λ, where I is the identity matrix
and λ satisfies λ ≥ max{−Ai,i , i ∈ S}. The rate λ is the rate of the Poisson
process {Nu , u ≥ 0}, independent of Z, that counts the number of transitions
of process {ZNu , u ≥ 0} over [0, t). It is well-known that the processes {ZNu }
and X are stochastically equivalent. We consider a partition S = U ∪ D,
U ∩ D = ∅, of the state space S and we study the occupation time in the
subset U.
7
3.1
The Discrete Time Case
For the Markov chain Z = {Zn , n ≥ 0}, the random variable Vn is the total
number of states of U visited during the n first transitions of Z, that is
n
X
Vn =
k=0
1{Zk ∈U } .
The following theorem gives the backward equations for the behavior of the pair
(Vn , Zn ). For every i ∈ S, the notation Pi denotes the conditional probability
given that Z0 = i, that is Pi {.} = P{. | Z0 = i}.
Theorem 3.1 For n ≥ 1, and 1 ≤ k ≤ n, we have that
for i ∈ U, Pi {Vn ≤ k, Zn = j} =
for i ∈ D, Pi {Vn ≤ k, Zn = j} =
X
l∈S
X
Pi,l Pl {Vn−1 ≤ k − 1, Zn−1 = j},
l∈S
Pi,l Pl {Vn−1 ≤ k, Zn−1 = j}.
Proof. By using the Markov property and the homogeneity of Z, we get that
Pi {Vn ≤ k, Zn = j} =
=
X
l∈S
X
l∈S
Pi,l Pi {Vn ≤ k, Zn = j | Z1 = l}
Pi,l Pl {Vn−1 ≤ k − 1{i∈U } , Zn−1 = j}.
The following theorem gives the forward equations for the probabilities
associated with the pair (Vn , Zn ).
Theorem 3.2 For n ≥ 1, and 1 ≤ k ≤ n, we have that
for j ∈ U, Pi {Vn ≤ k, Zn = j} =
for j ∈ D, Pi {Vn ≤ k, Zn = j} =
X
l∈S
X
l∈S
Pi {Vn−1 ≤ k − 1, Zn−1 = l}Pl,j ,
Pi {Vn−1 ≤ k, Zn−1 = l}Pl,j .
Proof. By the same arguments, we get that
P{Vn ≤ k, Zn = j, Z0 = i} = P{Vn−1 ≤ k − 1{j∈U }, Zn = j, Z0 = i}
=
X
l∈S
=
X
l∈S
=
X
l∈S
P{Vn−1 ≤ k − 1{j∈U }, Zn = j, Zn−1 = l, Z0 = i}
P{Vn−1 ≤ k − 1{j∈U }, Z0 = i | Zn−1 = l}P{Zn = j, Zn−1 = l}
P{Vn−1 ≤ k − 1{j∈U }, Zn−1 = l, Z0 = i}Pl,j .
8
We thus obtain the desired relation by conditioning on Z0 .
For n ≥ 0, and k ≥ 0, we introduce the matrix F (n, k) = {Fi,j (n, k)}
defined by
Fi,j (n, k) = P{Vn ≤ k, Zn = j | Z0 = i}.
The results of Theorems 3.1 and 3.2 can be easily expressed in matrix notation.
We decompose the matrices P and F (n, k) with respect to the partition {U, D}
as



PU PU D 
P =


PDU PD

FU (n, k) FU D (n, k) 
and F (n, k) = 

.
FDU (n, k) FD (n, k)
The result of Theorem 3.1 can now be written as
FU (n, k) FU D (n, k)

PU PU D
=
FDU (n, k) FD (n, k)
or
=
PDU PD

0
0

 F (n − 1, k − 1) + 

0

 FU (n, k) 


FDU (n, k)
 =
FD (n, k)
or
F (n − 1, k − 1) 

 =

F (n − 1, k) 
PDU 0

PD
0
0 (PD )
n
Note that for all k ≥ n + 1, F (n, k) = P n .
9


 , for n ≥ 0.

 0 PU D 
 + F (n − 1, k) 

 0


0 

F (n, 0) = 
PDU

 PU
F (n, k) = F (n − 1, k − 1) 

 PU D 

The initial conditions are

 F (n − 1, k).
 PU 

 FU D (n, k) 
0 
PDU PD
In the same way, Theorem 3.2 can be written as

F (n − 1, k)

 PU PU D 
F (n, k) = 
F (n − 1, k − 1)
0
PD
.
3.2
The Continuous Time Case
We consider the Markov process X = {Xt , t ≥ 0} and the occupation time Wt
of the subset U in [0, t), that is
Z
Wt =
0
t
1{Xu ∈U } du.
That random variable represents the time spent by the process X in the subset
U during the interval [0, t). The joint distribution of the pair (Wt , Xt ) is given
by the following theorem.
Theorem 3.3 For every i, j ∈ S, for t > 0, and s ∈ [0, t), we have that
P{Wt ≤ s, Xt = j | X0 = i}


n
n X
n s k s n−k
−λt (λt)


=
e
1−
P{Vn ≤ k, Zn = j | Z0 = i}. (4)
n! k=0 k
t
t
n=0
∞
X
Proof. For s < t, we have that
P{Wt ≤ s, Xt = j | X0 = i}
=
=
=
=
=
=
=
∞
X
n=0
∞
X
n=0
∞
X
n=0
∞
X
n=0
∞
X
n=0
∞
X
n=0
∞
X
n=0
Pi {Wt ≤ s, Nt = n, Xt = j}
Pi {Wt ≤ s, Nt = n, Zn = j} since {Xt } and {ZNt } are equivalent
Pi {Nt = n}Pi {Wt ≤ s, Zn = j | Nt = n}
P{Nt = n}Pi {Wt ≤ s, Zn = j | Nt = n}
P{Nt = n}
P{Nt = n}
P{Nt = n}
n+1
X
l=0
n+1
X
l=0
n
X
l=0
Pi {Wt ≤ s, Vn = l, Zn = j | Nt = n}
Pi {Vn = l, Zn = j}Pi {Wt ≤ s | Vn = l, Zn = j, Nt = n}
Pi {Vn = l, Zn = j}Pi {Wt ≤ s | Vn = l, Zn = j, Nt = n}.
The fourth and sixth equalities follow from the independence of the processes
{Zn } and {Nt } and the fact that X0 = Z0 . The last equality follows from the
10
fact that if l = n + 1, we trivially have that Vn = n + 1 and Nt = n imply that
Wt = t. We so get P{Wt ≤ s | Vn = n + 1, Zn = j, Nt = n} = 0, since s < t.
Let us consider now the expression Pi {Wt ≤ s | Vn = l, Zn = j, Nt = n}. For
fixed i, j ∈ S and 0 ≤ l ≤ n, we define the set
Gi,j
l,n =



zb = (i, z1 , . . . , zn−1 , j) ∈ S n+1


l entries of zb are in U and
n + 1 − l entries of zb are in D





and we denote by Zb the random vector (Z0 , . . . , Zn ). We then have
Pi {Wt ≤ s | Vn = l, Zn = j, Nt = n}
X
=
P{Wt ≤ s | Zb = zb, Vn = l, Nt = n}Pi {Zb = zb | Vn = l, Zn = j, Nt = n}
z ∈Gi,j
l,n
b
=
X
P{Wt ≤ s | Zb = zb, Vn = l, Nt = n}Pi {Zb = zb | Vn = l, Zn = j},
z ∈Gi,j
b
l,n
where the last equality follows from the independence of {Zn } and {Nt }. We
denote by T0 , T0 + T1 , . . ., T0 + T1 + · · · + Tn−1 , the first n instants of jumps of
the Poisson process {Nt } over [0, t) and we set Tn = t − (T0 + T1 + · · · + Tn−1 ).
Then,
P{Wt ≤ s | Zb = zb, Vn = l, Nt = n} = P{
= P{
l
X
j=1
l
X
j=1
Tij ≤ s | Zb = zb, Vn = l, Nt = n}
Tij ≤ s | Nt = n},
where the distinct indices i1 , . . . , il ∈ {0, 1, . . . , n} correspond to the l entries
of zb that are in U and the last equality is due to the independence of the
processes {Zn } and {Nt }. For l = 0, we obtain the correct result, which is
equal to 1 by using the convention
Pb
a (...)
= 0 if a > b. From relation (2) we
get, for l = 0, . . . , n,
P{Ti1 + · · · + Til ≤ s | Nt = n} = P{T0 + · · · + Tl−1 ≤ s | Nt = n}
=
n
X
k=l
11


n
k


k s
t
s
1−
t
n−k
.
Again, the convention
Pb
a (...)
= 0 for a > b allows us to cover the cases l = 0
and l = n + 1. Finally, we obtain that
Pi {Wt ≤ s | Vn = l, Zn = j, Nt = n}
n
X X
=


n


k s
t
s
1−
t
n−k
k
k=l
z ∈Gi,j
b
l,n
 
n
n s k X
s n−k
 
=
1−
.
k=l
k
t
P{Zb = zb | Vn = l, Zn = j, Z0 = i}
t
That is, since P{Nt = n} = e−λt (λt)n /n!,
P{Wt ≤ s, Xt = j | X0 = i}


n X
n
n s k (λt)n X
s n−k
 
=
e−λt
1−
P{Vn = l, Zn = j | Z0 = i}
n! l=0 k=l k
t
t
n=0
∞
X


n
n s k (λt)n X
 
e−λt
1−
=
n! k=0 k
t
n=0
 
∞
n
n X
n s k X
(λt)
 
e−λt
1−
=
n! k=0 k
t
n=0
∞
X
s
t
s
t
n−k X
k
l=0
n−k
P{Vn = l, Zn = j | Z0 = i}
P{Vn ≤ k, Zn = j | Z0 = i}.
4 JOINT DISTRIBUTION OF OCCUPATION TIMES
Next, we partition the state space S into m + 1 subsets B0 , B1 , . . . , Bm .
4.1
The Discrete Time Case
We consider the random variables Vni defined by
Vni =
n
X
k=0
1{Zk ∈Bi } .
The next theorem gives the backward equation for the joint distribution of the
Vni and Zn .
12
Theorem 4.1 For r = 1, . . . , m, n ≥ 1, and 0 ≤ k1 , . . . , km ≤ n (kr ≥ 1), we
have
for i ∈ Br , Pi {Vn1 ≤ k1 , . . . , Vnr ≤ kr , . . . , Vnm ≤ km , Zn = j} =
X
l∈S
1
r
m
Pi,l Pl {Vn−1
≤ k1 , . . . , Vn−1
≤ kr − 1, . . . , Vn−1
≤ km , Zn−1 = j},
for i ∈ B0 , Pi {Vn1 ≤ k1 , . . . , Vnm ≤ km , Zn = j} =
X
l∈S
1
m
Pi,l Pl {Vn−1
≤ k1 , . . . , Vn−1
≤ km , Zn−1 = j}.
Proof. By Vcn and kb we denote the vectors (Vn1 , . . . , Vnm ) and (k1 , . . . , km )
respectively and by ei , i = 1, . . . , m, the unit row vector of dimension m whose
ith entry is 1. The proof follows the same steps as that of Theorem 3.1. We
have
b Z = j | Z = i} =
P{Vcn ≤ k,
n
0
=
X
l∈S
X
l∈S
b Z = j | Z = l}
Pi,l Pi {Vcn ≤ k,
n
1
b
Pi,l Pl {Vd
n−1 ≤ k − er 1{i∈Br } , Zn−1 = j}.
The theorem that follows gives the forward equation for the joint distribution of the Vni and Zn .
Theorem 4.2 For r = 1, . . . , m, n ≥ 1, and 0 ≤ k1 , . . . , km ≤ n (kr ≥ 1), we
have
for j ∈ Br , Pi {Vn1 ≤ k1 , . . . , Vnr ≤ kr , . . . , Vnm ≤ km , Zn = j} =
X
l∈S
1
r
m
Pi {Vn−1
≤ k1 , . . . , Vn−1
≤ kr − 1, . . . , Vn−1
≤ km , Zn−1 = l}Pl,j ,
for j ∈ B0 , Pi {Vn1 ≤ k1 , . . . , Vnm ≤ km , Zn = j} =
X
l∈S
1
m
Pi {Vn−1
≤ k1 , . . . , Vn−1
≤ km , Zn−1 = l}Pl,j .
Proof. Using the notation of the proof of Theorem 4.1, we follow the same
steps as in Theorem 3.2. We have that
b Z = j, Z = i} = P{Vd ≤ k
b−e 1
P{Vcn ≤ k,
n
0
n−1
r {j∈Br } , Zn = j, Z0 = i}
=
X
l∈S
b
P{Vd
n−1 ≤ k − er 1{j∈Br } , Zn = j, Zn−1 = l, Z0 = i}
13
=
X
l∈S
=
X
l∈S
b
P{Vd
n−1 ≤ k − er 1{j∈Br } , Z0 = i | Zn−1 = l}P{Zn = j, Zn−1 = l}
b
P{Vd
n−1 ≤ k − er 1{j∈Br } , Zn−1 = l, Z0 = i}Pl,j .
By conditioning on Z0 , we obtain the desired relation.
By F (n, k1 , . . . , km ), for n ≥ 0 and kr ≥ 0, we denote the matrix with (i, j)
entry
Fi,j (n, k1 , . . . , km ) = P{Vn1 ≤ k1 , . . . , Vnm ≤ km , Zn = j | Z0 = i}.
The results of Theorems 4.1 and 4.2 can be conveniently expressed in matrix
notation. We first decompose the matrices P and F (n, k1, . . . , km ) with respect
to the partition {B0 , B1 , . . . , Bm } of the state space S as
P = {PBr Bh }0≤r,h≤m and F (n, k1, . . . , km ) = {FBr Bh (n, k1 , . . . , km )}0≤r,h≤m .
The result of Theorem 4.1 can then be written as
FBr Bh (n, k1 , . . . , km ) =
m
X
l=0
PBr Bl FBl Bh (n − 1, k1 , . . . , kr − 1{r6=0} , . . . , km ),
and that of Theorem 3.2 as
FBr Bh (n, k1 , . . . , km ) =
m
X
l=0
FBr Bl (n − 1, k1 , . . . , kh − 1{h6=0} , . . . , km )PBl Bh .
The initial conditions are given by


 0
F (n, 0, . . . , 0) = 
0
0 (PB0 B0 )n

 , for n ≥ 0.
Note that in the case k1 +· · ·+km ≥ n+1, with ki ≤ n, for i = 1, . . . , m, the mdimensional joint distribution of Vn1 , . . . , Vnm can be expressed as a combination
of the h-dimensional joint distributions of the Vni for h = 1, . . . , m − 1. That
observation is based on the following general result.
For any random variables U1 , . . . , Um and any event A, we have
P{U1 ≤ x1 , . . . , Um ≤ xm , A} =
X
(−1)m−|E|+1P{Ul ≤ xl ; l ∈ E, A}
E⊂{1,...,m}
m
+ (−1) P{U1 > x1 , . . . , Um > xm , A}, (5)
14
where the inclusion is strict, that is E 6= {1, . . . , m}, and where, for convenience, we set P{Ul ≤ xl ; l ∈ ∅, A} = P{A}. For the random variables Vni ,
P{Vn1 > k1 , . . . , Vnm > km , Zn = j | Z0 = i} = 0, if k1 + · · · + km ≥ n + 1,
so, in that case, we get the desired result,
X
b Z = j} =
Pi {Vcn ≤ k,
n
(−1)m−|E|+1 Pi {Vnl ≤ kl ; l ∈ E, Zn = j}. (6)
E⊂{1,...,m}
4.2
The Continuous Time Case
We consider the random variables Wti , i = 1, . . . , m, defined by
Z
Wti =
0
t
1{Xu ∈Bi } du,
whose joint distribution with Xt is given in the next theorem.
Theorem 4.3 For every i, j ∈ S, for every t > 0, and s1 , . . . , sm ∈ [0, t) such
that s1 + s2 + · · · + sm < t, we have
P{Wt1 ≤ s1 , . . . , Wtm ≤ sm , Xt = j | X0 = i} =
∞
X
−λt (λt)
e
n=0
n!
n
X
t;s1 ,s2 ,...,sm
n!θn;k
Pi {Vn1 ≤ k1 , . . . , Vnm ≤ km , Zn = j} (7)
1 ,k2 ,...,km
k1 ≥ 0, . . . , km ≥ 0,
k1 + · · · + km ≤ n
where
t;s1 ,s2 ,...,sm
θn;k
=
1 ,k2 ,...,km
s1
t
k1 s2 k 2
sk k m
s1 + · · · + sm n−(k1 +···+km )
···
1−
t
t
t
.
k1 !k2 ! · · · km !(n − (k1 + k2 · · · + km ))!
Wt = (Wt1 , . . . , Wtm ), Vcn = (Vn1 , . . . , Vnm ), and
Proof. We define the vectors d
sb = (s1 , . . . , sm ). Inequality between vectors means component-wise inequality.
For n ≥ 0, we define the set En by
En = {lb = (l1 , l2 , . . . , lm ) ∈ Nm | l1 + l2 + · · · + lm ≤ n}.
15
We have that
d ≤ sb, X = j | X = i}
P{W
t
t
0
=
=
=
=
=
∞
X
n=0
∞
X
n=0
∞
X
n=0
∞
X
n=0
∞
X
n=0
=
∞
X
n=0
d ≤ sb, N = n, X = j}
Pi {W
t
t
t
d ≤ sb, N = n, Z = j} since {X } and {Z } are equivalent
Pi {W
t
t
n
t
Nt
d ≤ sb, Z = j | N = n}
Pi {Nt = n}Pi {W
t
n
t
d ≤ sb, Z = j | N = n}
P{Nt = n}Pi {W
t
n
t
P{Nt = n}
P{Nt = n}
X
bl∈En
X
bl∈En
b Z = j | N = n}
d ≤ sb, V
c = l,
Pi {W
t
n
n
t
b Z = j, N = n}.
d ≤ sb | V
c = l,
Pi {Vcn = l,b Zn = j}Pi {W
t
n
n
t
The fourth and last equalities follow from the independence of the processes
{Zn } and {Nt } and the fact that X0 = Z0 . In the fifth equality, the summation
in lb should be over En+1 but it can be restricted to En . If l1 +l2 +· · ·+lm = n+1,
then Vcn = bl and Nt = n imply that Vn1 + · · · + Vnm = n + 1 and so that
bN =
d ≤ sb | V
c = l,
Wt1 + · · · + Wtm = t. As s1 + · · · + sm < t that implies P{W
t
n
t
d ≤ sb | V
c = b
l, Zn = j, Nt = n}.
n} = 0. Now consider the expression Pi {W
t
n
For bl = (l1 , l2 , . . . , lm ) ∈ En , and i, j ∈ S, we define the set
Gbi,j
=
l,n













zb = (i, z1 , . . . , zn−1 , j) ∈ S n+1
l entries of zb are in B , . . . ,
1
1
lm entries of zb are in Bm and
n + 1 − (l1 + · · · + lm ) are in B0













,
b We have that
and we denote the random vector (Z0 , . . . , Zn ) by Z.
b Z = j, N = n}
d ≤ sb | V
c = l,
Pi {W
t
n
n
t
=
=
X
z ∈Gi,j
b
bl,n
X
b Z = j, N = n}
d ≤ sb | Z
b = zb, V
c =b
P{W
l, Nt = n}Pi {Zb = zb | Vcn = l,
t
n
n
t
b Z = j}
d ≤ sb | Z
b = zb, V
c =b
P{W
l, Nt = n}Pi {Zb = zb | Vcn = l,
t
n
n
z ∈Gi,j
b
bl,n
16
where the last equality follows from the independence of the processes {Zn }
and {Nt }. It follows that
b N = n} = P{Tb (l)
b ≤ sb | Z
b N = n},
d ≤ sb | Z
b = zb, V
c = l,
b = zb, V
c = l,
P{W
t
n
t
n
t
where

b =
Tb (l)
l1
X
j=1
Tij ,
l1X
+l2
j=l1 +1
Tij , . . . ,

l1 +l2X
+···+lm
j=l1 +l2 +···+lm−1 +1
Tij  .
Again using the independence of {Zn } and {Nt } and relation (3), we obtain
b ≤ sb | Z
b N = n} = P{Tb (l)
b ≤ sb | N = n}
b = zb, V
c = l,
P{Tb (l)
n
t
t
X
=
t;s1 ,s2 ,...,sm
n!θn;k
.
1 ,k2 ,...,km
k1 ≥ l1 , k2 ≥ l2 , . . . , km ≥ lm ,
k1 + k2 + · · · + km ≤ n
b
Note that if one of the li ’s is zero, the corresponding entry of the vector Tb (l)
becomes zero and the preceding formula still holds. Indeed, suppose for simplicity that lm = 0, then
X
t;s1 ,s2 ,...,sm
n!θn;k
1 ,k2 ,...,km
X
=
k1 ≥ l1 , . . . , km ≥ 0,
k1 ≥ l1 , . . . , km−1
k1 + · · · + km ≤ n
n−(k1 +···+km−1 )
X
t;s1 ,...,sm
n!
θn;k
1 ,...,km
km =0
≥ lm−1 ,
k1 + · · · + km−1 ≤ n
X
=
t;s ,...,s
n!θn;k11 ,...,km−1
.
m−1
k1 ≥ l1 , . . . , km−1 ≥ lm−1 ,
k1 + · · · + km−1 ≤ n
b are zero and the
Note also that if all the li ’s are zero, all the entries of Tb (l)
formula still holds since
X
t;s1 ,s2 ,...,sm
n!θn;k
= 1.
1 ,k2 ,...,km
k1 ≥ 0, k2 ≥ 0, . . . , km ≥ 0,
k1 + k2 + · · · + km ≤ n
Putting these results together, we obtain
b Z = j, N = n}
d ≤ sb | V
c = l,
Pi {W
t
n
n
t
=
X
X
z ∈Gi,j
b
bl,n
k1 ≥ l1 , k2 ≥ l2 , . . . , km ≥ lm ,
t;s1 ,s2 ,...,sm
b Z = j}
n!θn;k
Pi {Zb = zb | Vcn = l,
n
1 ,k2 ,...,km
k1 + k2 + · · · + km ≤ n
17
X
=
t;s1 ,s2 ,...,sm
n!θn;k
1 ,k2 ,...,km
k1 ≥ l1 , k2 ≥ l2 , . . . , km ≥ lm ,
k1 + k2 + · · · + km ≤ n
X
=
X
Pi {Zb = zb | Vcn = l,b Zn = j}
z ∈Gi,j
b
bl,n
t;s1 ,s2 ,...,sm
n!θn;k
,
1 ,k2 ,...,km
k1 ≥ l1 , k2 ≥ l2 , . . . , km ≥ lm ,
k1 + k2 + · · · + km ≤ n
and so,
d ≤ sb, X = j | X = i}
P{W
t
t
0
=
∞
X
n=0
(λt)n X
n! b
∞
X
n
e−λt
X
t;s1 ,s2 ,...,sm
b Z = j}
n!θn;k
Pi {Vcn = l,
n
1 ,k2 ,...,km
l∈En k1 ≥ l1 , k2 ≥ l2 , . . . , km ≥ lm ,
k1 + k2 + · · · + km ≤ n
=
e−λt
n=0
(λt) X
t;s1 ,s2 ,...,sm
n!θn;k
1 ,k2 ,...,km
n! b
k∈En
X
Pi {Vcn = bl, Zn = j}
l1 ≥ k1 , l2 ≥ k2 , . . . , lm ≥ km ,
l1 + l2 + · · · + lm ≤ n
∞
X
n
(λt) X
t;s1 ,s2 ,...,sm
b
n!θn;k
P{Vcn ≤ l}
1 ,k2 ,...,km
n!
n=0
b
k∈En
∞
n
X
X
(λt)
t;s1 ,s2 ,...,sm
=
e−λt
n!θn;k
Pi {Vcn ≤ l,b Zn = j}.
1 ,k2 ,...,km
n!
n=0
=
e−λt
k1 ≥ 0, k2 ≥ 0, . . . , km ≥ 0,
k1 + k2 + · · · + km ≤ n
From relation (7) the distribution Pi {Wt1 ≤ s1 , . . . , Wtm ≤ sm , Xt = j} is
differentiable with respect to t and also with respect to s1 , s2 , . . . , sm for t > 0,
s1 , . . . , sm ∈ (0, t), and s1 + · · ·+ sm ∈ (0, t). Moreover, if s1 + s2 + · · ·+ sm ≥ t,
then trivially
P{Wt1 > s1 , . . . , Wtm > sm , Xt = j | X0 = i} = 0,
so that relation (6) applies by replacing the Vnl and the kl by the Wtl and the
sl respectively.
18
5 WEIGHTED SUMS OF OCCUPATION TIMES
A constant performance level or reward rate ρ(i) is associated with each state
i of S. We consider the random variable Yt defined by
Z
Yt =
0
t
ρ(Xu )du.
We denote by m + 1 the number of distinct rewards and their values by
r0 < r1 < · · · < rm−1 < rm .
We then have Yt ∈ [r0 t, rm t] with probability one. Without loss of generality,
we may set r0 = 0. That can be easily done by considering the random
variable Yt − r0 t instead of Yt and the reward rates ri − r0 instead of ri . As in
Section 4, the state space S is partitioned into subsets B0 , . . . , Bm . The subset
Bl contains the states with reward rate rl , that is Bl = {i ∈ S|ρ(i) = rl }. With
this notation,
Yt =
m
X
l=1
Z
rl
t
0
1{Xu ∈Bl } du =
m
X
l=1
rl Wtl .
(8)
As the distribution of each Wtl has at most two jumps at 0 and t, the
distribution of Yt has at most m + 1 jumps at the points r0 t = 0, r1 t, . . . ,
rm t. For t > 0, the jump at point x = rl t is equal to the probability that the
process X, starting in subset Bl , stays in the subset Bl during all of [0, t), that
is
P{Yt = rl t} = αBl eABl Bl t 1Bl for t > 0,
where 1Bl is the column vector of dimension |Bl | with all components equal to
1. For every i, j ∈ S, and t > 0, we define the functions Fi,j (t, x) by
Fi,j (t, x) = P{Yt > x, Xt = j | X0 = i},
and we introduce the matrix F (t, x) = {Fi,j (t, x)}. Using the partition Bm ,
Bm−1 , . . ., B0 , the matrices A, P , and F (t, x) can be written as
A = {ABu Bv }0≤u,v≤m ; P = {PBu Bv }0≤u,v≤m ; F (t, x) = {FBu Bv (t, x)}0≤u,v≤m .
19
Note that for t > 0, and 0 ≤ l ≤ m,


 (eABl Bl t )i,j
if i, j ∈ Bl ,

otherwise,
P{Yt = rl t, Xt = j | X0 = i} = 
0
that is
P{Yt = rl t, Xt = j | X0 = i} =
∞
X
e−λt
n=0
(λt)n
(PBl Bl )ni,j 1{i,j∈Bl} .
n!
(9)
The distribution Fi,j (t, x) can be obtained from relation (8), using the joint
distribution of the Wtl obtained in Section 4.2. From relation (7), Fi,j (t, x) is
differentiable with respect to x and t in the domain
E = {(t, x) ; t > 0 and x ∈
m
[
(rl−1 t, rl t)}.
l=1
The initial conditions are given, for t > 0, by
Fi,j (t, 0) = P{Xt = j | X0 = i} − P{Yt = 0, Xt = j | X0 = i},
that is, in matrix notation,
FBu Bv (t, 0) = (eAt )Bu Bv − eAB0 B0 t 1{u=v=0} ,
which can also be written as
FBu Bv (t, 0) =
5.1
∞
X
n=0
e−λt
i
(λt)n h n
(P )Bu Bv − (PB0 B0 )n 1{u=v=0} .
n!
(10)
Backward and Forward Equations
In what follows, we derive backward and forward equations satisfied by the
distribution of the pair (Yt , Xt ). First, we recall some well-known and useful
results in the following lemma. Remember that {N(t)} is a Poisson process
of rate λ, independent of the Markov chain Z. We denote by N(t, t + s) the
number of transitions during the interval [t, t + s).
20
Lemma 5.1
P{N(t, t + s) = 0 | Xt = j} = e−λs
(11)
P{Xt+s = j, N(t, t + s) = 1 | Xt = i} = Pi,j λse−λs
(12)
P{N(t, t + s) ≥ 2 | X0 = i} = o(s).
(13)
The following theorem establishes the forward equation for the pair (Yt , Xt ).
Theorem 5.2 For t > 0, i, j ∈ S, 1 ≤ h ≤ m, and x ∈ (rh−1 t, rh t), we have
∂Fi,j (t, x)
∂Fi,j (t, x) X
= −ρ(j)
+
Fi,k (t, x)Ak,j .
∂t
∂x
k∈S
(14)
Proof. By conditioning on the number of transitions in [t, t + s), we have
Pi {Yt+s > x, Xt+s = j} = Pi {Yt+s > x, Xt+s = j, N(t, t + s) = 0}
+ Pi {Yt+s > x, Xt+s = j, N(t, t + s) = 1}
+ Pi {Yt+s > x, Xt+s = j, N(t, t + s) ≥ 2}.
We separately consider these three terms. For the first term, since Xt+s = j
and N(t, t + s) = 0 is equivalent to Xt = j and N(t, t + s) = 0, we have
Pi {Yt+s > x, Xt+s = j, N(t, t + s) = 0} = Pi {Yt+s > x, Xt = j, N(t, t + s) = 0}
= Pi {Yt+s > x | Xt = j, N(t, t + s) = 0}Pi {Xt = j, N(t, t + s) = 0}
= Pi {Yt > x − ρ(j)s | Xt = j, N(t, t + s) = 0}Pi {Xt = j, N(t, t + s) = 0}
= Pi {Yt > x − ρ(j)s | Xt = j}Pi {Xt = j, N(t, t + s) = 0}
= Pi {Yt > x − ρ(j)s, Xt = j}Pi {N(t, t + s) = 0 | Xt = j}
= P{N(t, t + s) = 0 | Xt = j}Fi,j (t, x − ρ(j)s)
= e−λs Fi,j (t, x − ρ(j)s)
= (1 − λs)Fi,j (t, x − ρ(j)s) + o(s).
The second equality follows from the fact that, if Xt = j and N(t, t+s) = 0, we
have Yt+s = Yt + ρ(j)s. The third and fifth follow from the Markov property,
21
and the sixth from relation (11). For the second term denoted by G(s), we
define
Gk (s) = Pi {Yt+s > x | Xt = k, Xt+s = j, N(t, t + s) = 1}.
We then have
G(s) = Pi {Yt+s > x, Xt+s = j, N(t, t + s) = 1}
=
X
k∈S
Gk (s)Pi {Xt = k, Xt+s = j, N(t, t + s) = 1}
Let us define ρmin = min{ρ(i)} and ρmax = max{ρ(i)}.
As Yt + ρmin s ≤ Yt+s ≤ Yt + ρmax s, we get
Pi {Yt > x − ρmin s | Xt = k, Xt+s = j, N(t, t + s) = 1} ≤ Gk (s),
and
Gk (s) ≤ Pi {Yt > x − ρmax s | Xt = k, Xt+s = j, N(t, t + s) = 1}.
Using the Markov property,
Pi {Yt > x − ρmin s | Xt = k} ≤ Gk (s) ≤ Pi {Yt > x − ρmax s | Xt = k}.
We thus obtain
X
Fi,k (t, x − ρmin s)Uk,j (s) ≤ G(s) ≤
k∈S
X
Fi,k (t, x − ρmax s)Uk,j (s),
k∈S
where Uk,j (s) = P{Xt+s = j, N(t, t + s) = 1 | Xt = k}. From relation (12),
Uk,j (s)
= λPk,j ,
s−→0
s
lim
so we obtain
X
G(s)
=λ
Fi,k (t, x)Pk,j .
s−→0
s
k∈S
lim
For the third term, we have by relation (13),
Pi {Yt+s > x, Xt+s = j, N(t, t + s) ≥ 2} ≤ Pi {N(t, t + s) ≥ 2} = o(s).
22
Combining the three terms, we obtain
Fi,j (t + s, x) − Fi,j (t, x)
(1 − λs)Fi,j (t, x − ρ(j)s) − Fi,j (t, x) G(s) o(s)
=
+
+
s
s
s
s
=
Fi,j (t, x − ρ(j)s) − Fi,j (t, x)
G(s) o(s)
− λFi,j (t, x − ρ(j)s) +
+
s
s
s
If now s tends to 0, we get
X
∂Fi,j (t, x)
∂Fi,j (t, x)
= −ρ(j)
− λFi,j (t, x) + λ
Fi,k (t, x)Pk,j .
∂t
∂x
k∈S
Since P = I + A/λ, we obtain that
∂Fi,j (t, x)
∂Fi,j (t, x) X
= −ρ(j)
+
Fi,k (t, x)Ak,j .
∂t
∂x
k∈S
Corollary 5.3 For t > 0, 0 ≤ p ≤ m, i ∈ Bp , j ∈ S, 1 ≤ h ≤ m, and
x ∈ (rh−1 t, rh t), we have
Fi,j (t, x) =
XZ
t
k∈S 0
Fi,k (t − u, x − ρ(j)u)λe−λu duPk,j + e−λt 1{h≤p} 1{i=j} . (15)
Proof. Consider equation (14) and the functions ϕi,j defined by
ϕi,j (u) = Fi,j (t − u, x − ρ(j)u)e−λu .
Differentiating with respect to u yields
"
ϕ0i,j (u)
−λu
=e
#
∂Fi,j
∂Fi,j
− ρ(j)
−
(t−u, x−ρ(j)u)−Fi,j (t−u, x−ρ(j)u)λe−λu
∂t
∂x
which, by (14) and the relation A = −λ(I − P ), gives
ϕ0i,j (u) = −
= −
X
Fi,k (t − u, x − ρ(j)u)Ak,j e−λu − Fi,j (t − u, x − ρ(j)u)λe−λu
k∈S
X
k∈S
Fi,k (t − u, x − ρ(j)u)λe−λu Pk,j
Integrating that expression between 0 and t gives
ϕi,j (t) − ϕi,j (0) = −
XZ
k∈S 0
t
Fi,k (t − u, x − ρ(j)u)λe−λu duPk,j .
23
Finally, we have ϕi,j (0) = Fi,j (t, x) and
ϕi,j (t) = Fi,j (0, x − ρ(j)t)e−λt = e−λt 1{x−ρ(j)t<0} 1{i=j} = e−λt 1{h≤p} 1{i=j}.
We next derive the backward equation for the evolution of the pair (Yt , Xt ).
Theorem 5.4 For t > 0, 0 ≤ p ≤ m, i ∈ Bp , j ∈ S, 1 ≤ h ≤ m, and
x ∈ (rh−1 t, rh t),
Fi,j (t, x) =
X
Z
Pi,k
k∈S
t
0
Fk,j (t − u, x − ρ(i)u)λe−λu du + e−λt 1{h≤p} 1{i=j}. (16)
Proof. Let T1 be the sojourn time in the initial state. We have
Z
Fi,j (t, x) =
∞
0
P{Yt > x, Xt = j | T1 = u, X0 = i}λe−λu du.
If u ≥ t and X0 = i, we have Yt = ρ(i)t = rp t and P{Xt = j | T1 = u, X0 =
i} = 1, if i = j and 0 otherwise. Moreover, as rp t > x is equivalent to rp t ≥ rh t,
that is h ≤ p, we obtain
Z
Fi,j (t, x) =
t
0
P{Yt > x, Xt = j | T1 = u, X0 = i}λe−λu du + e−λt 1{h≤p} 1{i=j}.
Now,
P{Yt > x, Xt = j | T1 = u, X0 = i}
=
X
k∈S
P{Yt > x, Xt = j | Xu = k, T1 = u, X0 = i}P{Xu = k | T1 = u, X0 = i}.
For the second factor in the summand, we have that
P{Xu = k | T1 = u, X0 = i} = P{XT1 = k | T1 = u, X0 = i}
= P{Z1 = k | T1 = u, Z0 = i}
= Pi,k .
For the first factor, T1 = u and X0 = i imply Yu = ρ(i)u, so that
24
P{Yt > x, Xt = j | Xu = k, T1 = u, X0 = i}
= P{
= P{
Z
t
u
t
Z
u
ρ(Xv )dv > x − ρ(i)u, Xt = j | Xu = k, T1 = u, X0 = i}
ρ(Xv )dv > x − ρ(i)u, Xt = j | Xu = k}
= P{Yt−u > x − ρ(i)u, Xt−u = j | X0 = k}
= Fk,j (t − u, x − ρ(i)u),
where the second equality follows from the Markov property and the third by
homogeneity. Combining these results, we obtain relation (16).
Corollary 5.5 For t > 0, i, j ∈ S, 1 ≤ h ≤ m and x ∈ (rh−1 t, rh t) we have
∂Fi,j (t, x)
∂Fi,j (t, x) X
= −ρ(i)
+
Ai,k Fk,j (t, x).
∂t
∂x
k∈S
(17)
Proof. Consider equation (16) with i ∈ Bp , 0 ≤ p ≤ m and j ∈ S. Differentiating Fi,j (t, x) with respect to t, we obtain
Z t
X
∂Fi,j (t, x)
∂Fk,j (t − u, x − ρ(i)u) −λu
Pi,k
=
λe du
∂t
∂t
0
k∈S
+
X
k∈S
Pi,k Fk,j (0, x − ρ(i)t)λe−λt − λe−λt 1{h≤p}1{i=j}
Next, differentiating Fi,j (t, x) with respect to x, we get
∂Fi,j (t, x) X
=
Pi,k
∂x
k∈S
Z
t
0
∂Fk,j (t − u, x − ρ(i)u) −λu
λe du.
∂x
Consider the functions ψk,j and ϕk,j defined by
ψk,j (u) = Fk,j (t − u, x − ρ(i)u),
and
ϕk,j (u) = ψk,j (u)e−λu .
Note that ψi,j (t) = 1{h≤p} 1{i=j}, so (16) can be written as
ϕi,j (0) = λ
X
k∈S
Z
Pi,k
t
0
ϕk,j (u)du + ϕi,j (t)
Differentiating ψk,j with respect to u, we get
0
ψk,j
(u) = −
∂Fk,j (t − u, x − ρ(i)u)
∂Fk,j (t − u, x − ρ(i)u)
− ρ(i)
.
∂t
∂x
25
(18)
We thus obtain
X
∂Fi,j (t, x)
∂Fi,j (t, x)
+ ρ(i)
=λ
Pi,k ϕk,j (t) −
∂t
∂x
k∈S
= λ
X
X
k∈S
= λ
X
t
0
Pi,k ϕk,j (0) − λ
k∈S
= λ
Z
2
0
ψk,j
(u)e−λu du − λϕi,j (t)
Z
0
t
ϕk,j (u)du − λϕi,j (t)
Pi,k ϕk,j (0) − λϕi,j (0)
Pi,k Fk,j (t, x) − λFi,j (t, x)
k∈S
=
X
k∈S
Ai,k Fk,j (t, x),
where the second equality is obtained by integration by parts and the third
follows from relation (18).
Let D be the diagonal matrix with the reward rates ρ(i) on the diagonal and
F (t, x) the matrix {Fi,j (t, x)}. In matrix notation, the forward and backward
equations (14) and (17) become
∂F (t, x)
∂F (t, x)
=−
D + F (t, x)A,
∂t
∂x
(19)
∂F (t, x)
∂F (t, x)
= −D
+ AF (t, x).
∂t
∂x
(20)
and
These are hyperbolic partial differential equations having a unique solution
on the domain E with the initial condition given by relation (10), see for
instance [6].
5.2
Solutions
The solution to equation (19) is given by the following theorem:
Theorem 5.6 For every t > 0, and x ∈ [rh−1 t, rh t), for 1 ≤ h ≤ m,
∞
X


n
n
(λt)n X
  xk (1 − xh )n−k C (h) (n, k),
F (t, x) =
e−λt
h
n! k=0 k
n=0
26
(21)
x − rh−1 t
(h)
and the matrices C (h) (n, k) = CBu Bv (n, k)
0≤u,v≤m
(rh − rh−1 )t
are given by the recurrence relations
where xh =
for 0 ≤ u ≤ m, and h ≤ v ≤ m :
(1)
(h)
(h−1)
for n ≥ 0 : CBu Bv (n, 0) = (P n )Bu Bv , CBu Bv (n, 0) = CBu Bv (n, n), for h > 1,
for 1 ≤ k ≤ n :
(h)
CBu Bv (n, k)=
m
rv − rh (h)
rh − rh−1 X
(h)
CBu Bv (n, k−1)+
C
(n−1, k−1)PBw Bv ,
rv − rh−1
rv − rh−1 w=0 Bu Bw
(22)
for 0 ≤ u ≤ m, and 0 ≤ v ≤ h − 1 :
(m)
(h)
(h+1)
for n ≥ 0 : CBu Bv (n, n) = 0Bu Bv , CBu Bv (n, n) = CBu Bv (n, 0), for h < m,
for 0 ≤ k ≤ n − 1 :
(h)
CBu Bv (n, k)
m
rh−1 − rv (h)
rh − rh−1 X
(h)
=
CBu Bv (n, k + 1) +
CBu Bw (n − 1, k)PBw Bv .
rh − rv
rh − rv w=0
(23)
Proof. For t > 0, and x ∈ (rh−1 t, rh t), 1 ≤ h ≤ m, we write the solution of
equation (19) as

∞
X

n
n
(λt)n X
  xk (1 − xk )n−k C (h) (n, k),
F (t, x) =
e−λt
h
n! k=0 k
n=0
and we establish the relations that the matrices C (h) (n, k) must satisfy. So,


n
∞
n
X
λ
(λt)n X
∂F (t, x)
  xk (1 − xh )n−k
= −λF (t, x) +
e−λt
h
∂t
rh − rh−1 n=0
n! k=0 k
h
i
× rh C (h) (n + 1, k) − rh−1 C (h) (n + 1, k + 1) ,
and


n
∞
n
X
∂F (t, x)
(λt)n X
λ
  xk (1 − xh )n−k
=
e−λt
h
∂x
rh − rh−1 n=0
n! k=0 k
h
i
× C (h) (n + 1, k + 1) − C (h) (n + 1, k) .
27
Since A = −λ(I − P ), we obtain F (t, x)A = −λF (t, x) + λF (t, x)P, that is,

∞
X

n
n
(λt)n X
  xk (1 − xh )n−k C (h) (n, k)P.
F (t, x)A = −λF (t, x) + λ
e−λt
h
n! k=0 k
n=0
It follows that if the matrices C (h) (n, k) satisfy
C (h) (n + 1, k + 1)[D − rh−1 I] = C (h) (n + 1, k)[D − rh I] + (rh − rh−1 )C (h) (n, k)P,
(24)
then equation (19) is satisfied. For every 1 ≤ h ≤ m, and 0 ≤ u ≤ m, the
recurrence relation (24) can also be written as follows:
If h ≤ v ≤ m, then
(h)
CBu Bv (n, k)=
m
rv − rh (h)
rh − rh−1 X
(h)
CBu Bv (n, k−1)+
C
(n−1, k−1)PBw Bv ,
rv − rh−1
rv − rh−1 w=0 Bu Bw
and if 0 ≤ v ≤ h − 1, then
(h)
CBu Bv (n, k)
m
rh−1 − rv (h)
rh − rh−1 X
(h)
=
CBu Bv (n, k + 1) +
CBu Bw (n − 1, k)PBw Bv .
rh − rv
rh − rv w=0
To get the initial conditions for the C (h) (n, k), we consider the jumps of
F (t, x). We first consider the jump at x = r0 t = 0. For t > 0, at x = 0, that
is, for h = 1, relation (21) yields that
F (t, 0) =
∞
X
n=0
e−λt
(λt)n (1)
C (n, 0).
n!
It follows from (10) that for 0 ≤ u, v ≤ m,
(1)
CBu Bv (n, 0) = (P n )Bu Bv − (PB0 B0 )n 1{u=v=0} .
(25)
In particular, that implies that for every 0 ≤ u ≤ m,
(1)
CBu Bv (n, 0) = (P n )Bu Bv , for 1 ≤ v ≤ m.
Next, we consider the jumps at x = rh t, 1 ≤ h ≤ m − 1. For t > 0, 1 ≤ h ≤
m − 1, and i, j ∈ S, we have that
Fi,j (t, rh t) = lim Fi,j (t, x) − P{Yt = rh t, Xt = j | X0 = i}.
<
x−→rh t
28
From (21) and (9), we obtain
(h+1)
(h)
CBu Bv (n, 0) = CBu Bv (n, n) − (PBh Bh )n 1{u=v=h} .
(26)
In particular, that implies that for every 0 ≤ u ≤ m,
(h)
(h−1)
CBu Bv (n, 0) = CBu Bv (n, n), for 1 < h ≤ v ≤ m,
and
(h)
(h+1)
CBu Bv (n, n) = CBu Bv (n, 0), for 0 ≤ v ≤ h − 1 < m − 1.
Finally, we consider the jump at x = rm t, that is, for h = m. For t > 0,
0 = Fi,j (t, rm t) = lim Fi,j (t, x) − P{Yt = rm t, Xt = j | X0 = i},
<
x−→rm t
which, as in the preceding case, leads to
(m)
CBu Bv (n, n) = (PBm Bm )n 1{u=v=m} .
(27)
That implies that for every 0 ≤ u ≤ m,
(m)
CBu Bv (n, n) = 0, for 0 ≤ v ≤ m − 1.
Corollary 5.7 For 1 ≤ h ≤ m, n ≥ 0, and 0 ≤ k ≤ n, the matrices
(h)
C (h) (n, k) = CBu Bv (n, k)
0≤u,v≤m
satisfy the following recurrence relations
for h ≤ u ≤ m, and 0 ≤ v ≤ m :
(1)
(h)
(h−1)
for n ≥ 0 : CBu Bv (n, 0) = (P n )Bu Bv , CBu Bv (n, 0) = CBu Bv (n, n), for h > 1,
for 1 ≤ k ≤ n :
(h)
CBu Bv (n, k)=
m
ru − rh (h)
rh − rh−1 X
(h)
CBu Bv (n, k−1)+
PB B C
(n−1, k−1),
ru − rh−1
ru − rh−1 w=0 u w Bw Bv
for 0 ≤ u ≤ h − 1, and 0 ≤ v ≤ m :
(m)
(h)
(h+1)
for n ≥ 0 : CBu Bv (n, n) = 0Bu Bv , CBu Bv (n, n) = CBu Bv (n, 0), for h < m,
29
for 0 ≤ k ≤ n − 1 :
(h)
CBu Bv (n, k) =
m
rh−1 − ru (h)
rh − rh−1 X
(h)
CBu Bv (n, k + 1) +
PBu Bw CBw Bv (n − 1, k).
rh − ru
rh − ru w=0
Proof. The proof is the same as that of Theorem 5.6 using equations (20)
and (21). We thus obtain that the matrices C (h) (n, k) satisfy the relation
[D − rh−1 I]C (h) (n + 1, k + 1) = [D − rh I]C (h) (n + 1, k) + (rh − rh−1 )P C (h) (n, k).
(28)
For every 1 ≤ h ≤ m, and 0 ≤ v ≤ m, the relation (28) may also be written
as follows:
If h ≤ u ≤ m, then
(h)
CBu Bv (n, k)=
m
ru − rh (h)
rh − rh−1 X
(h)
CBu Bv (n, k−1)+
PBu Bw CBw Bv (n−1, k−1),
ru − rh−1
ru − rh−1 w=0
and if 0 ≤ u ≤ h − 1, then
(h)
CBu Bv (n, k) =
m
rh−1 − ru (h)
rh − rh−1 X
(h)
CBu Bv (n, k + 1) +
PBu Bw CBw Bv (n − 1, k).
rh − ru
rh − ru w=0
As in the proof of Theorem 5.6, we consider the jumps of F (t, x). Relation (25)
implies that, for every 0 ≤ v ≤ m,
(1)
CBu Bv (n, 0) = (P n )Bu Bv , for 1 ≤ u ≤ m.
Relation (26) implies that, for every 0 ≤ v ≤ m,
(h)
(h−1)
CBu Bv (n, 0) = CBu Bv (n, n), for 1 < h ≤ u ≤ m,
and
(h)
(h+1)
CBu Bv (n, n) = CBu Bv (n, 0), for 0 ≤ u ≤ h − 1 < m − 1.
Finally, (27) implies that, for every 0 ≤ v ≤ m,
(m)
CBu Bv (n, n) = 0, for 0 ≤ u ≤ m − 1.
30
The following corollary gives an upper bound for the matrices C (h) (n, k).
If M and K are square matrices of the same dimension, the notation M ≤ K
means element-wise inequality.
Corollary 5.8 For every n ≥ 0, 0 ≤ k ≤ n, and 1 ≤ h ≤ m,
0 ≤ C (h) (n, k) ≤ P n .
Proof. The proof is by a two-stage induction; first over n, then, for fixed
n, over k, by using the recurrence relation in Theorem 5.6, or equivalently in
Corollary 5.7. The result clearly holds for n = 0. Note that in (22), that is,
for h ≤ v, we have
0≤
rv − rh
rh − rh−1
=1−
≤ 1,
rv − rh−1
rv − rh−1
and in (23), that is, for v ≤ h − 1, we have
0≤
rh−1 − rv
rh − rh−1
=1−
≤ 1.
rh − rv
rh − rv
Consider first the case v ≤ h − 1. The result holds for the pair (n, n), since
(m)
CBu Bv (n, n) = 0. Suppose the result holds for n − 1 and for the pair (n, k + 1),
(h)
then from (23), we get CBu Bv (n, k) ≥ 0, and
(h)
m
rh−1 − rv (h)
rh − rh−1 X
(h)
CBu Bv (n, k + 1) +
C
(n − 1, k)PBw Bv
rh − rv
rh − rv w=0 Bu Bw
m
rh−1 − rv n
rh − rh−1 X
≤
(P )Bu Bv +
(P n−1 )Bu Bw PBw Bv
rh − rv
rh − rv w=0
rh−1 − rv n
rh − rh−1 n
=
(P )Bu Bv +
(P )Bu Bv
rh − rv
rh − rv
= (P n )Bu Bv .
CBu Bv (n, k) =
The same argument is used in the case h ≤ v from relation (22). Moreover,
the relations
(h)
(h−1)
CBu Bv (n, 0) = CBu Bv (n, n), for 1 < h ≤ v ≤ m,
31
and
(h)
(h+1)
CBu Bv (n, n) = CBu Bv (n, 0), for 0 ≤ v ≤ h − 1 < m − 1,
are used to account for both cases v ≤ h − 1 and h ≤ v.
In numerical procedures, that result is particularly useful in avoiding overflow problems.
References
[1] H. A. David. Order statistics. John Wiley & sons, Inc., New York London - Sidney - Toronto, 1981.
[2] E. de Souza e Silva and H. R. Gail. Calculating cumulative operational
time distributions of repairable computer systems. IEEE Trans. Computers, 35, April 1986.
[3] E. de Souza e Silva and H. R. Gail. An algorithm to calculate transient
distributions of cumulative rate and impulse based reward. Communications in Statistics - Stochastic Models, 14(3), 1998.
[4] S. Karlin and H. W. Taylor. A second course in stochastic processes.
Academic Press, New York - San Francisco - London, 1981.
[5] H. Nabli and B. Sericola. Performability analysis: A new algorithm. IEEE
Trans. Computers, 45(4), April 1996.
[6] I. G. Petrovsky. Lectures on partial differential equations. Interscience
publishers, New York, 1962.
[7] S.M. Ross. Stochastic Processes. John Wiley and Sons, 1983.
[8] G. Rubino and B. Sericola. Interval availability distribution computation. In 23th IEEE International Symposium on Fault Tolerant Computing
(FTCS’23), Toulouse, France, June 1993.
32
[9] G. Rubino and B. Sericola. Interval availability analysis using denumerable Markov processes. Application to multiprocessor subject to breakdowns and repair.
IEEE Trans. Computers. Special Issue on Fault-
Tolerant Computing, 44(2), February 1995.
[10] B. Sericola. Closed-form solution for the distribution of the total time
spent in a subset of states of a homogeneous markov process during a
finite observation period. J. Appl. Prob., 27, 1990.
33