1. Consider a Markov process with states 0, 1 and 2 and with the following transition rate
matrix Q:


−λ
λ
0

Q=
 µ −λ − µ λ 
µ
0
−µ
where λ > 0 and µ > 0.
a. Derive the parameters vi and Pij for this Markov process.
b. Determine the expected time to go from state 1 to state 0.
2.
3.
4.
5.
Exercise
Exercise
Exercise
Exercise
6.1.
6.3.
6.5.
6.6 (a), (b).
1
1 1.
a. We have
v0 = λ,
v1 = λ + µ,
v2 = µ,
and
P01 = P20 = 1,
P10 = 1 − P12 =
µ
.
λ+µ
b. Let Ti (i = 1, 2) denote the time to go from state i to 0. Then E[T2 ] = 1/µ and
E[T1 ] =
1
λ
1
+
· E[T2 ] = .
λ+µ λ+µ
µ
6.1. 2. Let us assume that the state is (n, m). Male i mates at a rate λ with female j, and
therefore it mates at a rate λm. Since there are n males, mating occurs at a rate λnm.
Therefore
vn,m = λnm.
Since any mating is equally likely to result in a female as in a male, we have
1
P(n,m);(n+1,m) = P(n,m);(n+1,m) = .
2
6.3. 3. This is not a birth and death process since we need more information than just the
number working. We must also know which machine is working. We can analyze it by
letting the states be
b
1
2
d1
d2
:
:
:
:
:
both machines are working
1 is working, 2 is down
2 is working, 1 is down
both are down, 1 is being repaired
both are down, 2 is being repaired.
Then
v b = µ1 + µ2 ,
v1 = µ1 + µ,
v2 = µ2 + µ,
and
µ2
,
µ1 + µ2
µ
= 1 − P1,d2 =
,
µ1 + µ
µ
= 1 − P2,d1 =
,
µ + µ2
Pd1,1 = Pd2,2 = 1.
Pb,1 = 1 − Pb,2 =
P1,b
P2,b
1
vd1 = vd2 = µ,
6.5.4.
a. Yes.
b. It is a pure birth process.
c. If there are i infected individuals then since a contact
! " will involve an infected and
an uninfected individual with probability i(n − i)/ n2 , it follows that the birth rates
! "
are λi = λi(n − i)/
n
2
, i = 1, . . . , n. Hence,
n-1
n
n(n − 1) #
1
E[time all infected] =
.
2λ
i=1 i(n − i)
6.6. 5. Starting with E[T0 ] =
1
λ0
= λ1 , employ the identity
E[Ti ] =
1
µi
+ E[Ti−1 ]
λi λi
to succesively compute E[Ti ] for i = 1, 2, 3, 4.
a. E[T0 ] + · · · + E[T3 ].
b. E[T2 ] + E[T3 ] + E[T4 ].
2
2
1. Consider the following queueing model: customers arrive at a service station according
to a Poisson process with rate λ. There are c servers; the service times are exponential with
rate µ. If an arriving customer finds c servers busy, then he leaves the system immediately.
a. Model this system as a birth and death process.
b. Suppose now that there are infinitely many servers (c = ∞). Again model this system
as a birth and death process.
3
2. In Example 6.11 it is shown, using the backward equations, that
!
P00
(t) = µ − (µ + λ)P00 (t).
a. Derive this result using the forward equations.
b. Derive a differential equation for P11 (t) in two ways: using the forward and backward
equations.
c. Suppose the machine is working at time 0. What is the probability that the machine
is also working at time t?
3. Exercise 6.8.
4. Exercise 6.10 (but you do not have to verify that the transition probabilities satisfy the
forward and backward equations).
1
2
1.
a. The state space is {0, 1, . . . , c − 1, c}, and the birth rates are
λn = λ,
n = 0, 1, . . . , c − 1,
and the death rates
µn = nµ,
n = 1, 2, . . . , c.
Hence the transition rate matrix Q is given by

Q=












−λ
λ
0
0
0
···
0
µ −(λ + µ)
λ
0
0
···
0
0
2µ
−(λ + 2µ)
λ
0
···
0
..
..
...
...
...
...
...
.
.
0
···
0
(c − 2)µ −(λ + (c − 2)µ)
λ
0
0
···
0
0
(c − 1)µ
−(λ + (c − 1)µ) λ
0
···
0
0
0
cµ
−cµ
b. Now the state space does not stop at c, i.e., it is {0, 1, . . .} and the birth rates are
λn = λ,
n = 0, 1, . . . ,
and the death rates
µn = nµ,
3
n = 1, 2, . . . .
2.
a. The forward equations are
!
P00 (t) = µP01 (t) − λP00 (t)
!
P01 (t) = λP00 (t) − µP01 (t).
Using P01 (t) = 1 − P00 (t), we obtain the following differential equation,
!
P00 (t) = µ − (µ + λ)P00 (t).
b. The backward equations are
!
P11 (t) = µP01 (t) − µP11 (t)
!
P01 (t) = λP11 (t) − λP01 (t).
Multiplying these equations by λ and µ, respectively, and then adding them, we
obtain
!
!
λP11 (t) + µP01 (t) = 0.
1













Hence, for some c,
λP11 (t) + µP01 (t) = c.
Substituting t = 0 yields c = λ. Using the above equation, we get
!
P11 (t) = λ − (λ + µ)P11 (t).
The forward equations are
!
P11 (t) = λP10 (t) − µP11 (t)
!
P10 (t) = µP11 (t) − λP10 (t).
By substituting P10 (t) = 1−P11 (t), we obtain the same differential equation as before.
c. Solving the differential equation for P00 (t) with initial condition P00 (0) = 1 yields
P00 (t) =
µ
λ −(λ+µ)t
+
e
,
λ+µ λ+µ
t ≥ 0.
6.8. 3. The number of failed machines is a birth and death process with
λ0 = 2λ,
λ1 = λ,
λn = 0, n > 1,
µ1 = µ2 = µ,
µn = 0, n #= 1, 2.
Now substitute into the backward equations.
6.10.4. Let
Ij (t) =
!
0,
1,
if machine j is working at time t,
otherwise.
Also, let the state be (I1 (t), I2 (t)). This is clearly a continuous-time Markov chain with
v(0,0) = λ1 + λ2 ,
v(0,1) = λ1 + µ2 ,
v(1,0) = µ1 + λ2 ,
v(1,1) = µ1 + µ2 ,
λ(0,0);(0,1)
λ(0,1);(0,0)
λ(1,0);(0,0)
λ(1,1);(0,1)
= λ2 ,
= µ2 ,
= µ1 ,
= µ1 ,
λ(0,0);(1,0)
λ(0,1);(1,1)
λ(1,0);(1,1)
λ(1,1);(1,0)
= λ1 ,
= λ1 ,
= λ2 ,
= µ2 .
By the independence assumption we have
P(i,j),(k,l) (t) = Pi,k (t)Qj,l (t),
(1)
where Pi,k (t) is the probability that the first machine is in state k at time t given that it
was at state i at time 0; Qj,l (t) is defined similarly for the second machine. By example
4.11 we have
µ1
λ1
P0,0 (t) =
+
e−(λ1 +µ1 )t ,
λ 1 + µ1 λ 1 + µ1
µ1
µ1
P1,0 (t) =
−
e−(λ1 +µ1 )t ,
λ 1 + µ1 λ 1 + µ1
2
and by the same argument,
λ1
µ1
+
e−(λ1 +µ1 )t ,
λ 1 + µ1 λ 1 + µ1
λ1
λ1
−
e−(λ1 +µ1 )t .
P0,1 (t) =
λ 1 + µ1 λ 1 + µ1
P1,1 (t) =
Of course, similar expressions for the second machine are obtained by replacing (λ1 , µ1 ) by
(λ2 , µ2 ). We then get P(i,j),(k,l) (t) by formula (??). For instance,
P(0,0),(0,0) (t) = P0,0 (t)Q0,0 (t) =
µ1 + λ1 e−(λ1 +µ1 )t µ2 + λ2 e−(λ2 +µ2 )t
·
.
λ 1 + µ1
λ 2 + µ2
3
6.12.1.
a. If the state is the number of individuals at time t, we get a birth and death processs
with
λn = nλ + θ,
n < N,
λn = nλ
n ≥ N,
µn = nµ.
b. Let Pi be the long-run probability that the system is in state i. Since this is also the
proportion of time the system is in state i, we are looking for
∞
!
Pi .
i=3
We have
Pk µk = Pk−1 λk−1 ,
k = 1, 2, . . . .
This yields
θ
1
P0 = P0 ,
µ
2
θ+λ
1
1
=
P1 = P1 = P0 ,
2µ
2
4
θ + 2λ
1
1
=
P2 = P2 = P0 ,
3µ
2
8
P1 =
P2
P3
and for k ≥ 3,
Pk =
Hence
Since
(k − 1)λ
k−11
3
Pk−1 =
Pk−1 = · · · =
kµ
k 2
k
∞
!
∞
!
3
Pk = P3
k=3
k=3 k
" #k−3
1
2
∞
!
1
= 24P3
k=3 k
" #k−3
1
2
" #k
1
2
∞
!
1 k
x = − log(1 − x),
k=1 k
we get
∞
!
k=3
Pk = P3 (24 log 2 − 15).
Because all probabilities add up to 1, we have
P0
"
#
1 1 1
1 + + + (24 log 2 − 15) = 1.
2 4 8
1
.
P3 .
So
1
P0−1 = 3 log 2 − ,
8
and thus finally,
∞
!
Pk =
k=3
24 log 2 − 15
≈ 0.105.
24 log 2 − 1
6.13.2. With the number of customers in the shop as the state, we get a birth and death process
with
λ0 = λ1 = 3,
µ1 = µ2 = 4.
Therefore
" #2
3
P1 = P0 ,
4
And since P0 + P1 + P2 = 1, we get
3
3
P2 = P 1 =
4
4
P0 =
P0 .
16
.
37
a. The average number of customers in the shop is
P1 + 2P2 =
30
.
37
b. The proportion of the customers that enter the shop is
λ(1 − P2 )
28
= 1 − P2 = .
λ
37
c. Now µ = 8, so
64
.
97
So the proportion of the customers that enter the shop is
P0 =
1 − P2 =
88
.
97
The rate of added customers is therefore
"
#
"
#
88
28
λ
−λ
≈ 0.45.
97
37
The business he does would improve by 0.45 customers per hour.
6.15.3. With the number of customers in the system as the state, we get a birth and death
process with
λ0 = λ1 = λ2 = 3,
λi = 0, i ≥ 4,
2
µ1 = 2, µ2 = µ3 = 4.
Therefore the balance equations reduce to
3
P1 = P0 ,
2
3
9
P2 = P1 = P0 ,
4
8
And therefore,
!
3 9 27
P0 = 1 + + +
2 8 32
3
27
P3 = P2 = P0 .
4
32
"−1
=
32
.
143
a. The fraction of potential custoemrs that enter the system is
λ(1 − P3 )
116
= 1 − P3 =
.
λ
143
b. With a server working twice as fast we would get
! "2
3
P1 = P0 ,
4
and
3
3
P2 = P1 =
4
4
#
! "2
3
3
P0 = 1 + +
4
4
+
So that now
1 − P3 =
P0 ,
P3 =
! "3 $−1
3
4
=
! "3
3
4
P0 ,
64
.
175
148
.
175
6.17.4. Say the state is 0 if the machine is up, say it is in state i when it is down due to a type
i failure, i = 1, 2. The balance equations for the limiting probabilities are as follows.
λP0
µ1 P1
µ2 P2
P0 + P1 + P2
=
=
=
=
µ1 P1 + µ2 P2 ,
λpP0 ,
λ(1 − p)P0 ,
1.
These equations are easily solved to give the results
P0 = (1 + λp/µ1 + λ(1 − p)/µ2 )−1 ,
P1 = λpP0 /µ1 ,
3
P2 = λ(1 − p)P0 /µ2 .
6.18.
1. There are k + 1 states: state 0 means the machine is working, state i means that it is
in repair phase i, i = 1, . . . , k. The balance equations for the limiting probabilities are
λP0 = µk Pk ,
µ1 P1 = λP0 ,
µi Pi = µi−1 Pi−1 ,
i = 2, . . . , k,
and the normalization equation is
P0 + · · · + Pk = 1.
To solve, note that
µi Pi = µi−1 Pi−1 = µi−2 Pi−2 = · · · = λP0 .
Hence,
Pi = (λ/µi )P0 ,
and, upon summing,
1 = P0
Therefore,
!
k
"
λ
P0 = 1 +
i=1 µi
#−1
,
!
#
k
"
λ
1+
.
i=1 µi
Pi = (λ/µi )P0 ,
i = 1, . . . , k.
a. Pi .
b. P0 .
2. The number in the system is a birth and death process with parameters
6.22.
λn = λ/(n + 1),
n ≥ 0,
µn = µ,
n ≥ 1.
From the equation above (6.20),
1/P0 = 1 +
∞
"
(λ/µ)n /n! = eλ/µ
n=1
and from (6.20),
Pn = P0 (λ/µ)n /n! = e−λ/µ (λ/µ)n /n!,
n ≥ 0.
6.23.
3. Let the state denote the number of machines that are down. This yields a birth and
death process with
λ0 =
3
,
10
λ1 =
2
,
10
λ2 =
1
1
,
10
λi = 0,
i ≥ 3,
and
1
µ1 = ,
8
The balance equations reduce to
2
µ2 = ,
8
2
µ3 = .
8
3/10
12
P0 =
P0 ,
1/8
5
4
48
2/10
=
P1 = P1 =
P0 ,
2/8
5
25
1/10
4
192
=
P2 =
P2 =
P0 .
2/8
10
250
P1 =
P2
P3
Hence, using P0 + P1 + P2 + P3 = 1, yields
!
P0 = 1 +
12 48 192
+
+
5
25 250
"−1
=
250
.
1522
a. Average number not in use is
P1 + 2P2 + 3P3 =
2136
1068
=
.
1522
761
b. Proportion of time both repairmen are busy is
P2 + P3 =
4.
a. π0 = 13 .
b. π0 = 1.
√
√
c. π0 = ( 3 − 1)/ 2.
2
336
.
761