10. Semi-Markov Processes
10.1 Introduction
10.2 Theoretical Developments
A. Finding Uij (t): Probability of being in state
B. Finding wij (t): Probability of leaving state
C. Marginal Probabilities
D. Matrix Relations and Solutions
E. Matrix Note
10.3 Simplified Model
345
10.4 First Passage Time Problems
A. Time from X0 = i to Absorption
B. Variance of Time to Absorption
C. Time to Go from any State j to being Absorbed Conditional on
X0 = i.
D. First Passage Time to Go from X0 = i to an Arbitrary
Absorbing State
E. Other First Passage Time Problems
10.5 Alternate Model: Semi-Markov II
346
10.1 Introduction
Example: Clinical Trial. Consider a clinical trial in which a patient can
be in one of several states when being followed in time; i.e.
-
Initial State
Partial Remission
Complete Remission
Progressive Disease
Relapse
Death
Death
Relapse
Remission
Initial State
X
>
0
347
Time
Example: Idealized Natural History(Chronic Disease)
S0 : Disease Free
Sp : Pre-clinical disease
Sc : Clinical disease
Sd1 : Death with disease
Sd2 : Death without disease
S0
−→
Sp
−→
↓
.
Sd2
348
Sc
↓
Sd1
Example: Early Detection (Progressive Disease Model)
S0
−→
Sp
−→
Sc (Clinical Disease)
Sp (Pre-clinical Disease)
Sc
X
S0 (Disease Free)
>
Time
Ex. Infectious Disease
S0 :
Disease free
SI : Infectious Disease
S0 ←→ SI
SI
S0
>
Time
349
Example: AIDS
S0 : Disease Free; SH : HIV Positive; SR : Remission
SA : AIDS; SD : Death
S0
−→
SH
−→
↓
&
SR
−→
SA
↓
SD
SD
SR
SA
SH
S0
X
>
Time
0
t = 0 corresponds to beginning of observation period.
Note person was in SH before beginning of observation period.
350
At time t, X(t) is defined as the state of the system. X(t) is right
continuous, i.e.
X(t) = lim X(t + ) for > 0
→
Also note that the system has an embedded Markov Chain with possible
transition probabilities P = (pij ). We will take pii = 0 for transient
states.
The system starts in a state X(0), stays there for a length of time, moves
to another state, stays there for a length of time, etc. This system or
process is called a Semi-Markov Process.
351
10.2 Theoretical Developments
Consider process as having n transient states and m absorbing states
or
Xn :
state at nth transition
X(t) :
state at time t
Tn :
Sojourn time for nth transition
History of process
Hn = {X0 , t0 ; X1 , t1 ; . . . ; Xn , tn }
Problems
Probability distribution of reaching a state
Probability distribution of returning to state
Probability distribution of time in state
First passage time probabilities, etc.
352
Assume:
P {Xn = j, t < Tn ≤ t+∆t|Hn−1 } = P {Xn = j, t < Tn ≤ t+∆t|Xn−1 }
Joint probability depends on History only through previous state
P (Xn = j, t < Tn ≤ t + ∆t|Xn−1 = i)
∆t→0
∆t
Since πij (t) does not depend on n (nth transition) the process is time
homogeneous.
Z ∞
πij (t)dt = P {Xn = j|Xn−1 = i} i 6= j, where pii = 0
pij =
πij (t) = lim
0
..˙
{Xn , n ≥ 0} form a Markov chain
P {t < Tn ≤ t + ∆t|Xn = j, Xn−1
353
πij (t)dt
= i} =
= qij (t)dt
pij
qij (t) : pdf of sojourn time in state j conditional on coming from state i.
πij (t)dt = pij qij (t)dt
πij (t) = pij qij (t)
qij (t) is a pdf of time spent in j conditional on previous transition being
in i. qij (t) may be defined by:
qij (t) = qj (t)
qij (t) = qi (t)
qij (t) = q(t)
If qij (t) = λij e−λij t the stochastic process is called a Markov Process.
Many authors define a Markov Process when λij = λj .
354
Remark: Sojourn time in state depends on current and previous state.
Note: Two Time Scales — Internal Time and External Time.
External Time: Time leaving or entering state
Internal Time: Time in a state or time to return to state
Def.
wij (t)dt = P {system leave state j during (t, t + dt)|X0 = i}
wij (t) is defined by external time
Z T
wij (t)dt = Expected no. of times system leaves state
Nij (T ) =
0
j conditional on X0 = i over (0, T )
Nij = lim Nij (t)
T →∞
= Expected no. of times system leaves state j over [0, ∞)
conditional on X0 = i.
355
Definition:
qoi (t): pdf of time spent in the initial state X0 = i
qij (t): pdf of time spent in state j conditional on coming from state i.
Qoi (t) = P {T0 > t|X0 = i} =
Z
∞
qoi (x)dx
t
Qij (t) = P {Tn > t, xn = j|x0 = i} =
Z
∞
qij (x)dx
t
Def.: Uij (t) = P {system in state j at time t|X0 = i}
Uij (t) is a function of external time; qoi (t), qij (t) are functions of
internal time.
356
A. Finding Uij (t)
Suppose x0 = j. Then to be in j at time t either:
(a) system never left x0 = j
j
0
Uii (t) = δij Uij (t) = Q0i (t)δij
t
>Time
or
(b) system left state k at time τ (wi (τ )), entered state j (pkj ) and stayed
at least (t − τ ) units of time Qkj (t − τ ); i.e. wik (τ )dτ pkj Qkj (t − τ ).
Then integrate over possible values of τ (0, t) and sum over all transient
states
Z t
n
X
pkj
wik (τ )Qkj (t − τ )dτ
k=1
0
357
Since (a) and (b) describe mutually exclusive events
Uij (t) = Q0i (t)δij +
n
X
pkj
k=1
Z
t
wik (τ )Qkj (t − τ )dτ
0
i, j = 1, 2, . . . , n
Taking LaPlace Transforms
∗
Uij
(s) = Q∗oi (s)δij +
n
X
∗
pkj wik
(s)Q∗kj (s)
k=1
for i, j = 1, 2, . . . , n (only over transient states)
358
B. Finding wij (t)dt
If x0 = j then to leave state j in interval (t, t + dt) either:
(a) system never left state i before time t and emerged for first time at
(t, t + dt); i.e. P {t < T0 ≤ t + dt|x0 = j}
wii (t) = δij wij (t) = qoi (t)dtδij
or
(b) system leaves state k at time τ (wik (τ )dτ ); enters state j (pkj ) and
stays there for (t − τ, t − τ + dt) time units (qkj (t − τ )dt). Hence
wij (t)dt
= qoi (t)dtδij +
n
X
k=1, k6=j
359
pkj
Z
t
wik (τ )qkj (t − τ )dτ dt
0
wij (t)
= qoi (t)δij +
n
X
pkj
k=1
Z
t
wik (τ )qkj (t − τ )dτ
0
i, j = 1, 2, . . . , n
Taking Laplace Transforms
∗
∗
wij
(s) = qoi
(s)δij +
n
X
∗
∗
pkj wik
(s)qkj
(s)
k=1
i, j = 1, 2, . . . , n
Remark: The equations in the time domain are two sets of coupled
integral equations. However in the S domain, we have two sets of linear
equations.
360
C. Marginal Probabilities
(1) Uij (t) = δij Qoi (t) +
t
X
pkj
k=1
(2) wij (t) = δij qoi (t) +
n
X
k=1
pkj
Z
Z
t
wik (τ )Qkj (t − τ )dτ
0
t
wik (τ )qkj (t − τ )dτ
0
i, j = 1, 2, . . . , n (Transient states only)
Suppose ai = P {X0 = i}. Then we can find
Uj (t) =
wj (t) =
n
X
ai Uij (t)
i=1
n
X
ai wij (t)
}
Marginal
Probabilities
i=1
361
Multiplying (1) by ai and summing over i
Uj (t) =
n
X
δij ai Qoi (t) +
i=1
Since
n
X
n
X
k=1
pkj
Z
t
wk (τ )Qkj (t − τ )dτ
0
δij ai Qoi (t) = aj Qoj (t)
i=1
Uj (t) = aj Qoj (t) +
n
X
pkj
k=1
Z
t
wk (τ )Qkj (t − τ )dτ, j = 1, 2, . . . , n
0
Similarly
wj (t) = aj qoj (t) +
n
X
k=1
pkj
Z
t
wk (τ )qkj (t − τ )dτ, j = 1, 2, . . . , n
0
0
wi∗ = D(q0∗ )ei + π ∗ wi∗
362
Consider n = 3 and i = 2


 

∗
∗
0
q01 (s)
wi1 (s)
0
0


 

∗
 
w∗ (s) =  0
0 
q02 (s)

  1
 i2 
∗
∗
0
(s)
(s)
wi3
0
0
q03

 
∗
∗
∗
wi1
0
p21 q21 (s) p31 q31

 
∗
∗
 ∗

+ p12 q12 (s)
0
p32 q32 
 wi2 
∗
∗
∗
p13 q13
(s) p23 q23
(s)
0
wi3


P3
∗
∗
j=1 pj1 qj1 (s)w2j (s)


P3
∗
∗
∗
∗

wi = q02 (s) + j=1 pj2 qj2 (s)w2j (s)

P3
∗
∗
p
q
(s)w
j3
j3
2j (s)
j=1
Similarly
0
Ui = D(Q∗0 )ei + Φ∗ wi∗
363
D. Matrix Relations and Solutions
Consider the S - domain equations:
∗
Uij
(s) = Q∗oi (s)δij +
n
X
∗
pkj wik
(s)Q∗kj (s)
k=1
∗
∗
(s) = qoi
(s)δij +
wij
n
X
∗
∗
pkj wik
(s) qkj
(s)
k=1
364
Define:
Ui∗
=
∗
∗
∗
(Ui1
(s), Ui2
(s), . . . , Uin
(s))0
wi∗
=
∗
∗
(wi1
(s), . . . , win
(s))0
Q∗o
=
(Q∗o1 (s), . . . , Q∗on (s))0
P
=
(pij ), Φ∗ij (s) = pij Q∗ij (s), Φ∗ = (Φ∗ij )
∗
(s) =
πij
ei
=
∗
pij qij
(s), π ∗ = (πij (s))
(0, 0, . . . , 1, 0 . . . 0)0 1 in ith position
0
Ui∗ = D(Q∗0 )ei + Φ∗ wi∗
⇒
wi∗
0
wi∗ = D(q0∗ )ei + π ∗ wi∗
=
D(q0∗ )ei
∗0
+ π w∗
∗0
(I − π )wi∗ = D(qo∗ )ei
365
0
Def. N ∗ = (I − π ∗ )−1
wi∗
∗0 −1
D(q0∗ )ei = N ∗ D(q0∗ )ei
Z ∞
=
e−st wij (t)dt
= (I − π )
∗
wij
(s)
∗
(0)
wij
0
=
Z
∞
wij (t)dt =
0
Expected no. of visits to j
conditional on x0 = i
..˙ putting s = 0 in: wi∗ , D(q0∗ ), N ∗
wi∗ (0) = N ∗ (0)ei
as
∗
q0i
(0) = 1, N ∗ (0) = (I − P 0 )−1
i.e.
0
N ∗ (0) = [I − π ∗ (0)]−1
,
∗
π ∗ (s) = (pij qij
(s))
π ∗ (0) = P, π ∗ (0)0 = P 0
..˙ N (0) = (I − P 0 )−1
366
Define
Nij
ei
N = (Nij )
= mean number of times j is visited where x0 = i
= column vector of zeros except the ith element is unity
= (I − P 0 )−1
and
e0i N = (Ni1 Ni2 . . . Nin ) = Ni0
Therefore

N
 i1

 Ni2
∗
0 −1
wi (0) = (I − P ) ei = 
 ..
 .

Nin
367




 = Ni



wi∗ = N ∗ D(q0∗ )ei
Ui∗
∗
= D(Q∗0 )ei + Φ0 wi∗ , Φ∗ (pij Q∗ij (s))
∗
= D(Q∗0 )ei + Φ0 N ∗ D(q0∗ )ei
∗
Ui∗ = [D(Q∗0 ) + Φ0 N ∗ D(q0∗ )]ei
Recall:
∗
Uij
(s)
∗
(0)
Uij
=
Q∗oi (0) =
Z
Z
=
Z
∞
Uij (t)e−st dt
0
∞
Uij (t)dt = mean time spent in j conditional on X0 = i
0
∞
Qoi (t)dt = moi
0
Φ∗ (s) = (pij Q∗ij (s)), Φ∗ (0) = (pij mij ), where mij is mean of qij (t).
368
Ex. n = 3, X0 = 2, e2 = (0 1 0)0
∗
U2∗ (0) = [D(m0 ) + Φ0 (0)N ∗ (0)]e2

∗
(0)
U21



U ∗ (0)
 22 
∗
(0)
U23

m01

=
 0
0

0
m02
0
0
p21 m21

+
p12 m12
p13 m13
as
N ∗ (0)ei =

N11

N12

N13
 
0
0
 
 
0 
 1
m03
0
N21
N22
N23
0
p23 m23
p31 m31

N21




p32 m32  N22 

0
N23
  

N21
0
N31
  





N32  1 = N22 

0
N33
N23
369



3
X
 pj1 mj1 N2j 


j=1


∗
0
U21
(0)
X

 mean time in state j = 1,2,3
 
  3

U ∗ (0) = m02  +  pj2 mj2 N2j  =

 
 22  
starting from X0 = 2


j=1
∗


0
U23 (0)
3
X

 p m N 




j3
j3
j=1
370
2j
Interpretation
∗
U21
(0) =
3
X
pj1 mj1 N2j = mean time spent in state 1 if X0 = 2.
j=1
mj1 N2j = (meantime in 1 coming from j)
×(Expected no. of visits from 2 to j)
pj1 mj1 N2j = E[Total no. of visits to j starting from state 2]
×Probability {j → 1}
×mean time in state 1 coming from j
∗
(0) = m02 +
U22
3
X
pj2 mj2 N2j
j=1
Note additional term because X0 = 2.
371
Homework
Consider the 3 state model with transition probabilities

0



P = β


0
α
0
0
1−α




1−β 


1
States 1 and 2 are transient states and state 3 is an absorbing state.
Define qij (t) = qj (t) = λj e−λj t
Find Uij (t) and wij (t) for i, j = 1, 2.
372
E. Matrix Note
All results depended on the inversion of [I −
∗0
πij (s)].
Does inverse exist?
Consider matrix (n × n) A.
Ln
= (I − A)(I + A + A2 + . . . + An−1 )
= (I + A + . . . + An−1 ) − (A + A2 + . . . + An ) = I − An
If An → 0 as n → ∞ ⇒ Ln → I and (I − A)−1 = I + A + A2 + . . .
exists.
∗0
∗
∗
) = (pij qij
(s))
Now consider (I − π ), π ∗ = (πij
Z ∞
∗
∗
∗
(s) =
e−st qij (t)dt, |qij
(s)| ≤ 1 ..˙ |πij
| ≤ pij
Since qij
0
(n)
(pij ) →
But P n =
0 as n → 0 as there exists at least one absorbing
state and with prob = 1, system will eventually be in absorbing state.
373
10.3 Simplified Model
Suppose qij (t) = qj (t) and q0j (t) = q(j(t); i.e. sojourn time only
depends on the state process is in. Then
πij (t)
= pij qij (t) = pij qj (t)
Φij (t)
= pij Qij (t) = pij Qj (t)

0
p12 q2 (t) p13 q3 (t)


0
p23 q3 (t)
 p21 q1 (t)
=
..


.

π(t) = πij (t))
pn1 q1 (t)
pn2 q2 (t)
374

...
p1n qn (t)
...


p2n qn (t)




0
...


π(t) = P D(q), D(q) = 



∗
∗

and π = P D(q )

Similarly
q1 (t)
0 ...
0
..
.
q2 (t) . . .
0
...
0


0 




qn (t)
Φ(t) = P D(Q), Φ∗ (s) = P D(Q∗ )
π ∗ = P D(q ∗ ),
Φ∗ = P D(Q∗ )
D(q0∗ ) = D(q ∗ ), D(Q∗0 ) = D(Q∗ )
∗
Since wi∗ = (I − π 0 )−1 D(q0∗ )ei
we can write
wi∗ = [I − D(q ∗ )P 0 ]−1 D(q ∗ )ei
375

Consider
[I − D(q ∗ )P 0 ]−1 D(q ∗ )
= [I + DP 0 + (DP 0 )2 + . . . ]D
= D[I + P 0 D + (P 0 D)2 + . . . ]
= D∗ (q ∗ )[I − P 0 D(q ∗ )]−1 = D(q ∗ )M ∗
where
M∗
= [I − P 0 D(q ∗ )]−1
..˙ wi∗ = D(q ∗ )M ∗ ei
Similarly
∗
Ui∗ = D(Q∗0 )e∗i + Φ0 wi∗
= D(Q∗ )ei + D(Q∗ )P 0 D(q ∗ )M ∗ ei
= D(Q∗ )[I + P 0 D(q ∗ )M ∗ ]ei
376
where
But
Ui∗
= D(Q∗ )[I + P 0 D(q ∗ )M ∗ ]ei
M∗
= [I − P 0 D(q ∗ )]−1
P 0 DM ∗
= P 0 D[I − P 0 D]−1 = P 0 D[I + P 0 D + (P 0 D)2 + . . . ]
P 0 DM ∗
= P 0 D + (P 0 D]2 + (P 0 D]3 + . . .
= [I − P 0 D]−1 − I = M ∗ − I
..˙ Ui∗ = D(Q)∗ M ∗ ei
..˙ When the sojourn time only depends on the state the process is in we
also have
wi∗ = D(q ∗ )M ∗ ei .
377

Ni1



 Ni2

∗
0 −1
Then wi (0) = (I − P ) ei = 
 .
 .
 .


Nin

m
 1


∗
Ui (0) = D(m)Ni = 

 0

0
m2
..
.
378






 = Ni









N
m N
  i1   1 i1 

 

  Ni2   m2 Ni2 

 

..
  ..  = 

 .  

.

 

Nin
mn Nin
mn
First Passage Time Problems
A. Time from X0 = i to being absorbed
To simplify the problem, suppose that there is only one absorbing state
which is n + 1.
Define Ti = random variable to Absorption conditional on X0 = i.
Gi (t) = P {Ti > t|X0 = i} = P {TX0 > t|X0 = i}
Ui,n+1 (t) = Prob. system is in state n + 1 at time t.
1 − Ui,n+1 (t) = Prob. system is not in (n + 1) at t
⇒
Gi (t) = 1 − Ui,n+1 (t) =
n
X
Uik (t)
k=1
as
n
X
Uik (t) + Ui,n+1 (t) = 1
k=1
379
Gi (t) =
n
X
Uik (t)
k=1
G∗i (s) =
n
X
∗
Uik
(s)
k=1
Since G∗i (0) =
Z
∞
Gi (t)dt =
0
..˙
∗
Uij
(0) = mean time to Absorption
j=1
But we had found
∗
(0)
Uij
n
X
= mean time in j starting from X0 = i
n
X
pkj mkj Nik
= δij moi +
G∗i (0) =
k=1
n X
n
X
pkj mkj Nik + moi
j=1 k=1
380
If mkj = mj ,
G∗i (0) =
n
X
mj
j=1
Recall
∗
(s)
wij
∗
(0)
wij
or equivalently
Nij
pkj Nik + moi
k=1
∗
= qoi
(s)δij +
= δij +
=
n
X
n
X
n
X
∗
∗
pkj wik
(s)qkj
(s)
k=1
∗
pkj wik
(0)
k=1
pkj Nik i 6= j
k=1
Nii
n
X
=1+
n
X
pki Nik
k=1
G∗i (0) =
n
X
j=1
mj
n
X
k=1
381
pkj Nik + moi
If mkj
= mj
n
X
=
pkj Nik i 6= j
Nij
k=1
=1+
Nii
..˙
G∗i (0) =
n
X
pkj Nik
k=1
" n
#
n
n
X
X
X
mj
pkj Nik + mi
pki Nii + moi
j=1,j6=i
=
X
k=1
k=1
mj Nij + mi [Nii − 1] + moi
j6=i
G∗i (0) =
n
X
mj Nij + (moi − mi )
j=1
If moi = mi ,
G∗i (0) =
n
X
mj Nij
j=1
382
B. Variance of Time to Absorption
G∗i (s) =
n
X
∗
Uij
(s), Gi (t) = P {Ti > t|X0 = i}
j=1
If gi (t) is pdf of time to Absorption with X0 = i, we know that
s2
1 − [1 − sm1 + m2 + · · · ]
∗
1
−
g
s
(s)
i
∗
2
=
= m1 − m2 + O(s2 )
Gi (s) =
s
s
2
m2
dG∗i (s)
=−
+ O(s)
ds
2
∗ dGi (s)
and taking s = 0
⇒ m2 = −2
ds
s=0
..˙ V ar Ti = m2 − m21 = −2G∗i (0) − [G∗i (0)]2
We have found
s=0
G∗i (0),
dG∗t (s)
evaluated at
it is only necessary to find −2
ds
383
To simplify problem we will consider the special case
mkj = mj , moj = mj
Then
Ui∗ = D(Q∗ )M ∗ ei , M ∗ = [I − P 0 D(q ∗ )]−1
∗
d
dUi∗
∗
∗
∗ dM
= −2 D(Q )M ei − 2D(Q )
ei
−2
ds
ds
ds
dQ∗j (s)
=second moment of qj (t) = (σj2 + m2j ), s = 0
Recall −2
ds


σ12 + m21




2
2
σ
+
m
0


d
2
2
∗
2
2


..˙ −2 D(Q )s=0 = D(σ +m ) = 

.
ds
.


.
0


σk2 + m2k
384
dM ∗
∗
∗−1
, consider M M
To evaluate
=I
ds
−1
d
dM ∗ ∗−1
M
+ M∗ M∗ = 0
ds
ds
∗
−1
dM
d
= −M ∗
M∗
M∗
ds
ds
!
∗−1
dM
∗−1
0
∗
Since M
= I − P D(q ),
= +P 0 D(m)
ds
s=0
∗
−1
d ∗
dM
= −M ∗
M
M∗
ds
ds
and setting s = 0
∗
dM
= −(I − P 0 )−1 P 0 D(m)(I − P 0 )−1 = −N P 0 D(m)N
ds 0
385
dUi∗
d
∗
∗
∗ dM
= −2 D(Q )M , − 2D(Q )
..˙ − 2
ds
ds
ds
∗
ei
Setting s = 0




Now N ei = 



Ni1
Ni2
..
.
Nik
= [D(σ 2 + m2 )N + 2D(m)N P 0 D(m)N ]ei








dUi∗
= [D(σ 2 + m2 ) + 2D(m)N P 0 D(m)]Ni
−2
ds
Also G∗i (0) =
X
Ui∗ (0) =
N
X
Ni mi = Ni0 m
1
386
..˙ V arTi = −2
=
n
X
X dU ∗ (s) i
ds
σj2 Nij +
j=1
N
X
− m0 Ni Ni0 m
s=0
m2j Nij + 2m0 [N − I]D(m)Ni − m0 Ni Ni0 m
j
2
as N P 0 = (I − P 0 )−1 P 0 = P 0 + P 0 + . . . = N − I
..˙ V arT0 =
X
σj2 Nij +
X
m2j Nij + 2m0 [N − I]D(m)Ni − m0 Ni Ni0 m
j
Simplifying m0 [N − I]D(m)Ni
387
D(m)Ni
m0 [N − I]D(m)Ni
= D(Ni )m
= m0 N D(Ni )m − m0 D(Ni )m
Note: m0 D(Ni )m =
n
X
m2j Nij
j=1
V arTi
=
=
V arTi =
X
P
P
σj2 Nij + 2m0 N D(Ni )m −
− m0 Ni Ni0 m
j
P
j
m2j Nij
σj2 Nij + m0 [2N − I]D(Ni )m − m0 Ni Ni0 m
σj2 Nij + m0 {[2N − I]D(Ni ) − Ni Ni0 }m
j
388
C. Time to go from any state j to being absorbed conditional on X0 = i
Earlier we had found the prob. distribution of being absorbed starting out
from X0 = i at time 0.
Now we wish to generalize the result of finding the time to Absorption for
an arbitrary state j with X0 = i.
Define
gj,n+1 (t) = gj (t) = pdf of time to being absorbed beginning with the
time the system enters state j.
389
There are two ways of going from j to the absorbing state n + 1; i.e.
(a) The state after j is the absorbing state. Hence the time to being
absorbed is Tj (time spent in j), or
(b) The state after j is another transient state r and the time to being
absorbed is Tj + Tr,n+1 where Tr,n+1 is the time to Absorption from
state r.
Assume that the state prior to entering state j is state k.
gj,n+1 (t) = gj (t)
=
n
X
pkj qkj (t)pj,n+1 +
k=1 r=1
k=1
gj (t) = pj,n+1 qj (t) +
n
X
r=1
where qj (t) =
n
X
n X
n
X
pjr
Z
pkj
Z
t
qkj (τ )pjr gr (t − τ )dτ
0
t
qj (τ )gr (t − τ )dτ
0
pkj qkj (t)
k=1
390
gj (t) = pj,n+1 qj (t) +
n
X
r=1
where qj (t) =
n
X
pjr
Z
t
qj (τ )gr (t − τ )dτ
j = 1, . . . , n
0
pkj qkj (t).
k=1
Note that before entering j the process was in state k. Thus the pdf of the
stay in j is qkj (t). However it came from state k with prob. pkj . It is also
necessary to sum over all possible values of k. This leads to the marginal
distribution qj (t).
Taking LaPlace Transforms results in
gj∗ (s) = pj,n+1 qj∗ (s) +
n
X
r=1
for j = 1, . . . , n.
391
pjr qj∗ (s)gr∗ (s)
Substituting pj,n+1 = 1 −
n
X
pjr , subtracting 1 from both sides and
r=1
multiplying by 1/s results in
1−
gj∗ (s)
s
or
=
1−
qj∗ (s)
s
G∗j (s) = Q∗j (s) + qj∗ (s)
+ qj∗ (s)
n ∗
X
1 − g (s)
r
r=1
n
X
s
pjr
pjr G∗r (s)
r=1
where we have written G∗j (s) for G∗j,n+1 (s)
Writing G∗n+1 = (G∗1 (s), G∗2 , . . . , G∗n (s))0 we have
G∗n+1 = Q∗ + D(q ∗ )P G∗n+1
or [I − D(q ∗ )P ]G∗n+1 = Q∗
Earlier we had defined M ∗ = [I − P 0 D(q ∗ )]−1
∗
..˙ G∗n+1 = [I − D(q ∗ )P ]−1 Q∗ = M 0 Q∗
392
∗
G∗n+1 = M 0 Q∗
,
∗
M 0 = [I − D(q ∗ )P ]−1
Since mj,n+1 = mean time to go from j to absorbing state we can write
G∗n+1 (0) = mn+1 = (I − P )−1 m
where mn+1 = (m1,n+1 , m2,n+1 , . . . , mn,n+1 )0
and m = (m1 , m2 , . . . , mn ).
Furthermore (I − P )−1 = (Nij ) = (Expected no. of visits from i to j ).
Therefore
mj,n+1 =
n
X
Njr mr
r=1
Thevariance
of the time to be absorbed from j can be found from
dG∗
− G∗ (0)2
−2
ds s=0
393
D. First passage time to go from X0 = i to an arbitrary Absorption state
Consider n transient states and m absorbing states. The transient states
will be written as the first n states. Assume X0 = i and it is desired to
find the first passage time starting from X0 = i to being absorbed by state
r, conditional on reaching state r.
Define
fir = Prob. of being absorbed by state r
fir
conditional on X0 = i
n
X
X
X
= pir +
pij pjr +
pik pkj pjr +
pil plk pkj pjr
= pir +
j=1
X
j
pij pjr +
j,k
X
j
j,k,l
X
(2)
(3)
pij pjr +
pij pjr + . . .
j
394
fir = pir +
n
X
pij pjr +
j=1
(m)
where pij
(m)
pij
X
(2)
pij pjr
j
+
X
(3)
pij pjr + . . .
j
= Prob. of i → j in m steps
is i, j th element of P m .
Hence if
fr =
(f1r , f2r , . . . , fnr )0
R=
(p1r , p2r , . . . , pnr )0
P =
(pij ) i, j = 1, 2, . . . , n
fr =
R + P R + P 2 R + . . . = (I + P + P 2 + . . . )R
fr = (I − P )−1 R
i.e. fir =
n
X
Nij pjr
j=1
fir = e0i (I − P )−1 R = R0 (I − P 0 )−1 ei
e0i (I − P )−1 = (Ni1 , Ni2 , . . . , Nim )
395
Define
gir (t) : pdf of time to being absorbed in state r with X0 = i,
and conditional on being absorbed in state r.
gir (t) =
n
X
wij (t)pjr /fir = wi0 (t)R/fir
j=1
where wi (t) = (wi1 (t), wi2 (t), . . . , win (t))0
R = (p1r , p2R , . . . , pnr )0
∗
gir
(s) = wi∗ (s)0 R/fir = R0 wi∗ (s)/fir
where wi∗ (s) = N ∗ D(q0∗ )ei
n
0 ∗
R
wi (0) X Nij pjr
∗
Note: gir (0) =
=
=1
fir
f
ir
j=1
396
E. Other First Passage Time Problems
1. Suppose i, j ∈ T (T = Transient States)
Define T 0 = (n − 1) transient states omitting j. A is set of absorbing
states.
fij = Prob. of i → j (Probability of eventually reaching j from i)
X
X
pik pkj +
pik pkl plj + . . . .
fij = pij +
k∈T 0
k,l∈T 0
Partition the transition matrix P
P =
T0
j
A
T0
P̄
α
γ
j
β0
0
δ
A
0
0
I
α0 = (p1j , p2j , . . . , pnj )
,
omit pjj
β 0 = (pj1 , pj2 , . . . , pjn )
397
Define e0i = (n − 1) × 1 vector with i in the ith place and 00 s
everywhere else.
pij
= e0i α
fij
= e0i α + e0i P̄ α + e0i P̄ 2 α + . . .
= e0i (I − P̄ )−1 α
gij (t) =
X
wik (t)pkj /fij
k∈T 0
398
2. First passage time to return to j
X
gjj (t) =
wjk (t)pkj /fjj
k∈T 0
fjj = Prob. of visiting j after initial visit
X
X
pjk pkj +
pjk pkl plj + . . .
fjj =
k∈T 0
k,l,∈T 0
fjj = β 0 α + β 0 P̄ α + β 0 P̄ 2 α + . . . = β 0 (I − P̄ )−1 α
399
10.5 Alternate Model: Semi-Markov II
Suppose time in state depends on current state and the next state. Our
models up to now have assumed time in state depends on current state and
state before current state; i.e.
P {t < TXn ≤ t + dt, Xn = j|Xn−1 = i} = πij (t)dt.
Now assume
P {t < TXn ≤ t + dt, Xn+1 = j|Xn = i} = πij (t)dt
P {t < TXn ≤ t + dt|Xn = i, Xn+1 = j} = qij (t)dt
πij (t) = pij qij (t)
qij (t): refers to the pdf of time in i where j is the next transition.
400
(0)
Suppose X0 = i. If next state is k, then time spent in i is qik (t) where
the (0) indicates the initial state with probabiblity pik .
Define Uij (t), wij (t) as before where i, j ∈ Transient states.
Uij (t) =
+
n X
n Z
X
"
t
n
X
(0)
#
pik Qik (t) δij
k=1
wik (τ )pkj Qjl (t − τ )pjl dτ
k=1 l=1 0
Initial state X0 = i: If j = i, δij = 1 and Q0ik (t) where k is next state
with Prob. = pik .
Other: At time t, system is in state j. Hence at time < t, it entered j from
a state k and left state k at time τ (τ < t). Also the state after j is l with
Prob. pj l .
401
Uij (t) =
"
n
X
(0)
#
pik Qik (t) δij +
k=1
n
X
pkj
k=1
Z
t
wik (τ )
0
" n
X
#
pjl Qjl (t − τ ) dτ
l=1
Define
Qi0 (t) =
n
X
(0)
pik Qik (t)
Qj (t) =
k=1
Then
X
pj l Qj l (t)
l=1
Uij (t) = δij Qi0 (t) +
n
X
pkj
k=1
Z
t
wik (τ )Qj (t − τ )dτ
0
These are same equations as our other model except the sojourn time need
only be marginal distributions.
Taking LaPlace Transforms
∗
Uij
(s) = δij Q∗i0 (s) + Q∗j (s)
n
X
k=1
402
∗
pkj wik
(s)
Similarly we have
wij (t) = δij qi0 (t) +
X
pkj
k
Z
t
wik (τ )qj (t − τ )dτ
0
resulting in
∗
(s) + qj∗ (s)
wij (s) = δij qi0
n
X
∗
pkj wik
(s)
k=1
Hence in matrix notation
wi∗ = D(q0∗ )ei + D(q ∗ )P 0 wi∗
[I − D(q ∗ )P 0 ]wi∗ = D(q0∗ )ei
wi∗ = N ∗ D(q0∗ )ei ), N ∗ = [I − D(q ∗ )P 0 ]−1
403
wi∗ = N ∗ D(q0∗ )ei ), N ∗ = [I − D(q ∗ )P 0 ]−1
∗
Uij
(s) = δij Q∗io (s) + Q∗j (s)
n
X
∗
pkj wik
(s)
k=1
or in matrix notation Ui∗ = D(Q∗0 )ei + D(Q∗ )P 0 wi∗
Substituting for w ∗
Ui∗ = D(Q∗0 )ei + D(Q∗ )P 0 N ∗ D(q0∗ )ei
Now if Q∗0i (s) = Q∗i (s)
Ui∗ = [D(Q∗ ) + D(Q∗ )P 0 N ∗ D(q ∗ )]ei
Ui∗ = D(Q∗ )[I + P 0 N ∗ D(q ∗ )]ei
404
Since
P 0 N ∗ D(q ∗ ) = P 0 [I + DP 0 + (DP 0 )2 + . . . ]D
= P 0 D + (P 0 D)2 + . . . = (I − P 0 D)−1 − I = M ∗ − I
Ui∗ = D(Q∗ )M ∗ ei , M ∗ = [I − P 0 D(q ∗ )]−1
Ui∗ = D(Q∗ )M ∗ ei , M ∗ = [I − P 0 D(q ∗ )]−1
Now
wi∗ = N ∗ D(q0∗ )ei ,
N ∗ = [I − D(q ∗ )P 0 ]−1
Consider
N ∗D
= [I + DP 0 + (DP 0 )2 + . . . ]D
= D[I + P 0 D + . . . ] = D(I − P 0 D)−1
N ∗ D(q ∗ )
Therefore
= D(q ∗ )M ∗
wi∗ = D(q ∗ )M ∗ ei
405
∗
when q0i
= qi∗