STA 348
Introduction to
Stochastic Processes
Lecture 19
1
CMC Transition Probability
Function

CMC described by two sets of quantities:



vi : exponential rate of leaving state i
Pij : probability of going from state i to state j,
after leaving state i
Based on these, want to find probability of
going from i to j after some time t, called the
transition probability function Pij(t)
Pij (t )  P  X (t )  j | X (0)  i 
 P  X (t  s)  j | X ( s )  i  , s
2
CMC Transition Probability
Function
Probability of being in state j
at time t, starting from i :
Pij (t )
j
i
t
0
● What is

j
(time)
Pij (t ) ?
3
Pure Birth Process Transition
Probability Function

For pure birth process, transition probability
function is straightforward to calculate:


Birth rates λi=vi , death rates µi=0 → Pi,i+1=1
Let Ti be the iid Exp(λi) time it takes for process to
go from state i to i+1
Ti
Ti 1
i2
i 1
i
0
(time)
4
Pure Birth Process Transition
Probability Function

Starting from state i, process will be in some
state ≤ j ( j ≥ i) at time t, only if there are less
than j−i # transitions between time [0,t]

j

Thus, P  X (t )  j | X (0)  i   P  k i Tk  t  , and from


this we can readily find Pij(t)
Ti
Ti 1
Tj
j
…
i
0
t
(time)
5
Example

Find the transition probability function for a
Poisson process with rate λ
6
Chapman-Kolmogorov
Equations


For general CMC, need to solve a set of
differential equations to find Pij(t)
Start with Chapman-Kolmogorov equations
Pij (t  s )   k Pik (t )  Pkj ( s ) , i, j & s, t  0

Proof:
7
Instantaneous Transition
Rates

Define quantities qij, called the instantaneous
transition rates of the CMC, as qij  vi Pij , i, j


They represent the rate at which the process
switches states over a small (~0) period of time
For any instantaneous rates qij, we have




qij   j vi Pij  vi
j
qij

j
qij

vi Pij
vi
 Pij
→ rates uniquely determine the CMC
8
Instantaneous Transition
Rates

We can show (somewhat informally) that
1  Pii (h)
 vi
h
Pij ( h)
 lim
 qij
h 0
h
 lim
h 0
9
10
Kolmogorov’s Backward
Equations

For all states i, j and times t ≥ 0, we have
Pij(t )   k i qij Pkj (t )  vi Pij (t )

dPij (t ) 


 where Pij (t ) 

dt


Proof:
11
Example

Find backward eqn’s of Birth & Death process
12
Example

Machine works for Exponential(λ) time until it
breaks down, and it takes Exponential(µ)
time to fix it. If machine is working at time 0,
find probability it will be working at time 10.
13
Example (cont’d)
14
Example (cont’d)
15
Kolmogorov’s Forward
Equations

For all states i, j and times t ≥ 0, we have
Pij (t )   k  j Pik (t )qkj  Pij (t )v j


Note: Forward equations don’t hold for all CMC’s, but do
hold for all Birth & Death and finite state-space CMC’s
Proof:
16
Example

Find Pij(t) for machine with Exp(λ) work time
& Exp(µ) repair time using forward eqn’s
17
Example

Find forward eqn’s of Birth & Death process
18
Example

Find forward eqn’s for pure birth process,
and show that
 Pii (t )  e  it , i  0


 jt t  j s
 Pij (t )   j 1e 0 e Pi , j 1 ( s )ds , j  i  1
19
Example (cont’d)
20