CS723 - Probability
and
Stochastic Processes
Lecture No. 33
Markov Chains
Yn is our total holdings after n
rounds of coin-toss
Current state of the chain is given by
Yn  100  X1  X2    Xn
Markov Chains
The situation near zero state is
different and is given below
Markov Chains
The state transition diagram for a
stochastic process with 2 states
Markov Chains
The state transition diagram for a
stochastic process with 2 states
Markov Chains
The state transition diagram for
year-after-year growth of
descendents of a famous person
Markov Chains
The state transition diagram for a
year-after-year status of a
Muslim aspiring to perform Hajj
Markov Chains
Typical values of state transition
probabilities for a Muslim
aspiring to perform Hajj
Markov Chains
The state transition diagram, initial,
and transitional probabilities
P(1,1)  Pr( X n  1, X n1  1)  1  p
P(0,1)  Pr( X n  0, X n 1  1)  1
P(1,0)  Pr( X n  1, X n 1  0)  p
(0)  Pr( X 0  0)
(1)  Pr( X 0  1)
Markov Chains
Pr(X n1  0)  Pr(X n  0, X n1  0)
 Pr(X n  1, X n1  0)
 Pr(X n  0) P(0,0)
 Pr(X n  1) P(1,0)
 Pr(X n  1)(p)
 1  Pr(X n  0)(p)
Markov Chains
Pr(X 1  0)  1  (0)(p)  p  p(0)
Pr(X 2  0)  1  Pr(X 1  0)(p)  1  p  p(0)(p)
 p  p 2  p 2 (0)
Pr(X 3  0)  1  Pr(X 2  0)(p)


 1   p  p  p (0) (p)
2
2
 p  p  p  p (0)
2
3
3
n1
Pr(X n  0)  (p)n (0)  p (p)k
k 0
Markov Chains
n1
Pr( X n  0)  ( p)n (0)  p ( p)k
k 0
n

1  ( p) 
n
 ( p) (0)  p 

 1 p 
p
p 
n

 ( p)  (0) 

1 p
1 p 

1
1 
n
Pr( X n  1) 
 ( p)  (1) 

1 p
1 p 
