Markov Chains
Ali Jalali
Basic Definitions
Assume Si s as states and Cis as happened states.

 
P Ci  Si Ci 1  S j , Ci 2  Sk ,..., C1  S z  P Ci  Si Ci 1  S j
For a 3 state Markov model, we construct a transition matrix.
 P At | At 1 
Ttt1   P At | Bt 1 
 P At | Ct 1 
PBt | At 1 
PBt | Bt 1 
PBt | Ct 1 
PCt | At 1  
PCt | Bt 1 
PCt | Ct 1 

Second Order Probability
Two Level Transition Probabilities would be
P Ci  Si | Ci 2  Sk   P Ci  Si | Ci 1  S1 .P Ci 1  S1 | Ci 2  Sk  
P Ci  Si | Ci 1  S2 .P Ci 1  S2 | Ci 2  Sk  
P Ci  Si | Ci 1  S3 .P Ci 1  S3 | Ci 2  Sk 
So two level transition matrix would be
Ttt2
 P  At | At 1 
  P  At | Bt 1 
 P  At | Ct 1 
T2
PBt | At 1 
P Bt | Bt 1 
P Bt | Ct 1 
P Ct | At 1    P  At 1 | At 2 
PCt | Bt 1   P  At 1 | Bt 2 
P Ct | Ct 1   P  At 1 | Ct 2 
P Bt 1 | At 2 
PBt 1 | Bt 2 
P Bt 1 | Ct 2 
PCt 1 | At 2  
P Ct 1 | Bt 2 
PCt 1 | Ct 2 
nth Order Markov Transition Matrix
t
n
Transition matrix of nth order would be Tt n  T
So for the probability of being in a certain state, we have
P Ci  Si   P Ci  Si C1  S1 P C1  S1  
P Ci  Si C1  S 2 P C1  S 2  
P Ci  Si C1  S3 P C1  S3 

P n   T n .P 0
Some Linear Algebraic Definitions
 Eigen Values
v0 Av  v

A  I v  0

det A  I   0
 Eigen Vectors
Those v' s !
 Characteristic Polynomial
Pn    det A  I 
 Matrix Representation 1 0 ... 0   v1 
0  ... 0   v 
2
 2 
A  V  [ v1 v 2 ... v n ]
...
 ... 

 
0
0
...

n v n 

V  I  A2  VV  VI  V2 
State Probabilities
By means of eigen values & vectors, It is true that
T  V
T n  Vn 

It could be easily proven that

i
 

i2  1   j i 2j  1

n  cte

T n  cte
Finally, we have
PCi  Si   lim n Tiin  lim n T n  cte
Thanks for Patients