Markov Chains
P{ X n +1 = j| X n = i , X n −1 = in −1 , X n − 2 = in −1 ,...., X1 = i1 , X 0 = io } = Pij
for all io …,in-1, i, j,
Pij, 0 ≤ i ≤M, 0 ≤ j ≤ N, are called transition probabilities of the Markov chain
Pij ≥ 0
M
∑P
ij
=1
i = 0,1,2,.... M
j =1
:
P00
P10
M
PM0
P01
P11
L
L
P0 M
P1M
PM1 L PMM
P{Xn = in, X n-1 = in-1,…,X1=i1,X0=io}
= P{Xn = in|Xn-1=in-1,…,X0=i0}P{Xn-1 = i n-1,…,X0=i0}
= Pi ,i P{X n −1 = i n −1 ,...,X 0 = i o}
n-1
n
= Pin−1 ,in Pin−2 ,i n−1 L Pi1 ,i 2 Pi o ,i1 P{ X 0 = i0 }
Example:
Suppose that whether or not it rains tomorrow depends on previous weather
conditions only through whether or not it is raining today. Suppose further tha if it is
raining today, then it will rain tomorrow with probability α, and if it is not rasining
today, then it will rain tomorrow with probability β.
If we say that the system is in state ) when it rains and state 1 when it does not, then
the above is a two-state Markov chain having transition probability matrix
α 1− α
β 1− β
That is, P00= α =1-P 01, P10= β =1-P 11
Example:
Consider a gambler who, at each play of the game, either wins 1 unit with probability
p or loses 1 unit with probability 1-p. If we suppose that the gambler will quit
playing when his fortune hits either 0 or M, then the gambler's sequence of fortunes is
a Markov chain having transition probabilities
Example:
The physicists P. and T. Ehrenfest considered a conceptual model for the movement of
molecules in which M molecules are distributed among 2 urns. At each time point one
of the molecules is chosen at random, and is removed from its urn and placed in the
other one. If we let Xn denote the number of molecules in the first urn immediately
after the nth exchange, then {X0, X1,…} is a Markov chain with transition
probabilities
M −i
M
i
Pi ,i −1 =
M
Pij = 0
Pi ,i +1 =
0≤i ≤ M
0≤i ≤ M
if j- i > 1
Thus, for a Markov chain, Pij represents the probability that a system in state I will
enter state j at the next transition. We can also define the two-stage transition
(2 )
probability, Pij , that a system, presently in state I, will be in state j after two
additional transition. That is,
Pij( 2) = P{ X m+ 2 = j| X m = i}
(2 )
The Pij
can be computed from the Pij as follows:
Pij( 2 ) = P{ X 2 = j| X 0 = i}
M
= ∑ P{ X 2 = j , X 1 = k | X 0 = i}
k=0
M
= ∑ P{ X 2 = j|, X 1 = kX 0 = i}P{ X 1 = k | X 0 = i}
k =0
M
= ∑ Pkj Pik
k =0
(n)
In general, we define the n-stage transition probabilities, denoted as Pij , by
Pij( n ) = P{ X n +m = j| X m = i}
(n)
This leads us to the Chapman-Kolmogorov equations, which shows how the Pij can
be computed
Chapman-Kolmogorov Equations .
M
(n)
ij
P
= ∑ Pik( r ) Pkj( n −r )
For all 0 < r < n
k=0
Proof:
Pij( n ) = P{ X n = j| X 0 = i}
= ∑ P{ X n = j , X r = k | X 0 = i}
k
= ∑ P{ X n = j|, X r = kX 0 = i}P{ X r = k | X 0 = i}
k
M
= ∑ Pkj( n −r ) Pik( r )
k=0
Example
A Random Walk. An example of a Markov chain having a countable infinite state
space s the so-called random walk, which tracks a particle as it moves along a one
dimensional axis. Suppose that at each point in time the particle will either move one
step to the right or one step to the left with respective probabilities p and 1-p. That is,
suppose the particle's path follows a Markov chain with transition probabilities
Pi ,i +1 = p = 1 − Pi ,i −1
i = 0,±1,...
If the particle is at state i, then the probability it will be at state j after n transitions is the
probability that (n-i+j)/2 of these steps are to the right and n-[(n-i+j/2]=(n+i-j)/2 are to the
left. As each step will be to the right, independently of the other steps, with probability, it
follows that the above is just the binomial probability.
n
n
Pij = n − i + j p ( n− i + j ) / 2 (1 − p) ( n +i − j )/ 2
2
F
GH
I
JK
nI
F
where G J is taken to equal 0 when x is not a nonnegative integer less than or equal to n.
H xK
FG 2n IJ p (1 − p)
Hn + kK
2n + 1 I
F
=G
H n + k + 1JK p (1 − p)
Pi 2,i n+2 k =
Pi n,i ++12 k -1
( n+ k )
n −− k
n +k +1
k = 0, ± 1,... ± n
n− k
k = 0,±1, ..... , ± n,-(n +1)
P{ X n = j} = ∑ P { X n = j| X o = i} P{ X 0 = i}
i
= ∑ Pij( n) P{ X 0 = i}
i
A sufficient condition for a Markov chain to possess this property is that for some n>0
Pij( n) >0 for all I,j,= 0,1,….,M
Markov chains that satisfy this equation are said to be ergodic.
Since the Chapman-Kolmogorov equations yield
M
= ∑ Pik( n) Pkj
( n +1 )
ij
P
k =0
it follows, by letting n→∞ , that for egodic chains
M
Π j = ∑ Π j Pkj
k =0
Furthermore, simce 1 =
M
∑P
( n)
ij
j =0
M
∑Π
j
, we also obtain, by letting n →∞ ,,
=1
j =0
In fact, it can be shown that the Π j, 0 ≤ j≤ M, are the unique nonnegative solutions of the above
last two equations
Theorem
For an ergodic Markov chain
Π j = lim Pij( n)
n →∞
Exists, and the Π j , 0 ≤ j ≤ M are the unique solutions of
M
Π j = ∑ Πk Pkj
k =0
M
∑Π
j =0
j
=1
Example:
Consider the example from above, in which we assume that if it rains today, then it will rain
tomorrow with probability α, and if it does not rain today, then it will rain tomorrow with
probability β. From the above theorem it follow that the limiting probabilities of rain and of no
rain, Πo and Π1 are given by
Πo = αΠ0 + βΠ1
Π1 = (1 − α )Πo + (1 − β) Π1
Π0 + Π1 = 1
which yields
Π0 =
β
1+ β −α
Π1 =
1−α
1+ β − α
For instance, if α=.6,β=.3, then the limiting probability of rain on the nth day is Π0 = 3/7.
The quantity Πj is also equal to the long run proportion of time that the Markov chain is in state
j, j=0,….,M. To intuitively see why this might be so, let P j denote the long run proportion of
time the chain is in state j. (It can be proven, using the string law of large numbers, that for an
ergodic chain such long run proportions exist and are constant). Now, since the proportion of
time the chain is in state k is Pk and since, when in state k, the chain goes to state j with
probability Pkj , it follows that the proportion of time the Markov chain is entering state j from
state k is equal to Pk Pkj . Summing over all k shows that Pj, the proportion of time the Markov
Pj = ∑kP k Pkj
chain is entering state j, satisfies
Since clearly it is also true that
∑P
j
= 1 it thus follows, since by our theorem the Πj j=0,…,M
j
are the unique solution of the above, that pj = Π j j=0,…,M .
Example
Suppose, in the example above that we are interested in the proportion of time there are j
molecules in urn 1, j-0,…,M. By the Chaoman-Kolmolorov theorem, these quantities will be the
unique solution of
1
M
M − j +1
j +1
Π j = Π j −1 ×
+ Π j +1 ×
M
M
1
Π M = Π M −1 ×
M
Π 0 = Π1 ×
M
∑Π
j
=1
j =0
However, as it it easily checked that
Πj =
FG M IJ ( )
H jK
1 M
2
,
j = 0,..., M.
j = 1,..., M .
Review of Markov Chains
When the process is in state i ⇒ there is a fixed probabilty, P ij, that it will next be in state j.
i.e.
Pij = P{ X n+1 = j| X n = i, X n−1 = in−1,..., X 0 = to}
for all states i0 , i1 ,.... in −1. i, j and all n > 0
" A Markov chain can be described by the conditional distribution of any
future state, Xn+1 , given the past states X0, X1, …,Xn-1 , and the present state
Xn. This conditional probability is independent of the past states and
depends only on the present state".
Pij ≥ 0
∞
∑P
j =1
ij
=1
i, j ≥ 0
i = 1,2,.....
Assume a sequence of experiments with the following properties:
1 Outcome of each experiment is one of a finite number of possible outcomes a1,
a2, …an.
2 Probability of outcome aj of any given experiment is not necessarily independent
of previous outcomes of experiments, but depends at most on outcome of
immediately preceding experiment.
3 Assume we can assign number, Pij, which represents probability of outcome aj,
given ai occurred. Note: outcome a1, a2, …,ai, ar ⇒ states, and pij ⇒ transition
probabilities.
4 If the process is assumed to begin in some particular state, we have enough
information to determine the tree measure (flow of events) of the process and
calculate the probability of statement relating to the experiments
1) Square array (transition matrix)
Fp
P=Gp
GH p
p12
p22
p32
11
21
31
p12
p23
p33
I
JJ
K
or
2) Transition diagram. i.e. given 3 possible state:
½
1
a1
a2
½
a3
1/3
2/3
The matrix associated with diagram is;
a1 a 2
F
GG
H
a3
I
JJ
K
a1 0 1 0
P = a2 0 12 21
a3 13 0 23
Example
Suppose that the chance of rain tomorrow depends only if it is raining today (not on past weather.
And ,if it is raining today, then it will rain tomorrow with probability α. If it does not rain today,
it will rain tomorrow with probability β. We say that the process is in state 0 when it rains, and
in state 1 when it does not rain. This is a two-state Markov chain whose transition probability
matrix is
α
L
P= M
Nβ
OP
Q
1−α ←Probabiltyvector
1− β ←Probabiltyvector
Example
Gary is either cheerful, C, so-so, S or glum, G.
1) If he is cheerful today then he will be C, S, or G tomorrow with probability
.5, .4, or.1, respectively.
2) If he is so-so today then he will be C, S, or G tomorrow with probability .3,
.4. or .3 , respectively.
3) If he is glum today then he will be C, S, or G tomorrow with probability .2,
.3. or .5 , respectively.
If Xn →Gary's mood on the nth day, {Xn, n ≤0} is a 3stage Markov chain. If
we designate C=0, S=1, and G=2 the probability matrix is:
Fp
P=Gp
GH p
00
10
20
p01
p11
p21
I
JJ
K
F
GG
H
I
JJ
K
p02
.5 .4 .1
p12 = P = .3 .4 .3
p22
.2 .3 .5
(n )
The n transitional probability matrix Pij
Pij(n )
Fp
=Gp
GG p
H
( n)
11
( n)
21
( n)
31
p12(n )
p22(n )
p32(n )
p13(n)
p23(n)
p33(n)
I
JJ
JK
Given
a1 a2
F
P = a G0
G
a H
a3
I
JJ
K
a1 0 1 0
2
3
1
3
1
2
1
2
0
2
3
So,
a2
½
½
a1
1
a2
½
a3
a2
½
2/3
a3
a3
1/3
a1
.
P13( 3) = probability of going to state 3 from state 1 after 3 steps
= sum of the probabilities assigned to all paths thru tree which end at state a 3
i.e
1 1
1 2 7
P13( 3) = 1 × × + 1 × × =
2 2
2 3 12
Similarly,
1 1 1
× =
2 2 4
1 1 1
P11( 3) = 1 × × =
2 3 6
1 1 7
∴ P1(i 3) =
6 4 12
P12( 3) = 1 ×
FG
H
We also can represent this as follows,
IJ
K
½
a2
½
a2
½
½
2/3
½ a2
a2
½
a3
2/3
a3
1/3
2/3
3
a1
1/3
a3
1/3
a3
a3
a1
a
a1
1
a2
P21( 3) = 12 × 23 × 13 + 12 × 12 × 13 =
7
36
P22( 3) = 12 × 13 × 1 + 12 × 12 × 12 =
7
24
P23( 3) = 12 × 12 × 12 + 12 × 12 × 23 + 12 × 23 × 23 =
P2(i3 ) = (
7 7 37
, , )
36 24 72
37
72
1/3
a1
1
a2
a3
2/3
1/3
1
a1
1/3
a3
2/3
a2
a3
2/3
1/3
a3
2/3
P31(3) = 23 × 23 × 13 =
a2
a1
a3
4
27
P32(3) = 13 × 1 × 12 + 23 × 13 × 1 = 187
P33(3) = 13 × 1 × 12 + 23 × 23 × 23 =
P3(i3) = ( 274 , 187 , 25
54 )
F
=a G
G
a H
a1
P( 3)
2
3
a1 a 2
a3
1
6
1
4
7
12
7
36
4
27
7
24
7
18
37
72
25
54
I
JJ
K
25
54
Example:
Suppose that we are interested in studying the way in which a given state votes in a series of
national elections. We shall base our prediction only on past history of the outcomes of the
elections (Republican or Democrat). As a first approximation we assume that knowledge of the
past, beyond the last election, would not cause us to change the probability for the outcome on the
next election. So we have a markov process , or chain with 2 states, Republican ( R ) and
Democrtat ( D ).
FG
DH
R
R 1− a
b
D
IJ
1 − bK
a
where a can be estimated from past results and represents the fraction of years in which there was
a change from R to D., and b represents the fraction of years in which there was a change from D
to R.
We can obtain a better approximation from past results of two previous elections.
Our states now are RR, RD,DR,DD, and if next election is a Democrat, RR -> RD, etc. The first
letter of the new state must be the second letter of the old state. So, if election outcomes for a
series of elections is DDDRDRR our process moves DD→DD → DR → RD → DR → RR etc
Hence
F
GG
RD G
G
DD H
RR
RR 1 − a
DR b
0
0
DR
RD DD
0
0
a
1− b
1− c
d
0
0
where a,b,c,d, must be estimated.
I
JJ
c J
J
1 − dK
0
0
Regular Transition Matrix Properties:
i)
ii)
iii)
iv)
v)
Pn → W
Each row of W is same probability vector w.
Components of w are positive
If p is any probaboility vector, p Pn → w
w is the unique fixed point probability vector
of P. (i.e. a probability vector w, is a fixed
point of the matrix P if w=wP)
The fixed probabilty vestor is
FG bd
H bd + 2ad + ca
ad
bd + 2 ad + ca
ad
bd + 2ad + ca
∴ probability of being in state RR and DR =
ca
bd + 2 ad + ca
bd + ad
bd + 2ad + ca
IJ
K
Example:
Consider
F IJ and if w = (.6 .4) we see that
P=G
H K
F IJ = (.6,.4) = w
(.6,.4 )G
H K
F I × F I = FG .611 .389IJ
P =G
H JK GH JK H .583 .417K
F .611 .389IJ FG IJ = FG .602 .398IJ
P =G
H .583 .417K H K H .597 .403K
F.6 .4IJ
→G
H.6 .4K
2
3
1
3
1
2
1
2
2
3
1
2
2
3
1
2
2
1
3
1
2
1
3
1
2
2
3
1
2
1
3
1
2
2
3
1
2
3
Pn
1
3
1
2
w=w
FG w IJ = (w , w )LM OP
Hw K
N Q
1
1
2
2
w1 = 23 w1 + 12 w2
w2 = 21 w1 + 12 w2
w1 = 23 w2
2
3
1
2
1
3
1
2
Define Pijn
Pijn = P{ X n + m = j | X m = i}
Pijn + m = P{ X n + m = j| X 0 = i} =
n ≥ 0,
∑ P{ X
∞
0
n+ m
i, j ≥ 0
= j, X n = k | X o = i}
∞
=
∑ P{X
∞
n +m
= j| X n = k , X 0 = i}P{ X n = k | X 0 = i} =
k =0
m+ n
ij
P
∑P P
m n
kj ik
k= 0
∞
= ∑ Pikn Pkjm
k =0
Note: Pikn Pkjm → probability that starting in i the process will go to state j
in n + m steps through a path that takes it into state k at the nth transition.
Formally,
If P(n) denotes the matrix of n-step transition probabilolity,
P ( n + m) = P ( n ) ⋅ P ( m )
i. e.
P ( 2) = P ( 1) ⋅ P (1) = P 2
In general,
P ( n) = P ( n−1+ 1) = P ( n −1) ⋅ P (1) = P n
Pijn , then
Example:
Consider a two stage Markov Chain, with transition probability matrix as α =.7, and β=.4,
.7 .3O
L
P=M
N.4 .6PQ
Calculate the probability that it will rain four days from today, given that it rained today.
P( 2 ) = P2 =
P( 4 ) = P( 2 )
P004
Pijn
2
FG.7 .3IJ × FG.7 .3IJ = FG .61 .39IJ
H.4 .6K H.4 .6K H.52 .48K
F .61 .39IJ FG.61 .39IJ = FG.5749
=P =G
H.52 .48K H.52 .48K H .5668
4
IJ
K
.425
.433
= probability of going from state 0 (rain) to state 0 (rain) in 4 steps = 5749
= probability that state at time n is j, given that the initial state at time 0 is i.
If we want the unconditional distribution of the state at time n, we need to specify t
he probability distribution of the initial state. α i = P{ X 0 = i} where i ≥ 0 and ∑ α i = 1
i
Example:
Let X = { X n ; n ∈ N } be a Markov Chain with states a,b,c.
(state space, E={a,b,c}) and transition matrix
LM1 / 2
P 1/ 3
MM3 / 5
N
OP
P
0 PQ
1/ 4 1/ 4
0 1/ 3
2/ 5
Then
P[ X 1 = b, X 2 = c, X 3 = a, X 4 = c, X 5 = a , X 6 = c, X 7 = b| X 0 =
= P (c, b) P (b, c) P (c, a ) P (a, c) P (c, a ) P (a, c) P (c, b)
=
2 1 3 1 3 1 2
5 3 5 4 5 4 5
=
3
2500
The two step transition probabilities are
P2
LM1 / 2
= M1 / 3
MN3 / 5
OPLM
1 / 3PM1 / 3
0 PQMN3 / 5
OP LM17
1 / 3P = M 8 /
0 PQ MN12
1/ 4 1/ 4 1 / 2 1 / 4 1 / 4
0
2 /5
0
2/5
So that, for example
P{ X n+ 2 = c| X n = b} = P 2 (b, c) = 1 / 6
We can write
∞
P{ X n = 5} = ∑ P{X n = j| X 0 = c} P{Xi = c}
i =0
∞
= ∑ Pijnα i
i =0
For example, considering the Markov Chain from the previous problem, given
unconditional probability that it will rain 4 days from now is
P[ X 4 = 0} =.4 P004 +.6 P104
=.4(.5749) +.6(.5668)
=.5700
α 0 =.4; α 1 =.6 , then the
Let X be a Markov Chain with state space E=[1,2], initial
distribution = (1/3,2/3) and transition matrix
.5 .5
P=
.3 .7
P
2
.4
L
=M
N.36
.6
.64
OP
Q
P
3
.38
L
=M
N.372
LM
N
OP
Q
.62 O
.628PQ
P
4
.376
L
=M
N.3744
In general, it can be shown that
P
m
L
=M
MN
3
8
3
8
+ 58 (.2) m
− 38 (.2) m
5
8
5
8
OP
Q
.624
.6256
− 58 (.2) m
+ 38 (.2) m
OP
PQ
P[ X1 = 2, X 4 = 1, X 6 = 1, X18 = 1| X 0 = 1]
= P(1, 2) P3 ( 2,1) P2 (11
, ) P12 (11
, ) = (.5)(.372 )(.4)( 38 + 85 (.2 )12 )
= 0.26100
And
P[ X 2 = 1, X 7 = 2, X 9 = 2| X 0 = 1]
= P 2 (11
, ) P5 (1,2 ) P2 ( 2,2 ) = (.4 )( 58 − 58 (.2)5 )(.64 )
= 015995
.
P[ X 2 = 1, X 7 = 2] =
∑ P{ X
0
= i}P{ X 2 = 1, X 7 = 2| X 0 = i}
i
= π (1) P2 (11
, ) P5 (1, 2) + π ( 2 ) P 2 ( 2,1) P5 (1,2 ) = 1 / 3(.4 )( 58 − 58 (.2 )5 + 2 / 3(.36 )( 58 − 58 (.2
= 0.23326
Accessible States
State j is accessible from state I if
Pijn > 0 for some n ≥ 0.
Two states that are accessible to each other are said to be
communicate, i ↔ j.
Any state communicates with itself since Pii0 = P[ X 0 = i | X 0 = i ] = 1
Communication satisfies the following three properties:
i.
State I communicates with state I, for all i ≥0.
ii. If state I communicates with state j, then state j
communicates with state i.
iii. If state I communicates with state j, and state j
communicates with state k, then state I communicates with
state k.
Two states that communicate are of the same class. A chain is
irreducible if there is only one class (i.e. all states communicate with
each other).
Example:
Gambler's Ruin
.
You have A dollars, and the gambler has B dollars. You bet $1 on each game and play until
one of you is ruined. p > 1/2. Find the probability that you will be ruined.
Let N= A+B - total amount of dollars in the game. As states for the chain we choose
1,2,….,N. At any one moment the position of the chain is the amount of money you have.
The initial position is:
Your money.
His money.
O
A
B
At any step there is a probability p of moving one step to right and probability (1-p)=q of
moving 1 step to the left. We have a: Markov Chain! If the chain reaches 0 or N we stop.
These are called ABSORBING STATES, since once they are entered they are never left. We
are interested in the probability of your ruin (reaching 0).
Find the ruin-probability for every possible starting position.
Let Xi be the probability of your ruin when you start at position i We solve for N=5, so we have unknown
X1, X2, X3, X4, X5, we start at position 2, the chain moves to 3 with probability p, or to 1 with probability
q.
P{ruin/start at 2}=P{ruin/start at 3}*p + P{ruin/start at 1}*q
But once it reaches stage 3, a Markov Chain behaves like it had been started there. So,
P{ruin/start at 3} = X3
P{ruin/start at 1} = X1
Therefore we can write
X2 = pX3 + qX1
Substituting p+q=1
X2 = (p+q)X2 = pX3 + qX 1
P(X2 -X3) = q(X1 -X2)
Hence,
X1 - X2 = r(X2 - X3)
Where r = p/q, and hence, r<1.
We can write this for each of the four ordinal positions
We have our absorbing states X0 = 1 and X5 = 0. If we substitute X5 into the above
X 0 − X1 = r( X1 − X2 )
X 0 − X1 = r( X1 − X2 )
X 0 − X1 = r( X1 − X2 )
X1 − X 2 = r( X 2 − X 3)
X 2 − X3 = r ( X3 − X 4 )
X 3 − X4 = r ( X4 − X 5)
X 3 - X4 = rX4
X 4 = 1⋅ X 4
X3 − X4 = r ⋅ X4
1
2
X2 − X3 = r2 ⋅ X4
3
X1 − X 2 = r 3 ⋅ X 4
4
X 0 − X1 = r 4 ⋅ X 4
5
1 − r2
X 3 = (1 + r ) X 4 =
1− r5
X 0 = (1 + r + r 2 + r 3 + r 4 ) X 4
(1 − r 5 )
But X 0 = 1, and (1 + r + r + r + r ) =
(1 − r )
1− r
4
1
−
r
X4 =
X1 =
1− r5
1 − r5
2
1 − r3
X2 =
1 − r5
3
4
X4 =
1− r
1 − r5
Ruin Probability
1− rB
XA =
1− rN
Suppose, for example, that a gambler wants to have
probability .999 of ruining you. He must make sure that rB < .001.
For example, if p = .495, the gambler needs $346 to have a
probability .999 of ruining you, even if you are a millionaire. If p
= .48, he needs only $87. And even for the almost fair game with p
= .499, $1727 will suffice..
There are two ways that gamblers achieve this goal. Small
gambling houses will fix the odds quite a bit in their favor, making
r much less than 1. Then even a relatively small bank of B dollars
suffices to assure them of winning. Larger houses, with B quite
sizable, can afford to let you play nearly fair games
SUMMARIZING
Markov Chain
Consider a random process or stochastic process, { Xn, n = 0,1,2
i.e.
P{ X n +1 = j| X n = i , X n −1 = in−1 .... X 0 − i0 } = Pij
for all states i0 ,i1 ,...in −1 ,i , j , and alln > 0
Pij ≥ 0
i, j ≥ 0
∞
∑ Pij = 1
j =1
Absorbing State Markov Chain.
A state, Si, is absorbing if Pii = 1 and Pij =0 for j≠i.
Consider 5 states, 2 absorbing
LM 1
0
M
P = MP
MM P
MN P
31
41
51
0
1
P32
P42
P52
0
0
P33
P43
P53
OP
PP L
PP MN
PQ
0 0
0 0
I 0
P34 P35 =
R Q
P44 P45
P54 P55
OP
Q
Let m ij = the mean number of times the system is in state Sj, given it started from
SI, where Si and Sj are non absorbing states.
mij = δ ij + ∑ Pik mkj ;
δ ij =
k
RS1
T0
i=j
i≠j
δij accounts for the fact that if chains goes to absorbing state in one step, it was in
state Si one time. We can write this for all states;
M == I + QM
M = I −Q
mi = ∑k mik
−1
- over all non - absorbing states
Consider now the problem of absorption into various absorbing states. Let
a ij = Probability of system absorbed in state Sj, given it started in state Si
(Si non-absorbing state and Sj absorbing state)
The system could move to Sj in one step, or it could move to a non-absorbing state,
Sk, and be absorbed from there
∴ aij = pij + ∑ pik akj
k
non - absorbing states
∴ A = R + QA
or
A = ( I - Q) -1 R = MR
Example:
The manager of a plant has employees working at level I and level II. New employees
mat enter either level.. At the end of each year the performance of employees are
evaluated. They can be either reassigned to level I or II jobs, terminated, or promoted to
level III, in which case they may never come back to levels I or II.
We can treat this as a Markovian process, with two absorbing states
Employment at level I - S3
Employment at level II - S4
Records over long periods of time indiocate the following probabilites as valid
A1
LM 1
0
M
P = M− −
MM .2
N .1
A2
A3
A4
OP
PP LM
PP N
.5 Q
0 | 0
0
1 | 0
0
I 0
−− −− −− =
R Q
.1 | .2
.5
.3 |
.1
OP
Q
Thus, if employee enters levek I, the probabiltiy is .5 that she will jump to level II and
.2 that she is terminated.
Find: The expected number of evaluations an employee must go through in this plant
i.e. we want to find the expected number of steps to absorbtion from non-absorbing
state i, then sum over all i.
M = ( I − Q) −1 and from P, we have R =
L.8 .1OP = LM
=M
N.1 .3Q N
−1
10
7
2
7
10
7
16
7
OP
Q
LM.2 .1OP and Q = LM.2 .5OP
N.1 .3Q
N .1 .5Q
Hence,
m1 = m11 + m12 = 20/7
m2 = m21 + m22 = 18/7
Which concludes that new employees can expect to remain there through 20/7 evaluations
while in level I.
Also
L
O
.2 .1O
L
A = MR = M
P
M
P
.
1
.
3
N QN Q
L
O
∴A= M
P
N Q
10
7
2
7
3
7
2
7
10
7
16
7
4
7
5
7
That is, an employee entering at level I has a probability 4/7 of reaching level II, whereas, at
level II raises the probability to 5/7.
.
P{ X i = S k | X i −1 = S j } = Pjk
LM P
M
P=M
MM
NP
11
P12 L
m1
P1m
M
L Pmm
OP
PP
PQ
Let X0 be starting state of the system with probability given by
Pk( 0) = P ( X 0 = S k )
Let P{being in Sk after n steps} =
P
M
(0 )
Pk(n ).
These are conviently displayed as;
= [ P1(0 ) , P2(0 ) .... Pm(0 ) ]
P (n ) = [ P1(n ) , P2(n ) .... Pm(n ) ]
The relationship between these probabilities can be ascertain by considering the
following:
P(1) = probability of being in various states after 1 step = P (0) = [ P1(0) , P2(0) .... Pm(0) ] . If
we multiply this vector by the transition probability matrix,
P(2) =[probability of being in state k after 1 step][probability of going to a next
state]
Hence, we get P
(n )
= P(n −1) P
.
Definition: If some power of P (Pn) has all positive entries,
P is said to be regular. So, one can get from state
Sj to state Sk eventually, for any (j,k)
Definition: If P is regular, the Markov chain has a
stationary distribution that gives the probability of
being in the respective states after many transitions.
i.e. Pj(n) must have a limit, Πj as n →∝, which
satisfies Π = Π P
Example:
Market stocks 3 brands of coffee, A, B, C, and it has been noticed that customers switch from brand
to brand according to the following transition matrix:
LM3 / 4
P=M 0
MN1 / 4
1/ 4
0
2 / 3 1/ 3
1/ 4 1/ 2
OP
PP
Q
where S1 → purchase of A
S2 → purchase of B
S3 → purchase of C
So, 3/4 of A today switches to A tomorrow (stays with A), 1/4 of A today switches to B tomorrow,
and 0 of A toad switches to C tomorrow (no one switches).
(a) Let's find the probability a customer switches to A today will swithc to A two weeks from now,
assuming she buys coffee once/week
Customer starts with purchase of A:. then
P (1) = P ( 0) P; (1,0,0)[ D]
3 1
0)
=(
4 4
3/ 4 1/ 4 0
3
1
P(2 ) = (
0) 0
2 / 3 1/ 3
4 4
1/ 4 1/ 4 1/ 2
=
FG 9
H 16
LM
MM
N
17
48
1
12
IJ
K
OP
PP
Q
(b)In the long run, what fraction of customers will buy the respective brands? We need the staionary
distribution, Π = ΠP
FΠ
GG Π
GH Π
1
2
I
LM3 / 4
JJ = bΠ Π Π gM 0
JK
MN1 / 4
LM Π + Π OP
=M Π + Π + Π P
MN Π + Π PQ
1
3
2
3
4
1
4
1
1
3
1
2
3
2
3
1
4
2
1
2
3
1
4
OP
P
1 / 2 PQ
1/ 4
0
2 / 3 1/ 3
1/ 4
3
3
Π1 + Π2 + Π 3 = 1 (must be somewhere).
From (1) equation (1 − 3 / 4) Π1 = 1 / 4Π3
∴ Π1 = Π3
Using (2) or (3) 1 / 2Π = 1 / 3Π
∴ Π2 = 3 / 2 Π3
3
2
Substituting this into Π + Π + Π = 1 , we have
1
2
3
Π 3 + 3 / 2Π 3 + Π 3 = 1
∴ 7 / 2Π3 = 1
And
Π 3 = 2 / 7, Π 1 = 2 / 7 , Π 2 = 3 / 7
Solve this , along with the constraint that
We therefor conclude that the store should stock more B than A or C coffee
Discrete-Time Markov Chains
The Markov chain Assumptions
Let Xt denote the number in stock at the end of the tth month. If
Xt falls below a fixed critical level, more disks are ordered. Xt+1
depends on two quantities: Xt and the number sold between time t and
time t+1. Since the number sold is random, Xt does not completely
determine Xt+1. But the basic assumption is that given the value of Xt,
in trying to predict what Xt+1 will be, all the past records Xt-1,Xt-2,…,X0
are irrelevant; Xt+1 depends only on the number Xt now in stock at
time t and the number sold during this month. This is the Markov
assumption that the future is independent of the past given the state of
affairs in the present.
Stochastic Process {X } . The location X t is measured only at the discrete times
t=0,1,2,…. X0 is the starting location at time 0.
∞
t t =0
Markov Chain Assumptions
Let i , j , it −1 , K , i0 ∈ S .Then for any time t,
P ( X t +1 = j | X t = i , X t −1 = it −1 , K , X 0 = i0 ) = P ( X t +1 = j | X t = i )
That is, the future (time t+1), given the present (time t), is independent of the past (times t1,…,0). The probability above is the transition or jump probability from state i to state j.
P ( X t +1 = j | X t = i ) = Π i, j
This assumption is called homogeneity in time.
A (homogeneous) Markov chain in discrete time consists of a particle that jumps at each
unit of time between states in a state space S. Xt denotes the state occupied at time t for
t=0,1,2,…. If the particle is in state i at time t, it will be in state j at time t+1 regardless of
the states occupied before time t with probability
Π i, j = P ( X t+1 = j | X t = i )
Andrei Andreevich Markov (1856-1922) was Chebyshev's most
outstanding student. Markov was most involved in proving limit
theorem and the law of large numbers of Chapter 10. Although
Markov originally used Markov chains in abstract settings to
prove various theorems, they were quickly applied. Markov
himself illustrated the ideas by examining vowel-consonant
interchanges in 20,000 letters of Pushkin's poetry.
Example
Let
S = (0,1) and the transition probabilities
1
2
1
3
Π 0 ,0 =
Π 0 ,1 =
Π 1, 0 =
Π 1,1 =
3
3
4
4
be
Figure 1, below, depicts this.
1/3
3/4
2/3
.
.
1
1/4
0
Figure 1
A table or matrix that displays the jump probabilities is
1 / 3 2 / 3
Π=

1 / 4 3 / 4
There is a standard way of writing the jump probabilities Π i, j as a table. This is called
the transition matrix Π . The element in the ith row of and jth column is Π i, j - the probability
to that the particle jumps from i to j.
 Π 0,0
Π
1, 0
Π=
 L

Π N ,0
Π 0,1
Π1,1
L
Π N ,1
Π 0, 2
Π1, 2
L
Π N ,2
L
L
L
L
Π 0 ,N 
Π1,N 

L 

Π N ,N 
Using matrix terminology, the ij entry of Π, is Π i, j . Note that the ith row of Πdisplays the
jump probabilities from state i; the jth column displays the jump probabilities to state j. For
example, if the third column consisted of all 0's, that is, if
Π i ,3 = 0
for all states i in the state space S, the particle could never enter state 3, then Π i 0,3 > 0 .
Let Π be a Markov chain transition matrix.
Then
1.
0 ≤ Π i , j ≤ 1 for all i,j in the state
2.
space S.
Π has row sums 1:
N
∑Π
j =0
N
i, j
= ∑ P( X t +1 = j | X t = i ) = 1
j =0
Example
In this chain there are three states; S={0,1,2}. From state 0
the particle jumps to states 1 or 2 with equal probability
1/2. From state 2 the particle must next jump to state 1.
State 1 is absorbing; that is, once the particle enters state
1, it cannot leave. Draw the diagram and write down the
transition matrix.
1
.
1
1/2
1
.
0
.
2
1/2
Figure 2
Solution:
Fig. 2 depicts the transition probabilities. The zeroth row of Π consists of the jump
probabilities from state 0 and similarly for the other two rows. Check that

0
Π = 0

0

1
2
1
1
1
2
0

0

State i is absorbing if Π i ,i = 1.
Example: A Random Walk on S ={0,1,2,…,N}
For 1 ≤ i ≤ N − 1 .
Π i ,i +1 = p
Π i ,i −1 = q
Π i, j = 0
for j ≠ i ± 1
Case 1. Both absorbing,
Π 0,0 = 1
Π N ,N = 1
1 0 0 0 LL 0
 q 0 p 0 LL 0 


0 q 0 p LL 0 
Π=

LLLLLLL 
0 0 0 0 L q 0 p 


0 0 0 0 LL L 1 
r
s
1-r
. .
0
q
p
p
1-s
. . . . .
i
1
Figure 3
N-1
N
Case 2. Reflecting,
Π 0 ,0 = 1
Π N ,N −1 = 1
Case 3. Partially reflecting, as depicted in Fig. 3
Π 0 ,0 = r
Π 0,1 = 1 − r
 r 1-r 0 0 0 K . 0 
q 0 p 0 0 K . 0 


0 q 0 p 0K . 0 
Π=

..........
..........
..........
........


 0 0 0 0 0 .. q 0 p 


0
0
0
0
0
.....
1
-s
s


Π N ,N = s
Π N , N −1 = 1 − s
Example A Renewal Process
. The state space is S={0,1,2,…} and the state of the system is the age of the
component presently installed.
 q p 0 0 .....
 q 0 p 0 0 ...

Π=
 q 0 0 p 0 ...


.................... 
Note that the state space S, while discrete, has infinitely many states. (This model is
also called a birth or disaster model
THE TTH -ORDER TRANSITION MATRIX
P ( X t +3 = j | X t = i )
. The one-step probabilities Π i, j are the entries of the matrixΠ . From these, how can one find
the three-step probabilities, and more generally the t-step probabilities?
Definition
The tth -order transition matrix is Π , whose
ijth entry is
t
Π ti , j = P( X t = j | X 0 = i )
which is the probability of jumping from i
to j in t steps.
Homogeneity in time implies that regardless of the time u ≥ 0 ,
P( X t +u = j | X u = i ) = Π ti , j
To find the (t+1)st-order transition matrix from the tth , use the basic Markov assumptions:
Π ti +, j1 = P( X t +1 = j | X 0 = i )
N
= ∑ P( X t +1 = j and X t = k | X 0 = i )
k =0
N
= ∑ P( X t +1 = j | X t = k and X 0 = i )P( X t = k | X 0 = i )
k =0
N
= ∑ P( X t +1 = j | X t = k )P ( X t = k | X 0 = i )
k =0
N
= ∑ Π k , j Π ti , k
k =0
N
= ∑ Π ti ,k Π k , j
k =0
P ( A ∩ B | C ) = P ( A | B ∩ C )P (B | C )
Π t +1 = Π t Π
Generalizing:
(
)
Π t +2 = Π t +1Π = Π t ⋅ Π Π = Π t Π 2
Chapman-Kolmogorov Equation
Let times t , s ≥ 0 . Then for all states i,j
N
Π ti,+js = ∑ Π ti ,k Π sk , j
k =0
In terms of matrix multiplication, the (t+s)thorder transition matrix is the product of the tth
and the sth:
Π t +s = Π t Π s
Example
Convert the jump probability diagram of Fig. 4 into the corresponding Markov chain and
find the probability that the particle will be state 1 after three jumps given it started in
state 1.
0.9
.
.
0
0.7
0.8 0.2
0.1
0.3
.
1
Figure 4
2
Solution
 0 .1 .9 
Π = .8 0 .2 


.7 .3 0 
.71 .27 .02
Π 2 = Π 1Π 1 = .14 .14 .72


.24 .07 .69
.230 .077 .693
Π = Π Π = .616 .230 .154


.539 .231 .230
3
2
Consequently,
P( X 3 = 1 | X 0 = 1) = Π 13,1 = .230
The Probability Vector
ρt
Let the initial probability vector be defined
ρ = (ρ 0 , ρ1 ,K , ρ N )
Note that 0 ≤ ρ i ≤ 1 for all states i in the state space S and
ρ 0 + ρ1 + K + ρ N = 1
ρ t = (ρ t 0 , ρ t1 ,K , ρ t N )
ρ tj = P( X t = j | initial probability vector ρ )
where
t
That is, ρ j is the chance that the particle will be found in state j given that at time 0 it started in
ρ0 = ρ
and
N
∑ρ
That is, for each t,
ρ
j =0
N
t
j
= ∑ P (X t = j ) = 1
j =0
t
is a probability vector.
Definition
A probability vector ρ = (ρ 0 , ρ 1 ,K , ρ N ) satisfies
0 ≤ ρi ≤ 1 for each i = 0,1,…,N
2. ρ 0 + ρ 1 + K + ρ N = 1
1.
N
ρ tj = ∑ ρi Π ti , j
i= 0
Expressed as multiplication between a vector and a
matrix, the probability vector at time t is
ρ t = ρΠ t = ρ 0 Π t
ρ tj = P ( X t = j )
= ∑ P( X t = j | X 0 = i )P ( X 0 = i )
N
i =0
N
= ∑ Π ti , j ρ i
i =0
Example
For the Markov chain in Fig. 5 find the chance that the particle will be in state 0 at
time 3 if it started in state 0 with probability 1/3 and in state 1 with probability 2/3 at
time 0.
Solution
1

Π = 4

 1
and
3
4


0 
ρ = (1 / 3, 2 / 3)
Consequently
 13

Π 2 = Π ⋅ Π =  16
1

4
 25

Π 3 = Π ⋅ Π 2 =  64
13

 16
3
16 
3

4
39 
64 
3

16 
1 2
 129 63 
ρ 3 = ρΠ 3 =  , Π 3 = 
,

 3 3
 192 192 
P ( X 3 = 0) = ρ 03 =
129
192
1/4
3/4
. .
0
1
Figure 5
1
ρ 0 = (0, K,0,1,0,K ,0)
ith entry
ith entry
ρ t = (0, K,0,1,0,K ,0)Π t
= (Π ti, 0 , Π ti,1 , Π ti , 2 ,K , Π ti, N ,)
P ( X t = j ) = ρ tj = Π ti , j
The Steady State Probability Vector
N
ni = ∑ n j Π j ,i
j =0
N n
ni
j
= ∑ Π j ,i
n
j=0 n
N
φi = ∑ φ j Π j ,i
j= 0
1
that is, if φ = φ ⋅ Π = φ .
φ 1 = φΠ = φ
φ 2 = φ 1 Π = φΠ = φ
φ 3 = φ 2 Π = φΠ = φ
and in general,
φ t = φ t −1Π = φΠ = φ
Procedure for Finding the Steady-State
Probability Vector
1.
Set up and solve these equations:
N
φ j = ∑ φi Π i , j
i =0
for j=0,1,2,…,N or alternatively, in matrix
notation
φ = φΠ
2.
Normalize by insisting that
N
∑φ
i =0
i
=1
Example
Find the steady-state probability vector of the Markov chain in Fig. 6
Solution
φ = φΠ
(φ0 , φ1 ) = (φ0 , φ1 )Π = (φ0 , φ1 )
1 / 2 1 / 2
0 
 1
1/2
1/2
.
.
0
1
1
Figure 6
which yields two equations in the two unknowns φ 0and φ1 :
1
φ + φ1
2 0
1
φ1 = φ0
2
φ0 =
φ1 =
φ0
2
1 = φ0 + φ1 = φ0 +
φ0
φ
=3 0
2
2
Thus
φ0 =
2
3
φ1 =
1
3
. A simulation was run in which 66 particles were started in state 0 and 33 in state 1.
Each jumped according to the transition probabilities.
n
Fraction in state 0
1
2
3
4
5
6
7
8
.667
.667
.717
.636
.677
.667
.647
.707
Example
Consider the Markov chain in Fig. 7 Find the steady-state probability vector φ.
2/3
1
1/32
.
.
2/3
1/3
1/2
1/2
.
0
Figure 7
Solution
 0 1 / 2 1 / 2
Π =  0 2 / 3 1 / 3


1 / 3 2 / 3 0 
 0 1 / 2 1 / 2
(φ0 , φ1 , φ2 ) = (φ0 , φ1 , φ2 ) 0 2 / 3 1 / 3
1 / 3 2 / 3 0 
results in these equations:
1
φ
3 2
1
2
2
φ1 = φ0 + φ1 + φ2
2
3
3
1
1
φ2 = φ0 + φ1
2
3
φ0 =
Clearing the fractions and combining factors of each φ:i
− 3φ0
+ φ2 = 0
3φ0 − 2φ1 + 4φ2 = 0
3φ0 + 2φ1 − 6φ2 = 0
Adding the second and third equations results in
6φ 0
− 2φ 2 = 0
which is essentially the same as the first equation. Hence the third equation is
redundant. From the first two equations
φ2 = 3φ0
1
15
φ1 = (3φ0 + 4φ2 ) = φ0
2
2
Now use the normalization condition
23
 15

1 = φ 0 + φ1 + φ2 = 1 +
+ 3 φ0 =
φ0
2
2


Consequently,
2
23
15
15
φ1 = φ 0 =
2
23
6
φ 2 = 3φ 0 =
23
φ0 =
Example
Consider the cyclic process depicted in Fig. 8. There are N+1 states 0,1,2,…,N. For
each state i, 0<qi<1. The particle remains in state i with probability qi. Otherwise, it
jumps to the next state i+1 with probability pi=1-qi . If i=N, then i+1 is taken to be
state 0. That is, there is a "wraparound" from state N to state 0 .
q0
q1
q2
p0
p1
.
.
.
0
1
p2
2
Figure 8
Solve for the steady-state probability vector φ .
qN
. .
...
3
.
N
Solution
0 
q0 p0 0 K
0 q p 0 K 0 
1
1


Π = 0 0 q 2 p2 K 0 


K
K
K
K
K
K
K
K


 p N 0 KKK 0 q N 
φ = φΠ implies that
(φ0 , φ1 , K , φN ) = (φ0 , φ1 , K , φ N )Π
φ0 = q0φ0
or
φ1 = p 0φ0
φ2 =
KKKK
p0φ0 = p N φ N
p1φ1 = p0φ0
p2φ2 = p1φ1
LL
+ p N φN
+ q1φ1
p1φ1 + q2φ2
φ1 =
p0
φ
p1 0
p1
φ =
p2 1
p
φ3 = 2 φ 2 =
p3
L
p
φN = 0 φ0
pN
φ2 =
p0
φ
p2 0
p0
φ
p3 0
Normalizing
1 = φ0 + φ1 + L + φN

=  1 +


=  1 +

Let
C=
Then
p0 p0
p 
+
+ L + 0 φ 0
p1 p2
pN 
1
1
1 
+
+L+
p φ
p1 p 2
p N  0 0
1
1
1
1
+
+
+L +
p 0 p1 p2
pN
1 = Cp 0φ 0 , which determines φ0 .
For the cyclic process,
φi =
1
Cp i
where
N
C=∑
j =0
1
pj
Example 10
Let
0 
1 / 2 1 / 2 0
1 / 2 1 / 2 0
0 

Π=
 0 1 / 3 1 / 3 1 / 3


0
0
1 
 0
Here are two distinct steady-state probability vectors:
1 1

φ =  , ,0,0,0 
2 2

ψ = (0,0,0,0,1)
These can be verified simply by checking that φΠ = φ and
ψΠ = ψ .
Example
Consider a random walk on the integers S={0,1,2,…} in which transitions only to the
right can occur as in Fig. 9. Assume 0<p<1 and q=1-p.
q
q
p
q
p
.
.
0
1
Figure 9
p
.
.
2
...
3
 q p 0 0 0 L
0 q p 0 0 L 

Π =
0 0 q p 0 L 
LLLLLL


φ = φΠ implies that
qφ0 = φ0
pφ0 + qφ1 = φ1
pφ1 + qφ2 = φ2
LLLLLL
φ0 = 0
φ1 = 0
φ2 = 0
Example
With
0 1 / 2 1 / 2
Π = 1 0
0 


1 0
0 
it is straightforward verification to check that φ = (1 / 2, 1 / 4, 1 / 4) is a steady-state
probability vector, and in fact it is the only one. For a large number of particles, there
is a steady state: If n/2 start in state 0 and n/4 in each of states 1 and 2, there will be
approximately this many in each of the states at all subsequent times. But for one
particle, there is no steady state. If it starts in state 0, it must jump to {1,2}, from
which …. At even times it will be in state 0; at odd times in either 1 or 2. There is a
periodicity problem.
Birth and Death Processes in Discrete Time: Random Walks
Definition
A birth and death process in discrete time is a
Markov chain with this property:
From any state i ∈ S transitions in one jump to i-1,
i, or i+1 only can occur.
Set
bi = Π i , i +1
ri = Π i , i
bi + ri + d i = 1
The transition matrix is
r0 b0 0 0 0 LLLL
d r b 0 0 LLLL
 1 1 1
Π = 0 d 2 r2 b2 0 LLLL

LLLLLLLLLL
LLLLLLL 0 d N rN







d i = Π i , i −1
r0
.
ri
di
b0
..…..
0
..
rN
1
i
i-1
dN
bi
i
..…..
i+1
N-1
N
Figure 10
. The number of these that will jump next to state i+1 is therefore (nφ i )bi . But in the steady state
this must be compensated for an equal number of particles moving down from i+1. This is
(nφi+ )d i+1 . Otherwise, particles would "bunch up" to the left or to the right of state i and the
system would not be in the steady state. Consequently (after canceling the factor n),
φi +1d i +1 = φi bi
Procedure for finding the Steady-State
Probability Vector φ
Use the equation
φi +1 =
to solve for each
normalization
bi
φi
d i +1
φ i in terms ofφ0 . Then use
1 = φ0 + φ1 + L + φ N
Example
Solve for φ for the Markov chain in Fig. 11
1/2
1/10
1/3
5/10
1/4
2/3
4/10
1/2
0
3/4
1
2
Figure 11
Solution
Using the procedure
φ1 =
b0
1/ 2
2
φ0 =
φ0 = φ0
d1
1/ 3
3
b1
2/3
4
4 2
φ1 =
φ1 = φ1 = ⋅ φ0 = 2φ0
d2
5 / 10
3
3 3
b
4 / 10
8
8
16
φ3 = 2 φ 2 =
φ 2 = φ2 = ⋅ 2φ0 = φ0
d3
1/ 4
5
5
5
φ2 =
3
Use normalization
16 
77
 3
1 = φ0 + φ1 + φ 2 + φ 3 = 1 + + 2 + φ0 = φ 0
5
10
 2
Thus
φ0 =
10
77
φ2 = 2φ 0 =
3φ 0 15
=
2
77
16φ0 32
φ3 =
=
5
77
φ1 =
20
77
Example: The Random Walk on {0,…,N} with Partially Reflecting Boundaries
For i=1,…,N-1, bi=b, ri =r, and di =d are constant as in Fig. 12. b0=b and dN=d.
1-b
r
d
b
0
1-d
b
..…
i
d
...
Figure 12
Note that bi=b for i=0,1,…,N-1 and di=d for i=1,2,…,N.
Thus
φi +1 =
bi
b
φ i = φi
d i +1
d
for i=0,1,…,N-1. Using this relation recursively,
2
i
b
 b
b
φ i = φi −1 =   φ i − 2 = L =   φ 0
d
d
d 
N
Use normalization
1 = φ0 + φ1 + L + φ N
2
N


b
b
b




= 1 + +   + L +   φ 0

d d 
 d  

There are two cases:
Case 1: The asymmetric random walks, b ≠ d . Then this
normalization sum is a geometric series whose sum we know from Section 4.3.
1 − (b / d )
1=
1−b /d
N +1
φ0
With constant
C=
1− b/d
1 − (b / d )N +1
i
 b
φi =   C
d
for i = 0,1,2, K , N
Case 2: The symmetric random walk, b=d. Then the
normalization sum is
1 = (1 + 1 + 12 + L + 1N )φ0 = (N + 1)φ 0
Consequently,
φ0 =
1
N +1
i
1
b
φi =   φ0 = 1i φ0 = φ0 =
N +1
d 
STEADY-STATE PROBABILITYφi
State i
0
1
2
3
4
5
6
7
8
9
10
b/d=1 b/d=1.2 b/d=1.5 b/d=2 b/d=5
.0909
.0909
.0909
.0909
.0909
.0909
.0909
.0909
.0909
.0909
.0909
.0311
.0373
.0448
.0537
.0645
.0774
.0929
.1115
.1337
.1605
.1926
.0058
.0088
.0132
.0197
.0296
.0444
.0666
.0999
.1499
.2248
.3372
.0005
.0010
.0020
.0039
.0078
.0156
.0313
.0625
.1251
.2501
.5002
.0000
.0000
.0000
.0000
.0001
.0003
.0013
.0064
.0320
.1600
.8000
Aperiodicity, Indecomposability, and the Approach to the Steady State
Definition
Suppose that the sets A 1,A 2,…,A r partition the
state space S. That is, suppose that
S = U Ai
r
i=1
disjointly. Assume that a transition from A i to
Ai+1 must occur in one jump (where r+1 is taken to
be 1: The particle "wraps around" to A 1 from Ar )pictorially
A1 → A2 → L → Ar
That is, if at time t the particle is in one of
the states of A i, then at time t+1 it must be in
one of the states of Ai+1 . Then the Markov chain
is said to be periodic with period r. If the chain
is not periodic, it is said to be aperiodic.
Example
(a) The random walk infinite in both directions is periodic with period 2: Let A1=set
of even integers and A2=set of odd integers. Then
A1 → A2 → A1 → A2 → L
(b) The random walk with a boundary at 0 is periodic with period 2 if the reflecting
probability at 0 is r=1. If r<1, then the chain is aperiodic. This is the case since state
0 can be in one and only one partition set.
(c) The random walk with completely reflecting boundaries at 0 and N is periodic
with period 2 just as in part (a).
In looking for conditions implying existence of a steady state, we ought to exclude
periodic chains. Fortunately, there's an easy criterion to use to determine if a Markov
chain is aperiodic:
Suppose that
Π k ,k > 0
for any state k. Then the chain
is aperiodic.
This may not always apply: that is, there are chains with Π k ,k = 0 for all states k
which are periodic. But it is easy to spot entries Π k,k along the main diagonal of Π
which are nonzero; if there are any, the chain must be aperiodic.
To see why the boxed result is true, suppose that A1,…,A r is a periodic decomposition
of the state space S; thus the Ai 's are disjoint. But k ∈ Ai for some i: if Π k , k > 0 ,
then k ∈ Ai +1 , which contradicts the fact that Ai I Ai +1 = 0/ .
Definition
A Markov chain is indecomposable if for
every pair i,j of states, either i → j or
j → i or both.
Example
The random walk on S={0,1,2,…,N} with absorbing barriers 0 and N is not
indecomposable since neither 0 → N nor N → 0 . If N is partially reflecting, the
chain is indecomposable since N → 0 , although 0 →
/ N still.
Example
Is the chain in Fig. 13, (a) aperiodic? (b) indecomposable? (The arrows indicate
positive transition probabilities.)
Solution
(a) Note that i →
/ 2 for all states i. Let us start at state 2 and see where it leads:
{2} → {1} → {0,3} → {0,3,4} → {0,1,3, 4} → {0,1,3,4} → L
where the next set is obtained by placing into it all the states that the states in the
previous set can jump to in one transition. Thus the chain is aperiodic.
0
.
.
. .
1
3
.
2
4
Figure 13
(b) The chain is also indecomposable since for all pairs i,j either i → j or j → i
/ 2 for every i.
even though i →
Limit Theorem
Assume that the vector equation
φΠ = φ
has a solution φ that is a probability vector with
0 < φi < 1
N
∑φ
i
for all i ∈ S
=1
i =0
Then the following hold:
1. φ is unique: it is the only probability vector satisfying φ Π = φ .
2. Regardless of the initial probability vector ρ ,
ρ t = ρΠ t → φ
t→∞
as
.
3. If the initial probability vector is φ , then for all t,
φ t = φΠ t = φ
P ( X t = j | initial vector ρ ) → φ j
as t → ∞ for all j ∈ S . And property 3 states
for all t and all j ∈ S .
P ( X t = j | initial vector φ ) = φ j
Example
The general Two-State Process of Fig. 14
α 
1 − α
Π=
1 − β 
 β
Case 1: α = 0 and β = 0 . The particle will not move; the chain is decomposable.
Case 2: α = 1 and β = 1 . The particle jumps back and forth; the chain is periodic
with period 2.
Case 3: Either 0 < α < 1 or 0 < β < 1 or both.
1−α
(φ0 , φ1 )
 β
α 
= (φ 0 , φ1 )
1 − β 
implies that
φ 0 (1 − α ) + φ1 β = φ0
φ 0α + φ1 (1 − β ) = φ1
Both of these are the same equation:
αφ0 = βφ 1
Using normalization,
 α
1 = φ 0 + φ1 = 1 +
β


φ0

Therefore,
φ0 =
β
α+β
φ1 =
1− β
1 −α
. .
α
α+β
α
α
β
0
1
Figure 14
Corollary to the Limit Theorem
Under the same assumptions, the tth-order transition
t
matrix Π has a limit matrix
lim Π t = M
t →∞
where M has constant rows each equal to the
steady-state probability vector φ .
Example
For the Markov chain of the example earlier, successive powers of the transition
matrix Π yield
 0 1 / 2 1 / 2
Π =  0 2 / 3 1 / 3


1 / 3 2 / 3 0 
. 0876 .6524 .2600
Π 4 = . 0872 .6523 .2605


.0861 .6519 .2620
.1667 .6667 . 1667
Π 2 = . 1111 .6667 . 2222


 0
.6111 . 3889
.0870 .6522 .2609
Π 8 = .0870 .6522 .2609


.0870 .6522 .2609
which agrees exactly to four significant digits with the steady-state probability vector
φ found in another example (that is, each row ofΠ t approaches φ ).
Example 20
Consider this cyclic process - a special case , see Fig. 15
0 
1 / 10 9 / 10
Π= 0
3/ 4 1/ 4


 5 / 7
0
2 / 7
Using the notation of Example 9 yields
9
1
5
p1 =
p2 =
10
4
7
−1
−1
−1
C = p 0 + p1 + p 2 = 6.511
1
φ0 =
= .1706
Cp 0
p0 =
(
1
= .6143
Cp1
1
φ2 =
= .2150
Cp2
φ1 =
)
Computing powers successively implies
.765 .225 
 .01

Π = .1786 .5625 .2589
.2755 .6429 .0816
.1712 .6133 .2154
Π 4 = .1710 .6142 .2148
.1692 .6156 .2152
2
.1706 .6143 .2150
Π 8 = .1706 .6143 .2150
.1706 .6143 .2150
A final note about the steady-state probability vectorφ :
1/10
3/4
9/10
0 .
.
1
5/7
1/4
. 2/7
2
Figure 15
Existence Theorem
An aperiodic and indecomposable Markov chain with
finite state space has one and only one steady-state
probability vector.