Applications of Linear Algebra: Markov Chains I
A probability vector is a vector with nonnegative entries which sum to 1:
n
X
(i) pi ≥ 0
and
(ii)
pi = 1.
i=1
A Markov chain is a system which moves among the elements of a set (the state
space) in the following way: when at state x, the next position is chosen according to
a fixed probability distribution/vector P (x, ·). We write this probability vector as the
xth row of a square matrix:

 

p11 p12 . . . p1n
P (1, 1) P (1, 2) . . . P (1, n)
 P (2, 1) P (2, 2) . . . P (2, n)   p21 p22 . . . p2n 
= .
P =
.. . .
.. 
..
..
..
..
  ..

.
.
.
. 
.
.
.
P (n, 1) P (n, 2) . . .
pn1 pn2 . . .
P (n, n)
pnn
P is called the transition matrix, since P (x, y) is the probability that the system
makes a transition from state x to state y. If Xn is the state of the system at time n,
then we can write this as a conditional probability:
P (x, y) = P r(X1 = y | X0 = x) = P r(Xn+1 = y | Xn = x),
for all n = 0, 1, 2, . . . .
The key property of Markov chains is that next state depends ONLY on the current
state: the previous history is irrelevant. Mathematically, this means that the probabilities governing the next transition are independent of everything except the current
location:
P r(Xn+1 = y | Xn = x, Xn−1 = xn−1 , . . . , X1 = x1 , X0 = x0 ) = P r(Xn+1 = y | Xn = x).
1. Which of the following are probability vectors?
a) 12 13 23
b) 0 1 0
c) 14 61 13 14
2. Consider the matrices


0.2 0.8 0.0
0.3 0.7
a)
b)  0.3 0.5 0.2 
0.4 0.6
0.1 0.7 0.2
c)
0 1
1 0
d)
1
5
2
5
1
10
2
10


0.3 0.2 0.1
d)  0.4 0.0 0.3 
0.2 0.8 0.6
Which of (a)–(d) can be transition matrices of a Markov chain?
3. Determine the value of each missing entry (denoted by ) so that the matrix
will be a transition matrix of a Markov process. There may be more than one
right answer.




0.4 0.3
0.2 0.3
a)  0.3 0.5 
b)  0.3 0.5 
0.2 0.2
2
Applications of Linear Algebra
Suppose that µ0 is a probability vector which gives the state of the system at time
n = 0. Then µ1 := µ0 P is the vector whose xth entry is the likelihood that the system
is in state x at time n = 1. In general, µn := µn−1 P = µn−2 P 2 = · · · = µ0 P n is the
vector whose xth entry is the likelihood that the system is in state x at time n.
4. Even though the system evolves randomly, there are many applications where
you can start the chain in a particular chosen state. For a Markov chain with 6
states, give the vector µ0 = [
] which corresponds to starting
the system in the third state.
5. Suppose a frog has two lily pads e and w (east and west). If the frog
up on the east lily pad, then he hops to the west with probability p and
there. Otherwise, he spends the night on the east lily pad. If the frog
up on the west lily pad, then he hops to the east with probability q and
there. Otherwise, he spends the night on the west lily pad.
wakes
sleeps
wakes
sleeps
a) What is the transition matrix for this Markov chain, for a given p and q?
b) Suppose the frog starts on the east lily pad. Plot the function f (n) =
P r(Xn = e) as a function of n, for n = 0, 1, . . . , 20 for each of (i) p = q = 21 ,
(ii) p = 0.2, q = 0.1, (iii) p = 0.95, q = 0.7, and (iv) p = q = 1.
c) For p, q < 1, the plots in (b) suggest the existence of a limiting steady-state
vector µ̄ = limn→∞ µn . Such an “equilibrium distribution” must be a fixed
point of P : µ̄ = µ̄P . From µ̄ = µ̄P , compute the entries of µ̄ in terms of
p and q.
d) Now we will show that µn tends to µ̄ for ANY starting state µ0 , provided
0 < p, q < 1. Define ∆n = µn (e) − µ̄(e).
(i) Why is it true that limn→∞ µn = µ̄ if and only if limn→∞ ∆n = 0?
(ii) Use the definition of ∆n and µn+1 to see that ∆n+1 = (1 − p − q)∆n .
(iii) Use ∆n+1 = (1 − p − q)∆n to see that limn→∞ ∆n = 0.
(Hint: for a constant a ∈ R, limn→∞ an = 0 if and only if |a| < 1.)
e) Write out the transition matrix P for this chain. Find two numbers λ such
that P x = λx has nontrivial solutions.
(Hint: λ must satisfy det(λI − P ) = 0.)
f) The numbers λ found in part (d) are called eigenvalues of P . For each λ,
solve P x = λx to obtain an eigenvector of P corresponding to λ.
g) What do the eigenvalues λ have to do with convergence of the Markov
chain? What do the corresponding eigenvectors x have to do with convergence of the Markov chain? What is the determinant of P ?
(Hint: write the eigenvector so that the denominator of the entries is p+q.)