Solutions to Problem Set #8
Section 11.2
6. In the Land of Oz example (also known as the “weather in Hanover” example from
class), change the transition matrix by making R an absorbing state. This gives


1
0
0
P =  1/2 0 1/2  .
1/4 1/4 1/2
Find the fundamental matrix N , and also N C and N R. Interpret the results.
0 1/2
By re-arranging P , it is easy to see that Q =
.
1/4 1/2
1/2 1/2
4/3 4/3
1
−1
, i.e. N =
.
Therefore, N = (I − Q) = (1/2)(1)−(1/2)(1/4)
1/4
1
2/3
8/3
8/3
1
Then, N c =
, and N R =
.
10/3
1
Interpretation: The matrix N c tells us that if today is a nice day, then we expect that,
after 8/3 days, it will always rain. If today is a snowy day, then we expect permanent rain
forecasts after 10/3 days. The matrix N R tells us that, regardless of whether today is a nice
day or a snowy day, the weather will eventually be permanently rainy with probability 1.
15. Consider the game of tennis where deuce is reached. If a player wins the next point,
he has advantage. On the following point, he either wins the game or the game returns to
deuce. Assume that for any point, player A has probability .6 of winning the point and player
B has probability .4 of winning the point.
(a) Set this up as a Markov chain with state 1: A wins; 2: B wins; 3: advantage A; 4:
deuce; 5: advantage B.
The re-arranged form

0 .4 0 .6
 .6 0 .4 0

P =
 0 .6 0 0
 0 0 0 1
0 0 0 0
of P looks like:

0
0 

.4 
.
0 
1
(b) Find the absorption probabilities.


1.462 0.769 0.308
N =  1.154 1.923 0.769  .
0.692 1.154 1.462




2.539
.877 .123
Then, t = N c =  4.226  , and N R =  .692 .308  .
3.308
.415 .585
1
Solutions to Problem Set #8
(c) At deuce, find the expected duration of the game and the probability that B will win.
From our matrix t, we see that when the score is deuce (i.e. when we’re in state 4), the
expected number of points to be played is 4.226. The probability that B wins (i.e. that the
process ends in state 2) is .16.
Section 11.3
2. Consider the Markov chain with transition matrix


1/2 1/3 1/6
P =  3/4 0 1/4  .
0
1
0
(a) Show that this is a regular

1/2 1/3
3

9/16 1/4
Note that P =
3/8 1/2
the Markov chain is regular.
Markov chain.

1/6
3/16  .Since all of the entries of P 3 are positive, then
3/24
(b) The process is started in state 1; find the probability that it is in state 3 after two
steps.
(2)
p13 = p11 · p13 + p12 · p23 + p13 · p33 =
1
1 1 1 1 1
· + · + ·0= .
2 6 3 4 6
6
(c) Find the limiting probability vector w.
To find the limiting probability vector, you need to solve the equation w · P = w. This
yields 4 equations with 3 unknowns:
w1 + w2 + w3 = 1
1/2w1 + 3/4w2 + 0w3 = w1
1/3w1 + 0w2 + 1w3 = w2
1/6w1 + 1/4w2 + 0w3 = w3 .
Solving this system yields w1 = 1/2, w2 = 1/3, w3 = 1/6.
3. Consider the Markov chain with general 2 × 2 transition matrix
1−a
a
P =
.
b
1−b
(a) Under what conditions is P absorbing?
2
Solutions to Problem Set #8
P will be absorbing when at least one of its rows looks like a row of the 2 × 2 identity
matrix, i.e. when a = 0 or when b = 0.
(b) Under what conditions is P ergodic but not regular?
In order for P to be ergodic, we need to have p12 6= 0 and p21 6= 0 (otherwise, it won’t
be possible to go from state 1 to state 2 or from state 2 to state 1). In order for P not to
be regular, we will need to have some 0’s in the matrix (this will prevent P from having
all positive entries). If there is only one 0 in the matrix (say p11 = 0) then P 2 will have
all positive entries. Thus, we need to have exactly two 0’s in the matrix (since we already
established that, in order for P to be ergodic, p12 and p21 must both be nonzero). This forces
us to take a = 1 and b = 1.
(c) Under what conditions is P regular?
P will be regular if either P consists only of positive entries or if P contains exactly one
0, in which case P 2 has only positive entries (as mentioned in part (b)). It’s important to
notice that, if a = 0, then the entry p12 will always be 0 for every power of P . Similarly, if
b = 0, then the entry p21 will be 0 for all powers of P . This reasoning yields the following
three scenarios in which P is regular:
1) 0 < a < 1 and 0 < b < 1
2) a = 1 and 0 < b < 1
3) 0 < a < 1 and b = 1
12. Consider Example 11.13 (Drunkard’s Walk). Assume that if the walker reaches state
0, he turns around and returns to state 1 on the next step and, similarly, if he reaches 4 he
returns on the next step to state 3. Is this new chain ergodic? Is it regular?
The new chain is ergodic but not regular. It’s easy to see that the matrix is ergodic (just
draw the matrix and check the possibilities!). To see that it’s not regular, we will appeal
to the “alternate” condition for regularity. Notice that if we start in state 0, then after any
even number of steps, we’ll be in states 0, 2 or 4, and after any odd number of steps, we’ll
be in states 1 or 3. So, there is no n for which it is possible to go from any state to any state
in exactly n steps.
3