ID
NAME
SCORE
MATH 564/STAT 555 Applied Stochastic Processes
Homework 3, October 9, 2015
Due October 23, 2015
1. (3 points) A fair die is thrown repeatedly. Let Xn denote the sum of the first n throws.
Find
lim P(Xn is a multiple of 13)
n→∞
quoting carefully any general theorem you use.
Solution: Let Yn = Xn (mod13). Then Y = (Yn )n≥0 is a Markov chain with state space
S = {0, 1, . . . , 12}. For each state i ∈ S only six successive states can be reached in one
step. Hence, each row of the transition matrix P has six entries equal to 1/6 and seven
entries equal to 0. Same argument shows that each column of P has six entries equal to
1/6 and seven entries equal to 0. Therefore, P is doubly stochastic. It follows that the
unique stationary distribution of Y is π = (1/13, 1/13, . . . , 1/13). Clearly Y is irreducible
(3)
(4)
and aperiodic (e.g. p00 > 0 and p00 > 0). Thus all assumptions of the Theorem 1.8.3 are
satisfied. It follows that
lim P(Xn is a multiple of 13) = lim P(Yn = 0) = π0 =
n→∞
n→∞
1
.
13
2. (3 points) Each morning a student takes one of the three books he owns from his shelf.
The probability that he chooses book i is αi , where 0 < αi < 1 for i = 1, 2, 3, and choices
on successive days are independent. In the evening he replaces the book at the left-hand end
of the shelf. If pn denotes the probability that on day n the student finds the books in the
order 1,2,3, from left to right, show that, irrespective of the initial arrangement of the books,
pn converges as n → ∞, and determine the limit.
Solution: For simplicity denote the permutations of 1,2,3 as A = 123, B = 132, C = 213,
D = 231, E = 312 and F = 321. The order of books on the shelf at the end of the day can
be modeled with a Markov chain (Xn )n≥0 with the state space S = {A, B, C, D, E, F } which
has the transition matrix


α1 0 α2 0 α3 0
 0 α1 α2 0 α3 0 


 α1 0 α2 0 0 α3 


P =

α
0
0
α
0
α
1
2
3


 0 α1 0 α2 α3 0 
0 α1 0 α2 0 α3
Note that pn = P(Xn = A). The chain is clearly irreducible and aperiodic. Since the
state space is finite, it is recurrent. Hence it has a unique stationary distribution π =
(πA , πB , πC , πD , πE , πE ). By the convergence to equilibrium theorem,
lim pn = lim P(Xn = A) = πA .
n→∞
n→∞
1
α1 α2
It is straightforward, (perhaps somewhat tedious), to compute that πA = 1−α
.
1
3. (3 points) Let X be a (λ, P ) Markov chain with state space S, and Y an independent
(µ, P ) Markov chain with the same state space. Assume that the product chain W = (X, Y )
is recurrent. Prove that
lim |P(Xn = j) − P(Yn = j)| = 0
n→∞
for all j ∈ S.
Solution: The proof is a modification of the proof of Theorem 1.8.3 (Convergence to equilibrium). The first step in the that proof was to show that T := min{n ≥ 0 : Xn = Yn = i0 } =
min{n ≥ 0 : Wn = (i0 , i0 )} is finite P-a.s. Here this is immediate since the product chain is
assumed to be recurrent. The rest of the proof is the same.
Note: In order to have that P(T < ∞) = 1 one should apply Theorem 1.5.7 which requires
W to be irreducible. The problem was not stated correctly - what I wanted to assume is
that if T = min{n ≥ 0 : Xn = Yn } is the coupling time, then P (T < ∞) = 1. Then the
second part of the proof of Theorem 1.8.3 (Steps 2 and 3) apply, proving that limn→∞ |P(Xn =
j) − P(Yn = j)| = 0 for all j ∈ S. The point is that now neither P(Xn = j) nor P(Yn = j)
need not converge.
4. (3 points) An opera singer is due to perform a long series of concerts. Having a fine
artistic temperament, she is liable to pull out each night with probability 1/2. Once this has
happened she will not sing again until the promoter convinces her of his high regard. This
he does by sending flowers every day until she returns. Flowers
costing x thousand dollars,
√
0 ≤ x ≤ 1, bring about a reconciliation with probability x. The promoter stands to make
$750 from each successful concert. How much should he spend on flowers?
Solution: Let S = {0, 1} with 1 - the opera singer performs, 0 - she does not sing. The
dynamics is modeled by a Markov chain(Xn )n≥0 with transition matrix (firt row 0, second
row 1)
√ √ x
1− x
,
P =
1/2
1/2
√
2 x
√ . Let
The stationary distribution π = (π0 , π1 ) of P is given by π0 = 1+21√x , π1 = 1+2
x
f : S → R be the cost function of the promoter: f (0) = −1000x, f (1) = 750. By the ergodic
theorem
√
√
n−1
2 x
−1000x + 1500 x
1X
1
√ + 750
√ =
√
lim
f (Xk ) = f (0)π0 + f (1)π1 = −1000x
n→∞ n
1+2 x
1+2 x
1+2 x
k=0
The expression on the right hand side has its maximum at x = 1/4 so the promoter should
spend $250 for flowers. With this amount used for flowers, his long run earning is $250 per
concert.
5. Happy Harry used to play semipro basketball where he was a defensive specialist. His
scoring productivity per game fluctuated between three states: 1 (scored 0 or 1 points), 2
(scored between 2 and 5 points), 3 (scored more than 5 points). Inevitably, if Harry scored a
lot of points in one game, his jealous teammates refused to pass the ball in the next game, so
his productivity it next game was nil. The team statistician, Mrs. Doc, upon observing the
transitions between states, concluded these transitions could be modelled by a Markov chain
with transition matrix


0 31 23
P =  13 0 32  .
1 0 0
2
(a) (2 points) What is the long run proportion of games that Harry had high scoring game?
(b) (1 point) The salary structure in the semipro leagues includes incentives for scoring. Harry
was paid $ 40/game for a high scoring performance, $ 30/game when he scored between 2 and
5 points, and only $ 20/game when he score nil. What was Harry’s long run earning rate?
Solution: (a) We first compute the stationary distribution π = (π1 , π2 , π3 ) which solves the
system π = πP . The solution is π = (9/20, 3/20, 8/20). The long run proposition of high
scoring games is according to the ergodic theorem equal to
n−1
2
1X
1(Xk =3) = π3 = a.s.
lim
n→∞ n
5
k=0
(b) Again by the ergodic theorem
n−1
1X
59
lim
a.s.
f (Xk ) = 20π1 + 30π2 + 40π3 =
n→∞ n
2
k=0
6. (5 points) In each of the following cases determine whether the stochastic matrix P , which
you may assume irreducible, is reversible:


0
p
1−p
1−a
a
0
p ;
(a)
;
(b)  1 − p
b
1−b
p
1−p
0
(c) S = {0, 1, . . . , N } and pij = 0 if |j − i| ≥ 2;
(d) S = {0, 1, , . . . } and p01 = 1, pi,i+1 = p, pi,i−1 = 1 − p for i ≥ 1;
(e) pij = pji for all i, j ∈ S.
Solution: By Theorem 1.9.3, the stochastic matrix P is reversible if and only if there exists a
distribution λ such that P and λ are in detailed balance - λi pij = λj pji for all i, j.
(a) P is reversible - take λ = (b, a) and normalize to probability.
(b) The detailed balance equations λ1 p = λ2 (1 − p), λ2 p = λ3 (1 − p) and λ3 p = λ1 (1 − p) have
a solution if and only if p = 1/2. Hence, P is reversible only when p = 1/2.
(c) This is similar to Example 1.9.5. The detailed balance equations are λi pi,i+1 = λi+1 pi+1,i
and have a unique solution
Qk−1
pi,i+1
λ0 , k = 1, . . . , N.
λk = Qi=0
k
i=1 pi,i−1
This can be normalized to a probability distribution, hence P is reversible.
(d) A unique solution to the detailed balance equations is λi = (1/q)(p/q)i λ0 , i ≥ 0. In case
p < q, this measure can be normalized, hence P is reversible. When p ≥ q, P is not reversible.
(e) The measure λi = 1, all i ∈ S, is a unique solution of the detailed balance equations. If
the state space is finite, it can be normalized, and then P is reversible. If the state space id
infinite, there is no finite measure in detailed balance with P , so P is not reversible.
3