Homework set 7 - Solutions
Math 495 – Renato Feres
Problems
1. (Text, Exercise 7.1, page 170.) Show that every irreducible, discrete-time, two-state Markov chain is reversible
with respect to its invariant probability.
Solution.
Ã
The general form of an irreducible, discrete-time, two-state transition probability matrix is P =
1−p
p
!
.
q
1−q
Let {1.2} be the set of states. The stationary probability distribution,
³
´ characterized as the unique probability
p
q
vector π that satisfies π(I − P ) = 0, is easily shown to be π = p+q , p+q . Observe now that
π(1)P (1, 2) =
qp
pq
=
= π(2)P (2, 1).
p +q p +q
Therefore, the Markov chain is reversible with respect to π.
♦
2. (Text, Exercise 7.2, page 170.) Suppose X t is a continuous-time Markov chain with state space S = {1, . . . , N } and
symmetric rates α.
(a) Show that for all t ≥ 0 and all x,
P{X t = x|X 0 = x} ≥
[Hint: write P{X t = x|X 0 = x} =
P
y∈S P{X t /2
1
.
N
= y|X 0 = x}2 .]
(b) Give an example of a non-symmetric chain whose invariant probability distribution is uniform such that
(7.5) does not hold for some x ∈ S, t > 0. (You may use R if you like.)
Solution. (a) For each x and t let us write a y = P(X t /2 = y|X 0 = x) = p t /2 (x, y). Because the Markov chain is
symmetric in α, it follows that the transition probabilities are also symmetric, so p t /2 (y, x) = p t /2 (x, y) = a y . It
follows that
p t (x, x) =
X
p t /2 (x, y)p t /2 (y, x) =
y
X
y
a 2y .
Thus we want to show that the set of non-negative numbers a y satisfying the constraint equation G(a 1 , . . . , a N ) =
P 2
P
y a y = 1 has the property F (a 1 , . . . , a N ) = y a y ≥ 1/N . This is the typical problem for which we can use the idea
of Lagrange multipliers. Note that the function F to be minimized is quadratic and all the second derivatives
in a y are positive, so the point of minimum is given by the unique critical point of F under constraint G if the
critical point has non-negative coordinates adding up to 1. By the method of Lagrange multipliers,
∂F
∂G
=λ
∂a y
∂a y
holds for all y at a critical point. This amounts to 2a y = λ for all y, so a y = 1/N . This implies that (1/N , . . . , 1/N )
is the point of minimum. At this point
F (1/N , . . . , 1/N ) =
1
1
1
+···+ 2 = .
2
N
N
N
This establishes the desired inequality.
(b) Consider the Markov with 3 set of states S = {1, 2, 3} and transition rates α(1, 2) = α(2, 3) = α(3, 1) = 1 and all
the other rates equal to zero. This is clearly not symmetric. (The states rotate in the order 1, 2, 3, 1, 2, 3, 1, . . . .)
This is an irreducible, recurrent chain with rate matrix

1
−1

A= 0
0

1 .
−1
1
0

−1
It is easily checked that the uniform distribution π = (1/3, 1/3, 1/3) satisfies πA = 0, so it is the unique stationary
measure. To see that the inequality is not satisfied, we obtain the transition probabilities from P t = e t A using the
exponential of matrix function expm in R. (This requires the R-package expm.) The following computation gives
P t (1, 1) for t = 2.5. The resulting value is less than 0.325. A similar chain, in which the states alternate in cyclic
fashion, but with more states, would show the breakdown of the inequality in more dramatic form.
A=matrix(0,3,3)
A[1,]=c(-1,1,0)
A[2,]=c(0,-1,1)
A[3,]=c(1,0,-1)
> expm(2.5*A)
[,1]
[,2]
[,3]
[1,] 0.3245549 0.3489727 0.3264724
[2,] 0.3264724 0.3245549 0.3489727
[3,] 0.3489727 0.3264724 0.3245549
Note that 0.3246 < 1/3. This means that the transition from
♦
3. (Text, Exercise 7.3, page 170.) Let X n be an aperiodic, discrete-time Markov chain on S = {1, . . . , N } whose transition probability is symmetric. Show that for all x ∈ S and all non-negative integers n,
P{X 2n |X 0 = x} ≥
1
.
N
Does this hold if 2n is replaced with 2n + 1?
Solution. The proof of the inequality is the same used in the previous problem. However, the main trick only
applies when the number of steps is even:
p 2n (x, x) =
X
p n (x, y)p n (y, x) =
y
X
y
2
p n (x, y)2 .
At this point the same argument using Lagrange multipliers can be used to obtain the same conclusion.
To see that the inequality breaks down for transitions in an odd number of steps, consider the two-state chain
of the first problem, where p = q is greater than 1/2. Then the one-step transition from 1 to 1 is 1 − p < 1/2. This
shows that this symmetric irreducible chain violates the inequality when the number of steps is not even.
♦
4. (Text, Exercise 7.9, page 171.) Find the eigenvalues of the N × N matrix A from example 2, Section 7.2,


−1


A(i , j ) = 1/2



0
if i = j
if |i − j | ≡ 1(mod N )
otherwise.
[Hint: any eigenvector with eigenvalue λ can be considered as a function f (n) on the integers satisfying
λ f (n) =
1
1
f (n + 1) + f (n − 1) − f (n),
2
2
and the periodicity condition f (n) = f (n + N ), for each n. Find the general solution of the difference equation
and then use the periodicity condition to put restrictions on the λ.]
Solution. The N -periodic functions on the integers (modulo N ) form a vector space of dimension N . This is
because we only need to specify the values of the periodic function on the set of integers 0, 1, . . . , N − 1 for the
function to be fully defined. A basis for this space is given by the functions f 0 (n), f 1 (n), . . . , f N −1 (n) such that
f k (n) := e
2πi kn
N
Replacing f k into the right-hand side of the above difference equation gives
!
à 2πi k
−2πi k
2πi k(n+1)
2πi k(n+1)
1
1 2πi k(n+1) 1 2πi k(n−1)
1
e N +e N
N
N
N
f k (n + 1) + f k (n − 1) − f k (n) = e
−1 e N
+ e
−e
=
2
2
2
2
2
That is,
2πi k(n+1)
1
1
f k (n + 1) + f k (n − 1) − f k (n) = (cos(2πk/N ) − 1) e N
2
2
This means that the f k are themselves a complete set of eigenfunctions associated to the eigenvalues
λk = cos(2πk/N ) − 1
for k = 0, 1, . . . , N − 1.
♦
5. (Text, Exercise 7.6, page 171.) COMPUTER SIMULATION. Let M be a matrix chosen uniformly from the set of
50 × 50 matrices with entries 0 and 1 such that no two 1s are together (see Section 7.3). Use a Markov chain
simulation as described in Section 7.3 to estimate the probability that the M(25, 25) entry of this matrix is a 1.
Solution. The following R script does the experiment.
n=50
3
M=matrix(0,nrow=n+2,ncol=n+2)
#Only entries (i,j), i, j between 2 and 51 can be changed.
#The others form a margin of zeros framing the matrix.
N0=10000
N=10000000
X=0*c(1:N)
#Elements of X record the entry ((N+2)/2,(N+2)/2)
#of the random matrix at each step of the Markov
#chain.
for (s in 1:N){
i=sample(2:(n+1),1,replace=TRUE)
j=sample(2:(n+1),1,replace=TRUE)
a=M[i,j-1]+M[i,j+1]+M[i-1,j]+M[i+1,j]
if (a==0){
M[i,j]=1*(M[i,j]==0)
}
X[s]=M[(n+2)/2,(n+2)/2]
}
f=sum(X[N0:N])/(N-N0+1)
To see that it is working let us consider the case of two-by-two matrices first. In this case we can easily count
7 allowed matrices in total and 2 matrices having 1 on the (1, 1)-entry. So the probability of a 1 there is 2/7 =
0.2857143. One run of the simulation with 107 sampled matrices gave me the value 0.2849455.
There are 63 allowed 3-by-3 matrices and 21 among them with 1 on the (1, 1) entry. So the fraction is 21/63 =
0.3333. One run of the simulation with 107 sampled matrices gave me the value 0.3329. So the program seem to
be working OK.
For n = 50 (50-by-50 matrices) I obtained f = 0.2193 for the fraction of matrices with 1 in the (25, 25)-entry. So
the asked for probability seems to be approximately 0.22.
♦
4