Classical (and Useful) Markov
Chains
Markov Chains Seminar, 9.11.2016
Tomer Haimovich
Outline
1.
2.
3.
4.
5.
Gambler’s Ruin
Coupon Collecting
Hypercubes and the Ehrenfest Urn Model
Random Walks on Groups
Random Walks on ℤ
Gambler’s Ruin
Introduction
• A gambler bets on the outcomes of independent fair coin tosses
• If it’s heads, she wins 1. if it’s tails, she loses 1
• If she reaches 0 or 𝑛, she stops gambling
• When will the gambler reach one of the two outcomes?
• What are the probabilities of the two possibilities?
0
K-1
k
K+1
n
Definitions
• Let 𝑋𝑡 be gambler’s fortune at time 𝑡
• Let 𝜏 be the time to reach one of 𝑛 or 0
• Assume that 𝑋0 = 𝑘, where 0 ≤ 𝑘 ≤ 𝑛
• 𝑃𝑘 {𝑋𝜏 = 𝑛}: the probability of reaching 𝑛 after 𝜏 steps, when starting
in 𝑋0 = 𝑘
• 𝐸𝑘 𝜏 : the expected time to reach one of 𝑛 or 0
Probability of Reaching 𝑛
• Denote by 𝑝𝑘 the probability that the gambler ended at 𝑛
• 𝑝0 = 0, 𝑝𝑛 = 1
• For 1 ≤ 𝑘 ≤ 𝑛 − 1:
1
1
𝑝𝑘 = 𝑝𝑘−1 + 𝑝𝑘+1
2
2
• And when we solve the system of linear equations:
𝑘
𝑝𝑘 =
𝑛
Expected Time to be Absorbed
• Define 𝑓𝑘 ≔ 𝐸𝑘 (𝜏), the expected time to be absorbed, starting at
position 𝑘
• 𝑓0 = 𝑓𝑛 = 0
• For 1 ≤ 𝑘 ≤ 𝑛 − 1:
1
1
𝑓𝑘 = 1 + 𝑓𝑘+1 + 1 + 𝑓𝑘−1
2
2
• And when we solve the system of linear equations:
𝑓𝑘 = 𝑘 𝑛 − 𝑘
Coupon Collecting
Introduction
• 𝑛 different types of coupons. Gotta catch ‘em all!
• Each new coupon is equally likely to be each of the 𝑛 types
• How many coupons must be obtained to collect all 𝑛 types?
Why is it even a Markov Chain?
• Let 𝑋𝑡 denote the number of different types the collector has, after
obtaining 𝑡 coupons
• 𝑋0 = 0
• 𝑃 𝑘, 𝑘 + 1 = 𝑃 𝑋𝑡+1 = 𝑘 + 1 | 𝑋𝑡 = 𝑘 =
• 𝑃 𝑘, 𝑘 = 𝑃 𝑥𝑡+1 = 𝑘 | 𝑋𝑡 = 𝑘 =
𝑘
𝑛
𝑛−𝑘
𝑛
How many steps until collecting all types?
• Let 𝜏 be the number of coupons collected when the set first contains
every type
• We’ll see that:
𝑛
1
𝐸 𝜏 =𝑛
𝑘
𝑘=1
Proof
• Let 𝜏𝑘 be the number of collected coupons when the set first contains 𝑘
types
• By definition, 𝜏 = 𝜏𝑛
• Also:
𝜏𝑛 = 𝜏1 + 𝜏2 − 𝜏1 + ⋯ + 𝜏𝑛 − 𝜏𝑛−1
• 𝜏𝑘 − 𝜏𝑘−1 is a geometric random variable with success probability
• Therefore:
𝑛
𝐸 𝜏 =
𝑛
𝐸 𝜏𝑘 − 𝜏𝑘−1 =
𝑘=1
𝑘=1
𝑛
=𝑛
𝑛−𝑘+1
𝑛
𝑘=1
𝑛−(𝑘−1)
𝑛
1
=𝑛
𝑛−𝑘+1
𝑛
𝑘=1
1
𝑘
Hypercubes and the Ehrenfest
Urn Model
The 3-dimensional Hypercube
Lazy Random Walk
• To fix the periodicity issue, we define a lazy random walk on the
hypercube:
• Remain at current position with probability ½, otherwise pick a
coordinate uniformly and flip the bit
A Slight Detour: Reversible Chains
• A distribution 𝜋 is reversible if:
∀𝑖, 𝑗 𝜋𝑖 𝑃𝑖𝑗 = 𝜋𝑗 𝑃𝑗𝑖
• This is called the detailed balance property
• A Markov chain is reversible if it has a reversible distribution
• Lemma: A reversible distribution is a stationary distribution
Proof
𝑃11 𝑃12 𝑃13 𝑃14
𝑃21 𝑃22 𝑃23 𝑃24
𝜋1 , 𝜋2 , 𝜋3 , 𝜋4
𝑃31 𝑃32 𝑃33 𝑃34
𝑃41 𝑃42 𝑃43 𝑃44
= 𝜋1 𝑃11 + 𝜋2 𝑃21 + 𝜋3 𝑃31 + 𝜋4 𝑃41 , … , … , …
= 𝑃11 𝜋1 + 𝑃12 𝜋1 + 𝑃13 𝜋1 + 𝑃14 𝜋1 , … , … , …
= 𝜋1 (𝑃11 + 𝑃12 + 𝑃13 + 𝑃14 ), … , … , …
= (𝜋1 , … , … , … )
Stationary Distribution
• Any Markov chain which is defined by an undirected graph has the
stationary distribution
𝑑1 𝑑2
𝑑𝑛
𝜋=
,
,…,
2𝑚 2𝑚
2𝑚
• Proof:
𝑑𝑖 1
1 𝑑𝑗
1
𝜋𝑖 𝑃𝑖𝑗 = 𝜋𝑗 𝑃𝑗𝑖 ⟺
=
=
2𝑚 𝑑𝑖 𝑑𝑗 2𝑚 2𝑚
Back to Hypercubes
• The Hypercube graph is 𝑛-regular: the degree of any node is 𝑛
• So the stationary distribution is uniform
The Ehrenfest Urn
• There are 𝑛 balls, distributed among 2 urns
• At time 𝑡, a ball is selected uniformly at random and moved from its
current urn to the other
• Let 𝑋𝑡 denote the number of balls in urn 1 at time 𝑡
• The transition matrix:
• 𝑃 𝑘, 𝑘 + 1 =
• 𝑃 𝑘, 𝑘 − 1 =
𝑛−𝑘
𝑛
𝑘
𝑛
Stationary Distribution
• We’ll show that the stationary distribution is binomial with
parameters 𝑛 and ½:
𝑛
𝑛 1
𝜋𝑘 =
𝑘 2
• By verifying the useful detailed balance property
• (at least one case)
Detailed Balance
• Assuming 𝑘 = 𝑗 + 1, we need to show that:
𝜋𝑗 𝑃𝑗𝑘 = 𝜋𝑘 𝑃𝑘𝑗 ∀𝑖, 𝑗
• So:
𝑛
𝑗
𝑛
𝑛
1 𝑛−𝑗
𝑛 1 𝑘
=
2
𝑛
𝑘 2 𝑛
𝑛!
𝑛−𝑗
𝑛!
𝑗+1
⋅
=
⋅
𝑗! 𝑛 − 𝑗 !
𝑛
𝑗+1 ! 𝑛−𝑗−1 ! 𝑛
(𝑛 − 1)!
(𝑛 − 1)!
=
𝑗! 𝑛 − 𝑗 − 1 ! 𝑗! 𝑛 − 𝑗 − 1 !

Tying it all together
• Let (𝑋𝑡 ) be the simple random walk on the 𝑛-dimensional hypercube
• Let 𝑊𝑡 = 𝑊(𝑋𝑡 ) be the Hamming weight of 𝑋𝑡
• When 𝑊𝑖 = 𝑗:
• 𝑊𝑖+1 = 𝑗 + 1 when one of the 𝑛 − 𝑗 coordinates with value 0 is selected
• 𝑊𝑖+1 = 𝑗 − 1 when one of the 𝑗 coordinates with value 1 is selected
• Surprise:
• 𝑃 𝑘, 𝑘 + 1 =
• 𝑃 𝑘, 𝑘 − 1 =
𝑛−𝑘
𝑛
𝑘
𝑛
Random Walks on Groups
Groups
• A group (𝐺, ⋅) is a set with an associative operation ⋅ ∶ 𝐺 × 𝐺 → 𝐺
and an identity member 𝑖𝑑 ∈ 𝐺 such that:
• For all g ∈ 𝐺, 𝑖𝑑 ⋅ 𝑔 = 𝑔 ⋅ 𝑖𝑑 = 𝑔
• For all 𝑔 ∈ 𝐺, there exists an inverse 𝑔−1 ∈ 𝐺 s.t 𝑔 ⋅ 𝑔−1 = 𝑔−1 ⋅ 𝑔 = 𝑖𝑑
• Given a probability distribution 𝜇 on a group 𝐺, ⋅ , we define a
Markov chain with state space 𝐺 and transition matrix
𝑃 𝑔, ℎ𝑔 = 𝜇 ℎ
• This is called the random walk on 𝐺 with increment distribution 𝜇
Stationary Distribution
• The uniform probability distribution 𝑈 is a stationary distribution
• Proof:
For any 𝑔 ∈ 𝐺:
ℎ∈𝐺
1
𝑈 ℎ 𝑃(ℎ, 𝑔) =
|𝐺|
𝑘 = 𝑔ℎ
𝑘∈𝐺
−1
𝑃 𝑘
−1
1
𝑔, 𝑔 =
|𝐺|
𝑘∈𝐺
1
𝜇(𝑘) =
= 𝑈(𝑔)
|𝐺|
Generating Sets
• A subset 𝐻 ⊂ 𝐺 generates 𝐺 if every element of 𝐺 can be written as a
product of elements of 𝐻 and their inverses
• We’ll prove that:
𝑃𝜇 is irreducible ⇔ S = 𝑔 ∈ 𝐺 | 𝜇 𝑔 > 0 generates 𝐺
• We limit ourselves to finite groups
Proof (⇒)
• For any 𝑔 ∈ 𝐺, since the random walk is irreducible, there exists 𝑟
> 0 such that 𝑃𝑟 𝑖𝑑, 𝑔 > 0
• Therefore, there is a sequence such that 𝑔 = 𝑠𝑟 𝑠𝑟−1 … 𝑠1 and 𝑠𝑖 ∈ 𝑆
• So any 𝑔 ∈ 𝐺 can be expressed as a product of elements of 𝑆
Proof (⇐)
• For any 𝑎, 𝑏 ∈ 𝐺, 𝑏𝑎−1 can be written as a product of elements in S
and their inverses
• Since every element of 𝐺 has finite order, any inverse 𝑠 −1 can be
rewritten as 𝑠 𝑘 for some 𝑘
• This sequence defines the set of steps which can be done to reach
from 𝑎 to 𝑏
Symmetric Probability Distribution
• 𝜇 is a symmetric probability distribution if for every 𝑔 ∈ 𝐺:
𝜇 𝑔 = 𝜇 𝑔−1
• Proposition: the random walk 𝑃𝜇 is reversible if 𝜇 is symmetric
• Proof: We know that the uniform distribution 𝑈 is stationary. From
the detailed balance property:
𝑈 𝑔 𝑃 𝑔, ℎ =
𝜇 ℎ𝑔−1
|𝐺|
and 𝑈 ℎ 𝑃 ℎ, 𝑔 =
• These are equal if and only if 𝜇 ℎ𝑔−1 = 𝜇 ℎ𝑔−1
−1
𝜇 𝑔ℎ −1
|𝐺|
Random Walks on ℤ
Introduction
• We’ll focus on nearest-neighbor random walk on ℤ:
𝑃 𝑘, 𝑘 + 1 = 𝑥,
𝑃 𝑘, 𝑘 = 𝑦,
where 𝑥 + 𝑦 + 𝑧 = 1
• Simple walk is defined as 𝑥 = 𝑧 =
1
,𝑦
2
=0
𝑃 𝑘, 𝑘 − 1 = 𝑧
Will the Chain Reach 0?
• Denote the first time the walk hits zero by 𝜏𝑜 = min 𝑡 ≥ 0 ∶ 𝑋𝑡 = 0
• We’ll prove:
12𝑘
𝑃𝑘 𝜏0 > 𝑟 ≤
𝑟
For any integers 𝑘, 𝑟 > 0
Lemma 1 (Reflection Principal)
• For any positive integers 𝑗, 𝑘, 𝑟:
𝑃𝑘 𝜏0 < 𝑟, 𝑋𝑟 = 𝑗 = 𝑃𝑘 𝑋𝑟 = −𝑗
• When starting from 𝑘, the probability of reaching 0 in 𝑟 steps and
finishing in 𝑗 is equal to the probability of reaching −𝑗
−𝑗
0
𝑗
𝑘
Proof
• Assume we reached 0 in exactly 𝑠 steps
𝑃𝑘 𝜏0 = 𝑠, 𝑋𝑟 = 𝑗 = 𝑃𝑘 𝜏0 = 𝑠 𝑃0 𝑋𝑟−𝑠 = 𝑗
• Symmetry when starting at 0:
= 𝑃𝑘 𝜏0 = 𝑠 𝑃0 𝑋𝑟−𝑠 = −𝑗
= 𝑃𝑘 𝜏0 = 𝑠, 𝑋𝑟 = −𝑗
• And sum over all 𝑠 < 𝑟 to obtain 𝑃𝑘 𝜏0 < 𝑟, 𝑋𝑟 = 𝑗 = 𝑃𝑘 𝑋𝑟 = −𝑗
• Sum over all 𝑗:
𝑃𝑘 𝜏0 < 𝑟, 𝑋𝑟 > 0 = 𝑃𝑘 𝑋𝑟 < 0
Lemma 2
• For any 𝑘 > 0:
𝑃𝑘 𝜏0 > 𝑟 = 𝑃0 −𝑘 < 𝑋𝑟 ≤ 𝑘
• The probability to reach 0 in more than 𝑟 steps when starting from 𝑘,
equals the probability to end in −𝑘, 𝑘 after 𝑟 steps when starting in
0
Proof
𝑃𝑘 𝜏0 > 𝑟 = 𝑃0 −𝑘 < 𝑋𝑟 ≤ 𝑘
• Condition on the time to reach 0:
𝑃𝑘 𝑋𝑟 > 0 = 𝑃𝑘 𝑋𝑟 > 0, 𝜏0 ≤ 𝑟 + 𝑃𝑘 𝜏0 > 𝑟
• From Lemma 1:
𝑃𝑘 𝑋𝑟 > 0, 𝜏0 ≤ 𝑟 = 𝑃𝑘 𝑋𝑟 < 0
• From the symmetry of the walk:
𝑃𝑘 𝑋𝑟 < 0 = 𝑃𝑘 𝑋𝑟 > 2𝑘
• So:
𝑃𝑘 𝜏0 > 𝑟 = 𝑃𝑘 𝑋𝑟 > 0 − 𝑃𝑘 𝑋𝑟 > 2𝑘 = 𝑃𝑘 0 < 𝑋𝑟 ≤ 2𝑘
= 𝑃0 −𝑘 < 𝑋𝑟 ≤ 𝑘
Lemma 3
𝑃0 𝑋𝑡 = 𝑘 ≤
3
𝑡
• We’ll prove by splitting to two cases: 𝑡 = 2𝑟 and 𝑡 = 2𝑟 + 1
The constant 3 is not very tight, but if it’s good enough for Levin, Peres
& Wilmer it’s definitely good enough for me…
Proof (Case 𝑡 = 2𝑟)
𝑃0 𝑋𝑡 = 𝑘 ≤
3
• 𝑋2𝑟 must be an even number: 𝑋2𝑟 = 2𝑘
• To get to 2𝑘 we have to take 𝑟 + 𝑘 “up” moves and 𝑟 − 𝑘 “down”
2𝑟
moves, and the probability is 𝑟+𝑘 2−2𝑟
• This is maximized at 𝑘 = 0, so:
𝑃0 𝑋2𝑟
≤
3
𝑡
2𝑟
2𝑟 !
−2𝑟
= 2𝑘 ≤
2
=
𝑟+𝑘
𝑟! 2 22𝑟
≤
Stirling′ s formula
8 1
⋅
𝜋 2𝑟
𝑡
Proof (Case 𝑡 = 2𝑟 + 1)
𝑃0 𝑋𝑡 = 𝑘 ≤
• 𝑋2𝑟+1 must be an odd number: 𝑋2𝑟+1 = 2𝑘 + 1
• Condition on the first step to get:
1
1
𝑃0 𝑋2𝑟+1 = 2𝑘 + 1 = 𝑃1 𝑋2𝑟 = 2𝑘 + 1 + 𝑃−1 𝑋2𝑟 = 2𝑘 + 1
2
2
1
1
1 8 1
1 8 1
= 𝑃0 𝑋2𝑟 = 2𝑘 + 𝑃0 𝑋2𝑟 = 2𝑘 + 2 ≤
⋅
+
⋅
2
2
2 𝜋 2𝑟 2 𝜋 2𝑟
=
8
1
2𝑟 + 1
⋅
⋅
𝜋 2𝑟 + 1
2𝑟
𝑡
𝑡−1
1
2
1
2
≤ 2
2 8
1
4
1
3
≤
⋅
≤
⋅
≤
𝜋
𝜋 2𝑟 + 1
𝑡
2𝑟 + 1
3
𝑡
Tying it all together
12𝑘
𝑃𝑘 𝜏0 > 𝑟
= 𝑃0 −𝑘 < 𝑋𝑟 ≤ 𝑘
≤
𝑟
Lemma 2
Lemma 3
The Ballot Theorem
• We have two candidates, 𝐴 and 𝐵
• The final vote count is 𝑎 and 𝑏 respectively, 𝑎 < 𝑏
• At each step, we process one vote
• What is the probability that the winning candidate was always ahead?
The A-B Grid
Analysis
• We know we started from 0,0 and reached (𝑎, 𝑏) after 𝑎 + 𝑏 votes
• Exactly
𝑎+𝑏
𝑏
such paths
• If 𝐵 was always ahead, we never reached the line 𝑥 = 𝑦 after (0,0)
• So the first step was up:
𝑎+𝑏−1
𝑏−1
paths
• How many of these reached the line later?
The Reflection Property
• Take a path which started with “up”, touched 𝑥 = 𝑦 and ended at
(𝑎, 𝑏)
• Reflect the portion after the first touch of 𝑥 = 𝑦
• Now a path from (0,0), starting “up”, ending at (𝑏, 𝑎)
• Any such path must cross 𝑥 = 𝑦
• Therefore:
𝑎+𝑏−1
𝑏
“bad” paths we need to subtract
Final Details
𝑎+𝑏−1
paths which started with “up” and reached 𝑎, 𝑏
𝑏−1
Out of them, 𝑎+𝑏−1
are “bad” paths we need to subtract
𝑏
The total number of paths is 𝑎+𝑏
𝑏
𝑎+𝑏−1
𝑎+𝑏−1
−
𝑎! 𝑏!
𝑎+𝑏−1 !
𝑎+𝑏−1 !
𝑏−1
𝑏
=
−
𝑎+𝑏
𝑎 + 𝑏 ! 𝑎! 𝑏 − 1 !
(𝑎 − 1)! 𝑏!
𝑏
• There are
•
•
𝑏
𝑎
𝑏−𝑎
=
−
=
𝑎+𝑏 𝑎+𝑏 𝑎+𝑏