Lecture 2
1
Martingales
We now introduce some fundamental tools in martingale theory, which are useful in controlling
the fluctuation of martingales.
1.1
Doob’s inequality
We have the following maximal inequality of Doob, which allows us to control the fluctuation
of the whole path (Xi )0≤i≤n by controlling the fluctuation of the end point Xn , when X is a
(sub-)martingale.
Theorem 1.1 [Doob’s maximal inequality]
Let (Xi )i∈N be a sub-martingale w.r.t. a filtration (Fi )i∈N . Let Sn = max1≤i≤n Xi be the
running maximum of Xi . Then for any l > 0,
1
1 +
E Xn 1{Sn ≥l} ≤ E[Xn+ ],
l
l
P(Sn ≥ l) ≤
(1.1)
where Xn+ = Xn ∨ 0. In particular, if Xi is a martingale and Mn = max1≤i≤n |Xi |, then
P(Mn ≥ l) ≤
1
1 E |Xn |1{Mn ≥l} ≤ E[|Xn |].
l
l
(1.2)
Proof. Let τl = inf{i ≥ 1 : Xi ≥ l}. Then
P(Sn ≥ l) =
n
X
P(τl = i).
(1.3)
i=1
For each 1 ≤ i ≤ n,
P(τl = i) = E[1{Xi ≥l} 1{τl =i} ] ≤
1
E[Xi+ 1{τl =i} ].
l
(1.4)
Note that {τl = i} ∈ Fi , and Xi+ is a sub-martingale because Xi itself is a sub-martingale
while φ(x) = x+ is an increasing convex function. Therefore
E[Xn+ 1{τl =i} |Fi ] = 1{τl =i} E[Xn+ |Fi ] ≥ 1{τl =i} E[Xn |Fi ]+ ≥ 1{τl =i} Xi+ ,
and hence
E[Xi+ 1{τl =i} ] ≤ E[Xn+ 1{τl =i} ].
Substituting this inequality into (1.4) and then summing over 1 ≤ i ≤ n then yields (1.1).
(1.2) follows by applying (1.1) to the sub-martingale |Xi |.
Corollary 1.2 Let X, S and M be as in Theorem 1.1. Then for any p ≥ 1, we have
P(Sn ≥ l) ≤
P(Mn ≥ l) ≤
1
lp
1
lp
1
E (Xn+ )p 1{Sn ≥l} ≤ p E[(Xn+ )p ],
l
1
p
E |Xn | 1{Mn ≥l} ≤ p E[|Xn |p ].
l
1
(1.5)
(1.6)
Proof. Apply Theorem 1.1 respectively to the sub-martingales (Xn+ )p and |Xn |p .
The bound we have on P(Sn ≥ l) allows us to obtain bounds on the moments of Sn+ .
Corollary 1.3 [Doob’s Lp maximal inequality]
Let Xi and Si be as in Theorem 1.1. Then for p > 1, we have
p p
E[(Sn+ )p ] ≤
E[(Xn+ )p ].
p−1
(1.7)
Proof. Note that by the layer-cake representation of an integral, we have
Z ∞
Z ∞
+ p
+ p
E[(Sn ) ] =
P((Sn ) > t)dt = p
lp−1 P(Sn ≥ l)dl
0
0
Z ∞
Z ∞
i
h
p−2
+
+
lp−2 1{Sn ≥l} dl
l E[Xn 1{Sn ≥l} ] dl = p E Xn
≤ p
0
0
=
p−1
1
p
p
E Xn+ (Sn+ )p−1 ≤
E[(Xn+ )p ] p E[(Sn+ )p ] p .
p−1
p−1
If E[(Sn+ )p ] < ∞, then (1.7) follows immediately. Otherwise, we can first replace Sn+ by Sn+ ∧N
and repeat the above estimates, and then send N → ∞ and apply the monotone convergence
theorem.
Pn
Example 1.4 Let Xn =
i=1 ξi be a random walk with X0 = 0, where (ξi )i∈N are i.i.d.
random variables with E[ξ1 ] = 0 and E[ξ12 ] = σ 2 < ∞. Then Xn is a square-integrable
martingale. Let Sn = max0≤i≤n Xi . By Doob’s inequalities, we have
S
E[X 2 ]
σ2
n
n
P √ ≥a ≤
=
na2
a2
n
and
h S 2 i 4
n
E √
≤ E[Xn2 ] = 4σ 2 .
n
n
(1.8)
Exercise 1.5 Show that when ξi are bounded, the tail bound for Sn in (1.8) can be improved
to a Gaussian tail bound as in the Azuma-Hoeffding inequality.
Exercise 1.6 Doob’s Lp maximal inequality fails for p = 1. Indeed, try to construct a nonnegative martingale Xn with E[Xn ] ≡ 1, and yet supn∈N E[Sn ] = ∞. To get a bound on E[Sn+ ],
we need a bit more than E[Xn+ ] < ∞.
Exercise 1.7 Let Xn be a sub-martingale and let Sn = max1≤i≤n Xi . By mimicking the proof
of Doob’s Lp maximal inequality and by using a log b ≤ a log a + b/e for a, b > 0, show that
E[Sn+ ] ≤
1 + E[Xn+ log+ (Xn+ )]
,
1 − e−1
where log+ x = max{0, log x}.
1.2
Stopping times
It is often useful to stop a stochastic process at a time which is determined from past observations of the process. Such times are called stopping times.
Definition 1.8 [Stopping time]
Given (Ω, F, P), let (Fn )n∈N be a filtration in F. A {0} ∪ N ∪ {∞}-valued random variable τ is
called a stopping time w.r.t. (Fn )n∈N , if for every n ≥ 0, the event {ω ∈ Ω : τ (ω) = n} ∈ Fn .
2
Remark 1.9 Intuitively, a stopping time τ is a decision on when to stop the stochastic process
(Xn )n∈N , using only information up to that time. Examples of stopping times include: τ := k
for some fixed k ≥ 0; τ := inf{n ≥ 0 : Xn ≥ a} for some a ∈ R, i.e., the first passage time of
level a for a process (Xn )n∈N adapted to (Fn )n∈N ; τ := τ1 ∧ τ2 or τ := τ1 ∨ τ2 , the minimum
or maximum of two stopping times τ1 and τ2 .
Definition 1.10 [Stopped σ-field]
Let τ be a stopping time w.r.t. the filtration (Fn )n∈N . The stopped σ-field Fτ associated with
τ is defined to be
Fτ := {A ∈ F : A ∩ {ω : τ (ω) = n} ∈ Fn for all n ≥ 0}.
(1.9)
Remark 1.11 Fτ can be interpreted as the information available up to the stopping time τ .
For each event A ∈ Fτ , we can determine whether it has occurred or not based on what we
have observed about the process up to (and including) time τ .
Exercise 1.12 Verify that Fτ is indeed a σ-algebra, and show that the definition of stopping
times and stopped σ-fields are unchanged if we replace τ (ω) = n by τ (ω) ≤ n, but not by
τ (ω) < n.
Theorem 1.13 [Stopped martingales are martingales]
Let (Xn )n∈N be a martingale, and τ a stopping time, then the stopped martingale (Xn∧τ )n∈N is
also a martingale. More generally, if θ is another stopping time and θ ≤ τ , then Xn∧τ − Xn∧θ
is a martingale. If X is a sub/super-martingale, then Xn∧τ − Xn∧θ is also a sub/supermartingale.
Proof. We will verify that Xn∧τ − Xn∧θ is a super-martingale when X is a super-martingale.
The rest then follows. Note that
Xn∧τ − Xn∧θ =
n
X
1{θ<i≤τ } (Xi − Xi−1 ),
i=1
where {θ < i ≤ τ } ∈ Fi−1 since {i ≤ τ } = {τ ≤ i − 1}c ∈ Fi−1 . Therefore, Xn∧τ − Xn∧θ is a
martingale transform of Xn , and
E[Xn∧τ − Xn∧θ |Fn−1 ] = X(n−1)∧τ − X(n−1)∧θ + 1{θ<n≤τ } E[Xn − Xn−1 |Fn−1 ]
≤ X(n−1)∧τ − X(n−1)∧θ .
Therefore Xn∧τ − Xn∧θ is a super-martingale when Xn is a super-martingale.
If the price of a stock evolves in time as a martingale, then Theorem 1.13 tells us that no
matter when do we decide to buy and sell the stock, as long as our strategy is only based on
past observations, the expected payoff will be zero.
1.3
Upcrossing inequality, almost sure Martingale Convergence, and Polya’s
urn
As an application of the notion of stopped martingales, we prove the upcrossing inequality.
Let (Xi )0≤i≤n be a super-martingale adapted to the filtration (Fi )0≤i≤n on a probability
space (Ω, F, P). Let a < b. An upcrossing by X over the interval (a, b) is a pair of indices
3
0 ≤ k < l ≤ n with Xk ≤ a and Xl ≥ b. We are interested in the number Un of complete
upcrossings X makes before time n. Define recursively
τ1 := inf{i : Xi ≤ a},
τ2 := inf{i ≥ τ1 : Xi ≥ b},
..
.
τ2k+1 := inf{i ≥ τ2k : Xi ≤ a},
τ2k+2 := inf{i ≥ τ2k+1 : Xi ≥ b},
..
. ,
where the infimum of an empty set is taken to be ∞. Note that τi are all stopping times, and
the number of completed upcrossings before time n is given by Un = max{k : τ2k ≤ n}.
Theorem 1.14 [Upcrossing inequality]
Let (Xi )0≤i≤n be a (super-)martingale and let Un be the number of complete upcrossings over
(a, b) defined as above. Then
E[Un ] ≤
E[(a − Xn )+ ]
|a| + E[|Xn− |]
≤
.
b−a
b−a
(1.10)
Proof. By Theorem (1.13), (Xi∧τ2k −Xi∧τ2k−1 )0≤i≤n is a super-martingale for each 1 ≤ k ≤ n.
Therefore
n
hX
i
E
Xn∧τ2k − Xn∧τ2k−1 ≤ 0.
(1.11)
k=1
On the other hand, note that


Xτ2k − Xτ2k−1 ≥ b − a



X − X
n
τ2Un +1 ≥ Xn − a
Xn∧τ2k − Xn∧τ2k−1 =

Xn − Xn = 0



X − X = 0
n
n
Therefore,
n
X
if 1 ≤ k ≤ Un ,
if k = Un + 1 and τ2Un +1 ≤ n,
if k = Un + 1 and τ2Un +1 > n,
if k ≥ Un + 2.
Xn∧τ2k − Xn∧τ2k−1 ≥ (b − a)Un + 1{τ2Un +1 ≤n} (Xn − a).
k=1
Taking expectation and combined with (1.11) then yields
(b − a)E[Un ] ≤ E[(a − Xn )1{τ2Un +1 ≤n} ] ≤ E[(a − Xn )+ ],
(1.12)
and hence (1.10). To summarize, the number of upcrossings has to be balanced out by Xn− if
X is a super-martingale, which explains why E[Un ] is controlled by E[|Xn− |].
Remark 1.15 Note that the number of upcrossings and downcrossings differ by at most 1.
When Xn is a sub-martingale, the expected number of upcrossings can be bounded in terms of
E[(Xn − a)+ ]. See Theorem 4.(2.9) in Durrett [1].
A consequence of the upcrossing inequality is the almost sure martingale convergence
theorem for L1 -bounded martingales.
4
Theorem 1.16 [Martingale convergence theorem]
Let (Xn )n∈N be a (super)-martingale with supn E[|Xn− |] < ∞. Then X = limn→∞ Xn exists almost surely, and E[|X|] < ∞. If Xn is a sub-martingale, then the condition becomes
supn E[Xn+ ] < ∞.
Proof. By the upcrossing inequality, for any a < b,
|a| + E[|Xn− |]
|a| + supn∈N E[|Xn− |]
≤
< ∞.
b−a
b−a
n∈N
sup E[Un ] ≤ sup
n∈N
(1.13)
Since Un almost surely increases to a limit U (a, b), which is the total number of upcrossings
by X over (a, b), by Fatou’s Lemma, E[U (a, b)] < ∞ and hence U (a, b) < ∞ almost surely.
Thus, almost surely, U (a, b) < ∞ for all a < b ∈ Q, which implies that
lim sup Xn = lim inf Xn ,
n→∞
n→∞
and hence X = limn→∞ Xn exists in [−∞, ∞] almost surely.
By Fatou’s Lemma,
E[|X − |] ≤ lim inf E[|Xn− |] < ∞
n→∞
by assumption. Similarly, by Fatou and the fact that Xn is a super-martingale,
E[X + ] ≤ lim inf E[Xn+ ] = lim inf (E[Xn ] + E[|Xn− |]) ≤ lim inf (E[X1 ] + E[|Xn− |]) < ∞.
n→∞
n→∞
n→∞
Therefore E[|X|] < ∞.
Corollary 1.17 [Almost sure convergence of a non-negative supermartingale]
If (Xn )n∈N is a non-negative super-martingale, then X = limn→∞ Xn exists a.s., and E[X] ≤
E[X1 ].
Remark 1.18 In Corollary 1.17, we could have E[X] < E[X1 ] since part of the measure
P
Xn dP could escape to ∞. For example, let Xn = X0 + ni=1 ξi be a symmetric simple random
walk on Z starting from X0 = 1, where ξi are i.i.d. with P(ξi = 1) = P(ξi = −1) = 21 . Let
τ = inf{n : Xn = 0} be the first hitting time of the origin. Then Xn∧τ is a non-negative
martingale which converges to a limit Xτ almost surely. The only possible limit for Xn∧τ is
either 0 or ∞. Since E[Xτ ] ≤ E[X1 ] = 1 by Corollary 1.17, we must have Xτ = 0 almost
surely. Note that this also implies that τ < ∞ almost surely, so that the symmetric simple
random walk always visits 0, which is a property called recurrence.
Exercise 1.19 Show by example that a non-negative sub-martingale need not converge almost
surely.
As an application of Corollary 1.17, we have the following dichotomy between convergence
and unbounded oscillation for martingales with bounded increments.
Theorem 1.20 Let (Xn )n∈N be a martingale with |Xn+1 − Xn | ≤ M < ∞ a.s. for all n ≥
0. Then almost surely, either limn→∞ Xn exists and is finite, or lim supn→∞ Xn = ∞ and
lim inf n→∞ Xn = −∞.
5
Proof. For L < 0, let τL = inf{n : Xn ≤ L}, which is a stopping time. By assumption,
XτL ≥ L − M . Therefore Xn∧τL − L + M is a non-negative martingale, which converges a.s. to
a finite limit. In particular, on the event {τL = ∞}, limn→∞ Xn exists and is finite. Letting
L → −∞, it follows that on the event {lim inf n→∞ Xn > −∞}, limn→∞ Xn exists and is finite.
Applying the same argument to −Xn implies that limn→∞ Xn exists and is finite on the event
that {lim supn→∞ Xn < ∞}. The theorem then follows.
Exercise 1.21 Construct an example where the conclusion in Theorem 1.20 fails if the bounded
P
increment assumption is removed. (Hint: Let Xn = ni=1 ξi , where ξi are independent with
mean 0 but not i.i.d., such that Xn → −∞ almost surely.)
P
Recall that the Borel-Cantelli lemma states: If (An )n∈N ∈ F satisfy ∞
i=1 P(An ) < ∞,
then almost surely, (An )n∈N occurs only finitely many times. If (An )n∈N are all independent,
P
then ∞
i=1 P(An ) = ∞ guarantees that almost surely, (An )n∈N occurs infinitely often. Using
Theorem 1.20, we give a proof of the second Borel-Cantelli lemma which allows dependence
of events.
Corollary 1.22 [Second Borel-Cantelli Lemma]
Let (Fn )n≥0 be a filtration on the probability space (Ω, F, P) with F0 = {∅, Ω}. Let An ∈ Fn
for n ≥ 1. Then
∞
∞
n
o n
o
X
X
1An (ω) = ∞ = ω :
P(An |Fn−1 ) = ∞
ω:
n=1
(1.14)
n=1
modulo a set of probability 0. When An are independent, we retrieve the classic Borel-Cantelli
lemma.
P
Proof. Let Xn = ni=1 1Ai (ω) − P(Ai |Fi−1 ) . Note that Xn is a martingale with bounded
increments. Therefore by Theorem 1.20, either Xn converges to a finite limit, in which case
P
P∞
and only if ∞
n=1 P(An |Fn−1 ) = ∞; or Xn oscillates between ±∞, in
n=1 1An (ω) = ∞ if
P
P∞
which case we have n=1 1An (ω) = ∞
n=1 P(An |Fn−1 ) = ∞.
We now study the example of Polya’s urn using martingale techniques.
Example 1.23 [Polya’s urn] Let an urn initially contain b black balls and w white balls.
Each time, we pick a ball from the urn with uniform probability, and we put back in the urn 2
balls of the same color. Obviously the number of balls in the urn increase by one each time. A
natural question is what is the fraction Xn of black balls in the urn after time step n? What
is the asymptotic distribution of Xn as n → ∞?
b
. It turns out that Xn is a martingale. To check this, assume that out
Note that X0 = b+w
of the b + w + n balls at time n, j balls are black and k = b + w + n − j balls are white. Then
j
Xn = j+k
, and Then
h
i
E Xn+1 |X1 , · · · , Xn =
=
j
j+1
k
j
·
+
·
j+k j+k+1 j+k j+k+1
j
= Xn .
j+k
Since Xn is non-negative, by the martingale convergence theorem, almost surely, Xn converges
to a limit X∞ ∈ [0, 1].
6
Polya’s urn has the special property of exchangeability. If Yn denotes the indicator event
that the ball drawn at the n-th time step is black, then the distribution of (Yn )n∈N is invariant
under finite permutation of the indices. In particular, if (Yi )1≤i≤n and (Ỹi )1≤i≤n are two
P
P
different realizations of Polya’s urn up to time n with ni=1 Yi = ni=1 Ỹi = m, then observe
that
P((Yi )1≤i≤n ) = P((Ỹi )1≤i≤n ) =
n
Y
i=1
m
n−m
Y
Y
1
(b + i − 1)
(w + i − 1).
b+w+i−1
i=1
i=1
Assume b = w = 1 for simplicity, then after the n-th time step, for each 0 ≤ j ≤ n,
n j!(n − j)!
1
j +1
=
=
.
P Xn =
n+2
n+1
j (n + 1)!
Therefore X∞ is uniformly distributed on [0, 1].
Check that for general b, w > 0, X∞ follows the beta distribution with parameters b, w and
density
Γ(b + w)
(1 − x)b−1 (1 − x)w−1 ,
(1.15)
Γ(b)Γ(w)
where Γ(x) is the gamma function.
References
[1] R. Durrett, Probability: Theory and Examples, 2nd edition, Duxbury Press, Belmont,
California, 1996.
[2] S.R.S. Varadhan, Probability Theory, Courant Lecture Notes 7, American Mathematical
Society, Providence, Rhode Island, 2001.
7