Discrete Probability
Tail inequalities
R. Inkulu
http://www.iitg.ac.in/rinkulu/
(Tail inequalities)
1 / 20
Motivation
The tail inqualities help in deriving bounds on probabilities when only the
mean and variance of a probability distribution are known.
(Tail inequalities)
2 / 20
Outline
1
Markov’s
2
Chebyshev’s
3
Chernoff’s
4
Hoeffding’s
5
Azuma-Hoeffding’s (martingales’ based)
(Tail inequalities)
3 / 20
Markov’s inequality
Let X be a random variable that assumes only nonnegative values. Then for all
a > 0, p(X ≥ a) ≤ E(X)
a .
Corollary: If X is a nonnegative random variable, then for all c ≥ 1,
p(X ≥ cE(X)) ≤ 1c .
(Tail inequalities)
4 / 20
Example: coin flipping
The probability of obtaining more than
flips is upper bounded by
(Tail inequalities)
3n
4
heads in a sequence of n fair coin
5 / 20
Example: coin flipping
The probability of obtaining more than
flips is upper bounded by 23 .
(Tail inequalities)
3n
4
heads in a sequence of n fair coin
5 / 20
Example: hatcheck problem
The probability of more than y number of people receiving their own hat is
upper bounded by
(Tail inequalities)
6 / 20
Example: hatcheck problem
The probability of more than y number of people receiving their own hat is
upper bounded by 1y .
(Tail inequalities)
6 / 20
Example: coupon collector’s problem
The probability that the number of coupons to collect before having at least
one type of each of the n types of coupons is greater than 2nHn is upper
bounded by
(Tail inequalities)
7 / 20
Example: coupon collector’s problem
The probability that the number of coupons to collect before having at least
one type of each of the n types of coupons is greater than 2nHn is upper
bounded by 12 .
(Tail inequalities)
7 / 20
Outline
1
Markov’s
2
Chebyshev’s
3
Chernoff’s
4
Hoeffding’s
5
Azuma-Hoeffding’s (martingales’ based)
(Tail inequalities)
8 / 20
Chebyshev’s inequality
For any a > 0,
p(|X − E[X]| ≥ a) ≤
Var(X)
a2
for large n, gives a significantly tighter bound than the Markov’s inequality
Corollary: For any a > 0,
(Tail inequalities)
p(|X − E[X]| ≥ cσX ) ≤
1
c2
9 / 20
Example: coin flipping
The probability of obtaining more than n4 and less than
of n fair coin flips together is lower bounded by
(Tail inequalities)
3n
4
heads in a sequence
10 / 20
Example: coin flipping
The probability of obtaining more than n4 and less than
of n fair coin flips together is lower bounded by 1 − 4n .
(Tail inequalities)
3n
4
heads in a sequence
10 / 20
An application: The weak law of large numbers 2
Let X1 , X2 , . . . , Xn be a sequence of i.i.d. random variables, each with mean µ
n
and variance σ 2 . Then, for any > 0, p(| X1 +...+X
− µ| ≥ ) → 0 as n → ∞.1
n
1
essentially, says that the sample mean is same as the mean whenever the sample large
enough
2
n
the strong law of large numbers states that with probability 1, X1 +X+2+...+X
→ µ as
n
n → ∞ — not proved
(Tail inequalities)
11 / 20
Outline
1
Markov’s
2
Chebyshev’s
3
Chernoff’s
4
Hoeffding’s
5
Azuma-Hoeffding’s (martingales’ based)
(Tail inequalities)
12 / 20
Chernoff bounds
Let X be a random variable.
tX ]
3
• For any t > 0, p(X ≥ a) ≤ E[e
eta .
In particular, p(X ≥ a) ≤ mint>0
E[etX ]
eta .
tX ]
• For any t < 0, p(X ≤ a) ≤ E[e
eta .
In particular, p(X ≤ a) ≤ mint<0
3
E[etX ]
eta .
E(etX ) is known as the moment generating function of X
(Tail inequalities)
13 / 20
Few notes
tX ]
• While the value of t that minimizes E[e
eta gives the best possible bounds,
often one chooses a value of t that gives a convenient form.
• Further, typically, Chernoff bounds yield tight (exponentially small)
bounds as compared with the polynomially small bounds via other two
tail inequalities.
(Tail inequalities)
14 / 20
Example
Suppose a gambler wins or looses 1 dollar on every play, independently of his
past results. The number of dollars after n plays being greater than a (for
a > 0) is upper bounded by
(Tail inequalities)
15 / 20
Example
Suppose a gambler wins or looses 1 dollar on every play, independently of his
past results. The number of dollars after n plays being greater than a (for
2
2
a > 0) is upper bounded by mint>0 e−ta ent /2 = e−a /2n .
(Tail inequalities)
15 / 20
Chernoff bounds: more than one random variable
involved
Let X be a sum of n independent random variables Xi , each with range in
[0, 1]. Let E(X) = µ. Then
n−k
• p(X ≥ k) ≤ ( µk )k ( n−µ
n−k )
≤
( µk )k ek−µ
for k > µ
for k > µ
p(X ≥ (1 + )µ) ≤ ( (1+)e (1+) )µ
n−k
• p(X ≤ k) ≤ ( µk )k ( n−µ
n−k )
≤
( µk )k ek−µ
p(X ≤ (1 − )µ) ≤ e−
for ≥ 0
for k < µ
for k < µ
µ2
2
for ∈ (0, 1)
—not proved in class
(Tail inequalities)
16 / 20
Outline
1
Markov’s
2
Chebyshev’s
3
Chernoff’s
4
Hoeffding’s
5
Azuma-Hoeffding’s (martingales’ based)
(Tail inequalities)
17 / 20
Hoeffding’s inequality4
Let X1 , . . . , Xn be independent random variables, where Xi ∈ [ai , bi ], for
i = 1, . . . , n. Then, for the random variable S = X1 + . . . + Xn and any η > 0,
we have p(|S − E(S)| ≥ η) ≤ 2e
(− Pn
2η 2
)
(bi −ai )2
i=1
.
—not proved in class
4
a generalization of Chernoff’s inequality
(Tail inequalities)
18 / 20
Outline
1
Markov’s
2
Chebyshev’s
3
Chernoff’s
4
Hoeffding’s
5
Azuma-Hoeffding’s (martingales’ based)
(Tail inequalities)
19 / 20
—– not taught —–
(Tail inequalities)
20 / 20