Probabilistic Inequalities
Dan A. Simovici
UMB
Dan A. Simovici (UMB)
Probabilistic Inequalities
1 / 14
1
Markov’s and Cebyshev’s Inequalities
2
Hoeffding’s Inequality
Dan A. Simovici (UMB)
Probabilistic Inequalities
2 / 14
Markov’s Inequality
Theorem
Let X be a non-negative r.v. such that E (X ) exists. For a > 0 we have
P(X > a) 6
E (X )
.
a
Let Y be the random variable
(
a
Y =
0
if X > a,
if X < a.
Since Y 6 X , we have E (Y ) 6 E (X ). The expected value of Y can be
also written as E (Y ) = aP(X > a), which yields the desired inequality.
Dan A. Simovici (UMB)
Probabilistic Inequalities
3 / 14
Chebyshev’s Inequality - I
Theorem
Let X be a random variable such that var(X ) exists. Then, for any
positive number a we have
P(|X − E (X )| > a) 6
var(X )
.
a2
For Y = (X − E (X ))2 we have P(Y > 0) = 1 and E (Y ) = var(X ). By
Markov inequality applied to Y and a2 we obtain
P(|X − E (X )| > a) = P(Y > a2 ) 6
Dan A. Simovici (UMB)
Probabilistic Inequalities
var(X )
.
a2
4 / 14
Chebyshev’s Inequality - II
A generalization of Chebyshev’s Inequality can be obtained if the
expectation of Z = (X − E (X ))k exists for k > 0. Namely, we have:
P(|X − E (X )| > a) = P(Z > ak ) 6
E ((X − E (X ))k )
ak
for every a > 0.
Dan A. Simovici (UMB)
Probabilistic Inequalities
5 / 14
Preliminary Results-I
Lemma
Let f : R≥0 −→ R be the function defined by
f (x) =
x2
+ px − ln(1 − p + pe x ),
8
where p ∈ [0, 1]. We have f (x) > 0 for every x > 0.
Proof: Note that f (0) = f 0 (0) = 0 because
f 0 (x) =
x
ex
+p−p
.
4
1 − p + pe x
x
(1−p)pe
2
Since f 00 (x) = 14 − (1−p+pe
x )2 , and 4ab 6 (a + b) , where a = 1 − p and
b = pe x , we have f 00 (x) > 0. Therefore, x > 0 implies f 0 (x) > f 0 (0) = 0,
which, in turn, implies f (x) > f (0) = 0.
Dan A. Simovici (UMB)
Probabilistic Inequalities
6 / 14
Preliminary Results-II
Lemma
Let X be a random variable such that E (X ) = 0 and a 6 X 6 b. We have
E (e tX ) 6 e
t 2 (b−a)2
8
.
Proof: If x ∈ (a, b), then x = λa + (1 − λ)b, where λ =
The convexity of the exponential implies
b−x
b−a
∈ [0, 1].
e tx 6 λe ta + (1 − λ)e tb
for every x ∈ [a, b] and, therefore,
b − E (X )
a
b
b − E (X ) ta
tX
e + 1−
e ta −
e tb .
e tb =
E (e ) 6
b−a
b−a
b−a
b−a
Dan A. Simovici (UMB)
Probabilistic Inequalities
7 / 14
Preliminary Results-III
Since E (X ) = 0 we have 0 6
x = t(b − a), we have:
−a
b−a
6 1. If we define p =
−a
b−a
and
E (e tX ) ≤ (1 − p)e at + pe bt
=
1 − p + pe t(b−a) e at
=
1 − p + pe t(b−a) e −pt(b−a)
= e ln(1−p+pe
t(b−a) )−pt(b−a)
= e ln(1−p+pe
x )−px
≤ e
x2
8
(by previous Lemma)
= e
Dan A. Simovici (UMB)
t 2 (b−a)2
8
,
Probabilistic Inequalities
8 / 14
Hoeffding’s Inequality
Theorem
Let X1 , . . . , Xn be n independent random variables such that ai 6 Xi 6 bi
for 1 6 i 6 n and let U be the random variable defined by:
n
X
(Xi − E (Xi )).
U=
i =1
The following inequalities hold:
P(U > ) 6 e
P(U 6 −) 6 e
Dan A. Simovici (UMB)
−22
Pn
(b −ai )2
i =1 i
−22
Pn
(b −ai )2
i =1 i
Probabilistic Inequalities
.
9 / 14
Proof of theorem
Let Yi be the rv defined by Yi = Xi − E (Xi ) for 1 6 i 6 n. The
independence of X1 , . . . , Xn implies that Y1 , . . . , Yn are independent.
E (Yi ) = 0;
ai 6 Xi 6 bi implies ai − E (Xi ) 6 Yi 6 bi − E (Xi );
t 2 (bi −ai )2
tYi
8
The second Lemma
.
Pnimplies E (e ) 6 e
For the r.v. U = i =1 Yi we have E (U) = 0. Since
P(U > ) = P(e tUn > e t ), by Markov’s inequality we have:
P(e tU > e t ) 6
Dan A. Simovici (UMB)
E (e tU )
.
e t
Probabilistic Inequalities
10 / 14
Proof of theorem (cont’d)
P
For the r.v. U = ni=1 Yi we have E (U) = 0. Since
P(U > ) = P(e tUn > e t ), by Markov’s inequality we have:
P(e tU > e t ) 6
E (e tU )
.
e t
Since
P
n
n
2
Y
Pn
Y
t2 n
t 2 (bi −ai )2
i =1 (bi −ai )
8
8
e
E (e tYi ) ≤
E (e tU ) = E e t i =1 Yi =
=e
,
i =1
i =1
it follows that
P(U > ) = P(e
Dan A. Simovici (UMB)
tU
t
>e )6e
Probabilistic Inequalities
−t+
t2
Pn
2
i =1 (bi −ai )
8
.
(1)
11 / 14
Proof of theorem (cont’d)
Similarly,
P(U 6 −) = P(e −tU > e t ) 6 e −t+
t2
Pn
2
i =1 (bi −ai )
8
Note that the exponent has a minimal value when t =
we obtain the inequalities of the theorem:
P(U > ) 6 e
P(U 6 −) 6 e
Dan A. Simovici (UMB)
Pn
4a
.
2
i =1 (bi −ai )
(2)
. Thus,
−22
Pn
(b −ai )2
i =1 i
−22
Pn
(b −ai )2
i =1 i
Probabilistic Inequalities
.
12 / 14
A Corollary
Corollary
Let X1 , . . . , Xn be n independent random variables such that Xi ∈ [0, 1] for
1 6 i 6 n and let Zn be the random variable defined by:
n
1X
Xi .
Zn =
n
i =1
The following inequalities hold:
P(Zn − E (Zn ) > ) 6 e −2n
2
2
P(Zn − E (Zn ) 6 −) 6 e −2n .
Dan A. Simovici (UMB)
Probabilistic Inequalities
13 / 14
Proof of the corollary
We have
U=
n
X
(Xi − E (Xi )) = n(Zn − E (Zn )).
i =1
Consequently, by Hoeffding’s Inequality, taking a1 = · · · = an = 0 and
b1 = · · · = bn = 1 we have
P(Zn − E (Zn ) > ) = P(U > n) 6 e
−2n2 2
n
2
= e −2n ,
The second inequality is obtained in a similar manner.
Dan A. Simovici (UMB)
Probabilistic Inequalities
14 / 14