Randomized Algorithms
CS648
Lecture 22
• Chebyshev Inequality
• Method of Bounded Difference
1
Chernoff Bound
Theorem : Suppose 𝑿𝟏 , 𝑿𝟐 , … , 𝑿𝒏 be 𝒏 independent Bernoulli random
variables with parameters 𝒑𝟏 , 𝒑𝟐 , … , 𝒑𝒏 , that is, 𝑿𝒊 takes value 1 with
probability 𝒑𝒊 and 0 with probability 𝟏 − 𝒑𝒊 . Let 𝑿 = 𝒊 𝑿𝒊 and 𝝁 = 𝑬[𝑿] =
𝒊 𝒑𝒊 .
For any 𝜹 > 𝟎,
𝝁
𝒆𝜹
𝐏 𝑿≥ 𝟏+𝜹 𝝁 ≤
(𝟏 + 𝜹) 𝟏+𝜹
𝐏 𝑿 ≥ 𝟏 − 𝜹 𝝁 ≤ 𝒆−𝝁𝜹
𝟐 /𝟐
Limitations:
• Works only for bounding sum of random variables.
• Requires independence among 𝑿𝟏 , 𝑿𝟐 , … , 𝑿𝒏 .
THREE EXAMPLES TO ILLUSTRATE THE
INAPPLICABILITY OF CHERNOFF BOUND
3
Red-blue balls out of bin
Randomized Experiment:
There are 𝒏 red and 𝒏 blue balls in a bag. We take out 𝒏 balls from the bag
uniformly randomly and without replacement.
𝒀: no. of red balls in the sample.
𝟏 if 𝒊th ball in the sample is red
𝒀𝒊 =
𝟎 otherwise
𝒀 = 𝒊 𝒀𝒊
𝒏
𝟏
=
=
= 𝒊 𝐏(𝒀𝒊 = 𝟏)
𝒊 𝟐
𝟐
 𝐄 𝒀 = 𝒊 𝐄[𝒀𝒊 ]
𝒏
Aim: To show 𝑿 is concentrated around 𝟐.
Question: Can we apply Chernoff bound ?
Answer: NO because 𝒀𝒊 ’s are NOT independent.
4
Balls into Bins
(number of empty bins)
1 2 3 4 5
1
2
3
…
…
m-1 m
…
n
𝒁 : random variable denoting the number of empty bins.
𝟏 if 𝒊th bin is empty
𝒁𝒊 =
𝟎 otherwise
𝒁 = 𝒊≤𝒏 𝒁𝒊
= 𝑖≤𝑛(1 − 𝑛1 )𝑚 = 𝑛(1 − 𝑛1 )𝑚 = 𝑛/𝑒 for 𝑚 = 𝑛
 𝐄[𝑿] = 𝑖≤𝑛 𝐄[𝑿𝒊 ]
𝒏
Aim: To show that 𝒁 is concentrated around 𝒆 .
Question: Can we apply Chernoff bound ?
Answer: NO because 𝒁𝒊 ’s are NOT independent.
5
Number of Triangles in a random graph
𝑮(𝒏, 𝒑) : A graph on 𝒏 vertices where each edge is present with probability 𝒑
independent of others.
𝒂
𝒅
𝒃
𝒑
𝒄
𝑻 : random variable denoting the number of triangles.
𝟏 if triangle 𝒒𝒓𝒕 is present in 𝑮
𝑻𝒒𝒓𝒕 =
𝟎 otherwise𝟑
𝒏
1
𝑛 3
=
𝑝
=
for
𝑝
=
 𝐄[𝑻] = 𝒒,𝒓,𝒕∈𝑽 𝐄[𝑻𝒒𝒓𝒕 ]
3
𝟒𝟖
2
Aim: To show that 𝑻 is concentrated around
𝒏𝟑
.
𝟒𝟖
Question: Can we apply Chernoff bound ?
Answer: NO because 𝑻𝒒𝒓𝒕 ’s are NOT independent.
6
CHEBYSHEV’S INEQUALITY
7
Chebyshev’s inequality
Let 𝑿 be a random variable defined over a probability space.
Question: How to capture deviation of 𝑿 from 𝐄 𝑿 ?
Define 𝑽 = 𝑿 − 𝐄 𝑿
Question: What is 𝐄 𝑽 ?
Answer: 𝟎
𝟐
Redefine 𝑽 = 𝑿 − 𝐄 𝑿
Question: What is 𝐄 𝑽 ?
Answer: 𝐄 𝑿𝟐 − 𝐄 𝐗
𝟐
Called variance of 𝑿
8
Chebyshev’s inequality
Let 𝑿 be a random variable defined over a probability space.
𝐏 𝑿−𝐄 𝐗 ≥𝒕
= 𝐏 𝑿 − 𝐄 𝐗 𝟐 ≥ 𝒕𝟐
Applying Markov Inequality,
𝐄[ 𝑿 − 𝐄 𝑿 𝟐 ]
≤
𝒕𝟐
𝐄 𝑿𝟐 − 𝐄 𝐗 𝟐
=
𝒕𝟐
variance of 𝑿
=
𝒕𝟐
Limitations:
•
•
•
Calculating 𝐄 𝑿𝟐 is sometimes difficult.
Usually gives bounds that are better than Markov Inequality but inferior to the
bound achieved by other methods.
Simple practice problems will be given to you on the use of Chebyshev Inequality.
9
METHOD OF BOUNDED DIFFERENCE (MOBD)
The most powerful method for bounding
the probability of deviation of a random variable from expected value
10
The Power of MOBD
• Tightest bound for Randomized Quick Sort was derived using MOBD.
𝑸𝒏 : number of comparisons during randomized quick sort on 𝒏 elements.
𝐏(𝑸𝒏 ≥ 𝟏 + 𝝐 𝐄 𝑸𝒏 = 𝒆−𝟐𝝐 (log log 𝒏 + log log log 𝒏)
• MOBD subsumes Chernoff bound.
• Based on theory of Martingales.
Note: Proof similar and almost as hard as the proof of Chernoff bound.
[Not part of the course]
11
Notations
𝒙𝟏 , … , 𝒙𝒏 : a sequence of 𝒏 random variables.
𝒇 = 𝒇(𝒙𝟏 , … , 𝒙𝒏 ) be a function of 𝒏 random variables.
Objective: to achieve a bound on the probability of deviation of 𝒇 from 𝐄[𝒇].
𝐏(|𝒇 − 𝐄 𝒇 | > ϵ 𝐄[𝒇]|) = ?
Notations:
𝑿𝒊 ∶ (𝒙𝟏 , … , 𝒙𝒊 )
𝑨𝒊 ∶ (𝒂𝟏 , … , 𝒂𝒊 )
“𝑿𝒊 = 𝑨𝒊 ” means “𝒙𝟏 = 𝒂𝟏 , … , 𝒙𝒊 = 𝒂𝒊”
12
A new perspective
The value of 𝒇 is well defined once the values taken by 𝒙𝟏 , … , 𝒙𝒏 is exposed.
View the process of exposing the values of 𝒙𝟏 , … , 𝒙𝒏 happening gradually in 𝒏 steps.
• In the beginning, when none of the 𝒙𝟏 , … , 𝒙𝒏 is revealed, all we can say about
value of 𝒇 is that its expected value is 𝐄[𝒇].
• In first step, 𝒙𝟏 is exposed. If 𝒙𝟏 takes value 𝒂𝟏 , all we can say about value of 𝒇 is
that its expected value is 𝐄[𝒇|𝑿𝟏 = 𝑨𝟏 ].
• In second step, 𝒙𝟐 is also exposed. If 𝒙𝟐 takes value 𝒂𝟐 , all we can say about value
of 𝒇 is that its expected value is 𝐄[𝒇|𝑿𝟐 = 𝑨𝟐 ].
…
The next slide will give a visual description of the process
mentioned above.
But ponder over this slide before pressing the next button.
13
The value of 𝒇 and
the gradual exposition of 𝒙𝒊 ’s
𝒙𝟑 = 𝒂𝟑
𝐄[𝒇|𝑿𝟐 = 𝑨𝟐 ]
𝐄[𝒇]
𝐄[𝒇|𝑿𝟏 = 𝑨𝟏 ]
…
𝒙𝟐 = 𝒂𝟐
…
…
…
𝒙𝟏 = 𝒂𝟏
…
𝒙𝒏 = 𝒂𝒏
𝐄[𝒇|𝑿𝒏−𝟏 = 𝑨𝒏−𝟏 ]
𝐄[𝒇|𝑿𝟑 = 𝑨𝟑 ]
𝒇(𝑨𝒏 )
Examples to illustrate the meaning of 𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ]
Algorithm : Quick sort
𝒇: no. of comparisons during quick sort on 𝒏 elements.
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ] : Given first 𝒊 pivot elements, the expected number of comparisons
during quick sort on 𝒏 elements.
14
The value of 𝒇 and
the gradual exposition of 𝒙𝒊 ’s
𝒙𝟑 = 𝒂𝟑
𝐄[𝒇|𝑿𝟐 = 𝑨𝟐 ]
𝐄[𝒇]
𝐄[𝒇|𝑿𝟏 = 𝑨𝟏 ]
…
𝒙𝟐 = 𝒂𝟐
…
…
…
𝒙𝟏 = 𝒂𝟏
…
𝒙𝒏 = 𝒂𝒏
𝐄[𝒇|𝑿𝒏−𝟏 = 𝑨𝒏−𝟏 ]
𝐄[𝒇|𝑿𝟑 = 𝑨𝟑 ]
𝒇(𝑨𝒏 )
Examples to illustrate the meaning of 𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ]
Stochastic Process : Ball-Bin problem
𝒇: no. of empty bins
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ]: Given the destination of first 𝒊 balls, the expected number of empty
bins.
15
Gradual exposition of 𝒙𝒊 ’s
𝒙𝟑 = 𝒂𝟑
𝐄[𝒇|𝑿𝟐 = 𝑨𝟐 ]
𝐄[𝒇]
𝐄[𝒇|𝑿𝟏 = 𝑨𝟏 ]
…
𝒙𝟐 = 𝒂𝟐
…
…
…
𝒙𝟏 = 𝒂𝟏
…
𝒙𝒏 = 𝒂𝒏
𝐄[𝒇|𝑿𝒏−𝟏 = 𝑨𝒏−𝟏 ]
𝐄[𝒇|𝑿𝟑 = 𝑨𝟑 ]
𝒇(𝑨𝒏 )
Examples to illustrate the meaning of 𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ]
Random Structure : Random graph
𝒇: no. of triangles in 𝑮(𝒏, 𝒑)
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ]: Given the presence/absence of 𝒆𝟏 , … , 𝒆𝒊 edges, the expected number
of triangles in the random graph.
16
THE INTUITION
UNDERLYING MOBD
17
Method of Bounded Difference
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜷]
𝒙𝟐 = 𝒂𝟐
…
…
…
𝒙𝟏 = 𝒂𝟏
…
𝒙𝒊+𝟏 = 𝜶
…
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ]
𝐄[𝒇]
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜶]
𝒇(𝑨𝒏 )
18
Method of Bounded Difference
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜷]
𝒙𝟐 = 𝒂𝟐
…
…
…
𝒙𝟏 = 𝒂𝟏
…
𝒙𝒊+𝟏 = 𝜶
…
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ]
𝐄[𝒇]
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜶]
𝒇(𝑨𝒏 )
For any 𝑨𝒊 , and any 𝜶, 𝜷 ∈ 𝒙𝒊+𝟏, if
if |𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜷] − 𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜶]| is small, then
Most probably 𝒇(𝑨𝒏 ) will be close to 𝐄[𝒇].
Think for a while over the above statement before proceeding further.
19
Method of Bounded Difference - I
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜷]
𝒙𝟐 = 𝒂𝟐
…
…
…
𝒙𝟏 = 𝒂𝟏
…
𝒙𝒊+𝟏 = 𝜶
…
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 ]
𝐄[𝒇]
Theorem 1:
𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜶]
𝒇(𝑨𝒏 )
If there are positive numbers 𝒄𝒊 ’s such that for any 𝑨𝒊 , and any 𝜶, 𝜷 ∈ 𝒙𝒊+𝟏 ,
|𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜷] − 𝐄[𝒇|𝑿𝒊 = 𝑨𝒊 , 𝒙𝒊+𝟏 = 𝜶]| < 𝒄𝒊
Then
𝟐𝒕𝟐
𝐏(|𝒇 − 𝐄 𝒇 | ≥ t) ≤ 𝐞𝐱𝐩 −
𝟐
𝒊 𝒄𝒊
Note: In order to get a meaningful bound using Therem1, you must have small 𝒄𝒊 ’s .
20
Method of Bounded Difference - II
Definition: function 𝒇(𝒙𝟏 , … , 𝒙𝒏 ) is said to satisfy Lipschitz condition with
parameters 𝒄𝒊 ’s if
𝒇 𝑨 − 𝒇 𝑨′ ≤ 𝒄𝒊
For all 𝑨, 𝑨′ that differ only at 𝒊th coordinate.
Theorem 2: If 𝒇 satisfies Lipschitz condition and 𝒙𝟏 , … , 𝒙𝒏 are independent,
then
𝟐𝒕𝟐
𝐏(|𝒇 − 𝐄 𝒇 | ≥ t) ≤ 𝐞𝐱𝐩 −
𝟐
𝒄
𝒊 𝒊
Remark: This form is easiest to use but requires independence.
21
MOBD SUBSUMES CHERNOFF BOUND
22
MOBD subsumes Chernoff Bound
𝒙𝟏 , … , 𝒙𝒏 are 0-1 independent random variables.
𝒇 = 𝒊 𝒙𝒊
 𝒇 satisfies Lipschitz condition with parameters 𝒄𝒊 = 𝟏
Hence applying Theorem 2
𝟐𝒕𝟐
𝐏(|𝒇 − 𝐄 𝒇 | ≥ t) ≤ 𝐞𝐱𝐩 −
𝟐
𝒄
𝒊 𝒊
𝟐𝒕𝟐
= 𝐞𝐱𝐩 −
𝒏
= 𝒆−𝟐 l𝑛 𝒏
= 𝒏−𝟐
for 𝒕 = 𝒏 ln 𝒏
for 𝒕 = 𝒏 ln 𝒏
23
PROBLEM 1
NO. OF EMPTY BINS
24
Balls into Bins
(number of empty bins)
1 2 3 4 5
1
2
3
…
…
n-1 n
…
n
𝒛𝒊 : random variable denoting the number of balls in 𝒊th bin.
𝒇(𝒛𝟏 , … , 𝒛𝒏 ) : number of empty bins.
Observation: 𝒛𝒊 takes value in range [𝟎, 𝒏].
Question: What is 𝒄𝟏 s.t. |𝐄[𝒇|𝒛𝟏 = 𝜶] − 𝐄[𝒇|𝒛𝟏 = 𝜷]| < 𝒄𝟏 for any given 𝜶, 𝜷?
Answer: For 𝜶 = 𝒏, 𝜷 = 𝟎,
𝒏
𝟏
𝒄𝟏 = …𝒏 − 𝟐 − (𝒏 − 𝟏) 𝟏 −
𝒏−𝟏
≈𝒏
𝒆−𝟏
𝒆
This will give very inferior bound
25
Balls into Bins
(number of empty bins)
1 2 3 4 5
1
2
3
…
…
n-1 n
…
n
𝒙𝒊 : random variable denoting the destination of 𝒊th ball.
𝒇(𝒙𝟏 , … , 𝒙𝒏 ) : number
empty
bins. our choice of random variables 𝒛
We of
failed
because
𝒊
Observation:
for defining 𝒇 was bad. Can you think of other
random variables such that 𝒇 is a function of them ?
• 𝒙𝟏 , … , 𝒙𝒏 are independent.
• 𝒇 satisfies Lipschitz condition with parameters 𝒄𝒊 = 𝟏
Hence
𝐏(|𝒇 − 𝐄 𝒇 | ≥ t) ≤ 𝐞𝐱𝐩 −
𝟐𝒕𝟐
𝟐
𝒊 𝒄𝒊
𝟐𝒕𝟐
= 𝐞𝐱𝐩 −
𝒏
≤ 𝒏−𝟐
for 𝒕 > 𝒏 ln 𝒏
26
Balls into Bins
(number of empty bins)
Theorem: If 𝒏 balls are thrown r.u.i. into 𝒏 bins, then the number of empty bins
𝒏
will be within range ± 𝒏 ln 𝒏 with probability at least 𝟏 − 𝒏−𝟐 .
𝒆
Lesson to be learnt from this exercise:
Be careful in selecting the base random variables 𝒙𝟏 , … , 𝒙𝒏 used in defining 𝒇.
27
PROBLEM 2
RED-BLUE BALLS OUT OF BIN
28
Red-blue balls out of bin
Randomized Experiment:
There are 𝒏 red and 𝒏 blue balls in a bag. We take out 𝒏 balls from the bag
uniformly randomly and without replacement.
𝒇: no. of red balls in the sample.
𝟏 if 𝒊th ball in the sample is red
𝒙𝒊 =
𝟎 otherwise
𝒇 = 𝒊 𝒙𝒊
𝒏
=
𝐄 𝒇 = 𝒊 𝐄[𝒀𝒊 ] 𝟐
Observation: 𝒙𝒊 are not independent
 Can apply Theorem 1 only.
Question: What is 𝒄𝒊 s.t.
|𝐄[𝒇|𝑿𝒊−𝟏 = 𝑨𝒊−𝟏 , 𝒙𝒊 = 𝜶] − |𝐄[𝒇|𝑿𝒊−𝟏 = 𝑨𝒊−𝟏 , 𝒙𝒊 = 𝜷]| < 𝒄𝒊 for any given 𝑨𝒊−𝟏 , 𝜶, 𝜷 ?
29
Red-blue balls out of bin
𝑨𝒊−𝟏
𝒏 − 𝒊 balls
𝒊th ball
…
…
…
Let 𝑨𝒊−𝟏 has 𝒌 red balls.
𝐄[𝒇|𝑿𝒊−𝟏 = 𝑨𝒊−𝟏 , 𝒙𝒊 = 𝟏] = ? 𝒌 + 𝟏 +
𝐄[𝒇|𝑿𝒊−𝟏 = 𝑨𝒊−𝟏 , 𝒙𝒊 = 𝟎] = ? 𝒌 +
𝒏−𝒌−𝟏
(𝒏 − 𝒊)
𝟐𝒏 − 𝒊
𝒏−𝒌
(𝒏 − 𝒊)
𝟐𝒏 − 𝒊
 |𝐄[𝒇|𝑿𝒊−𝟏 = 𝑨𝒊−𝟏 , 𝒙𝒊 = 𝟏] − 𝐄[𝒇|𝑿𝒊−𝟏 = 𝑨𝒊−𝟏 , 𝒙𝒊 = 𝟎]| < 𝟏
 𝒄𝒊 < 𝟏
Applying Theorem 1
𝐏(|𝒇 − 𝐄 𝒇 | ≥ t) ≤ 𝐞𝐱𝐩 −
𝟐𝒕𝟐
𝟐
𝒊 𝒄𝒊
𝟐𝒕𝟐
< 𝐞𝐱𝐩 −
𝒏
≤ 𝒏−𝟐
for 𝒕 > 𝒏 ln 𝒏
30
Red-blue balls out of bin
Theorem: There are 𝒏 red and 𝒏 blue balls in a bag. Suppose we take out 𝒏
balls from the bag uniformly randomly and without replacement. The
𝒏
number of red balls in the sample is within range ± 𝒏 ln 𝒏 with
𝟐
−𝟐
probability at least 𝟏 − 𝒏 .
31
PROBLEM 3
NO. OF TRIANGLES IN RANDOM GRAPH
Do it as exercise.
This problem will also be posted in practice sheet.
32