• Randomized quick sort
• Randomized maximum cut
2006/2/21
1
Sorting
• Input: n distinct numbers x1, x2, …, xn .
• Output: a sorted list of x1, x2, …, xn .
4
2006/2/21
3
5
8
1
9
2
6
7
2
Quick sort: divide & conquer
algorithm quicksort(X) f
if X is empty then
return;
select a number x;
let Y = fy 2 X j y < xg;
let Z = fz 2 X j z > xg;
call quicksort(Y );
print x;
call quicksort(Z);
g
2006/2/21
3
x divides X into Y and Z.
4
2
7
8
1
9
3
6
5
x
4
Y
2006/2/21
Z
4
The whole process
4
2
7
8
1
9
3
6
5
2
1
3
4
7
8
9
6
5
1
2
3
6
5
7
8
9
3
5
6
8
9
1
5
2006/2/21
9
5
Efficiency depends on …
algorithm quicksort(X) f
if X is empty then
return;
select a number x;
let Y = fy 2 X j y < xg;
let Z = fz 2 X j z > xg;
call quicksort(Y );
print x;
call quicksort(Z);
g
2006/2/21
Critical step
6
A bad case
9
8
7
6
5
4
3
2
1
8
7
6
5
4
3
2
1
9
7
6
5
4
3
2
1
8
6
5
4
3
2
1
7
5
4
3
2
1
6
4
3
2
1
5
3
2
1
4
2
1
3
1
2
(n)
1
2006/2/21
7
A good case
• Each x divides X evenly into Y and Z, i.e.,
when –1· |Y| – |Z| · 1.
• The running time is O(n log n).
O(log n)
2006/2/21
8
Median selection
F Input: a (multiple) set X of n numbers.
F Output: a number x in X with
¡1 · jfy 2 X j y < xgj ¡ jfz 2 X j z > xgj · 1.
• If the median-selection problem can be
solved in O(n) time, then algorithm
quicksort runs in O(n log n) time.
2006/2/21
9
How hard is median selection?
• [Blum et al. STOC’72 & JCSS’73]
– A “shining” paper by five authors:
•
•
•
•
•
Manuel Blum (Turing Award 1995)
Robert W. Floyd (Turing Award 1978)
Vaughan R. Pratt
Ronald L. Rivest (Turing Award 2002)
Robert E. Tarjan (Turing Award 1986)
– The number of comparisons required to find a
median is between 1.5n and 5.43n.
2006/2/21
10
Number of comparisonss
• Upper bound:
– 3n + o(n) by Schonhage, Paterson, and
Pippenger (JCSS 1975).
– 2.95n by Dor and Zwick (SODA 1995, SIAM
Journal on Computing 1999).
• Lower bound:
– 2n+o(n) by Bent and John (STOC 1985)
– (2+2-80)n by Dor and Zwick (FOCS 1996,
SIAM Journal on Discrete Math 2001).
2006/2/21
11
Question:
Perhaps not!
Do we really need all the
complication of median selection
to obtain an O(n log n)-time
quicksort?
2006/2/21
12
For example, x does not have to
be a median.
As long as
jY j
jZ j
= £(1), the resulting quicksort runs in O(n log n) time.
O(log n)
2006/2/21
13
How many good choices for x?
Observation For example, if we only aim for
1 · jY j ·
3;
j
j
3
Z
then at least half of the elements x in X are good.
2006/2/21
14
This leads to RandQS
隨機快排法
2006/2/21
15
Randomized quicksort
algorithm RandQS(X) f
if X is empty then
randomization
return;
select x uniformly at random from X;
let Y = fy 2 X j y < xg;
let Z = fz 2 X j z > xg;
call RandQS(Y );
print x;
call RandQS(Z);
g
2006/2/21
16
2 Questions for RandQS
•
Is RandQS correct?
– That is, does RandQS “always” output a
sorted list of X?
•
What is the time complexity of RandQS?
– Due to the randomization for selecting x, the
running time for RandQS becomes a
random variable.
– We are interested in the expected time
complexity for RandQS.
2006/2/21
17
Features for many randomized
algorithms
• Implementation is relatively simple,
– compared to their deterministic counterparts.
• The challenge for designing randomized
algorithms are usually in (asymptotic)
analysis for (time and space) complexity.
2006/2/21
18
Let C = #comparision.
• When we choose x uniformly at random from X,
the probability for each element of X being
chosen as x is exactly 1/|X|.
• The values of C for two independent executions
of RandQS are very likely to be different.
• So, C is a random variable. We are interested in
determining E[C], since the expected running
time is clearly O(E[C]).
2006/2/21
19
Computing E[C] from E[Ci;j ]
Suppose X consists of x1 ; : : : ; xn with x1 < ¢ ¢ ¢ < xn .
Let Ci;j denote the number of times that xi and xj
are compared during the execution of RandQS. By
the linearity of E[¢], we have
2
3
X
X
n
4
5
E[C] = E
Ci;j
i=1 j>i
=
X
X
n
E[Ci;j ]:
i=1 j>i
2006/2/21
20
Ci;j can only be 0 or 1.
4
2
7
8
1
9
3
6
5
2
1
3
4
7
8
9
6
5
1
2
3
6
5
7
8
9
3
5
6
8
9
1
5
2006/2/21
9
21
Therefore,
E[Ci;j ]
=
Pr[Ci;j = 1] ¢ 1 + Pr[Ci;j = 0] ¢ 0
=
Pr[Ci;j = 1]:
Q: What is the value of Pr[Ci;j = 1]?
Do you think Pr[Ci;j = 1] has a ¯xed value in di®erent executions of RandQS(X)?
2006/2/21
22
CLAIM
Pr[Ci;j
2
= 1] = ¡
:
j i+1
For example,
Pr[C1;n
2
2
= 1] = ¡
= :
n 1+1
n
Reason: x1 and xn are compared (once) if and
only if the ¯rst choice of x is either x1 or xn .
2006/2/21
23
Another example
For example,
Pr[C2;n
2
2
= 1] = ¡
= ¡ :
n 2+1
n 1
?
Yes: x2 and xn are compared (once) if and only if (a) the
¯rst choice of x is either x2 or xn , or (b) the ¯rst choice
of x is x1 and the second choice of x is either x2 or xn .
Therefore,
Pr[C2;n
2
1£ 2
2
= 1] = +
= ¡ :
¡
n n n 1
n 1
Wow!
2006/2/21
24
It seems hard to further
generalize the argument …
We need a more general
approach to prove the amazingly
simple claim.
2006/2/21
25
The subset Xi,j of X
Let Xi;j be the subset of X such that RandQS(Xi;j ) is
the ¯rst subroutine call such that its x belongs to the set
fx ; x ; : : : ; x g. Clearly, fx ; : : : ; x g µ X .
i
i+1
j
i
j
i;j
This speci¯c choice of x determines whether xi and xj will
ever be compared to each other throughout the execution
In particular, Ci;j = 1 if and only if this choice of x among
fx ; x ; : : : ; x g is either x or x . Therefore,
i
i+1
j
i
j
Pr[Ci;j
2006/2/21
2
= 1] = ¡
:
j i+1
26
For example,
X3,6 = X2,4 = X1,9
X1,3 = X1,2 = X2,3
2
7
4
8
1
9
3
6
5
3
1
2
4
8
9
7
6
5
1
2
3
5
6
7
8
9
3
5
6
8
9
1
5
2006/2/21
X8,9
9
27
Comments about Xi,j
• For each execution of RandQS(X), Xi,j is
well defined.
• However, the Xi,j for two independent
execution of RandQS(X) are (very likely to
be) different.
2006/2/21
28
So, E[C] = O(n log n)
E[C] =
X
X
n
E[Ci;j ]
i=1 j>i
=
=
X
X
n
i=1 j>i
X
X
n n¡ i
i=1µ
k=1
2
j¡i+1
2
k+1
· 2n 1 + 1 + 1 + ¢ ¢ ¢ + 1
2 3
n
= O(n log n)
2006/2/21
¶
29
In summary,
The expected running time of RandQS is O(E[C]) =
O(n log n).
algorithm RandQS(X) f
if X is empty then
return;
select x uniformly at random from X;
let Y = fy 2 X j y · xg;
let Z = fz 2 X j z > xg;
call RandQS(Y );
print x;
call RandQS(Z);
g
2006/2/21
30
Features for many randomized
algorithms
• Implementation is relatively simple,
– compared to their deterministic counterparts.
• The challenge for designing randomized
algorithms are usually in (asymptotic)
analysis for (time and space) complexity.
2006/2/21
31
A misunderstanding
• “The performance of randomized
algorithms depends on the distribution
of the input.”
• Clarification: Not at all (for this course)
– For example, our analysis for RandQS
did not assume that n! possible
permutations for the input are given to
RandQS with probability 1/(n!). More
precisely, the expected running time for
RandQS is O(n log n) on ANY input.
2006/2/21
32
RA  probabilistic analysis
• Probabilistic analysis for a (deterministic or
randomized) algorithm assumes a
particular probability distribution on the
input and then studies the expected
performance of the algorithm.
2006/2/21
33
Achilles’ heel of RA
• Is there a random number generator?
(That is, can computer throw a die?)
2006/2/21
34
Related to CMI’s
Millennium Problems
• Pseudo-random generator exists if and only
if one-way function exists.
• If one-way function exists, then P NP.
• Anybody proving or disproving P  NP gets
US$1,000,000 from Clay Math Institute,
which raised 7 open questions in year 2000,
imitating Hilbert’s 23 open questions raised
in year 1900.
2006/2/21
35
CMI Millennium Prize
• Clay Mathematics Institute (Cambridge,
MA, USA) offered US$100,000 for each of
seven open problems on May 24, 2000 at
Paris.
– | Birch and Swinnerton-Dyer Conjecture |
Hodge Conjecture | Navier-Stokes Equations |
P vs NP | Poincare Conjecture | Riemann
Hypothesis | Yang-Mills Theory |
2006/2/21
36
Simulate a fair coin by a
biased coin.
• By von Neumann (1951):
– biased “head” + biased “tail”  fair “head”.
– biased “tail” + biased “head”  fair “tail”.
– Otherwise, repeat the above procedure.
2006/2/21
37
Part 2
掃黑的藝術 (maximum cut)
2006/2/21
38
Maximum Cut Problem
• Input:
– A graph G
• Output:
– A partition of G’s
nodes into A and B
that maximizes the
number of edges
between A and B.
2006/2/21
39
Illustration
2006/2/21
40
Intractability
• The problem is NP-hard even if each node
in the input graph has no more than 3
neighbors.
Bad News
2006/2/21
41
How “hard” is NP-hard?
• So far, scientist only
knows how to solve
such a problem on an
n-node graph in O(cn)
time for some
constant c.
2006/2/21
42
That is, …
• “Impossible” to find an optimal solution for
all graphs with, say, 10000 nodes.
2006/2/21
43
On the bright side…
• If you can come up with an algorithm for
such a problem that runs in O(n1000000)
time, then you will be awarded Turing
Award for sure plus US$1,000,000 from
CMI.
2006/2/21
44
NP-completeness
= Dead end ?
2006/2/21
45
退一步 海闊天空
Approximation Algorithms
(近似演算法)
2006/2/21
46
近似演算法的哲學
放下對完美的堅持,
往往就可以找到新的出路。
知所進退, 則近道矣!
2006/2/21
47
A extremely simple randomized
approximation
For each node v of G, put v into A
with probability ½.
2006/2/21
48
Approximation Algorithm
• Criterion 1: feasibility
– output a feasible solution.
• Criterion 2: tractability
– runs in polynomial time.
• Criterion 3: quality
– The solution’s quality is provably not too far
from that of an optimal solution.
2006/2/21
49
Q1: feasibility
• Any partition is a feasible solution.
2006/2/21
50
Q2: tractability
• Runs in (deterministic) linear time, with the
help of a random number generator.
2006/2/21
51
Q3: quality?
• The expected approximation ratio is 2.
• More precisely, we can prove that the
expected cardinality of the output cut is at
least 0.5 times that of an optimal cut.
– Note that the proof for the expected
approximation ratio does not require knowing
the cardinality of an optimal in advance.
2006/2/21
52
Let C = the cardinality of the
output cut.
Let m be the number of edges in the input graph. Clearly,
the size of any cut, including any maximum cut, is at most
m.
We have
E[C] =
X
Pr[e is in the output cut]
edge e of G
=
m
:
2
So, the expected approximation ratio is at most 2.
2006/2/21
53
Interesting enough,
this simple algorithm was the
best previously known
approximation until…
M. Goemans and D. Williamson,
[ACM STOC 1994]
2006/2/21
54
A seminal paper
• 1/0.878-randomized approximation for
MAXCUT.
• Initiate a series of research in Operations
Research, Scientific Computing, and
Approximation Algorithms.
2006/2/21
55
Traditional approach
relaxation
Integer Linear Program
Linear Program
Approximation
Solver
Integral solution
Fractional solution
Rounding
2006/2/21
56
SDP-based approach
relaxation
Quadratic Program
Semi-definite Program
Approximation
Solver
Scalar solution
Vector solution
Rounding
2006/2/21
57
Remarks by 呂學一教授
• 勇於嘗試, 努力創新
• 一篇好作品的影響, 往往遠甚於許多普
通的論文. (跳高)
2006/2/21
58