Chapter 7: Quick Sort
Recursive sorting using Partition
Partition: algorithm that enable divide and conquer
Worst-case runtime of Quicksort = Q(n2)
Randomized Quicksort has expected runtime = Q(nlgn)
Randomized Quicksort is asymptotically optimal
Pseudocode Quicksort(A,p,r)
% to sort whole array p=1, r=length[A]
if p < r then
q  Partition(A,p,r)
% q is index that defines subarrarys
Quicksort(A,p,q-1)
Quicksort(A,q+1,r)
Partition: Algorithm for Divide and Conquer
Continue until j = r-1, then exchange A[i+1] and A[r]
Note: upper and lower sub-arrays are not the same size
and not sorted
Note: pivot is sorted relative elements in upper and lower
sub-arrays
Partition runtime: T(n) =Q(n) why?
Quicksort’s worst case
Quicksort(A,p,r])
if p < r then
q  Partition(A,p,r)
% q is the index of pivot
Quicksort(A,p,q-1)
Quicksort(A,q+1,r)
Worst-case partitioning: (always maximum imbalance)
Partition(A,p,r) always returns q = p
What type of input would make this happen?
A worst-case Quicksort(A,p,r])
if p < r then
p  Worst Partition(A,p,r)
Quicksort(A,p+1,r)
What happened to the lower sub-array?
What recurrence describes this worst case?
What is its solution?
CptS 350 Spring 2017
Homework Assignment 12: due 3/22/17
Worst-case runtime of Quicksort described by
T(n) = T(n-1)) + Q(n)
Show by tree analysis that T(n) = Q(n2) is a guess of the solution
Prove by structured substitution that T(n) has asymptotic
solution T(n) = Q(n2)
More rigorous worst-case analysis
We argued for T(n)=T(n-1)+Q(n) as worst case for Quicksort
based on a particular input (sorted in decreasing order).
A derive a rigorous recurrence for worst-case Quicksort:
Assume elements of A are indexed 0 to n-1
lower array contains elements 0 to q-1
upper array contains elements q+1 to n-1
On the 1st call to Partition, 0 < q < n-1
Find the value of q that maximizes T(n)
T(n) =
max
0  q  n 1
(T(q) + T(n-q-1)) + Q(n)
Show by the substitution method that solution is T(n)=O(n2).
T(n) = 0max
(T(q) + T(n-q-1)) + Q(n)
 q  n 1
Replace Q(n) by dn
Assume: T(q)=O(q2), T(n-q-1)=O((n-q-1)2)
Exist c>0 such that T(q)<cq2 and T(n-q-1)<c(n-q-1)2
T(n) < 0max
(cq2 + c(n-q-1)2) + dn
 q  n 1
< c ( 0max
(q2 + (n-q-1)2)) + dn
 q  n 1
f(q)=q2+((n-1)-q)2 df/dq=2q-2(n-q-1)=0 qm = (n-1)/2
d2f/dq2 = 4 therefore f(q) is minimum at qm
f(0)=(n-1)2 and f(n-1)=(n-1)2 so f(q) is maximum at q= 0 and n-1
If qmax= 0 or qmax= n-1 then T(n) < c(n-1)2 + dn < cn2
which is the same inequality we encounter in proof that T(n)=T(n-1)+dn
has asymptotic solution T(n)=O(n2)
Best case runtime of Quicksort described by
Use T(n) =
min
0 q  n 1
(T(q) + T(n-q-1)) + Q(n)
to show by structured substitution that T(n) has asymptotic
solution T(n)=W(nlgn).
This proves that Quicksort is asymptotically optimal
Therefore T(n)=W(nlgn) by definition
Suppose partition always returned split 9n/10 and n/10
Will the performance of Quicksort be O(n2)?
What recurrence describes this performance?
What is its solution?
T(n)=T(9n/10)+T(n/10)+Q(n)
Guess solution by imbalanced tree analysis.
CptS 350 Spring 2017
Homework Assignment 13: due 3/24/17
Prove by structured substitution that T(n)=T(9n/10)+T(n/10)+Q(n) has
asymptotic solution T(n)=Q(nlgn)
Summary:
Worst case Quicksort: T(n)=O(n2)
Best case Quicksort: T(n)=W(nlgn)
With imbalance other than 0 and n-1: T(n)=W(nlgn)
Is the worst case an outlier?
What is the runtime of Quicksort with a “typical” input?
Define a stochastic version of Quicksort: Randomized Quicksort
Find the expected value of the runtime
“Randomized” means For a given array of n elements to be sorted,
choose at random on the n! permutations.
Randomized Quicksort
Randomized by a random choice of pivot.
Randomized-Partition(A, p, r)
i  Random(p, r)
exchange A[r]  A[i]
return Partition
Randomized-Partition just calls Partition after randomly choosing a pivot.
Randomized-Quicksort(A, p, r)
if p < r then
q  Ramdomized-Partition(A, p, r)
Randomized-Quicksort(A, p, q-1)
Randomized-Quicksort(A, q+1, r)
Runtime of Randomized Quicksort is a random variable.
Random choice of pivot varies the degree of imbalance between
upper and lower subarrarys
Expected run time of Randomized Quicksort
What is the expected number of comparison from all calls to
Randomized-Partition in a complete execution of RandomizedQuicksort(A,1,n)?
Before we answer this question
Review probability theory (Appendix C)
Learn the technique of “indicator random variables”
Appendix C.2: Theory discrete probabilities
Discrete probabilities are defined in terms of a sample space S.
Usually S is a collection of elementary events that are outcomes of
independent experiments, such as flipping coins, throwing dice, pulling
cards, etc.
|S| = size of S (also called “cardinality” of S)
Example: S = set of outcomes from flipping 2 coins = {HH, HT, TH, TT}
Events are subsets of S. (S itself is called the “certain” set.)
The event of getting 1 head and 1 trail = {HT, TH}
The empty subset, , called the “null event”
Brief review of set theory (Appendix B.1, text p1158)
If x  A implies x  B, then A is a subset of B (written A  B)
If, in addition, A  B then A is a proper subset of B (written A  B)
A  B = intersection of sets A and B defined by {x: x  A and x  B}
A  B = union of sets A and B defined by {x: x  A or x  B}
A - B = difference of sets A and B defined by {x: x  A and x  B}
For laws obeyed by sets see text pp1159-1162
Definition of probability on sample space S
(1) Pr{A} > 0 for any event A
(2) Pr{S} = 1
(3) for any 2 mutually exclusive events (i.e. A  B = )
Pr{A  B} = Pr{A} + Pr{B}
By definition, elementary events are always mutually exclusive.
From the definition of discrete probabilities, it follows that
Pr{} = 0
If A  B then Pr{A} < Pr{B}
Pr{S – A} = 1- Pr{A}
S-A is the complement of A
For any two events (not necessarily mutually exclusive)
Pr{A  B} = Pr{A} + Pr{B} – Pr{A  B} < Pr{A} + Pr{B}
Discrete probability distributions
Defined over a finite or countable infinite sample space
Let s denote elementary events of S,
then for any event A
Pr{A} =  sA P{s}
If S is finite and every sS has probability 1/|S|,
then P(s) is uniformly distributed on S.
We call the experiment that gives rise to elementary events
“picking an event at random from S”
Example: in S = {HH, HT, TH, TT},
Pr{HH} = Pr{HT} = Pr{TH} = P{TT} = ¼
The probability of at least one head is
Pr{HH, HT, TH} = Pr{HH} + Pr{HT} + Pr{TH} = ¾
= 1 – Pr{TT} = 1 – ¼ = ¾
A “fair” coin flipped n times defines a uniform probability distribution on S
Elementary events are strings HHTHT…
For each of n positions in string we have 2 choices of letter
|S| = 2n
For n=2, S = {HH, HT, TH, TT},
What is the probability of event A = {exactly k heads and n-k tails}?
Pr{A} = (nk)/2n where
(nk) = n!/(k!(n-k)!)
= number ways to choose k items out of a total of n
(nk) are binomial coefficients (see text pp 1185-1186)
n
   2n

S is normalized
 
k 0 k 
n
Conditional probability: example
Given some knowledge about outcomes, we want the probability of an
outcome conditioned on our prior knowledge about it.
Suppose that someone flips 2 coins and tells us that at least one
shows heads. What is the probability that both coins are show heads?
S = {HH, HT, TH, TT}
Our prior knowledge eliminates elementary event TT.
The remaining 3 possibilities are equally likely.
Pr{HH} conditioned on at least 1 head is showing = 1/3.
Conditional probability: general definition
Probability of A conditioned on B, Pr{A|B}, is meaningful only
if Pr{B}  0.
Given that B occurs, the probability that A also occurs is related
to the set of outcomes in which both A and B occur.
 Pr{A|B} is proportional to Pr{AB}
If we normalize Pr{A|B} by dividing by Pr{B} (which ≠ 0), then
Pr{B|B} = Pr{BB}/Pr{B} = Pr{B}/Pr{B} = 1
Conditional probability: application of definition
In the experiment where 2 fair coins are flipped, calculate the probability
of 2 heads showing given that at least one head is showing using the
definition of conditioned probability.
Conditional probability: application of definition
Apply Pr{A|B} = Pr{AB}/Pr{B} to the problem
“probability of 2 heads showing given that at least one head is showing”
AB is the event with 2 heads showing and at least one head showing.
In S = {HH, HT, TH, TT}, HH is the only such event.
Pr{AB} = ¼
B is the event with at least one head showing.
3 out of 4 events in S have this property; therefore Pr{B} = ¾
Using the definition, Pr{A|B} = Pr{AB}/Pr{B}
Pr{A|B} = (1/4)/(3/4) = 1/3 > Pr{AB}
Additional knowledge increased probability
Conditional probability: application to independent events
Two events are independent if Pr{AB} = Pr{A}Pr{B}
If events A and B are independent and Pr{B}  0,
then Pr{A|B} = Pr{A}Pr{B}/Pr{B} = Pr{A}
Discrete random variables:
Mappings of finite or countable infinite sets of events onto the real numbers
For random variable X and real number x, the event X = x is the
collection of all elementary events {sS: X(s) = x}.
Pr{X = x} = {sS: X(s) = x} Pr{s}
f(x) = Pr{X = x} is the “probability density function” of random variable X
By axioms of probability f(x) > 0 and x f(x) = 1
In physical sciences more likely to encounter “continuous” random variables
defined for uncountable infinite sample spaces
In the continuous case, probability density function f(x) is defined such that
f(x)dx is probability that random variable X has a value between
x and x + dx.
Example of discrete random variable:
A pair of 6-sided dice is rolled.
Random variable X is the max of the 2 numbers showing.
What is Pr{X = 3}?
Use exhaustive enumeration
Elementary events with 3 as max of the 2 numbers showing
(1,3), (2,3), (3,3), (3,2), and (3,1).
The probability of each elementary event is (1/6)(1/6) = 1/36
 Pr{X = 3} = 5/36
Expected value of a random variable
discrete: E[X] = x xPr{X=x}
continuous: E[X] = x f(x)xdx
f(x) is probability density of X
If g(X) defines a new random variable,
then E[g(X)] = x g(x)Pr{X=x}
Special case: Given random variables X and Y
E[aX + bY] = aE[X] + bE[Y]
by property of sums and integrals
Called “linearity” of expectation values
For any two random variables X and Y
joint probability density function is Pr{X=x and Y=y}
for any fixed value of y Pr{Y=y} = x Pr{X=x and Y=y}
called “marginal” probability
Pr{X=x | Y=y} = Pr{X=x and Y=y}/Pr{Y=y}
called “conditional” probability
X and Y are independent if for all x and y
Pr{X = x and Y = y} = Pr{X = x}Pr{Y = y}
If X and Y are independent, then E[XY] = E[X]E[Y]
proof on p1198 of text
Indicator Random Variables
For any event A,
Indicator Random Variable I{A} = 1 if A occurs
= 0 if A does not occur
Zero and one are the only values an indicator random
variables can have
Lemma 5.1:
Let A be an event in sample space S, then the expected
value of I{A} = probability of event A.
Proof:
For any random variable X, E[X] = x x Pr{X = x}
For indicator random variables x = 0 or 1, only
The complement of event A in sample space S is S-A
E[I{A}] = (1)Pr(A) + (0)Pr(S - A) = Pr(A)
Structured solution to
“What is the expected number of heads for n coin flips?”
solved by indicator random variables
1. What is the random variable whose expectation value we want
to calculate? X = number of heads showing in n coins flips
2. What event defines the indicator random variable?
Ai = ith coin flip shows heads. Let I{Ai} = Xi
3. What is the probability that Ai occurs? E(Xi) = ½
4. How is random variable whose expectation we are
calculating related to the indicator random variable?
X = i=1 to n (Xi)
5. What is the expectation value of this random variable?
E[X] = E[i=1 to n (Xi)] = i=1 to n E[Xi] = i=1 to n (1/2) = n/2
Another example:
Randomized Hire-Assistant procedure
Randomize interview schedule
best  0 initialization
for i  1 to n
do interview candidate i
if candidate i is better than best
then best  i
hire candidate i
How does the expected total cost of finding the best assistant depend on
ci, the cost of interviewing, and ch, the cost of hiring, individual applicants?
total cost = cin + ch(number hired) is a random variable because number
hired depends on the order of candidates are interviewed, which is random.
Least total cost when best candidate is first.
Max total cost when interviewed in increasing order of quality
Structured solution:
1. What is the random variable whose expectation value we want
to calculate? Number of assistants hired
2. What event defines the indicator random variable?
Ak = kth candidate is hired. Let Xk = I{Ak}
3. What is the probability that this event occurs?
Since interview schedule randomized, kth candidate has probability 1/k of
being better than all previous candidates. E[Xk] = 1/k
4. How is random variable whose expectation we are calculating related to
the indicator random variable? X = k=1 to n Xk
5.What is the expectation value of this random variable?
E[X] = k=1 to n E[Xk] = k=1 to n (1/k) = ln(n) + O(1) (text p1147)
<total cost> = nci + ln(n)ch + O(1)
Cpt S 350 Spring 2017
Homework Assignment 14: due 3/29/17
Use indicator random variables to compute the expected value of the sum
of faces showing when n fair dice are tossed.
Give a structured solution.
Expected run time of Randomized Quicksort using
indicator random variables
What is the random variable whose expectation value we
wish to calculate?
X=number of comparison from all calls to RandomizedPartition in a complete execution of RandomizedQuicksort(A,1,n)?
What is the event that defines the indicator random
variable?
How is X related to the indicator random variable?
X=number of comparison from all calls to RandomizedPartition in a complete execution of RandomizedQuicksort(A,1,n)?
What is the event that defines the indicator random
variable?
Let Aij be the event when element i compared to j
Xij = I{Aij}
How is X related to the indicator random variable?
n 1
X =
n
X
ij
i 1 j  i 1
This is correct iff any pair of elements can be compared at
most once.
Last line of Partition stores the pivot between partitioned subarrays
Once used, a pivot is never involved in subsequent recursive calls
In sorting an array of length n, Partition called at most n times
Elements are compared to pivots only
Elements in separate subarrays are never compared
Rename the elements <a1, a2, ..., an> of array A by their order statistic
Call then z1, z2, ..., zn, where zi is the ith smallest element
z1 is the minimum element, zn is maximum element
Define subsets of elements Zij = {zi, zi+1, ..., zj} to be the elements (by size)
between zi and zj inclusively
Since elements are compared to pivots only and elements in separate
subarrays are never compared, zi and zj are compared, at most, once
when one or the other is a pivot.
n 1
n
  Xij
X=
is the correct relationship between X and I{Aij}
i 1 j  i 1
n 1
E(X) = 
n 1
n
 E[X ]
i 1 j  i 1
ij
=
n
  Pr{z i compared to zj}
i 1 j i 1
zi and zj are compared iff the 1st pivot chosen from the set
Zij={zi, zi+1,...zj} is either zi and zj.
If zi is the 1st pivot chosen from the set Zij then zj cannot be the
1st pivot chosen from that set
two possibilities leading to comparison of zi and zj are mutually
exclusive.
 Pr{ zi is compared to zj} = Pr{ zi is the 1st pivot chosen from Zij} +
Pr{ zj is the 1st pivot chosen from Zij}
= 2/(j – i + 1)
because all elements in Zij equally likely to be first chosen
n 1
E[X] =
n
  2/(j - i  1)
i 1 j  i 1
n 1 ni
E[X] =
E[X] <
  2 /( k  1)
i 1 k 1
n 1
n i
i 1
k 1
 2/ k
n 1
E[x] <
 O(lg(n))
i 1
E[x] = O(nlgn)
(by change of variable k = j-i)
k+1 -> k increases every term
extending inner sum to n gives harmonic sum
value, ln(n) + O(1), differs from lg(n) by constant