UCSD CSE 21, Spring 2014 [Section A00]
Mathematics for Algorithm and System
Analysis
ABK Guest Lecture
May 23, 2014
Class URL: http://vlsicad.ucsd.edu/courses/cse21-s14/
Agenda for Today
• Fibonacci – 3 ways
• Quantifying Better or Worse: Big-O, Big-Omega, Big-Theta
• Solving Linear Homogeneous Recurrences
• Solving Divide and Conquer Recurrences: Master Theorem
Basketball Before You Were Born
• No 3-point field goal
• Hypothetical game score: UCSD 75, UCLA 64
• Assuming no 3-pointers, in how many ways could
UCSD have accumulated 75 points?
• Notation:
– S(n)  # ways to score n points
• Small Cases:
–
–
–
–
S(0) = 1
S(1) = 1
S(2) = 2
2 or 1-1
S(3) = 3
2-1 or 1-2 or 1-1-1
Is this familiar?
Recurrence Relation
• S(n) = S(n-1) + S(n-2)
• What is S(75)?
– S(0) = 1, S(1) = 1, S(2) = 2, S(3) = 3, S(4) = 5, S(5) = 8, …
• Fibonacci numbers F(n): 1, 1, 2, 3, 5, 8, …
– S(75) = F(76) = 76th Fibonacci number
Choosing Between Solutions (Algorithms)
• Criteria:
–
–
–
–
Correctness
Time resources
Hardware resources
Simplicity, clarity (practical issues)
• Will need:
– Size (“n”, number of bits, …), Complexity measures
– Notion of “basic” (“unit-cost”) machine operation
Fibonacci Numbers
• fib1(n) if n < 2 then return n
else return fib1(n-1) + fib1(n-2)
– Analysis: T(n) = 1 if n<2; T(n) =T(n-1) + T(n-2) otherwise
T(n) = F(n) next two slides: this is  (1.64)n
Solving the Fibonacci Recurrence
DT Theorem 9, page DT-48
• Notation: F(n) = F(n-1) + F(n-2)
• Guess:
try F(n) = an for some a
an = an-1 + an-2


a2 = a + 1
a2 – a – 1 = 0
Roots of quadratic: a1 = (1 + sqrt(5))/2; a2 = (1 – sqrt(5))/2
What’s missing?
“Initial Conditions”: F(1) = 1, F(2) = 1
Solving the Fibonacci Recurrence
DT Theorem 9, page DT-48
• Guess:
try F(n) = an for some a
an = an-1 + an-2


a2 = a + 1
a2 – a – 1 = 0
Roots of quadratic: a1 = (1 + sqrt(5))/2; a2 = (1 – sqrt(5))/2
• Use all of the information
We know that F(1) = 1; F(2) = 1
(initial conditions)
• Theorem 9: Homogeneous linear recurrence:
any linear combination of (a1)n , (a2)n is a solution
– Set up two equations in two unknowns:
– c1 (a1)1 + c2 (a2)1 = F(1) = 1 ; c1 (a1)2 + c2 (a2)2 = F(2) = 1
 c1 = 1 / sqrt(5) , c2 = -1 / sqrt(5)
 F(n) = c1 (a1)n + c2 (a2)n
Fibonacci Numbers
• fib2(n) f [1] = 1; f [2] = 1;
for j = 3 to n do
f [j] = f [j – 1] + f [j – 2]
• Analysis: T(n) = n
– Saving your work (“caching”) can be useful !
• …. But… note that #bits in F(n) is ~0.694n
– #bits is linear in n because F(n) is exponential in n
– So, if we count bit operations, need quadratic number of
bitwise additions to get F(n)
Always need to understand “what is a unit-cost operation” !
• Fibonacci: Can we do “better” ?
Not Obvious, But Here Is A Shortcut…
• fib3(n)
– Consider 2x2 matrix M: m11 = 0, m12 = 1, m21 = 1, m22 = 1
– Observe: [F(k) F(k+1)]T = M x [F(k-1) F(k)]T
[F(n+1) F(n+2)]T = Mn x [F(1) F(2)]T = Mn x [1 1]T
[ ]x[]=[]
0 1
1 1
Fk-1
Fk
Fk
Fk+1
IDEA: (1) Keep squaring to get powers of 2 as
exponents. (2) Build an arbitrary exponent as the
sum of powers of 2. (Squaring = multiplying.
Need only “log” number of multiplications…)
– How does this help?
– Hint: 7610 = 10011002 , 7410 = 10010102
• M74 x [F1 F2]T = [F75 F76]T
M74 = M64 x M8 x M2
 fib3 uses “addition chains”
Agenda for Today
• Fibonacci – 3 ways
• Quantifying Better or Worse: Big-O, Big-Omega, Big-Theta
• Solving Linear Homogeneous Recurrences
• Solving Divide and Conquer Recurrences: Master Theorem
Quantifying “Better”, “Worse”
• Resources used in computation often depend on a
natural parameter, n, of the input
– search/sort list
– matrix mult
– traverse tree
# items
largest dim
# nodes
• Asymptotic Notation
x>y
x*y;x+y
follow ptr
“as n grows large”
– f  O(g) if c > 0, N s.t.  n > N, f(n)  cg(n)
e.g., 200n2  O(2n2.5)
e.g., 2n + 20  O(n2)
“f grows no faster than g”
– f  (g) if g  O(f)
– f  (g) if g  O(f) and f  O(g)
– [ f  o(g) iff limnf(n)/g(n) = 0 ]
Using “Big-Oh” Notation – Examples
Definition: f(n) is monotonically growing (non-decreasing)
if n1 n2 f(n1) f(n2)
• Fact: For all constants c > 0, a > 1, and for all monotonically
growing f(n), (f(n))c  O(af(n))
• Corollary (take f(n) = n):  c > 0, a > 1, nc  O(an)
– Any exponential in n grows faster than any polynomial in n
• Corollary (take f(n) = logan):  c > 0, a > 1,
(logan)c  O(alogan) = O(n)
– Any polynomial in log n grows slower than nc’, c’>0
• Exercise: f  O(s), g  O(r) f+g  O(s+r)
• Exercise: f  O(s), g  O(r) fg  O(sr)
Agenda for Today
• Fibonacci – 3 ways
• Quantifying Better or Worse: Big-O, Big-Omega, Big-Theta
• Solving Linear Homogeneous Recurrences
Unfortunately, we skipped this part due to time constraints. But, this is the same story as
finding the closed-form expression for F(n) (analysis of fib1(n), above).
• Solving Divide and Conquer Recurrences: Master Theorem
Theorem 9 (DT-48)
Theorem 9: Let a0, a1,…, an be a sequence of numbers.
Suppose there are constants b and c such that
an = ban-1 + can-2 for n ≥ 2.
Let r1 and r2 be the roots of the
characteristic equation r2 – br – c = 0.
If there are two distinct real roots, r1, r2:
an = αr1n + βr2n for n ≥ 0
where: a0 = α + β and a1 = r1α + r2β
If there is one repeated real root, r:
an = αrn + βnrn for n ≥ 0
where: a0 = α and a1 = rα + rβ
Also: characteristic polynomial
Theorem 9 (DT-48)
Theorem 9: Let a0, a1,…, an be a sequence of numbers.
Suppose there are constants b and c such that
an = ban-1 + can-2 for n ≥ 2.
Let r1 and r2 be the roots of the
characteristic equation r2 – br – c = 0.
If there are two distinct real roots, r1, r2:
an = αr1n + βr2n for n ≥ 0
where: a0 = α + β and a1 = r1α + r2β
If there is one repeated real root, r:
an = αrn + βnrn for n ≥ 0
where: a0 = α and a1 = rα + rβ
Also: characteristic polynomial
Example 1
• Find the exact solution to the recurrence equation:
an = an-1 + 2an-2, where a0 = 1 and a1 = 8
Example 1
• Find the exact solution to the recurrence equation:
an = an-1 + 2an-2, where a0 = 1 and a1 = 8
• Recall: the characteristic equation for an = ban-1 + can-2 is
r2 – br – c = 0
• Our characteristic equation is?
Example 1
• Find the exact solution to the recurrence equation:
an = an-1 + 2an-2, where a0 = 1 and a1 = 8
• Recall: the characteristic equation for an = ban-1 + can-2 is
r2 – br – c = 0
• Our characteristic equation is?
r2 – r – 2 = 0
Example 1
• Find the exact solution to the recurrence equation:
an = an-1 + 2an-2, where a0 = 1 and a1 = 8
• Recall: the characteristic equation for an = ban-1 + can-2 is
r2 – br – c = 0
• Our characteristic equation is?
r2 – r – 2 = 0
• Solving for the roots:
r2 – r – 2 = 0
(r – 2)(r + 1) = 0
r1 = 2, r2 = -1
Example 1 Continued
• Find the exact solution to the recurrence equation:
an = an-1 + 2an-2, where a0 = 1 and a1 = 8 (I.C.’s)
• Theorem 9 tells us that if there are two real roots:
an = αr1n + βr2n for n ≥ 0
where a0 = α + β, a1 = r1α + r2β
Here: r1 = 2, r2 = -1 
an = α2n + β(-1)n
Example 1 Continued
• Find the exact solution to the recurrence equation:
an = an-1 + 2an-2, where a0 = 1 and a1 = 8
• Theorem 9 tells us that if there are two real roots:
an = αr1n + βr2n for n ≥ 0
where a0 = α + β, a1 = r1α + r2β
Since here r1 = 2, r2 = -1  an = α2n + βn(-1)n
• Plugging into equations for a0 and a1, solving for α and β:
1=α+β
8 = 2α – β
Add: 9 = 3  α = 3  β = -2
• Putting it all together:
an = 3(2n) – 2(-1)n
Proof of Correctness by Induction on n
• Trying to prove: an = 3(2n) – 2(-1)n
Given: an = an-1 + 2an-2, a0 = 1, a1 = 8
• Base case (does our equation work for a0 and a1?):
a0 = 3(20) – 2(-1)0 = 1 ✔
a1 = 3(21) – 2(-1)1 = 8 ✔
• Inductive step (does it work for an, when an = an-1 + 2an-2?):
an = an-1 + 2an-2
= 3(2n-1) – 2(-1)n-1 + 2(3(2n-2) – 2(-1)n-2)
= 6(2n-2) – 4((-1)n-2) + 3(2n-1) – 2(-1)n-1
= 3(2n-1) + 4((-1)n-1) + 3(2n-1) – 2(-1)n-1
= 6(2n-1) + 2((-1)n-1)
= 3(2n) – 2((-1)n) ✔
Example 2: Gambler’s Ruin
• A gambler repeatedly bets a flipped coin will come up heads.
– If the coin is heads, the gambler wins $1.
– If the coin is tails, the gambler loses $1.
– If the gambler ever reaches $M he/she will stop.
• Let Pk = probability gambler loses all $k he/she has (= “ruin”)
Pk = P(H)*P(ruin|H) + P(T)*P(ruin|T)
Pk = ½P(ruin|H) + ½P(ruin|T) (since we’re flipping a coin)
Pk = ½P(ruin|win $1) + ½P(ruin|lose $1)
Pk = ½Pk+1 + ½Pk-1
-½Pk+1 = – Pk + ½Pk-1  Pk+1 = 2Pk – Pk-1  Pk = 2Pk-1 – Pk-2
Example 2: Gambler’s Ruin continued
• We just learned: Pk = 2Pk-1 – Pk-2
• Recall: the characteristic equation for an = ban-1 + can-2 is
r2 – br – c = 0
• Here our characteristic equation is:
r2 – 2r + 1 = 0
(r-1)(r-1) = 0
So we have the repeated root, r = 1
• We know P0 = 1 (if we start with $0 we’re already ruined)
• We know PM = 0 (if we start with $M we quit playing the game)
Example 2: Gambler’s Ruin continued
• We just learned: Pk = 2Pk-1 – Pk-2 and P0 = 1, PM = 0
• Theorem 9: if there is one repeated real root, r:
an = αrn + βnrn for n ≥ 0
where: a0 = α
a1 = rα + rβ
remember: we have r = 1
• 0 = PM
PM = αrM + βMrM
= P0 + βM
  = -1/M
 Pn = 1 – n/M
Example 2: Gambler’s Ruin continued
• Pn = 1 – n/M, so our probability of ruin, Pk = 1 – k/M
• If we have $10 and won’t stop playing unless we have
$100, what is the probability that we will lose our $10?
k = 10
M = 100
P10 = 1 – 10/100 = .90, or 90%!
• If we have $10 but only want to win $12, what is the
probability that we will lose our initial $10?
P10 = 1 – 10/12 = 1/6 = 0.1667, now only 16.67%
Agenda for Today
• Fibonacci – 3 ways
• Quantifying Better or Worse: Big-O, Big-Omega, Big-Theta
• Solving Linear Homogeneous Recurrences
• Solving Divide and Conquer Recurrences: Master Theorem
“Master Theorem for D/Q Recursions”
Theorem 8, GT-47
Recurrence T(n) ≤ aT(n/b) + O(nd)
work done at kth level is ak x O(n/bk)d = O(nd) x (a / bd)k
“Master Theorem for D/Q Recursions”
Theorem 8, GT-47
Recurrence T(n) ≤ aT(n/b) + O(nd)
1) if a < bd T(n) = O(nd)
2) if a = bd T(n) = O(nd log n)
3) if a > bd T(n) = O(nlogba)
work done at kth level is ak x O(n/bk)d = O(nd) x (a / bd)k
Type (3): long multiplication, matrix multiplication
Type (2): mergesort
Master Theorem Examples
• Mergesort T(n) = 2T(n/2) + (n)
T ( n )   ( n log n )
• Matrix Multiply T(n) = 8T(n/2) + (n2)
T ( n )   ( n log 2 8 )   ( n 3 )
Proof of “Master Theorem”
This is CSE 101 material: don’t worry for now, but you’ll need this in a year
Recurrence T(n) ≤ aT(n/b) + O(nd)
1) T(n) = O(nd) if a < bd
2) T(n) = O(nd log n) if a = bd
3) T(n) = O(nlogba) if a > bd
• Total work done at kth level is ak x O(n/bk)d = O(nd) x (a / bd)k
• As k goes from 0 (root) to logbn (leaves), have geometric series
with ratio a / bd
do you remember how to sum a geometric series?
• 1) a / bd < 1  sum is O(nd)
• 3) a / bd > 1  sum is given by last term, O(nlogba)
• 2) a / bd = 1  sum is given by O(log n) terms equal to O(nd)
Master Theorem, Extended a Bit …
a,b
f(n) (=nd)
T(n)
-----------------------------------a = 1
c
logdn
a = b
c
A 1 n + A2
a < b
cn
A1 n
a = b
cn
O(n logbn)
a > b
cn
O(n logba)
(From Tucker’s Applied Combinatorics text)
The MaxMin Problem
• Given a set S of n numbers, use divide-andconquer to find the maximum element and the
minimum element in S
Split S into two subsets of size n/2 each
Recursively find the max and min of each subset
Compare the two max’s, compare the two min’s
• Recurrence: T(n) = 2T(n/2) + 2
• Which case is this in the previous slide?
T(n) = 3n/2 – 2