9/3/2015
• Inductive argument shows that total number
of levels of the recursion tree is lg + 1
– base case = 1: tree has only one level; lg 1
0 lg + 1 is the correct number of levels
– Inductive hypothesis: number of levels of a
tree with 2 leaves is lg 2 + 1 = + 1
– We assume that the input size is a power of 2,
the next input size to consider is 2
– A tree with = 2
leaves has one more
level than a tree with 2 leaves, and so the
total number of levels is + 1) + 1 = lg 2
1
MAT-72006 AADS, Fall 2015
3-Sep-15
+
72
• To compute the total cost represented by the
recurrence, we simply add up the costs of all
the levels:
– The recursion tree has lg + 1 levels, each
costing , for a total cost of
(lg + 1) =
lg
– Ignoring the low-order term and the
constant gives the desired result of
( lg )
MAT-72006 AADS, Fall 2015
3-Sep-15
73
1
9/3/2015
3 Growth of Functions
• Order of growth of the running time
– gives a simple characterization of the
algorithm’s ef ciency
– allows us to compare the relative performance
of alternative algorithms
• Usually, an algorithm that is asymptotically
more ef cient will be the best choice for all
but very small inputs
MAT-72006 AADS, Fall 2015
3-Sep-15
74
3.1 Asymptotic notation
• For a function
set of functions
={
constants
1
) we denote by
)) the
there exist positive
1 2, and 0 such that
) for all
2
MAT-72006 AADS, Fall 2015
3-Sep-15
0}
75
2
9/3/2015
• A function ( ) belongs to
)) if there exist 1
and 2 such that it can be
“sandwiched” between
and 2
) for
1
suf ciently large
• Because
)) is a set,
we could write
“
))”
• We usually write
“
)=
)” instead
MAT-72006 AADS, Fall 2015
3-Sep-15
76
• When
)=
)) we say that
) is an
asymptotically tight bound for
)
• The de nition requires that every member
)) be asymptotically
nonnegative
–
) be nonnegative whenever
suf ciently large
is
• The function
) itself must be
asymptotically nonnegative, or else the set
( ( )) is empty
MAT-72006 AADS, Fall 2015
3-Sep-15
77
3
9/3/2015
• Let us use the formal de nition to show that
2
½
– We must determine positive 1, 2, and 0
2–
2 for all
such that 1 2
2
– Dividing by 2 yields 1
3/
2
– We can make the right-hand inequality hold
for any value of
1 by choosing 2
– Likewise, we can make the left-hand
inequality hold for any value of
7 by
choosing 1 1/14
– Thus, by choosing 1 1/14, 2 ½, and
2–
7 we can verify that
= ( 2)
MAT-72006 AADS, Fall 2015
3-Sep-15
0
78
• We can also use the formal de nition to verify
2)
that 3
• Suppose for the purpose of contradiction that
3
2 for all
2 and 0 exist such that
2
0
• But then dividing by 2 yields
2/6, which
cannot possibly hold for arbitrarily large ,
since 2 is constant
MAT-72006 AADS, Fall 2015
3-Sep-15
79
4
9/3/2015
-notation
•
-notation asymptotically bounds a function
from above and below
• When we have only an asymptotic upper
bound, we use -notation
• For a function
) we denote by
)) the
set of functions
)) = {
there exist positive
constants and 0 such that
0
( )
( ) for all
0}
MAT-72006 AADS, Fall 2015
3-Sep-15
80
3-Sep-15
81
• Note that ( ) =
)) implies
)=
)), since
-notation is a stronger
notion than -notation
• When we use notation to bound the
worst-case running
time, we have a bound
on the running time of
the algorithm on every
input
MAT-72006 AADS, Fall 2015
5
9/3/2015
-notation
•
–notation provides an asymptotic lower
bound on a function
• For a function
) we denote by
)) the
set of functions
)) = {
there exist positive
constants and 0 such that
0}
MAT-72006 AADS, Fall 2015
3-Sep-15
82
3-Sep-15
83
Theorem 3.1
For any two functions
) and
), we have
)=
))
if and only if
)=
)) and
))
Proof Exercises.
MAT-72006 AADS, Fall 2015
6
9/3/2015
)
• The running time of insertion sort is
• The running time of insertion sort therefore
2)
belongs to both
) and
2),
• E.g., the running time of insertion sort is not
since there exists an input for which insertion sort
runs in
) time
• It is not contradictory, however, to say that the
worst-case running time of insertion sort is
2), since there exists an input that causes
2) time
the algorithm to take
MAT-72006 AADS, Fall 2015
3-Sep-15
84
-notation
• The asymptotic bound provided by -notation
may or may not be asymptotically tight:
– The bound
– The bound
2
2
is asymptotically tight
2 ) is not
• Formally define
)) as the set
)) = {
for any positive
constant > 0 there exists 0 > 0 such that
)<
0}
MAT-72006 AADS, Fall 2015
3-Sep-15
85
7
9/3/2015
• For example, 2 = ( 2), but 2 2
( 2)
• The main difference between -notation and
-notation is that in
)=
)), the bound
holds for some constant > 0,
–
)=
)), the bound
holds for all constants > 0
–
)<
• Intuitively, in -notation, the function
)
becomes insigni cant relative to
) as
approaches in nity; i.e.,
)
lim
=0
( )
MAT-72006 AADS, Fall 2015
3-Sep-15
)
)
86
-notation
• We use -notation to denote a lower bound that
is not asymptotically tight
• One way to de ne it is by
)) if and
only if
))
• Formally we, however, we
)) as the set
)) = {
for any positive
constant > 0 there exists 0 > 0 such that
)<
) for all
0}
• E.g.,
2=
, but
2
)
MAT-72006 AADS, Fall 2015
3-Sep-15
87
8
9/3/2015
Comparing functions
Transitivity:
•
•
•
•
)=
)=
)=
•
)),
)),
)),
,
)=
)=
,
)=
))
))
)=
)=
)=
))
)=
MAT-72006 AADS, Fall 2015
Re exivity:
•
)=
•
)=
•
)=
))
))
))
))
3-Sep-15
88
3-Sep-15
89
))
))
))
Symmetry:
•
)=
)) iff
)=
))
Transpose symmetry:
•
)=
)) iff
)=
))
• ( ) = ( ( )) iff ( ) = ( ( ))
MAT-72006 AADS, Fall 2015
9
9/3/2015
We can draw an analogy between the
asymptotic comparison of functions and the
comparison of real numbers
•
)=
)) is like
•
)=
)) is like
•
)=
)) is like
•
)=
)) is like
•
)=
)) is like
MAT-72006 AADS, Fall 2015
3-Sep-15
90
3.2 Standard notations
and common functions
• For all real constants
lim
and
s.t.
=0
> 1,
• from which we can conclude that = o( )
• Thus, any exponential function with a base
strictly greater than 1 grows faster than any
polynomial function
MAT-72006 AADS, Fall 2015
3-Sep-15
91
10
9/3/2015
Exponentials
• Let to denote 2.71828 …, the base of the
natural logarithm, we have for all real ,
=1+ +
2!
+
3!
=
!
• For all real , we have inequality
1+ ,
where equality holds only when = 0
• When
1, we have the approximation
1+
1+ + 2
MAT-72006 AADS, Fall 2015
3-Sep-15
92
• When
0, the approximation of by 1 +
is quite good:
2)
=1+
• In this equation, the asymptotic notation is
used to describe the limiting behavior as
0 rather than as
• We have for all
lim 1 +
MAT-72006 AADS, Fall 2015
3-Sep-15
93
11
9/3/2015
Factorials
• Notation ! is de ned for integers
0 as
1
if = 0
!=
1 ! if > 0
• Thus, ! = 1 · 2 · 3 ···
• A weak upper bound is
, since each of
the terms in the factorial product is at most
• Stirling’s approximation gives a tighter upper
bound, and a lower bound as well,
1
!=
1+
MAT-72006 AADS, Fall 2015
3-Sep-15
94
Fibonacci numbers
• We de ne the Fibonacci numbers by the
following recurrence:
–
–
–
=0
1 = 1
0
for
2
• Thus, each Fibonacci number is the sum of
the two previous ones, yielding the sequence
0,1, 1, 2, 3, 5, 8, 13, 21, 34, 55,89,144,233,377, …
MAT-72006 AADS, Fall 2015
3-Sep-15
95
12
9/3/2015
• Fibonacci numbers are related to the golden
ratio and to its conjugate , which are the two
roots of the equation
+1
1+ 5
=
= 1.61803 …
2
5
=
.61803 …
2
• By induction we have ( = 0 5 = 0,
=
2 5 2 5 = 1, …)
=
5
MAT-72006 AADS, Fall 2015
3-Sep-15
96
< 1, we have
1
1
<
<
5
5 2
which implies
1
=
+
5 2
• Since
• Hence, the th Fibonacci number is equal
to / 5 rounded to the nearest integer
• Thus, Fibonacci numbers grow exponentially
MAT-72006 AADS, Fall 2015
3-Sep-15
97
13
9/3/2015
4 Divide-and-Conquer
• Three methods for obtaining asymptotic
bounds on the solution:
or
– In substitution method, we guess a bound
and then use induction to prove it correct
– recursion-tree method converts the
recurrence into a tree whose nodes represent
the costs incurred at various levels of the
recursion. We use techniques for bounding
summations to solve the recurrence
MAT-72006 AADS, Fall 2015
3-Sep-15
98
• The master method provides bounds for
recurrences of the form
)+
,
where
1, > 1, and
) is a given
function
• This characterizes an algorithm that creates
subproblems, each of which is 1/ the size
of the original problem, and in which the
divide and combine steps together take
)
time
MAT-72006 AADS, Fall 2015
3-Sep-15
99
14
9/3/2015
4.2 Strassen’s algorithm
for matrix multiplication
• Let =
and =
be square
matrices; in the product
, we de ne
the entry , for
= 1, 2, … , , by
=
• We must compute
matrix
entries, and each is the
sum of values
MAT-72006 AADS, Fall 2015
3-Sep-15
100
3-Sep-15
101
SQUARE-MATRIX-MULTIPLY
)
1.
2. let be a new
matrix
3. for
to
4.
for
to
5.
0
6.
7.
for
to
8. return
MAT-72006 AADS, Fall 2015
15
9/3/2015
• Each of the nested for loops runs exactly
iterations, execution of line 7 takes constant
time, the procedure takes
) time
• We have a way to multiply matrices in
time
• Strassen’s remarkable recursive algorithm for
multiplying
matrices runs in
time
• lg 7 lies between 2.80 and 2.81, Strassen’s
algorithm runs in
MAT-72006 AADS, Fall 2015
3-Sep-15
102
A simple divide-and-conquer algorithm
• Assume that is an exact power of 2 in each
of the
matrices
• Suppose that we partition each of , , and
into four 2 × 2 matrices
=
=
so that we rewrite
=
as
MAT-72006 AADS, Fall 2015
=
3-Sep-15
103
16
9/3/2015
• This corresponds to the four equations
• Each of these four equations speci es two
multiplications of 2 × 2 matrices and the
addition of their 2 × 2 products
• Based on them we can create a
straightforward, recursive, divide-andconquer algorithm
MAT-72006 AADS, Fall 2015
3-Sep-15
SQUARE-MATRIX-MULTIPLY-RECURSIVE( , )
1.
2. let be a new
matrix
3. if = 1
4.
5. else partition , , and
6.
RECURSE
) + RECURSE
7.
RECURSE
) + RECURSE
8.
RECURSE
) + RECURSE
9.
RECURSE
) + RECURSE
10. return
MAT-72006 AADS, Fall 2015
104
)
)
)
)
3-Sep-15
105
17
9/3/2015
• When = 1, we perform just the one
constant-time scalar multiplication in line 4
• When > 1, partitioning the matrices in line 5
takes (1) time, using index calculations
• Lines 6–9 recurse eight times, the time taken
by all recursive calls is
2)
• We also must account for the four matrix
additions in lines 6–9
• Each matrix contains
4 entries, and so
each of the four additions takes
) time
MAT-72006 AADS, Fall 2015
3-Sep-15
106
• The number of matrix additions is a constant,
the total time spent adding matrices in lines
6–9 is
)
• Total time for the recursive case is the sum of
– the partitioning time,
– the time for all the recursive calls, and
– the time to add the matrices resulting from the
recursive calls:
1 +8
=8
2) +
2) +
)
MAT-72006 AADS, Fall 2015
)
3-Sep-15
107
18
9/3/2015
• The recurrence for the running time of
SQUARE-MATRIX-MULTIPLY-RECURSIVE is
(1)
if = 1
=
2) if > 1
• We shall see that this recurrence has the
)
solution
• The approach is no faster than the
straightforward SQUARE-MATRIX-MULTIPLY
procedure
MAT-72006 AADS, Fall 2015
3-Sep-15
108
• In accounting for the eight recursive calls we
cannot just subsume the constant factor of 8
• We must say that together they take
2)
time, rather than just
2)
• If we were to ignore the factor of 8, the
recursion tree would just be linear, not
“bushy,” and each level would contribute only
one term to the sum
• Although asymptotic notation subsumes
constant multiplicative factors, recursive
notation such as
2) does not
MAT-72006 AADS, Fall 2015
3-Sep-15
109
19
9/3/2015
Strassen’s method
• Makes the recursion tree slightly less bushy
– Instead of performing eight recursive
multiplications, it performs only seven
– The cost is several new additions of
2 × 2 matrices, but only a constant
number of them
• Constant number of additions is subsumed
by -notation when we set up the recurrence
to characterize the running time
MAT-72006 AADS, Fall 2015
3-Sep-15
110
1. Divide matrices , , and into 2 × 2
submatrices in (1) time by index calculation
2. Create matrices
,…,
, each of which is
2 × 2 and is the sum or difference of two
matrices created in step 1
We can create all 10 matrices in
time
3. Using the matrices created, recursively
compute seven matrix products
,…,
Each matrix is 2 × 2
4. Compute the submatrices
by
adding and subtracting various combinations of
the matrices
We can compute all submatrices in
time
MAT-72006 AADS, Fall 2015
3-Sep-15
111
20
9/3/2015
• We have enough information to set up a
recurrence for the running time of the method
• Let’s assume that once the matrix size gets
down to 1, we perform a simple multiplication
• When > 1, steps 1, 2, and 4 take a total of
time, and step 3 requires us to perform
seven multiplications of 2 × 2 matrices.
• Hence, we obtain the following recurrence:
(1)
if = 1
=
2) if > 1
• By the master method this recurrence has the
)
solution ( ) = (
MAT-72006 AADS, Fall 2015
3-Sep-15
112
Matrices of Step 2
•
•
•
•
•
•
•
•
•
•
• we must add or
subtract 2 × 2
matrices 10 times, so
this step does take
time
MAT-72006 AADS, Fall 2015
3-Sep-15
113
21
9/3/2015
• In step 3, we recursively multiply 2 ×
matrices seven times to compute the
following 2 × 2 matrices:
•
•
•
•
•
•
•
MAT-72006 AADS, Fall 2015
3-Sep-15
2
114
• Step 4 adds and subtracts the matrices
created in step 3 to construct the four
2 × 2 submatrices of the product C
•
•
•
•
E.g.,
+
MAT-72006 AADS, Fall 2015
=
3-Sep-15
115
22
9/3/2015
4.3 The substitution method
• We can use the method to establish either
upper or lower bounds on a recurrence
• Let us, e.g., determine an upper bound on
the recurrence
2
• We guess the solution
)=
lg
• The substitution method requires us to prove
lg for an appropriate choice
that
of the constant > 0
MAT-72006 AADS, Fall 2015
3-Sep-15
116
• We start by assuming that this bound holds
for all positive
, in particular for
=
2 , yielding
2
2 lg( 2 )
• Substituting into the recurrence yields
2
2 lg
2
lg
lg
lg
lg
2
lg 2 +
(as long as
MAT-72006 AADS, Fall 2015
3-Sep-15
1)
117
23
9/3/2015
• We now need to show that our solution holds
for the boundary conditions
– Show that we can choose the constant large
enough so that the bound
lg works
for the boundary conditions as well
• Let us assume that 1 = 1 is the sole
boundary condition of the recurrence
• For = 1, we get (1)
lg 1 = 0, which is
at odds with 1 = 1
• Asymptotic notation requires us only to prove
the bound for
, where
is a constant
that we get to choose
MAT-72006 AADS, Fall 2015
3-Sep-15
118
• We keep the condition (1) = 1, but remove it
from consideration in the inductive proof
• First observe that for > 3, the recurrence does
not depend directly on (1)
• Thus, we replace (1) by (2) and (3) as the
base of inductive proof, letting
=2
• The recurrence yields 2 = 2 1 + 2 = 4 and
3 = 2 1 +3 =5
• To complete the proof, we choose so that
lg 2 and (3)
lg 3
• Any choice of
2 suf ces for the base cases
of = 2 and = 3 to hold
MAT-72006 AADS, Fall 2015
3-Sep-15
119
24
9/3/2015
Making a good guess
• There is no general way to guess the correct
solutions to recurrences
• If a recurrence is similar to one you have
seen before, then guessing a similar solution
is reasonable
• Another way to make a good guess is to
prove loose upper and lower bounds on the
recurrence and then reduce the range of
uncertainty
MAT-72006 AADS, Fall 2015
3-Sep-15
120
Subtleties
• Consider the recurrence
2
2 +1
• We guess
)=
), and try to show that
for an appropriate choice of
• Substituting our guess, we obtain
2
2 +1
+ 1,
which does not imply
for any
choice of
MAT-72006 AADS, Fall 2015
3-Sep-15
121
25
9/3/2015
• We could try a larger guess, e.g.,
)=
• The original guess of
)=
is correct
– We must make a stronger inductive
hypothesis
• We are off only by the lower-order term 1
• To overcome the dif culty subtract a lowerorder term from our previous guess
• Our new guess is
, where
is a constant
MAT-72006 AADS, Fall 2015
• We now have
=
2
+
+1
3-Sep-15
2
122
+1
as long as
1
• As before, we must choose large enough to
handle the boundary conditions
• The idea of subtracting a lower-order term
may seem counterintuitive
• Proving an upper bound by induction, it may
be more dif cult to show a weaker bound
MAT-72006 AADS, Fall 2015
3-Sep-15
123
26
9/3/2015
• To prove the weaker bound, we must use the
same weaker bound inductively in the proof
• There is more than one recursive term, and
we get to subtract out the lower-order term
once per recursive term
• We subtracted out the constant twice, once
for ( 2 ) and once for ( 2 )
• We ended up with the inequality
+ 1, and it was easy to nd values of to
make
+ 1 be less than or equal to
MAT-72006 AADS, Fall 2015
3-Sep-15
124
Avoiding pitfalls
• It is easy to err in asymptotic notation
• E.g., in an earlier recurrence we can falsely
guess
and argue
2
2
);
• wrong!!
• We have not proved the exact form of the
inductive hypothesis, that is, that
• We therefore will explicitly prove that
when we want to show that
)
MAT-72006 AADS, Fall 2015
3-Sep-15
125
27
9/3/2015
Changing variables
• A little algebraic manipulation can make an
unknown recurrence more familiar
• The recurrence
=2
+ lg looks
dif cult
• We can simplify it with a change of variables
• For convenience, we shall not worry about
rounding off values, such as , to be
integers
MAT-72006 AADS, Fall 2015
• Renaming
3-Sep-15
126
= lg yields
2 =2 2
• We can now rename
) = (2
to
produce the new recurrence
=2
2) +
which looks more familiar
• Indeed, this new recurrence has the solution
)=
lg )
• Changing back from
) to
), we obtain
) = (2 ) =
)=
lg )
= (lg lg lg )
MAT-72006 AADS, Fall 2015
3-Sep-15
127
28