Recurrences
Algorithms
Jay Urbain, PhD
[email protected]
Credits:
Discrete Mathematics and Its Applications, by Kenneth Rosen
The Design and Analysis of Algorithms, by Levitan
1
Overview
We are interested in determining the running time behavior
of algorithms expressed as bounds using Θ, O, or Ω
representation.
2
Recursive Algorithms
• Recurrence relations result naturally from the analysis of
recursive algorithms.
• Solving recurrence relations yields a closed-end formula
for calculation of run time.
• Example: the recursive n! algorithm:
int factorial (int n) {
if (n == 1) return 1;
return factorial (n-1) * n;
}
3
n!
int factorial (int n) {
if (n == 1) return 1;
return factorial (n-1) * n;
}
• The running time, T(n), can be defined as:
T(1) = 1*T(n) = T(n-1) + 1 for all n>1
• where 1 is the running time for each execution of the
factorial function.
4
n!
• Solving the recurrence by recognizing the summation leads to closedend formula:
T(n) = T(1) + 1 + ... + 1 = 1 + 1 + ... + 1 = n
• The run time for execution of factorial can now be directly computed
for a given n.
T(10) = 10.
• More importantly, when expressing the worst case run time bounds
we can then write:
T(n) = O(1) if n=1;
T(n) = O(n) if n>1
5
Sequences and Recurrence Relations
Sequence - a numerical, ordered list of numbers
• T - a recurrence function defining a sequence
• n - a parameter to function T
• T(n) - the sequence term generated by function T for parameter n
6
Sequences and Recurrence Relations
Is T(1) = 1 necessary?
7
Sequences and Recurrence Relations
• Inductive proofs on the recurrence:
T(n) = T(n-1) + 1
Typically assume the case for parameter n that generates a sequence
term for recurrence T(n) and prove: T(n+1)=T((n+1)-1)+1.
• For above recurrence: T(n) = T(floor(n/4)) + n
What parameter value generates the next sequence term following
T(n)?
8
Defining a sequence
• Either Tn or T(n) which stresses that a sequence is a function where
T(n) is the generic nth term.
• Defining a sequence (two ways)
– Closed-end formula of generic term as a function of n,
T(n) = 2n for n>=0
– Recurrence equation relating generic term to one or more other
sequence terms combined with one or more explicit values of first
term(s).
T(n) = T(n-1) + n
T(0) = 0
for n>1
// generic term defined by other
sequence terms, i.e., recurrence
//initial condition
9
Example: Fibonacci
• Fibonacci sequence definition:
int fibonacci (int n) {
if( n==0 || n==1 ) return n;
return fibonacci(n-1) + fibonacci(n-1);
}
• has the corresponding recurrence:
F(0)=0
Initial
F(1)=1
Initial
F(n) = F(n-1)+F(n-2) for n>1
Recurrence
10
Solve Recurrence Relation
Find closed-end formula for generic nth term of sequence that satisfies
both the recurrence term and initial condition or prove that the
sequence does not exist.
• Example
T(n) = T(n-1) + n
T(0) = 0
for n>1
B1
B2
• Solution of B1 subject to initial condition B2 is:
T(n) = n(n+1)
for n>=0
B3
2
11
Verify closed-end solution
•
Substitute solution into recurrence formula to check that equality holds
or use induction to prove. B1 must hold for every n>0:
12
Solution to Recurrence
• Particular solution to recurrence - a specific sequence that satisfies
recurrence.
– B3 is a particular solution for B1 and B2.
• General solution to recurrence - formula that specifies all
sequences, typically includes arbitrary constants. A general solution for
recurrence B1 is:
T(n) = c + n(n+1) for n>=0
2
• choosing different values for c gives all solutions to B1.
• No universal method but common techniques for a variety of
recurrences!
13
Forward substitution
• Forward substitution for finding exact closed-end equation for
T(n) - limited to simple recurrences.
• Start with initial term(s) given by initial conditions, generate first few
terms in hope of seeing a pattern that can be expressed by a closedend formula.
• Example
T(n) = 2T(n-1)+1 for n>1
B5
T(1)=1
B6
• Generate
T(1) = 1 = 21-1
T(2) = 2T(2-1)+1 = 2T(1)+1 = 2*1+1 = 3 = 22-1
T(3) = 2T(3-1)+1 = 2T(2)+1 = 2*3+1 = 7 = 23-1
T(4) = 2T(4-1)+1 = 2T(3)+1 = 2*7+1 = 15 = 24-1
• Hope
14
Observe pattern that suggests T(n) = 2n-1 for n=1,2,3,4
Forward substitution
• Prove T(n) = 2n-1 correct by substitution or induction
15
Backward Substituion
•
•
•
•
express T(n-1) as a function of T(n-2)
substitute result to give T(n) as a function of T(n-2)
repeating for T(n-2) gives T(n) as function of T(n-3)
hope to see a pattern to express T(n) as function of T(n-i) for i=1,2,
...
• select i to make n-i reach the initial condition
• closed-end formula
– use standard summation formula from Appendix A will often lead
to closed-end formula
– when lucky, get form similar to: T(n) = Base case + f(n), and can
solve by substitution
16
Backward Substituion
17
Backward Substitution
18
Common Recurrence Types
in Algorithmic Analysis
• Decrease-by-one algorithms exploits relationship between instance of
size n and smaller instance of size n-1.
• Decrease-by-a-constant factor algorithms reduce instance of
size n to an instance of size n/b (b=2 for many such algorithms),
solving the smaller instance recursively. If necessary, the solution of
the smaller instance is combined to a solution of the given instance.
• Divide-and-conquer divides a given instance into smaller instances,
solving each recursively. If necessary solutions to smaller instances
are combined to give a solution to a given instance.
19
Decrease-by-one algorithms
• Decrease-by-one algorithms exploits relationship between instance of
size n and smaller instance of size n-1.
20
Decrease-by-one algorithms
• Typical form of recurrence equation, where f(n) is the time to reduce an
instance to a smaller one is:
21
Specific functions - f(n)
22
Decrease-by-a-constant
factor algorithms
• Reduce instance of size n to an instance of size n/b (b=2 for many
such algorithms), solving the smaller instance recursively.
• If necessary, the solution of the smaller instance is extended to a
solution of the given instance.
23
Decrease-by-a-constant
factor algorithms – Binary Search
24
Divide and Conquer
Divides a given instance into smaller instances, solving each recursively. If
necessary solutions to smaller instances are combined to give a solution to a given
instance.
25
26
Merge Sort
27
Merge Sort
28
Merge Sort
29
General form of recurrence
equation
•
•
•
•
•
n is the problem size
B>=2 the number of problem divisions
n/b is the size of the smaller problem after division
A>=1 is the number of sub-problem solutions
f(n) is the time to divide an instance to a smaller ones and combine
the solutions
T(n) = aT(n/b) + f(n)
30