CSC 201 : Design and
Analysis of Algorithms
Introduction
Asymptotic notation
Proof By Induction
Mudasser Naseer
1
Thursday, Aug 15, 2010
The Course
• Purpose: a rigorous introduction to the design
and analysis of algorithms
 Not a lab or programming course
 Not a math course, either
• Textbook: Introduction to Algorithms, Cormen,
Leiserson, Rivest, Stein (2nd or 3rd Ed.)
 An excellent reference you should own
The Course
• Instructor: Dr. Mudasser Naseer
 [email protected],
[email protected]
 Office: Not yet allocated
 Office hours: 10:00 – 11:00 (Mon, Wed and Fri)
The Course
• Grading policy:




Homework + Assignments: 25%
Exam-1: 10%
Exam-2: 15%
Final: 50%
Formal Analysis of Algorithms
• A problem is a (infinite) set of instances.
• An instance is a triple <input, output, size>
 Less formally, we treat “input” as an instance.
 “Size” is related to the number of symbols needed to
represent the instance, but the exact convention is
problem-specific. (E.g., nxn matrix may be “size n”.)
• A decision problem is a problem in which every
instance’s output is “yes” or “no”.
Example: Sorting is a problem.
<5, 2, 17, 6> is an instance of sorting,
with output <2, 5, 6, 17> and size 4.
Formal Analysis of Algorithms
•
Algorithm: a step-by-step method of producing
“output” of instances given “input” OR
finite sequence of computational steps that transforms
the given input into the output for a given problem
 The steps are instructions for some model of computation.
 “Standard” model is the Random Access Machine (RAM)
Memory
ALU
All instructions are executed sequentially, with no concurrent operations.
Simple operations like addition, subtraction, assignments etc, takes one
step. One needs to be careful about what operations are allowed
- E.g. operations on very long numbers aren’t “one step”.
Formal Analysis of Algorithms
• An algorithm is considered to be correct, if
•
for every input instance, it generates the
correct output and gets terminated.
An algorithm can be specified in any
language e.g. English or a computer language,
or even as a hardware design.
Study of Algorithms
• Design of algorithms
 Divide and conquer
 Incremental approach
 Dynamic programming
• Algorithm Validation
 Checks the algorithm result for all legal set of
inputs.
• Analysis of Algorithms
 Space complexity
 Time complexity
• Algorithm Testing
Asymptotic Performance
• In this course, we care most about asymptotic
performance
 How does the algorithm behave as the problem
size gets very large?
o Running time
o Memory/storage requirements
o Bandwidth/power requirements/logic gates/etc.
Asymptotic Notation
• By now you should have an intuitive feel for
asymptotic (big-O) notation:
 What does O(n) running time mean? O(n2)?
O(n lg n)?
 How does asymptotic running time relate to
asymptotic memory usage?
• Our first task is to define this notation more
formally and completely
Analysis of Algorithms
• Analysis is performed with respect to a
•
computational model
We will usually use a generic uniprocessor
random-access machine (RAM)
 All memory equally expensive to access
 No concurrent operations
 All reasonable instructions take unit time
o Except, of course, function calls
 Constant word size
o Unless we are explicitly manipulating bits
Input Size
• Time and space complexity
 This is generally a function of the input size
o E.g., sorting, multiplication
 How we characterize input size depends:
o Sorting: number of input items
o Multiplication: total number of bits
o Graph algorithms: number of nodes & edges
o Etc
Running Time
• Number of primitive steps that are executed
 Except for time of executing a function call most
statements roughly require the same amount of
time
oy=m*x+b
o c = 5 / 9 * (t - 32 )
o z = f(x) + g(y)
• We can be more exact if need be
Analysis
• Worst case
 Provides an upper bound on running time
 An absolute guarantee
• Average case
 Provides the expected running time
 Very useful, but treat with care: what is “average”?
o Random (equally likely) inputs
o Real-life inputs
• Best Case
 Provides a lower bound on running time
Review: Induction
• Suppose
 S(k) is true for fixed constant k
o Often k = 0
 S(n)  S(n+1) for all n >= k
• Then S(n) is true for all n >= k
Proof By Induction
• Claim:S(n) is true for all n >= k
• Basis:
 Show formula is true when n = k
• Inductive hypothesis:
 Assume formula is true for an arbitrary n
• Step:
 Show that formula is then true for n+1
Induction Example:
Gaussian Closed Form
• Prove 1 + 2 + 3 + … + n = n(n+1) / 2
 Basis:
o If n = 0, then 0 = 0(0+1) / 2
 Inductive hypothesis:
o Assume 1 + 2 + 3 + … + n = n(n+1) / 2
 Step (show true for n+1):
1 + 2 + … + n + n+1 = (1 + 2 + …+ n) + (n+1)
= n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2
= (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2
Induction Example:
Geometric Closed Form
• Prove a0 + a1 + … + an = (an+1 - 1)/(a - 1) for
all a  1
 Basis: show that a0 = (a0+1 - 1)/(a - 1)
a0 = 1 = (a1 - 1)/(a - 1)
 Inductive hypothesis:
o Assume a0 + a1 + … + an = (an+1 - 1)/(a - 1)
 Step (show true for n+1):
a0 + a1 + … + an+1 = a0 + a1 + … + an + an+1
= (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1)
Induction
•
•
We’ve been using weak induction
Strong induction also holds
 Basis: show S(0)
 Hypothesis: assume S(k) holds for arbitrary k <= n
 Step: Show S(n+1) follows
•
Another variation:
 Basis: show S(0), S(1)
 Hypothesis: assume S(n) and S(n+1) are true
 Step: show S(n+2) follows
The End