Algorithmic Analysis
1
Analysis of Algorithms
Input
Algorithm
Output
An algorithm is a step-by-step procedure for
solving a problem in a finite amount of time.
2
How Fast is an Algorithm?
• Experimental
– Must actually implement the algorithm
– Results are system dependent (language, OS, hardware)
– Results are for the tested input only. Testing does not “prove”
anything about data that hasn’t been tested.
• Analytical
– Do not need to implement the algorithm
– Results are not dependent on a particular system
– Conclusions can be determined for all cases without performing
any tests!
3
Average Case vs. Worst Case
Running Timeof an algorithm
• An algorithm may run faster on certain data sets than on others.
• Finding the average case can be very difficult, so typically
algorithms are measured by the worst-case time complexity.
• Also, in certain application domains (e.g., air traffic control,
surgery, IP lookup) knowing the worst-case time complexity is of
crucial importance.
worst-case
5 ms
}
4 ms
average-case?
3 ms
best-case
2 ms
1 ms
A
B
C
D
Input
E
F
G
4
Measuring the Running Time
• How should we measure the running time of an algorithm?
• Approach 1: Experimental Study
– Write a program that implements the algorithm
– Run the program with data sets of varying size and composition.
– Use a method like System.currentTimeMillis() to get an accurate
measure of the actual running time.
t (ms)
60
50
40
30
20
10
5
0
50
100
n
Beyond Experimental Studies
• Experimental studies have several limitations:
– It is necessary to implement and test the algorithm in order to determine its
running time.
– Experiments can be done only on a limited set of inputs, and may not be
indicative of the running time on other inputs not included in the
experiment.
– In order to compare two algorithms, the same hardware and software
environments should be used.
6
Beyond Experimental Studies
• We will now develop a general methodology for
analyzing the running time of algorithms. In
contrast to the "experimental approach", this
methodology:
– Uses a high-level description of the algorithm instead of
testing one of its implementations.
– Takes into account all possible inputs.
– Allows one to evaluate the efficiency of any algorithm in a
way that is independent from the hardware and software
environment.
7
Pseudo-Code
• Pseudo-code is a description of an algorithm that is more structured
than usual prose but less formal than a programming language.
• Example: finding the maximum element of an array.
Algorithm arrayMax(A, n):
Input: An array A storing n integers.
Output: The maximum element in A.
currentMax  A[0]
for i 1 to n -1 do
if currentMax < A[i] then currentMax  A[i]
return currentMax
• Pseudo-code is our preferred notation for describing algorithms.
• However, pseudo-code hides program design issues.
8
What is Pseudo-Code ?
• A mixture of natural language and high-level programming concepts that
describes the main ideas behind a generic implementation of a data
structure or algorithm.
-Expressions: use standard mathematical symbols to describe
numeric and boolean expressions -use  for assignment (“=” in Java)
-use = for the equality relationship (“==” in Java)
-Method Declarations:
-Algorithm name(param1, param2)
-Programming Constructs: - decision structures:
if ... then ... [else ... ]
- while-loops:
- repeat-loops:
- for-loop:
- array indexing:
-Methods:
while ... do
repeat ... until ...
for ... do
A[i]
- calls:
object method(args)
- returns: return value
9
The Random Access Machine
(RAM) Model
• A CPU
• An potentially unbounded
bank of memory cells, each
of which can hold an
arbitrary number or character
2
0
1
• Memory cells are numbered and accessing any
cell in memory takes unit time.
10
Primitive Operations
• Basic computations performed
• Examples:
by an algorithm
– Evaluating an
• Identifiable in pseudocode
expression
• Largely independent from the
– Assigning a value
programming language
to a variable
• Exact definition not important
– Indexing into an
(we will see why later)
array
• Assumed to take a constant
– Calling a method
amount of time in the RAM
– Returning from a
model
method
11
Counting Primitive
Operations
• By inspecting the pseudocode, we can determine the
maximum number of primitive operations executed by an
algorithm, as a function of the input size
Algorithm arrayMax(A, n)
currentMax  A[0]
for i  1 to n  1 do
if A[i]  currentMax then
currentMax  A[i]
{ increment counter i }
return currentMax
# operations
2
2+n
2(n  1)
2(n  1)
2(n  1)
1
Total
7n  1
12
Estimating Running Time
• Algorithm arrayMax executes 7n  1 primitive
operations in the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
• Let T(n) be worst-case time of arrayMax. Then
a (7n  1)  T(n)  b(7n  1)
• Hence, the running time T(n) is bounded by two linear
functions
13
Analysis is Fun!
•
Each CPU operation takes time
–
–
–
–
–
–
–
arithmetic operations (+, -, /, *, %)
logical operations (!, &&, ||, ^)
comparisons (<,==,>,<=, >=, !=)
assignment
method calls and method returns
array access
member access
•
These operations may actually take different lengths of time (depending on the
CPU and operating system) but assume (for simplicity) that each operation
takes the same amount of time to complete.
•
The length of time a program will take to execute is given by the total number
of operations multiplied by the length of time per operation
TotalTimeprogram = Nop * Timeop
14
Asymptotic Notation
• A theoretical means of ordering functions
• Goal: to simplify analysis by getting rid of
unneeded information (like “rounding”
1,000,001≈1,000,000)
• We want to say in a formal way 3n2 ≈ n2
• Compares relative growth rates of functions - not
the functional values
• “Big-Oh” notation expresses a less-than-or-equalto relation between two functions
– given functions f(n) and g(n), we say that f(n)
is O(g(n) ) if and only if there are positive
constants c and n0 such that f(n)≤ c g(n) for n
≥ n0
15
The (Big) O Notation
O( g (n))  { f (n) : there exist positive constants c and n0 such that
0  f (n)  cg (n) for all n  n0}.
cg (n )
f (n )
g (n ) is an asymptotic upper bound for f (n ).
Examples:
n0
n
n 2  O(n 2 )
n  O(n 2 )
n
 O (n 2 )
1200
n 2 + n  O(n 2 )
Note: Since changing the base of a
log only changes the function by
a constant factor, we usually don’t
worry about log bases in
asymptotic notation.
n 2 + 1000n  O(n 2 )
5230n + 1000n  O(n )
2
2
n1.99999  O(n 2 )
n2
 O(n 2 )
log n
16
More Big-Oh Examples
7n-2
7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1
 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3 for n  n0
this is true for c = 4 and n0 = 21

3 log n + log log n
3 log n + log log n is O(log n)
need c > 0 and n0  1 such that 3 log n + log log n  c•log n for n  n0
this is true for c = 4 and n0 = 2
17
Big-Oh and Growth Rate
• The big-Oh notation gives an upper bound on the growth
rate of a function
• The statement “f(n) is O(g(n))” means that the growth rate of
f(n) is no more than the growth rate of g(n)
• We can use the big-Oh notation to rank functions according
to their growth rate
f(n) is O(g(n))
g(n) is O(f(n))
g(n) grows more
f(n) grows more
Yes
No
No
Yes
Same growth
Yes
Yes
18
Big-Oh Rules
• If is f(n) a polynomial of degree d, then f(n) is
O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
• Use the smallest possible class of functions
– Say “2n is O(n)” instead of “2n is O(n2)”
• Use the simplest expression of the class
– Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”
19
Common Growth Rates
Name
Big Oh Notation
Constant
O(1)
Logarithmic
O(log N)
Log-squared
O(log2 N)
Linear
O(N)
O(N log N)
Quadratic
O(N2)
Cubic
O(N3)
Exponential
O(2N)
20
Table of Common Functions
Growth Rates
n
log n
n^.5
n
n log n
n^2
n^3
2^n
2
1
1.4
2
2
4
8
4
4
2
2
4
8
16
64
16
8
3
2.8
8
24
64
512
256
16
4
4
16
64
256
4096
65536
32
5
5.7
32
160
1024
32768
4294967296
64
6
8
64
384
4096
262144
1.84E19
128
7
11
128
896
16384
2097152
3.4E38
256
8
16
256
2048
65536
16777216
1.15E77
512
9
23
512
4608
262144
134217728
1.34E154
21
Other Common Asymptotic
Notations
•  - “Big Omega”
– If T(n) is (f(n)) then T(n) is greater-than-or-equal-to f(n)
•  - “Big Theta”
– If T(n) is (f(n) )then T(n) is equal-to f(n)
• o – “Little-Oh”
– If T(n) is o(f(n)) then T(n) is strictly less-than f(n)
22
The  Notation
( g (n))  { f (n) : there exist positive constants c and n0 such that
0  cg (n)  f (n) for all n  n0}.
f (n )
cg (n )
g (n) is an asymptotic lower bound for f (n ).
Examples:
n0
n
n 2  (n 2 )
n 2 + n  (n 2 )
2311n 2 + 1000n  (n 2 )
n 2.0001  (n 2 )
n 2 lg lg n  (n 2 )
22  (n2 )
n
n  (n )
3
2
23
The  Notation
( g (n))  { f (n) : there exist positive constants c1, c2 , and n0
such that 0  c1g (n)  f (n)  c2 g (n) for all n  n0}.
c2 g (n)
f (n )
g (n ) is an asymptotic ally tight bound for f (n ).
c1g (n)
n0
n
Example:
n2
1
1
2
 2n  (n ), with c1  , c2  , and n0  8.
2
4
2
24
Relatives of Big-Oh
• big-Omega
– f(n) is (g(n)) if there is a constant c > 0
and an integer constant n0  1 such that
f(n)  c•g(n) for n  n0
• big-Theta
– f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and an integer
constant n0  1 such that c’•g(n)  f(n)  c’’•g(n) for n  n0
• little-oh
– f(n) is o(g(n)) if, for any constant c > 0, there is an integer constant n0  0
such that f(n)  c•g(n) for n  n0
• little-omega
– f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0  0
such that f(n)  c•g(n) for n  n0
25
Rules for Asymptotic Notation
• If T1(n) is O(f(n)) and T2(n) is O(g(n)) then
– T1(n) + T2(n) is Max(O(f(n)), O(g(n)))
– T1(n) * T2(n) is O(f(n)*g(n))
• If T(n) is a polynomial of degree k then
– T(n) is (nk)
• Logkn is O(n) for any constant k
26
Intuition for Asymptotic
Notation
Big-Oh
– f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n)
big-Omega
– f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n)
big-Theta
– f(n) is (g(n)) if f(n) is asymptotically equal to g(n)
little-oh
– f(n) is o(g(n)) if f(n) is asymptotically strictly less than g(n)
little-omega
– f(n) is (g(n)) if is asymptotically strictly greater than g(n)
27
Example Uses of the
Relatives of Big-Oh

5n2 is (n2)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1
such that f(n)  c•g(n) for n  n0
let c = 5 and n0 = 1
5n2 is (n)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1
such that f(n)  c•g(n) for n  n0
let c = 1 and n0 = 1
5n2 is (n)
f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0 
0 such that f(n)  c•g(n) for n  n0
need 5n02  c•n0  given c, the n0 that satisfies this is n0  c/5  0
28
Rules for Asymptotic Notation
• It is bad style to include constants or lowerorder terms when using asymptotic notation
• Example:
– If T(n) is O(10n3+5n-12) we should write that T(n) is O(n3)
29
Asymptotic Analysis of The
Running Time
• Use the Big-Oh notation to express the number of primitive
operations executed as a function of the input size.
• For example, we say that the arrayMax algorithm runs in O(n)
time.
• Comparing the asymptotic running time
-an algorithm that runs in O(n) time is better than one that runs in O(n2) time
-similarly, O(log n) is better than O(n)
-hierarchy of functions: log n << n << n2 << n3 << 2n
• Caution! Beware of very large constant factors. An algorithm
running in time 1,000,000 n is still O(n) but might be less efficient
on your data set than one running in time 2n2, which is O(n2)
30
Example of Asymptotic
Analysis
An algorithm for computing prefix averages
Algorithm prefixAverages1(X):
Input: An n-element array X of numbers.
Output: An n -element array A of numbers such that A[i] is the average of
elements X[0], ... , X[i].
Let A be an array of n numbers.
for i 0 to n - 1 do
a0
for j  0 to i do
a  a + X[j]
1 step
A[i]  a/(i+ 1)
return array A
• Analysis ...
i iterations
with
i=0,1,2...n-1
n iterations
31
Another Example
• A better algorithm for computing prefix averages:
Algorithm prefixAverages2(X):
Input: An n-element array X of numbers.
Output: An n -element array A of numbers such that A[i] is the average of
elements X[0], ... , X[i].
Let A be an array of n numbers.
s 0
for i  0 to n do
s  s + X[i]
A[i]  s/(i+ 1)
return array A
• Analysis ...
32
Analysis Example
Problem: Write an efficient method to compute xn where x is
a real number and n is a non-negative integer.
public double power(double x, int n) {
double result = 1;
for(int i=1; i<=n; i++) {
result *= x;
}
return result;
}
Is this fast
enough?
33
Analysis Example
Problem: Write an efficient method to compute xn where x is
a real number and n is a non-negative integer.
public double power(double x, int n) {
if(n == 0) return 1;
else if(n == 1) return x;
else if(n % 2 == 0) return power(x*x, n/2);
else return power(x*x, n/2) * x;
}
Am I good
or what?
Notice the following:
x3 = (x2)x
x4 = (x2)2
x5 = (x2)2x
x6 = (x2)3
x7 = (x2)3x
x8 = (x2)4
Show me the money!
34
Analysis Example
Problem: Write an efficient method to compute xn where x is
a real number and n is a non-negative integer.
public double power(double x, int n) {
if(n == 0) return 1;
else if(n == 1) return x;
else if(n % 2 == 0) return power(x*x, n/2);
else return power(x*x, n/2) * x;
}
Think, think,
think, ….!
Are the following modifications OK?
•
•
•
return power(power(x,2), n/2);
return power(power(x, n/2), 2);
return power(x, n/2) * power(x, n/2);
35
Analysis Example
Problem: Given an ordered set of (possibly negative)
integers A1, A2, A3, …, AN write an efficient method to find
the maximum value of
j
A
k
1  i  jN
k i
For convenience, the value is defined to be 0 if all the numbers are negative.
Example.
Given [-2, 11, -4, 13, -5, -2] what is the correct answer?
36
First Idea
public int maxSubSum1(int[] a) {
int maxSum = 0;
for(int i=0; i<a.length; i++){
for(int j=i; j<a.length; j++) {
int thisSum = 0;
for(int k=i; k<=j; k++) {
thisSum += a[k];
}
if(thisSum > maxSum) {
maxSum = thisSum;
}
}
}
return maxSum;
}
How “good” is this solution?
37
Second Idea
public int maxSubSum2(int[] a) {
int maxSum = 0;
for(int i=0; i<a.length; i++){
int thisSum = 0;
for(int j=i; j<a.length; j++) {
thisSum += a[j];
if(thisSum > maxSum) {
maxSum = thisSum;
}
}
}
return maxSum;
}
How “good” is this solution?
38
Third Idea
public int maxSubSum3(int[] a) {
int maxSum = 0, thisSum = 0;
for(int i=0; i<a.length; i++){
thisSum += a[i];
if(thisSum > maxSum) {
maxSum = thisSum;
} else if(thisSum < 0) {
thisSum = 0;
}
}
return maxSum;
}
How “good” is this solution?
39
Greatest Common Divisor
Problem: Compute the largest integer D that evenly divides two non-negative
integers M and N.
Example: Compute the greatest common divisor of 1989 and 1590
1989 = 1 * 3 * 3 * 13 * 17
1590 = 1 * 2 * 3 * 5 * 53
gcd(m,n) =
m
if n = 0
gcd(n, m%n)
otherwise
Example
Mr. Euclid
gcd(1440, 408) = gcd(408, 216)
gcd(408, 216) = gcd(216, 192)
gcd(216,192) = gcd(192, 24)
gcd(192, 24) = gcd(24, 0)
gcd(24, 0) = 24
40
Greatest Common Divisor
Problem: Compute the largest integer D that evenly divides two non-negative
integers M and N.
gcd(m,n) =
m
if n = 0
gcd(n, m%n)
otherwise
Implementation
int gcd(int m, int n) {
while(n != 0) {
int rem = m % n;
m = n;
n = rem;
}
return m;
}
Mr. Euclid
41
Summary
• Algorithmic analysis usually focuses on
– Determining the efficiency of an algorithm
• Quantified by growth rate using Big Oh notation
– Proving the correctness of an algorithm
• Detailed Examples
– Power
– Maximum subsequence
– Greatest common divisor
42