Analysis & Design of
Algorithms
(CSCE 321)
Prof. Amr Goneid
Department of Computer Science, AUC
Part 2. Types of Complexities
Prof. Amr Goneid, AUC
1
Types of Complexities
Prof. Amr Goneid, AUC
2
Types of Complexities
 Rules for Upper Bound
 Comparing Complexities
 Types of Complexities
Prof. Amr Goneid, AUC
3
1. Rules for Upper Bound
 If (k) is a constant, then:
O(k)  O(n) and
O(k f(n)) = O(f(n))
 e.g. an2 and bn2 are
both O(n2)
T(n)
O(kn)
O(n)
O(k)
n
Prof. Amr Goneid, AUC
4
Rules for Big O
 The growth rate of a sum of terms of is the
growth rate of its fastest growing term
O(f(n)) + O(g(n)) = max(O(f(n)) ,O(g(n)))
 e.g. f(n) = 2n = O(n), g(n) = 0.1 n3 = O(n3)
T(n) = max(O(n) , O(n3)) = O(n3)
2n
O(n3)
0.1 n3
Prof. Amr Goneid, AUC
5
Rules for Big O
 The product of big-O’s is the big-O of the
products:
O(f(n)) * O(g(n)) = O(f(n) * g(n))
 e.g. T1(n) = O(n), T2(n) = O(n2)
T(n) = T1 (n) * T2 (n) = O(n3)
Repeat n times
O(n2)
O(n3)
Prof. Amr Goneid, AUC
6
Rules for Big O
 Logarithms grow slower than powers:
O(loga n)  O(nk) for all a > 1 and k > 0
 e.g. O(log n) < O(n)
 Claim:
For all n ≥ 1, log n ≤ n.
Prove by induction on n.
Prof. Amr Goneid, AUC
7
Rules for Big O
 All logarithms grow at the same rate:
loga n =  (logb n) for all a , b > 1
 e.g. log2 n =
 (log3 n)
 Proof:
n  blogb n
loga n  loga b logb n  (loga b )logb n  c logb n   (logb n )
Prof. Amr Goneid, AUC
8
Rules for Big O
 For a polynomial of degree m,
T ( n )  Pm ( n )  a0  a1n  a2 n  ...  a m n
2
m
Then T ( n )  O( n m )
e.g.T ( n )  2  4n  3n  2n  O( n )
2
3
3
 Prove!
 O(nm-1)  O(nm) follows from above
Prof. Amr Goneid, AUC
9
Rules for Big O
 Exponential functions grow faster than powers
O(nk) < O(bn) for all k ≥ 0 and b > 1
 e.g. O(n3) < O(2n)
 Proof:
log nk  k log n
and log b n  n log b
Since log n  n then
O( nk )  O( b n )
for all k  0 and b  1
Prof. Amr Goneid, AUC
10
Summary of Rules for Big-O
Rule
Example
For constant k, O(k) < O(n)
O(7) = O(1) < O(n)
For constant k, O(kf) = O(f)
O(2n) = O(n)
O(|f|+|g|) = O(|f|) + O(|g|) =
Max (O(|f|) , O(|g|)
O(6n2+n)=O(6n2)+O(n) = O(n2)
Nesting of loop O(g) within a
loop O(f) gives O(f*g)
O(n4*n2)=O(n6)
O(nm-1) < O(nm)
O(n2) < O(n3)
O((log n)k) O(n)
O(log n) < O(n)
Prof. Amr Goneid, AUC
11
2. Comparing Complexities
Dominance:
 If lim(n->) f(n)/g(n) = 
then f(n) dominates (i.e. grows faster)
 If lim(n->) f(n)/g(n) = 0
little oh
then g(n) dominates.
In the latter case, we say that f(n) = o(g(n))
 If lim(n->) f(n)/g(n) = constant
then both grow at the same rate.
Prof. Amr Goneid, AUC
12
Comparing Complexities
Examples:
 if a > b then na dominates nb
Why?
 n2 dominates (3n+2)
Why?
 n2 dominates (n log n)
Why?
Prof. Amr Goneid, AUC
13
Using l’Hopital’s Rule (1696)
If lim f ( n )   and lim g( n )  
n 
f(n)
then lim
n  g( n )
n 
f '( n )
 lim '
n  g ( n )
d
where the prime means
dn
Example: Show that
f(n) = n(k+α) + nk (log n)2 and
g(n) = k n(k+α)
grow at the same rate.
Prof. Amr Goneid, AUC
14
Comparing Complexities
 Which grows faster:
f(n) = n3
or
f(n) = n 0.001 or
f(n) = 2n+1
or
f(n) = 2n
or
g(n) = n log n
g(n) = log n
g(n) = 2n
g(n) = 22n
Prof. Amr Goneid, AUC
15
Exercises
Which function has smaller complexity ?
 f = 100 n4
g = n5
 f = log(log n3)
g = log n
 f = n2
g = n log n
 f = 50 n5 + n2 + n
g = n5
 f = en
g = n!
Prof. Amr Goneid, AUC
16
3. Types of Complexities
 Constant Complexity
 T(n) = constant independent of (n)
 Runs in constant amount of time  O(1)
 Example: cout << a[0][0]
Prof. Amr Goneid, AUC
17
Types of Complexities
 Logarithmic Complexity
= m is equivalent to n=2m
 Reduces the problem to half  O(log2n)
 Example: Binary Search
T(n) = O(log2n)
Much faster than Linear Search which has
T(n) = O(n)
 Log2n
Prof. Amr Goneid, AUC
18
Linear vs Logarithmic Complexities
T(n)
O(n)
x
logx
O(log2n)
n
x
Prof. Amr Goneid, AUC
19
Types of Complexities
 Polynomial Complexity
 T(n) = amnm+…+ a2n2 + a1n1 + a0
 If m=1, then O(a1n+a0)  O(n)
 If m > 1, then  O(nm) as nm dominates
Prof. Amr Goneid, AUC
20
Polynomials Complexities
Log T(n)
O(n3)
O(n2)
O(n)
n
Prof. Amr Goneid, AUC
21
Types of Complexities
 Exponential
 Example:
List all the subsets of a set of n
elements {a,b,c}
{a,b,c}, {a,b},{a,c},{b,c},{a},{b},{c},{}
 Number of operations T(n) = O(2n)
 Exponential expansion of the problem 
O(an) where a is a constant greater than 1
Prof. Amr Goneid, AUC
22
Exponential Vs Polynomial
Log T(n)
O(2n)
O(n3)
O(n)
n
Prof. Amr Goneid, AUC
23
Types of Complexities
 Factorial time Algorithms
 Example:
 Traveling
salesperson problem (TSP):
Find the best route to take in visiting n
cities away from home. What are the
number of possible routes? For 3 cities:
(A,B,C)
Prof. Amr Goneid, AUC
24
Possible routes in a TSP
Home
Amsterdam
NY
Montreal
NY
Montreal
Amsterdam
Montreal
Amsterdam
NY
Montreal
NY
Montreal
Amsterdam
NY
Amsterdam
–Number of operations = 3!=6, Hence T(n) = n!
–Expansion of the problem  O(n!)
Prof. Amr Goneid, AUC
25
Exponential Vs Factorial
Log T(n)
O(nn)
O(n!)
O(2n)
n
Prof. Amr Goneid, AUC
26
Execution Time Example
 Example:
 For
the exponential algorithm of listing all
subsets of a given set, assume the set size
to be of 1024 elements
 Number of operations is 21024 about
1.8*10308
 If we can list a subset every nanosecond
the process will take 5.7 * 10291 yr!!!
Prof. Amr Goneid, AUC
27
P and NP – Times
 P (Polynomial) Times:
O(1), O(log n), O(log n)2, O(n), O(n log n),
O(n2), O(n3), ….
 NP (Non-Polynomial) Times:
O(2n) , O(en) , O(n!) , O(nn) , …..
Prof. Amr Goneid, AUC
28
P and NP – Times
 Polynomial time is
GOOD
Try to reduce the polynomial
power
 NP (e.g. Exponential) Time is
BAD
Prof. Amr Goneid, AUC
29