Algorithms and
Complexity
Lecture 3
Complexity

Space Complexity
Instruction Space
 Data Space
 Enviroment Space


Time Complexity
Operation count
 Asymptotic Notation

Instruction Space
Operation a+b+b*c+(a+b-c)7(a+b)+4
Data Space
Borland C++
Enviromental Space
template <class T>
Rsum(a,n)
T Rsum(T a[], int n)
Rsum(a,n-1)
{// Return sum of numbers a[0:n - 1].
Rsum(a,n-2)
if (n > 0)
...
return Rsum(a, n-1) + a[n-1];
Rsum(a,1)
return 0;
Rsum(a,0)
}
Recursive Program
Space Complexity
Space Required
Fixed Part
Variable Part
The Space Requirement S(P) = c + SP(instance characteristics)
Time Complexity


Compilation Time
Run Time
t P (n)  ca ADD(n)  cs SUB (n)  cm MUL(n)  cd DIV (n)  cc COM (n) 
n : instance characteristics
Example
max element
template <class T>
int Max(T a[], int n)
{// Locate the largest element in a[0:n-1].
int pos = 0;
for (int i = 1; i < n; i++)
if (a[pos] < a[i])
pos = i;
return pos;
}
Total n-1 comparisons
Example Polynomial Evaluation
n
P ( x )   ci x n
i 1
template <class T>
T PolyEval(T coeff[], int n, const T& x)
{// Evaluate the degree n polynomial with
// coefficients coeff[0:n] at the point x.
T y = 1, value = coeff[0];
for (int i = 1; i <= n; i++) {
// add in next term
y *= x;
value += y * coeff[i];
}
return value;
}
Number of addition : n
Number of Multiplication : 2n
Example (2)
Polynomial Evaluation
Horner’s rule
n
P ( x )   ci x n
i 1
P( x)  ( (cn  x  cn 1 )  x  cn  2 )  x  cn 3 )  x )  x  c0
template <class T>
T Horner(T coeff[], int n, const T& x)
{// Evaluate the degree n polynomial with
// coefficients coeff[0:n] at the point x.
T value = coeff[n];
for (int i = 1; i <= n; i++)
value = value * x + coeff[n - i];
return value;
}
Number of addition : n
Number of Multiplication : n
Example Rank
The rank of an element in a sequence is the number of smaller
elements in the sequence plus the number of equal elements
that appear to its left
Ex: a[4 3 9 3 7]
: rank r[2 0 4 1 3]
template <class T>
void Rank(T a[], int n, int r[])
{// Rank the n elements a[0:n-1].
for (int i = 0; i < n; i++)
r[i] = 0; //initialize
// compare all element pairs
for (int i = 1; i < n; i++)
for (int j = 0; j < i; j++)
if (a[j] <= a[i]) r[i]++;
else r[j]++;
}
The total number of comparison : 1 + 2 + …+ n-1 = (n-1)*n/2
Example Rank Sort
template <class T>
void Rearrange(T a[], int n, int r[])
{// Rearrange the elements of a into sorted order
// using an additional array u.
T *u = new T [n+1];
// move to correct place in u
for (int i = 0; i < n; i++)
u[r[i]] = a[i];
// move back to a
for (int i = 0; i < n; i++)
a[i] = u[i];
delete [] u;
}
The number of elements moved : 2n
Example Selection Sort
template <class T>
void SelectionSort(T a[], int n)
{// Sort the n elements a[0:n-1].
for (int size = n; size > 1; size--) {
int j = Max(a, size);
Swap(a[j], a[size - 1]);
}
}
The number of comparison = 1 + 2 + 3 +… + n-1 = (n-1)*n/2
The number of elements moved : 3 (n-1)
Example Buble Sort
template <class T>
void Bubble(T a[], int n)
{// Bubble largest element in a[0:n-1] to right.
for (int i = 0; i < n - 1; i++)
if (a[i] > a[i+1]) Swap(a[i], a[i + 1]);
}
template <class T>
void BubbleSort(T a[], int n)
{// Sort a[0:n - 1] using bubble sort.
for (int i = n; i > 1; i--)
Bubble(a, i);
}
The number of comparison = 1 + 2 + 3 +… + n-1 = (n-1)*n/2
The number of elements moved : 3 (n-1) (worsed case)
What is worst case what is best case
Sequential search
Best case : 1
Worst case : n

1 n
On average :  i   n  1 / 2
n i 1
Insertion into Sorted Array
template <class T>
void Insert(T a[], int & n, const T& x)
{// Insert x into the sorted array a[0:n-1].
// Assume a is of size > n.
int i;
for (i = n-1; i >= 0 && x < a[i]; i--)
a[i+1] = a[i];
a[i+1] = x;
n++; // one element added to a
}
1  n 1

 n  i   n   n / 2  n /(n  1)
On average :


n  1  i 0

Assumed Knowledge
Read the section on
 Existential and Universal Quantifiers,
g b Loves(b,g)
g b Loves(b,g)

Logarithms and Exponentials
Classifying Functions
Giving an idea of how fast a function grows
without going into too much detail.
Which are more alike?
Which are more alike?
Mammals
Which are more alike?
Which are more alike?
Dogs
Classifying Animals
Vertebrates
Fish
Reptiles
Mammals
Birds
Giraffe
Dogs
Classifying Functions
T(n) 10
100
1,000 10,000
6
9
13
amoeba
10
31
100
bird
n 10
100
1,000 10,000
n log n 30
600
9,000 130,000 my father
log n
3
n1/2 3
10,000 106
n2 100
109
n3 1,000 106
2n 1,024 1030
10300
108
1012
103000
human
elephant
dinosaur
the universe
Note: The universe contains approximately 1050 particles.
Which are more alike?
n1000
n2
2n
Which are more alike?
n1000
n2
Polynomials
2n
Which are more alike?
1000n2
3n2
2n3
Which are more alike?
1000n2
Quadratic
3n2
2n3
Classifying Functions?
Functions
Classifying Functions
Functions
Exp
Poly Logarithmic
Logarithmic
Constant
2
n5
(log n)5
5 log n
5
Polynomial
Exponential
2
25n
Double Exp
25n
n5
Classifying Functions?
Polynomial
Classifying Functions
Polynomial
?
5n2
Cubic
Quadratic
Linear
5n
5n3
5n4
Logarithmic



log10n = # digits to write n
log2n = # bits to write n
= 3.32 log10n
log(n1000) = 1000 log(n)
Differ only by a
multiplicative
constant.
Poly Logarithmic
(log n)5 = log5 n
Logarithmic << Polynomial
log1000 n << n0.001
For sufficiently large n
Linear << Quadratic
10000 n << 0.0001 n2
For sufficiently large n
Polynomial << Exponential
n1000 << 20.001 n
For sufficiently large n
Ordering Functions
Functions
25n << 2
n5
<<
<<
Double Exp
<<
<<
Exp
5 << 5 log n << (log n)5 << n5
<<
Exponential
<<
Polynomial
<<
Poly Logarithmic
Logarithmic
Constant
<<
25n
2
Which Functions are “Constant”?
Yes
Yes
Yes
No
No
Yes
9
7
•5
• 1,000,000,000,000
• 0.0000000000001
• -5
• 0
Lie in between
• 8 + sin(n)
Which Functions are Quadratic?
• n2
•…?
Which Functions are Quadratic?
• n2
• 0.001 n2
• 1000 n2
Some constant times n2.
Which Functions are Quadratic?
• n2
• 0.001 n2
• 1000 n2
• 5n2 + 3n + 2log n
Which Functions are Quadratic?
• n2
• 0.001 n2
Lie
in
between
2
• 1000 n
• 5n2 + 3n + 2log n
Which Functions are Quadratic?
• n2
• 0.001 n2
• 1000 n2
• 5n2 + 3n + 2log n
Ignore low-order terms
Ignore multiplicative constants.
Ignore "small" values of n.
Write Θ(n2).
Which Functions are Polynomial?
• n5
•…?
Which Functions are Polynomial?
• nc
• n0.0001
• n10000
n to some constant power.
Which Functions are Polynomial?
• nc
• n0.0001
• n10000
• 5n2 + 8n + 2log n
• 5n2 log n
• 5n2.5
Which Functions are Polynomial?
• nc
• n0.0001
Lie
in
between
10000
•n
• 5n2 + 8n + 2log n
• 5n2 log n
• 5n2.5
Which Functions are
Polynomials?
• nc
• n0.0001
• n10000
• 5n2 + 8n + 2log n
• 5n2 log n
• 5n2.5
Ignore low-order terms
Ignore power constant.
Ignore "small" values of n.
Write nΘ(1)
Which Functions are
Exponential?
• 2n
•…?
Which Functions are
Exponential?
• 2n
• 20.0001 n
• 210000 n
n times some constant power
raises to the power of 2.
Which Functions are
Exponential?
• 2n
• 20.0001 n
• 210000 n
• 8n
• 2n / n100 too small?
•2n · n100 too big?
Which Functions are
Exponential?
• 2n
• 20.0001 n
• 210000 n
3n
n
=
2
•8
0.5n
n
100
>
2
•2 /n
2n
n
100
<
2
•2 · n
Lie in between
Which Functions are
Exponential?
• 2n
• 20.0001 n
• 210000 n
3n
n
=
2
•8
0.5n
n
100
>
2
•2 /n
2n
n
100
<
2
•2 · n
20.5n > n100
2n = 20.5n · 20.5n > n100 · 20.5n
2n / n100 > n0.5n
Which Functions are
Exponential?
• 2n
• 20.0001 n
• 210000 n
• 8n
• 2n / n100
•2n · n100
Ignore base.
Ignore constant in front of n.
Ignore "small" values of n.
Ignore polynomial factors.
Write 2Θ(n)
Classifying Functions
Functions
Double Exp
2Θ(n)
Exp
Θ(log n) (log n)Θ(1) nΘ(1)
Exponential
Polynomial
Poly Logarithmic
Logarithmic
Constant
Θ(1)
nΘ(1)
2Θ(n)
2
2
Definition of Theta
f(n) = θ(g(n))
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
Definition of Theta
f(n) = θ(g(n))
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
f(n) is sandwiched between c1g(n) and c2g(n)
Definition of Theta
f(n) = θ(g(n))
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
f(n) is sandwiched between c1g(n) and c2g(n)
for some sufficiently small c1 (= 0.0001)
for some sufficiently large c2 (= 1000)
Definition of Theta
f(n) = θ(g(n))
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
For all sufficiently large n
Definition of Theta
f(n) = θ(g(n))
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
For all sufficiently large n
For some definition of “sufficiently large”
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
? ?
c1·n2  3n2 + 7n + 8  c2·n2
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
3 4
3·n2  3n2 + 7n + 8  4·n2
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
1
3·12  3·12 + 7·1 + 8  4·12
False
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
7
3·72  3·72 + 7·7 + 8  4·72
False
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
8
3·82  3·82 + 7·8 + 8  4·82
True
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
9
3·92  3·92 + 7·9 + 8  4·92
True
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
10
3·102  3·102 + 7·10 + 8  4·102
True
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
n8
3·n2  3n2 + 7n + 8  4·n2
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
n8
3·n2  3n2 + 7n + 8  4·n2
True
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
n8
3·n2  3n2 + 7n + 8  4·n2
7n + 8  1·n2
True 7 + 8/n  1·n
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
?
3·n2  3n2 + 7n + 8  4·n2
Definition of Theta
3n2 + 7n + 8 = θ(n2)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
8
n  8 3·n2  3n2 + 7n + 8  4·n2
Definition of Theta
3n2 + 7n + 8 = θ(n2)
True
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
3 4
8
n  8 3·n2  3n2 + 7n + 8  4·n2
Definition of Theta (BigOh)
n2 = θ(n3)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
Definition of Theta (BigOh)
n2 = θ(n3)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
? ?
c1 · n 3  n 2  c2 · n 3
Definition of Theta
n2 = θ(n3)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
0
0 · n 3  n 2  c2 · n 3
True, but ?
Definition of Theta
n2 = θ(n3)
False
 c1  0, c 2  0,  n0,  n  n0,...
0
Constants c1 and c2 must be positive!
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
? ?
c1·n  3n2 + 7n + 8  c2·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
3 100
3·n  3n2 + 7n + 8  100·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
?
3·n  3n2 + 7n + 8  100·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
100
3·100  3·1002 + 7·100 + 8  100·n
False
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
3 10,000
3·n  3n2 + 7n + 8  10,000·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
10,000
3·10,000  3·10,000 2 + 7·10,000 + 8  100·10,000
False
Definition of Theta
3n2 + 7n + 8 = θ(n)
What is the
reverse statement?
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
Definition of Theta
3n2 + 7n + 8 = θ(n)
The reverse statement
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
3 100
?
3n2 + 7n + 8 > 100·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
3 100
100
3·1002 + 7·100 + 8 > 100·n
True
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
3 10,000
?
3n2 + 7n + 8 > 10,000·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
3 10,000 10,000
3·10,0002 + 7·10,000 + 8 > 10,000·n
True
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
c1 c2
?
3n2 + 7n + 8 > c2·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
c1 c2
c2
3·c22 + 7 · c2 + 8 > c2·c2
True
Definition of Theta
3n2 + 7n + 8 = θ(n)
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
c1 c2 n 0 ?
3n2 + 7n + 8 > c2·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
True
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n ) or f ( n )  c2 g( n )
c1 c2 n0 max(c2,n0 )
3·c22 + 7 · c2 + 8 > c2·c2
True
Definition of Theta (BigOh)
3n2 + 7n + 8 = nθ(1)
 c1, c2, n0,  n  n0 g(n)c1  f(n)  g(n)c2
Definition of Theta
3n2 + 7n + 8 = nθ(1)
 c1, c2, n0,  n  n0 g(n)c1  f(n)  g(n)c2
? ?
nc1  3n2 + 7n + 8  nc2
Definition of Theta
3n2 + 7n + 8 = nθ(1)
 c1, c2, n0,  n  n0 g(n)c1  f(n)  g(n)c2
2 3
n2  3n2 + 7n + 8  n3
Definition of Theta
3n2 + 7n + 8 = nθ(1)
 c1, c2, n0,  n  n0 g(n)c1  f(n)  g(n)c2
2 3
?
n?
n2  3n2 + 7n + 8  n3
Definition of Theta
3n2 + 7n + 8 = nθ(1)
 c1, c2, n0,  n  n0 g(n)c1  f(n)  g(n)c2
2 3
5
n5
n2  3n2 + 7n + 8  n3
Definition of Theta
3n2 + 7n + 8 = nθ(1)
True
 c1, c2, n0,  n  n0 g(n)c1  f(n)  g(n)c2
2 3
5
n5
n2  3n2 + 7n + 8  n3
True
Order of Quantifiers
f(n) = θ(g(n))
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
 n0,  n  n0,  c1, c 2, c1 g( n )  f ( n )  c2 g( n )
?
Understand Quantifiers!!!
 g ,  b, loves( b, g )
One girl
 b,  g, loves( b, g )
Could be a separate
girl for each boy.
Sam
Mary
Sam
Mary
Bob
Beth
Bob
Beth
John
Marilin
Monro
John
Marilin
Monro
Fred
Ann
Fred
Ann
Order of Quantifiers
f(n) = θ(g(n))
 c1, c 2,  n0,  n  n0, c1 g( n )  f ( n )  c2 g( n )
 n0,  n  n0,  c1, c 2, c1 g( n )  f ( n )  c2 g( n )
No! It cannot be a different c1 and c2
for each n.
Other Notations
Theta
f(n) = θ(g(n))
f(n) ≈ c g(n)
BigOh
f(n) = O(g(n))
f(n) ≤ c g(n)
Omega
f(n) = Ω(g(n))
f(n) ≥ c g(n)
Little Oh
f(n) = o(g(n))
f(n) << c g(n)
Little Omega
f(n) = ω(g(n))
f(n) >> c g(n)
Big Oh (O)

Let f(n) and g(n) be such that
f(n)=O(g(n)) iff
lim
n 
f ( n)
c
g ( n)
upper bound notation
lim
n 
f ( n)
g ( n)
exists.
for some finite constant c.
Omega Notation ()

f(n) = (g(n)) iff positive constant c and n0
exists such that f(n)cg(n) for all n n  n0.
Definition of lower bound
Little Oh (o)

f(n) = o(g(n)) iff f(n) = O(g(n)) and f(n)(g(n)).

Upper bound
BigOh Notation


n2 = O(n3)
n3 = O(n2)
BigOh Notation


O(n2) = O(n3)
O(n3) = O(n2)
Odd Notation
f(n) = O(g(n))
Standard
f(n) ≤ O(g(n))
Stresses one function dominating another.
f(n)O(g(n))
Stress function is member of class.
The Time Complexity
of an Algorithm
Specifies how the running time
depends on the size of the input.
Purpose?
Purpose





To estimate how long a program will run.
To estimate the largest input that can reasonably be
given to the program.
To compare the efficiency of different algorithms.
To help focus on the parts of code that are executed
the largest number of times.
To choose an algorithm for an application.
Time Complexity of Algorithm
The time complexity of an algorithm is
the largest time required on any input
of size n.
O(n2): Prove that for every input of size n, the
algorithm takes no more than cn2 time.
Ω(n2): Find one input of size n, for which the
algorithm takes at least this much time.
θ (n2): Do both.
Time Complexity of Problem
The time complexity of a problem is
the time complexity of the fastest algorithm
that solves the problem.
O(n2): Provide an algorithm that solves the
problem in no more than this time.
Ω(n2): Prove that no algorithm can solve it faster.
θ (n2): Do both.