Tutorial Sheet 3 Question (1 (a)) Given,
T1 (n) = O(f (n))
T2 (n) = O(g(n))
Prove that,
τ (n) = T1 (n) + T2 (n) = max(O(f (n)), O(g(n)))
Solution
We can assume, without loss of generality, that,
max(O(f (n)), O(g(n))) = O(f (n))
(Because one has to be larger).
Therefore, some positive n0 exists, such that,
g(n) ≤ f (n), when n ≥ n0
Now,
T1 (n) = O(f (n))
T2 (n) = O(g(n))
It follows from the definition of “Big Oh” notation, that there must be positive
constants c1 , c2 , n1 and n2 such that,
T1 (n) ≤ c1 f (n), when n ≥ n1
T2 (n) ≤ c2 g(n), when n ≥ n2
Let nτ = max(n0 , n1 , n2 ). This is valid because nτ will always be the highest n,
preserving the inequality. Therefore,
T1 (n) ≤ c1 f (n), when n ≥ nτ
T2 (n) ≤ c2 g(n), when n ≥ nτ
T1 (n) + T2 (n) ≤ c1 f (n) + c2 g(n), when n ≥ nτ
But g(n) ≤ f (n), when n ≥ nτ
So, again using the (highly useful) inequality, we can say that
T1 (n) + T2 (n) ≤ c1 f (n) + c2 f (n), when n ≥ nτ
T1 (n) + T2 (n) ≤ (c1 + c2 )f (n), when n ≥ nτ
Let cτ = c1 + c2 . This is valid because it doesn’t matter for the purposes of this
calculation what the value of c is, just that there is one.
T1 (n) + T2 (n) ≤ cτ f (n), when n ≥ nτ
T1 (n) + T2 (n) = O(f (n))
But, from our general assumption above, we can say that O(f (n)) = max(O(f (n)), O(g(n)))
so,
T1 (n) + T2 (n) = max(O(f (n)), O(g(n)))
τ (n) = max(O(f (n)), O(g(n)))
Question 1 (b)
Given,
T1 (n) = O(f (n))
T2 (n) = O(g(n))
Prove that,
τ (n) = T1 (n) ∗ T2 (n) = O (f (n) ∗ g (n))
Solution.
By definition, there must be positive constants c1 , c2 , n1 and n2 such that,
T1 (n) ≤ c1 f (n) , when n ≥ n1
T2 (n) ≤ c2 g (n) , when n ≥ n2
Let nτ = max(n1 , n2 ).
Therefore,
T1 (n) ≤ c1 f (n) , when n ≥ nτ
T2 (n) ≤ c2 g (n) , when n ≥ nτ
Therefore,
T1 (n) ∗ T2 (n) ≤ (c1 f (n)) ∗ (c2 g (n)) , when n ≥ nτ
T1 (n) ∗ T2 (n) ≤ c1 ∗ c2 (f (n) ∗ g (n)) , when n ≥ nτ
Let cτ = c1 ∗ c2 .
Therefore,
T1 (n) ∗ T2 (n) ≤ cτ (f (n) ∗ g (n)) , when n ≥ nτ
T1 (n) ∗ T2 (n) = O (f (n) ∗ g (n))
τ (n) = O(f (n) ∗ g(n))
Question 2.1
Order the following functions by growth rate:
√
N,
N , N 1.5 , N 2 , N logN, N log log N, N log2 N, N log(N 2 ),
2 N N
, 2 , 2 2 , 37, N 2 log N, N 3 .
N
and indicate which functions grow at the same rate.
Solution
√
N
2
, 37, N , N, N log log N, N log N, N log N 2 , N log2 N, N 1.5 , N 2 , N 2 log N, N 3 , 2 2 , 2N
N
For most the functions in the “ordered list” it is easy to see that it grows
faster than its predecessor, but for some it is not so easy to see. For example N
versus N log log N . It is clear that both log log N and N increase with increasing
N . Therefore N log log N must increase more quickly.
It is also important to know that exponential functions, e.g. always 2n grow
more quickly than polynomial functions e.g. n5 .
The two functions which grow at the same rate are N log N and N log N 2
because,
N log N 2 = 2N log N
= O(N log N )
And of course N log N is also O(N log N ).
Question 2.2 Suppose T1 (N ) = O (f (N )) and T2 (N ) = O (f (N )). Which
of the following statements are true in general?
–
–
–
–
(a) T1 (N ) + T2 (N ) = O (f (n))
(b) T1 (N ) − T2 (N ) = o (f (n))
(N )
(c) TT21 (N
) = O(1)
(d) T1 (N ) = O (T2 (N ))
Solution
(a) - True.
The truth of statement (a) follows directly from the rule we proved true in
question 1(a) of Tute03.
(b) - False.
Consider the counterexample,
T1 (N ) = 2N, T2 (N ) = N, and f (N ) = N
Therefore,
T1 (N ) − T2 (N ) = N
= Θ(N )
= Θ (f (N ))
Therefore,
T1 (N ) − T2 (N ) 6= o (f (n))
(c) - False.
Consider the counterexample,
T1 (N ) = N 2 , T2 (N ) = N, and f (N ) = N 2
N2
T1 (N )
=
T2 (N )
N
=N
Clearly N 6= O(1)
(d) - False.
Consider the counterexample,
T1 (N ) = N 2 , T2 (N ) = N, and f (N ) = N 2
Clearly N 2 6= O(N )
Question 2.4
Prove that for any constant, k, logk N = o(N ).
Solution
We can assume, without loss of generality, that the log has base b > 1.
Clearly,
logkb 1 N = o(logkb 2 N ), if k1 < k2
It follows that if logk N = o(N ) is true for all positive integers it is true for
all real numbers also(positive or negative). This can be proved by induction.
Base case:
It is clear statement is true for k = 1. That is,
logb N = o(N )
Inductive Hypothesis We assume statement is true for k < l, where l is
some integer.
The next step is to show loglb N = o(N ) is true, given the previous
assumption
The relative growth rates of log l N and N can be determined by computing
logl N
limN →∞ Nb . (Note: See book page 43)
*****************************
L’Hopitals rule states that if the limN →∞ f (N ) = ∞ and limN →∞ g(N ) =
∞ then,
f (N )
f 0 (N )
= lim 0
N →∞ g(N )
N →∞ g (N )
lim
*****************************
Now,
0
l
(logbl N )0 = (logb N )
l−1
= l ∗ (logb N )
Clearly N 0 = 1, so,
∗
logb e
N
l ∗ logbl−1 N ∗ logNb e
(logbl N )0
lim
= lim
N →∞
N →∞
N0
1
l ∗ logb e ∗ logbl−1 N
= lim
N →∞
N
logbl−1 N
= l ∗ logb e lim
N →∞
N
But it follows from the inductive hypothesis that, limN →∞
(log l N )0
limN →∞ Nb 0
logl N
limN →∞ Nb
logbl−1 N
N
= 0.
Therefore
= 0 and therefore
= 0. Therefore
l
logb N = o(N ). Therefore, it follows that theorem is true for all real numbers.
QED
Question 2.7 For each of the following six program fragments, give an analysis of the running time(Big-oh will do).
– (1)
sum = 0;
for(i = 0; i < n; i++)
sum++;
Answer - O(n) There are
during each iteration of the
total number of operations
constant c can be discarded
a constant number of operations, c, performed
for loop, and it iterates n times. Therefore the
performed is c ∗ n = O(n) (Observe how the
when using Big-oh notation).
– (2)
sum = 0;
for(i = 0; i < n; i++)
for(j = 0; j < n; j++)
sum++;
Answer - O(n2 ) There are a constant number of operations, c, performed
during each iteration of the inner for loop, and it iterates n2 times. Therefore
the total number of operations performed is c ∗ n2 = O(n2 )
– (3)
sum = 0;
for(i = 0; i < n; i++)
for(j = 0; j < n * n; j++)
sum++;
Answer - O(n3 ) There are a constant number of operations, c, performed
during each iteration of the inner for loop, and it iterates n3 times. Therefore
the total number of operations performed is c ∗ n3 = O(n3 )
– (4)
sum = 0;
for(i = 0; i < n; i++)
for(j = 0; j < i; j++)
sum++;
Answer - O(n2 )
There are a constant number of operations, c, performed during each iteration of the inner for loop. For each value of i, the inner loop iterates i
times. As i ranges from 1, 2, ..., n, the average number of iterations is approximately n2 . Therefore the total number of times the inner loop is executed is
approximately,
n∗
n
n2
=
2
2
2
Therefore the total number of operations is approximately c∗ n2 = 2c ∗n2 . As
usual the constant can be discarded when using Big-oh notation. Therefore
the total number of operations performed is O(n2 ).
– (5)
sum = 0;
for(i = 0; i < n; i++)
for(j = 0; j < i * i; j++)
for(k = 0; k < j; k++)
sum++;
Answer - O(n5 )
The outer loop is executed n times. For each iteration of the outer loop
2
the middle loop, is executed, on average, approximately n2 times. For each
iteration of the middle loops, the inner loop is executed, on average, ap2
proximately, n4 times. Therefore the inner loop is executed approximately,
2
5
2
n ∗ n2 ∗ n4 = n8 times. Every time the inner loop is executed a constant
number of operations, c, are performed. Therefore, the total number of operations performed is approximately, 8c ∗ n5 = O(n5 ).
– (6)
sum = 0;
for(i = 1; i < n; i++)
for(j = 1; j < i * i; j++)
if( j % i == 0)
for(k = 0; k < j; k++)
sum++;
Answer - O(n4 )
Clearly the if statement is reached at most n3 times. Now consider how often
it is true, that is how often does j % i == 0. For each value of i, j ranges
from 1 to i2 , and j % i == 0 only when j is a multiple of i. Therefore j % i
== 0 only when j = i, 2i, 3i, ..., i2 . Thus the if statement is true i times for
each value of i. Thus it is true O(n2 ) times. Thus the innermost loop is only
reached O(n2 ) times. Every time it is reached it iterates j = O(n2 ) times.
Therefore the total number of operations performed is O(n4 ).
Question 3 Given,
f (0) = 1, f (1) = 1
and
f (n) = f (n − 1) + f (n − 2)
Prove that f (n) ≥
be true?
Solution
3 n
2
for “large” n. What is the base value of n for it to
0
3
f (0) = 1 <
2
1
3
f (1) = 1 <
2
2
3
f (2) = f (0) + f (1) = 2 <
2
3
3
2
4
3
f (4) = f (2) + f (3) = 2 + 3 = 5 <
2
5
3
f (5) = f (3) + f (4) = 3 + 5 = 8 ≥
2
f (3) = f (1) + f (2) = 1 + 2 = 3 <
Therefore the base case is n = 5. It is also easy to verify that theorem is true
for n = 6.
Assume statement is true for n = 5, ..., k.
Now, using this assumption, prove it is true for n = k + 1. That is, prove,
k+1
3
f (k + 1) ≥
2
f (k + 1) = f (k) + f (k − 1)
k k−1
3
3
≥
+
2
2
k k −1
3
3
3
≥
+
2
2
2
k k 3
3
2
≥
+
2
2
3
k
5 3
≥
3 2
But 53 > 32 .
Therefore
k
3 3
f (k + 1) ≥
2 2
k+1
3
f (k + 1) ≥
2