CS61B Lecture #15
Announcements:
• Please use bug-submit for code problems.
• Watch the newsgroup and class web site for updates, hints, useful
new utilities, etc.
Readings for Today:
Data Structures (Into Java), Chapter 1;
Readings for next Topics: Data Structures, Chapter 2–4
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
1
What Are the Questions?
• Cost is a principal concern throughout engineering:
“An engineer is someone who can do for a dime what any fool
can do for a dollar.”
• Cost can mean
– Operational cost (for programs, time to run, space requirements).
– Development costs: How much engineering time? When delivered?
– Costs of failure: How robust? How safe?
• Is this program fast enough? Depends on:
– For what purpose;
– What input data.
• How much space (memory, disk space)?
– Again depends on what input data.
• How will it scale, as input gets big?
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
2
Enlightening Example
Problem: Scan a text corpus (say 107 bytes or so), and find and print
the 20 most frequently used words, together with counts of how often
they occur.
• Solution 1 (Knuth): Heavy-Duty data structures
– Hash Trie implementation, randomized placement, pointers galore, several pages long.
• Solution 2 (Doug McIlroy): UNIX shell script:
tr -c -s ’[:alpha:]’ ’[\n*]’ < FILE | \
sort | \
uniq -c | \
sort -n -r -k 1,1 | \
sed 20q
• Which is better?
– #1 is much faster,
– but #2 took 5 minutes to write and processes 20MB in 1 minute.
– I pick #2.
• In most cases, anything will do: Keep It Simple.
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
3
Cost Measures (Time)
• Wall-clock or execution time
– You can do this at home:
time java FindPrimes 1000
– Advantages: easy to measure, meaning is obvious.
– Appropriate where time is critical (real-time systems, e.g.).
– Disadvantages: applies only to specific data set, compiler, machine, etc.
• Number of times certain statements are executed:
– Advantages: more general (not sensitive to speed of machine).
– Disadvantages: doesn’t tell you actual time, still applies only to
specific data sets.
• Symbolic execution times:
– That is, formulas for execution times or statement counts in
terms of input size.
– Advantages: applies to all inputs, makes scaling clear.
– Disadvantage: practical formula must be approximate, may tell
very little about actual time.
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
4
Asymptotic Cost
• Symbolic execution time lets us see shape of the cost function.
• Since we are approximating anyway, pointless to be precise about
certain things:
– Behavior on small inputs:
∗ Can always pre-calculate results some results.
∗ Times for small inputs not usually important.
– Constant factors (as in “off by factor of 2”):
∗ Just changing machines causes constant-factor change.
• How to abstract away from (i.e., ignore) these things?
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
5
Handy Tool: Order Notation
• Idea: Don’t try to produce specific functions that specify size, but
rather families of similar functions.
• Say something like “f is bounded by g if it is in g ’s family.”
• For any function g(x), the functions 2g(x), 1000g(x), or for any K >
0, K · g(x), all have the same “shape”. So put all of them into g ’s
family.
• Any function h(x) such that h(x) = K · g(x) for x > M (for some
constant M ) has g ’s shape “except for small values.” So put all of
these in g ’s family.
• If we want upper limits, throw in all functions that are everywhere
≤ some other member of g ’s family. Call this family O(g) or O(g(n)).
• Or, if we want lower limits, throw in all functions that are everywhere ≥ some other member of g ’s family. Call this family Ω(g).
• Finally, define Θ(g) = O(g) ∩ Ω(g)—the set of functions bracketed
by members of g ’s family.
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
6
Big Oh
• Goal: Specify bounding from above.
M =1
2g(x)
f (x)
g(x)
• Here, f (x) ≤ 2g(x) as long as x > 1,
• So f (x) is in g ’s upper-bound family, written
f (x) ∈ O(g(x)),
• . . . even though f (x) > g(x) everywhere.
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
7
Big Omega
• Goal: Specify bounding from below:
M =1
g(x)
f ′(x)
0.5g(x)
• Here, f ′(x) ≥ 12 g(x) as long as x > 1,
• So f ′(x) is in g ’s lower-bound family, written
f ′(x) ∈ Ω(g(x)),
• . . . even though f (x) < g(x) everywhere.
• In fact, we also have f ′(x) ∈ O(g(x)) and f (x) ∈ Ω(g(x)) and so we
can also write
Last modified: Mon Oct 6 16:08:02 2008
f (x), f ′(x) ∈ Θ(g(x)).
CS61B: Lecture #15
8
Using the Notation
• Can use this order notation for any kind of real-valued function.
• We will use them to describe cost functions. Example:
/** Find position of X in list L. Return -1 if not found */
int find (List L, Object X) {
int c;
for (c = 0; L != null; L = L.next, c += 1)
if (X.equals (L.head)) return c;
return -1;
}
• Choose representative operation: number of .equals tests.
• If N is length of L, then loop does at most N tests: worst-case
time is N tests.
• In fact, total # of instructions executed is roughly proportional
to N in the worst case, so can also say worst-case time is O(N ),
regardless of units used to measure.
• Use N > M provision (in defn. of O(·)) to handle empty list.
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
9
Why It Matters
• Computer scientists often talk as if constant factors didn’t matter
at all, only the difference of Θ(N ) vs. Θ(N 2).
• In reality they do, but we still have a point: at some point, constants
get swamped.
n
2
4
8
16
32
64
128
√
16 lg n
n
16
1.4
32
2
48
2.8
64
4
80
5.7
96
8
112
11
...
...
1, 024
160
32
220
320
...
...
...
...
n
2
4
8
16
32
64
128
n lg n
2
8
24
64
160
384
896
1, 024
10, 240
...
...
...
...
n3
8
64
512
4, 096
32, 768
262, 144
2.1 × 109
2n
4
16
256
65, 636
4.2 × 109
1.8 × 1019
3.4 × 1038
1.0 × 106 1.1 × 109
1.8 × 10308
n2
4
16
64
256
1024
4, 096
16, 384
...
...
...
...
...
...
1024 1.0 × 106 2.1 × 107 1.1 × 1012 1.2 × 1018 6.7 × 10315,652
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
10
Some Intuition on Meaning of Growth
• How big a problem can you solve in a given time?
• In the following table, left column shows time in microseconds to
solve a given problem as a function of problem size N .
• Entries show the size of problem that can be solved in a second,
hour, month (31 days), and century, for various relationships between time required and problem size.
• N = problem size
Time (µsec) for
problem size N
lg N
N
N lg N
N2
N3
2N
Last modified: Mon Oct 6 16:08:02 2008
1 second
300000
10
106
63000
1000
100
20
Max N Possible in
1 hour
1 month
1000000000
10
3.6 · 109
1.3 · 108
60000
1500
32
8·10
11
10
2.7 · 1012
7.4 · 1010
1.6 · 106
14000
41
1 century
9·10
14
10
3.2 · 1015
6.9 · 1013
5.6 · 107
150000
51
CS61B: Lecture #15
11
Careful!
• It’s also true that the worst-case time is O(N 2), since N ∈ O(N 2)
also: Big-Oh bounds are loose.
• The worst-case time is Ω(N ), since N ∈ Ω(N ), but that does not
mean that the loop always takes time N , or even K · N for some K .
• Instead, we are just saying something about the function that maps
N into the largest possible time required to process an array of
length N .
• To say as much as possible about our worst-case time, we should try
to give a Θ bound: in this case, we can: Θ(N ).
• But again, that still tells us nothing about best-case time, which
happens when we find X at the beginning of the loop. Best-case time
is Θ(1).
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
12
Effect of Nested Loops
• Nested loops often lead to polynomial bounds:
for (int i = 0; i < A.length; i += 1)
for (int j = 0; j < A.length; j += 1)
if (i != j && A[i] == A[j])
return true;
return false;
• Clearly, time is O(N 2), where N = A.length. Worst-case time is
Θ(N 2).
• Loop is inefficient though:
for (int i = 0; i < A.length; i += 1)
for (int j = i+1; j < A.length; j += 1)
if (A[i] == A[j]) return true;
return false;
• Now worst-case time is proportional to
N − 1 + N − 2 + . . . + 1 = N (N − 1)/2 ∈ Θ(N 2)
(so asymptotic time unchanged by the constant factor).
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
13
Recursion and Recurrences: Fast Growth
• Silly example of recursion:
/** True iff X is a substring of S */
boolean occurs (String S, String X) {
if (S.equals (X)) return true;
if (S.length () <= X.length ()) return false;
return
occurs (S.substring (1), X) ||
occurs (S.substring (0, S.length ()-1), X);
}
• In the worst case, both recursive calls happen.
• Consider a fixed size for X, say N0.
• Define C(N ) to be the worst-case cost of occurs(S,X) for S of
length N , measured in # of calls to occurs. Then




C(N ) = 
1,
if N ≤ N0,
2C(N − 1) if N > N0
• So C(N ) grows exponentially:
C(N ) = 2C(N −1) = 2·2C(N −2) = . . . = |2 · 2{z· · · 2} ·1 = 2N −N0 ∈ Θ(2N )
N −N0
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
14
Binary Search: Slow Growth
/** True X iff is an element of S[L .. U]. Assumes
* S in ascending order, 0 <= L <= U-1 < S.length. */
boolean isIn (String X, String[] S, int L, int U) {
if (L > U) return false;
int M = (L+U)/2;
int direct = X.compareTo (S[M]);
if (direct < 0) return isIn (X, S, L, M-1);
else if (direct > 0) return isIn (X, S, M+1, U);
else return true;
}
• Here, worst-case time, C(D), (as measured by # of string comparisons), depends on size D = U − L + 1.
• We eliminate S[M] from consideration each time and look at half the
rest. Assume D = 2k − 1 for simplicity, so:




0,
if D ≤ 0,


 1 + C((D − 1)/2), if D > 0.
= |1 + 1 +{z. . . + 1} +0
C(D) =
k
= k = ⌈lg D⌉ ∈ Θ(lg D)
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
15
Another Typical Pattern: Merge Sort
List sort (List L) {
if (L.length () < 2) return L;
Split L into L0 and L1 of about equal size;
L0 = sort (L0); L1 = sort (L1);
return Merge of L0 and L1
}































Merge (“combine into a single, ordered list”) takes
time proportional to size of
its result.
• Assuming that size of L is N = 2k , worst-case cost function, C(N ),
counting just merge time (∝ # items merged):
C(N ) =
=
=
=
=




1,
if N < 2;


 2C(N/2) + N, if N ≥ 2.
2(2C(N/4) + N/2) + N
4C(N/4) + N + N
8C(N/8) + N + N + N
N ·1+N
+ N +{z . . . + N}
|
k=lg N
= N + N lg N ∈ Θ(N lg N )
• In general, Θ(N lg N ) for arbitrary N (not just 2k ).
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
16
Amortization: Expanding Vectors
• When using array for expanding sequence, best to double size of
array to grow it. Here’s why.
• If array is size s, doubling its size and moving s elements to the new
array takes time ∝ 2s.
• Cost of inserting N items into array, doubling size as needed, starting with array size 1:
To Insert Resizing
Cost
Item #
1
0
2
2
3 to 4
4
8
5 to 8
...
...
2m+1
2m + 1 to 2m+1
Cumulative Resizing Cost
Array Size
Cost
per Item
After Insertions
0
0
1
2
1
2
6
1.5
4
14
1.75
8
...
...
...
2m+2 − 2
≈2
2m+1
• If we spread out (amortize ) the cost of resizing, we average about
2 time units on each item: “amortized insertion time is 2 units.”
• So even though worst-case time for adding one element to array of
N elements is 2N , time to add N elements is Θ(N ), not Θ(N 2).
Last modified: Mon Oct 6 16:08:02 2008
CS61B: Lecture #15
17