Algorithm Design and Analysis
LECTURE 7
Greedy Graph Algorithms
• Topological sort
• Shortest paths
Adam Smith
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
The (Algorithm) Design Process
1. Work out the answer for some examples
2. Look for a general principle
– Does it work on *all* your examples?
3. Write pseudocode
4. Test your algorithm by hand or computer
– Does it work on *all* your examples?
5. Prove your algorithm is always correct
6. Check running time
Be prepared to go back to step 1!
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Writing algorithms
• Clear and unambiguous
– Test: You should be able to hand it to any student in
the class, and have them convert it into working code.
• Homework pitfalls:
– remember to specify data structures
– writing recursive algorithms: don’t confuse the
recursive subroutine with the first call
– label global variables clearly
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Writing proofs
• Purpose
– Determine for yourself that algorithm is correct
– Convince reader
• Who is your audience?
– Yourself
– Your classmate
– Not the TA/grader
• Main goal: Find your own mistakes
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Exploring a graph
Classic problem: Given vertices s,t ∈ V, is there a
path from s to t?
Idea: explore all vertices reachable from s
How to convert these
Two basic techniques:
descriptions to precise
algorithms?
• Breadth-first search (BFS)
• Explore children in order of distance to start node
• Depth-first search (DFS)
• Recursively explore vertex’s children before exploring
siblings
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Adjacency-matrix representation
The adjacency matrix of a graph G = (V, E), where V = {1, 2, …,
n}, is the matrix A[1 . . n, 1 . . n] given by
1 if (i, j) ∈ E,
0 if (i, j)  E.
A[i, j] =
2
3
1
4
A
1
2
3
4
1
0
1
1
0
2
0
0
1
0
3
0
0
0
0
4
0
0
1
0
Storage: ϴ(V 2)
Good for dense graphs.
• Lookup: O(1) time
• List all neighbors: O(|V|)
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Adjacency list representation
• An adjacency list of a vertex v ∈ V is the list
Adj[v] of vertices adjacent to v.
2
3
1
4
Adj[1] = {2, 3}
Adj[2] = {3}
Adj[3] = {}
Adj[4] = {3}
Typical notation:
n = |V| = # vertices
m = |E| = # edges
For undirected graphs, | Adj[v] | = degree(v).
For digraphs, | Adj[v] | = out-degree(v).
How many entries in lists? 2|E| (“handshaking lemma”)
Total ϴ(V + E) storage — good for sparse graphs.
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
DFS pseudocode
• Maintain a global counter time
• Maintain for each vertex v
– Two timestamps:
• v.d = time first discovered
• v.f = time when finished
– “color”: v.color
• white = unexplored
• gray = in process
• black = finished
– Parent v.π in DFS tree
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
DFS pseudocode
• Note: recursive
function different
from first call…
• Exercise: change
code to set “parent”
pointer?
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
DFS example
•
•
•
•
9/10/10
T = tree edge (drawn in red)
F = forward edge (to a descendant in DFS forest)
B= back edge (to an ancestor in DFS forest)
C = cross edge (goes to a vertex that is neither ancestor nor descendant)
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
DFS with adjacency lists
• Outer code runs
once, takes time
O(n) (not counting
time to execute
recursive calls)
• Recursive calls:
– Run once per vertex
– time = O(degree(v))
• Total: O(m+n)
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Toplogical Sort
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Directed Acyclic Graphs
Def. An DAG is a directed graph that contains no directed cycles.
Ex. Precedence constraints: edge (vi, vj) means vi must precede vj.
Def. A topological order of a directed graph G = (V, E) is an ordering
of its nodes as v1, v2, …, vn so that for every edge (vi, vj) we have i < j.
v2
v6
v3
v5
v7
v4
v1
v2
v3
v4
v5
v1
a DAG
a topological ordering
v6
v7
Precedence Constraints
Precedence constraints. Edge (vi, vj) means task vi must occur before vj.
Applications.
Course prerequisite graph: course vi must be taken before vj.
Compilation: module vi must be compiled before vj. Pipeline of
computing jobs: output of job vi needed to determine input of job vj.
Getting dressed



shirt
mittens
socks
boots
underwear
jacket
pants
Recall from book
• Every DAG has a topological order
• If G graph has a topological order, then G is a
DAG.
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Review
• Suppose your run DFS on a DAG G=(V,E)
• True or false?
– Sorting by discovery time gives a topological order
– Sorting by finish time gives a topological order
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Shortest Paths
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Shortest Path Problem
• Input:
– Directed graph G = (V, E).
– Source node s, destination node t.
– for each edge e, length (e) = length of e.
– length path = sum of edge lengths
• Find: shortest directed path from s to t.
23
2
9
s
18
14
30
15
11
5
5
16
20
7
6
2
6
9/10/10
3
44
4
19
6
Length of path (s,2,3,5,t)
is 9 + 23 + 2 + 16 = 50.
t
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Rough algorithm (Dijkstra)
• Maintain a set of explored nodes S whose shortest path
distance d(u) from s to u is known.
• Initialize S = { s }, d(s) = 0.
• Repeatedly choose unexplored node v which minimizes
 (v) 
min
e  (u , v ) : uS
d (u)  (e) ,
• add v to S, and set d(v) = (v).
shortest path to some u in explored
part, followed by a single edge (u, v)
d(u)
(e)
v
u
S
s
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Rough algorithm (Dijkstra)
• Maintain a set of explored nodes S whose shortest path
distance d(u) from s to u is known.
Invariant: d(u) is known
• Initialize S = { s }, d(s) = 0.
for all vertices in S
• Repeatedly choose unexplored node v which minimizes
 (v) 
min
e  (u , v ) : uS
d (u)  (e) ,
• add v to S, and set d(v) = (v).
BFS with
weighted edges
shortest path to some u in explored
part, followed by a single edge (u, v)
d(u)
(e)
v
u
S
s
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
Graph with
nonnegative
edge lengths:
10
A
1 4
3
9/10/10
B
C
2
8
2
D
7 9
E
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm

B
Initialize:
10
0 A
Q: A B C D E
0



1 4
3

C

2
8
2

D
7 9
E

S: {}
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
“A”  EXTRACT-MIN(Q):
10
0 A
Q: A B C D E
0





B
1 4
3
C

2
8
2

D
7 9
E

S: { A }
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
Explore edges leaving A:
10
0 A
Q: A B C D E
0

10

3




10
B
1 4
3
C
3
2
8
2

D
7 9
E

S: { A }
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
“C”  EXTRACT-MIN(Q):
10
0 A
Q: A B C D E
0

10

3




10
B
1 4
3
C
3
2
8
2

D
7 9
E

S: { A, C }
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
Explore edges leaving C:
10
0 A
Q: A B C D E
0
9/10/10

10
7

3


11


5
7
B
1 4
3
C
3
2
8
2
11
D
7 9
E
5
S: { A, C }
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
“E”  EXTRACT-MIN(Q):
10
0 A
Q: A B C D E
0
9/10/10

10
7

3


11


5
7
B
1 4
3
C
2
8
2
3
11
D
7 9
E
5
S: { A, C, E }
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
Explore edges leaving E:
10
0 A
Q: A B C D E
0
9/10/10

10
7
7

3


11
11


5
7
B
1 4
3
C
2
8
2
3
11
D
7 9
E
5
S: { A, C, E }
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
“B”  EXTRACT-MIN(Q):
10
0 A
Q: A B C D E
0
9/10/10

10
7
7

3


11
11


5
7
B
1 4
3
C
2
8
11
D
7 9
2
3
E
5
S: { A, C, E, B }
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
Explore edges leaving B:
10
0 A
Q: A B C D E
0
9/10/10

10
7
7

3


11
11
9


5
7
B
1 4
3
C
9
D
2
8
7 9
2
3
E
5
S: { A, C, E, B }
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Demo of Dijkstra’s Algorithm
“D”  EXTRACT-MIN(Q):
10
0 A
Q: A B C D E
0
9/10/10

10
7
7

3


11
11
9


5
7
B
1 4
3
C
3
2
8
2
9
D
7 9
E
5
S: { A, C, E, B, D }
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Proof of Correctness
(Greedy Stays Ahead)
Invariant. For each node u  S, d(u) is the length of
the shortest path from s to u.
x
P'
Proof: (by induction on |S|)
s
• Base case: |S| = 1 is trivial.
u
S
• Inductive hypothesis: Assume for |S| = k  1.
P
y
v
– Let v be next node added to S, and let (u,v) be the chosen edge.
– The shortest s-u path plus (u,v) is an s-v path of length (v).
– Consider any s-v path P. We'll see that it's no shorter than (v).
– Let (x,y) be the first edge in P that leaves S,
and let P' be the subpath to x.
– P + (x,y) has length · d(x)+ (x,y)· (y)· (v)
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Review Question
• Is Dijsktra’s algorithm correct with
negative edge weights?
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Implementation
•For unexplored nodes, maintain  (v) min d (u)(e).
e(u ,v ):uS
– Next node to explore = node with minimum (v).
– When exploring v, for each edge e = (v,w), update
 ( w)min {  ( w),  (v)  (e)}.
Priority Queue
•Efficient implementation: Maintain a priority
queue of unexplored nodes, prioritized by (v).
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Priority queues
• Maintain a set of items with priorities (= “keys”)
– Example: jobs to be performed
• Operations:
–
–
–
–
Insert
Increase key
Decrease key
Extract-min: find and remove item with least key
• Common data structure: heap
– Time: O(log n) per operation
9/10/10
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Pseudocode for Dijkstra(G, )
d[s]  0
for each v  V – {s}
do d[v]  [v]  
S
QV
⊳ Q is a priority queue maintaining V – S,
keyed on [v]
while Q  
do u  EXTRACT-MIN(Q)
S  S  {u}; d[u]  [u]
for each v  Adjacency-list[u]
explore
do if [v] > [u] + (u, v)
edges
then [v]  d[u] + (u, v) leaving v
9/10/10
Implicit DECREASE-KEY
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne
Analysis of Dijkstra
n
times
while Q  
do u  EXTRACT-MIN(Q)
S  S  {u}
for each v  Adj[u]
degree(u)
do if d[v] > d[u] + w(u, v)
times
then d[v]  d[u] + w(u, v)
Handshaking Lemma  ·m implicit DECREASE-KEY’s.
9/10/10
PQ Operation
Dijkstra
Array
Binary heap
d-way Heap
Fib heap †
ExtractMin
n
n
log n
HW3
log n
DecreaseKey
Total
m
1
n2
log n
m log n
HW3
m log m/n n
1
m + n log n
† Individual ops are amortized bounds
A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne