Greedy Algorithms
CLRS CH. 16, 23, & 24
UNC Chapel Hill
Z. Guo
Overview
• Greedy algorithms <- optimization problems.
• Problems must have optimal substructure :
– an optimal solution must be made of subproblems
that are solved optimally.
• Problems must have greedy-choice property:
– A globally optimal solution can be arrived at by
making a locally optimal (greedy) choice.
• When we have a choice to make,
we can make the one that looks best right now.
• Demonstrate that a locally optimal choice
can be part of a globally optimal solution.
UNC Chapel Hill
Activity-selection Problem
• Input: Set S of n activities, a1, a2, …, an.
– si = start time of activity i.
– fi = finish time of activity i.
• Output: Subset A  S with a maximum number
of compatible activities.
– Two activities are compatible if their intervals don’t
overlap.
Example:
Activities in each line are compatible.
UNC Chapel Hill
Optimal Substructure
• Let’s assume that activities are sorted by finishing
times.
f1  f2  …  fn.
• Suppose an optimal solution includes activity ak.
– This generates two subproblems.
– Selecting from a1, …, ak-1, activities compatible with one
another, and that finish before ak starts (compatible with ak).
– Selecting from ak+1, …, an, activities compatible with one
another, and that start after ak finishes.
– The solutions to the two subproblems must be optimal.
• Prove using the cut-and-paste approach.
• Thus, we could use dynamic programming…
UNC Chapel Hill
Recursive Solution
• Let Sij = subset of activities in S that start after ai
finishes and finish before aj starts.
• Subproblems: Selecting maximum number of
mutually compatible activities from Sij.
• Let c[i, j] = size of maximum-size subset of
mutually compatible activities in Sij.
0
if Sij  


Recursive c[i, j ] 
max{ c[i, k ]  c[k , j ]  1} if Sij  
Solution:

ik  j
UNC Chapel Hill
Greedy-choice Property
• The problem also exhibits the greedy-choice property.
– There is an optimal solution to the subproblem Sij, that
includes the activity with the earliest finish time in set Sij.
– Proof: if any solution does not do the task with earliest
finishing time, then replace its first task with that.
• Hence, there is an optimal solution to S doing a1.
• So, make this greedy choice without first evaluating the
cost of subproblems.
• Then solve only the subproblem that results from making
this greedy choice.
• Combine the greedy choice and the subproblem solution.
UNC Chapel Hill
Recursive Algorithm
Recursive-Activity-Selector (s, f, i, j)
1. m  i+1
2. while m < j and sm < fi
3.
do m  m+1
4. if m < j
5.
then return {am} 
Recursive-Activity-Selector(s, f, m, j)
6.
else return 
Initial Call: Recursive-Activity-Selector (s, f, 0, n+1)
Complexity: (n)
This is just to make the algorithm look complicated;
The iterative algorithm is even easier. (See CLRS 16)
UNC Chapel Hill
Typical Steps
• Cast the optimization problem as one in which
we make a choice & are left with one
subproblem to solve.
• Prove that there exists an optimal solution that
makes the greedy choice, so the greedy choice is
always safe.
• Show that greedy choice and optimal solution to
subproblem  optimal solution to the problem.
• Make the greedy choice and solve top-down.
• May have to preprocess input to put it into
greedy order.
– Example: Sorting activities by finish time.
UNC Chapel Hill
Minimum Spanning Trees
CLRS CH. 23
Minimum Spanning Trees
• Given: Connected, undirected, weighted graph, G
• Find: Minimum - weight spanning tree, T
• Example:
b
5
a
7
c
1
3
Acyclic subset of edges(E) that connects
all vertices of G.
-3
11
d
0
e
2
f
b
5
a
1
3
d
UNC Chapel Hill
0
c
-3
e
f
Generic Algorithm
“Grows” a set A.
Invariant: A is subset of some MST.
Edge is “safe” adding it to A maintains the invariant.
A := ;
while A not complete tree do
find a safe edge (u, v);
A := A  {(u, v)}
od
UNC Chapel Hill
Definitions
no edge in the set crosses the cut
cut respects the edge set {(a, b), (b, c)}
a
5
b
d
0
e
2
a light edge crossing cut
(could be more than one)
c
1
3
11
7
-3
f
cut partitions vertices into
disjoint sets, S and V – S.
this edge crosses the cut
one endpoint is in S and the other is in V – S.
UNC Chapel Hill
Theorem 23.1
Theorem 23.1: Let (S, V-S) be any cut that respects A, and let (u, v)
be a light edge crossing (S, V-S). Then, (u, v) is safe for A.
Proof:
Let T be a MST that includes A.
Case: (u, v) in T. We’re done.
Case: (u, v) not in T. We have the following:
edge in A
(x, y) crosses cut.
Let T´ = T - {(x, y)}  {(u, v)}.
x
y
u
shows edges
in T
UNC Chapel Hill
v
cut
Because (u, v) is light for cut,
w(u, v)  w(x, y). Thus,
w(T´) = w(T) - w(x, y) + w(u, v)  w(T).
Hence, T´ is also a MST.
So, (u, v) is safe for A.
Corollary
In general, A will consist of several connected components.
Corollary: If (u, v) is a light edge connecting one CC in (V, A)
to another CC in (V, A), then (u, v) is safe for A.
UNC Chapel Hill
Kruskal’s Algorithm
• Starts with each vertex in its own component.
• Repeatedly merges two components into one by choosing a
light edge that connects them (i.e., a light edge crossing the
cut between them).
• Scans the set of edges in monotonically increasing order by
weight.
• Uses a disjoint-set data structure to determine whether an
edge connects vertices in different components.
UNC Chapel Hill
Prim’s Algorithm
• Builds one tree, so A is always a tree.
• Starts from an arbitrary “root” r .
• At each step, adds a light edge crossing cut (VA, V - VA) to A.
– VA = vertices that A is incident on.
UNC Chapel Hill
Prim’s Algorithm
• Builds one tree, so A is always a tree.
• Starts from an arbitrary “root” r .
• At each step, adds a light edge crossing cut (VA, V - VA) to A.
– VA = vertices that A is incident on.
•
•
•
•
Uses a priority queue Q to find a light edge quickly.
Each object in Q is a vertex in V - VA.
Key of v is minimum weight of any edge (u, v), where u  VA.
Then the vertex returned by Extract-Min is v such that there
exists u  VA and (u, v) is light edge crossing (VA, V - VA).
• Key of v is  if v is not adjacent to any vertex in VA.
UNC Chapel Hill
Prim’s Algorithm
Q := V[G];
Complexity:
for each u  Q do
Using binary heaps: O(E lg V).
key[u] := 
Initialization – O(V).
od;
Building initial queue – O(V).
key[r] := 0;
V Extract-Min’s – O(V lgV).
[r] := NIL;
E Decrease-Key’s – O(E lg V).
while Q   do
u := Extract - Min(Q);
Using Fibonacci heaps: O(E + V lg V).
for each v  Adj[u] do
if v  Q  w(u, v) < key[v] then (see book)
[v] := u;
key[v] := w(u, v)  decrease-key operation
fi
od
od
Note: A = {(v, [v]) : v  v - {r} - Q}.
UNC Chapel Hill
Example of Prim’s Algorithm
Not in tree
a/0
5
3
11
d/
UNC Chapel Hill
b/
0
e/
7
1
2
c/
-3
f/
Q=a b c d e f
0 

Example of Prim’s Algorithm
a/0
5
3
11
d/11
UNC Chapel Hill
b/5
0
e/
7
1
2
c/
-3
f/
Q=b d c e f
5 11  
Example of Prim’s Algorithm
a/0
5
3
11
d/11
UNC Chapel Hill
b/5
0
e/3
7
1
2
c/7
-3
f/
Q=e c d f
3 7 11 
Example of Prim’s Algorithm
a/0
5
3
11
d/0
UNC Chapel Hill
b/5
0
e/3
7
1
2
c/1
-3
f/2
Q=d c f
0 1 2
Example of Prim’s Algorithm
a/0
5
3
11
d/0
UNC Chapel Hill
b/5
0
e/3
7
1
2
c/1
-3
f/2
Q=c f
1 2
Example of Prim’s Algorithm
a/0
5
3
11
d/0
UNC Chapel Hill
b/5
0
e/3
7
1
2
c/1
-3
f/-3
Q=f
-3
Example of Prim’s Algorithm
a/0
5
3
11
d/0
UNC Chapel Hill
b/5
0
e/3
7
1
2
c/1
-3
f/-3
Q=
Example of Prim’s Algorithm
a/0
5
b/5
3
d/0
UNC Chapel Hill
0
e/3
c/1
1
-3
f/-3
Chapter 24: Single-Source Shortest Paths
• Given: A single source vertex in a weighted,
directed graph.
• Want to compute a shortest path for each
possible destination.
– Similar to BFS.
• We will assume either
– no negative-weight edges, or
– no reachable negative-weight cycles.
• Algorithm will compute a shortest-path tree.
– Similar to BFS tree.
UNC Chapel Hill
Outline
• General Lemmas and Theorems.
– CLRS now does this last. We’ll still do it first.
• Bellman-Ford algorithm.
• DAG algorithm.
• Dijkstra’s algorithm.
• We will skip Section 24.4.
UNC Chapel Hill
General Results (Relaxation)
Lemma 24.1: Let p = ‹v1, v2, …, vk› be a SP from v1 to vk. Then,
pij = ‹vi, vi+1, …, vj› is a SP from vi to vj, where 1  i  j  k.
So, we have the optimal-substructure property.
Bellman-Ford algorithm uses dynamic programming.
Dijkstra’s algorithm uses the greedy approach.
Let δ(u, v) = weight of SP from u to v.
Corollary: Let p = SP from s to v, where p = s
δ(s, v) = δ(s, u) + w(u, v).
p'
u v. Then,


Lemma 24.10: Let s  V. For all edges (u,v)  E, we have
δ(s, v)  δ(s, u) + w(u,v).
UNC Chapel Hill
Relaxation
Algorithms keep track of d[v], [v]. Initialized as follows:
Initialize(G, s)
for each v  V[G] do
d[v] := ;
[v] := NIL
od;
d[s] := 0
These values are changed when an edge (u, v) is relaxed:
Relax(u, v, w)
if d[v] > d[u] + w(u, v) then
d[v] := d[u] + w(u, v);
[v] := u
fi
UNC Chapel Hill
Properties of Relaxation
Consider any algorithm in which d[v], and [v] are first initialized
by calling Initialize(G, s) [s is the source], and are only changed by
calling Relax. We have:
Lemma 24.11: ( v:: d[v]  (s, v)) is an invariant.
Implies d[v] doesn’t change once d[v] = (s, v).
Proof:
Initialize(G, s) establishes invariant. If call to Relax(u, v, w)
changes d[v], then it establishes:
d[v] = d[u] + w(u, v)
 (s, u) + w(u, v)
, invariant holds before call.
 (s, v)
, by Lemma 24.10.
Corollary 24.12: If there is no path from s to v, then
d[v] = δ(s, v) =  is an invariant.
UNC Chapel Hill
Bellman-Ford Algorithm
Can have negative-weight edges. Will “detect” reachable negative-weight
cycles.
Initialize(G, s);
for i := 1 to |V[G]| –1 do
for each (u, v) in E[G] do
Relax(u, v, w)
Time
od
Complexity
od;
is O(VE).
for each (u, v) in E[G] do
if d[v] > d[u] + w(u, v) then
return false
fi
od;
return true
UNC Chapel Hill
Example
u

v
5

–2
6
–3
8
z 0
–4
2
7

x
UNC Chapel Hill
7
9

y
Example
u
6
v
5

–2
6
–3
8
z 0
–4
2
7
7
x
UNC Chapel Hill
7
9

y
Example
u
6
v
5
4
–2
6
–3
8
z 0
–4
2
7
7
x
UNC Chapel Hill
7
9
2
y
Example
u
2
v
5
4
–2
6
–3
8
z 0
–4
2
7
7
x
UNC Chapel Hill
7
9
2
y
Example
u
2
v
5
4
–2
6
–3
8
z 0
–4
2
7
7
x
UNC Chapel Hill
7
9
-2
y
Another Look
Note: This is essentially dynamic programming.
Let d(i, j) = cost of the shortest path from s to i that is at most j hops.
d(i, j) =
0

min({d(k, j–1) + w(k, i): i  Adj(k)}
 {d(i, j–1)})
j 0
1
2
3
4
UNC Chapel Hill
i
z
1
0
0
0
0
0
u
2

6
6
2
2
v
3


4
4
4
x y
4 5
 
7 
7 2
7 2
7 –2
if i = s  j = 0
if i  s  j = 0
if j > 0
Dijkstra’s Algorithm
Assumes no negative-weight edges.
Maintains a set S of vertices whose SP from s has been determined.
Repeatedly selects u in V–S with minimum SP estimate (greedy choice).
Store V–S in priority queue Q.
Initialize(G, s);
S := ;
Q := V[G];
while Q   do
Relax(u, v, w)
u := Extract-Min(Q);
if d[v] > d[u] + w(u, v) then
S := S  {u};
d[v] := d[u] + w(u, v);
for each v  Adj[u] do
[v] := u
Relax(u, v, w)
fi
od
od
UNC Chapel Hill
Example
u
v
1


10
2
s
3
9
4
0
7
5

x
UNC Chapel Hill
6
2

y
Example
u
v
1
10

10
2
s
3
9
4
0
7
5
5
x
UNC Chapel Hill
6
2

y
Example
u
v
1
8
14
10
2
s
3
9
4
0
7
5
5
x
UNC Chapel Hill
6
2
7
y
Example
u
v
1
8
13
10
2
s
3
9
4
0
7
5
5
x
UNC Chapel Hill
6
2
7
y
Example
u
v
1
8
9
10
2
s
3
9
4
0
7
5
5
x
UNC Chapel Hill
6
2
7
y
Example
u
v
1
8
9
10
2
s
3
9
4
0
7
5
5
x
UNC Chapel Hill
6
2
7
y
Correctness
Theorem 24.6: Upon termination, d[u] = δ(s, u) for all u in V
(assuming non-negative weights).
Proof:
By Lemma 24.11, once d[u] = δ(s, u) holds, it continues to hold.
We prove: For each u in V, d[u] = (s, u) when u is inserted in S.
Suppose not. Let u be the first vertex such that d[u]  (s, u) when
inserted in S.
Note that d[s] = (s, s) = 0 when s is inserted, so u  s.
 S   just before u is inserted (in fact, s  S).
UNC Chapel Hill
Proof (Continued)
Note that there exists a path from s to u, for otherwise
d[u] = (s, u) =  by Corollary 24.12.
 there exists a SP from s to u. SP looks like this:
p2
u
s
p1
x
S
UNC Chapel Hill
y
Proof (Continued)
Claim: d[y] = (s, y) when u is inserted into S.
We had d[x] = (s, x) when x was inserted into S.
Edge (x, y) was relaxed at that time.
By Lemma 24.14, this implies the claim.
Now, we have: d[y] = (s, y) , by Claim.
 (s, u) , nonnegative edge weights.
 d[u]
, by Lemma 24.11.
Because u was added to S before y, d[u]  d[y].
Thus, d[y] = (s, y) = (s, u) = d[u].
Contradiction.
UNC Chapel Hill
Complexity
Running time is
O(V2) using linear array for priority queue.
O((V + E) lg V) using binary heap.
O(V lg V + E) using Fibonacci heap.
(See book.)
UNC Chapel Hill