Greedy algorithm
叶德仕
[email protected]
1
Greedy algorithm’s
paradigm
Algorithm is greedy if
it builds up a solution in small steps
it chooses a decision at each step myopically
to optimize some underlying criterion
Analyzing optimal greedy algorithms by
showing that:
in every step it is not worse than any other
algorithm, or
every algorithm can be gradually transformed
to thegreedy one without hurting its quality
2
Interval scheduling
Input: set of intervals on the line, represented by
pairs of points (ends of intervals). In another
word, the ith interval, starts at time si and finish at
fi.
Output: finding the largest set of intervals such
that none two of them overlap. Or the maximum
number of intervals that without overlap.
Greedy algorithm:
Select intervals one after another using some rule
3
Rule 1
Select the interval which starts earliest
(but not overlapping the already chosen
intervals)
Underestimated solution!
OPT #4
Algorithm #1
4
Rule 2
Select the interval which is shortest (but
not overlapping the already chosen
intervals)
Underestimated solution!
OPT #2
Algorithm #1
5
Rule 3
Select the interval with the fewest conflicts
with other remaining intervals (but still not
overlapping the already chosen intervals)
Underestimated solution!
OPT #4
Algorithm #3
6
Rule 4
Select the interval which ends first (but still
not overlapping the already chosen
intervals)
Quite a nature idea: we ensure that our
resource become free as soon as possible
while still satisfying one request
Hurray! Exact solution!
7
f1 smallest
Algorithm #3
8
Analysis - exact solution
Algorithm gives non-overlapping intervals:
obvious, since we always choose an interval which
does not overlap the previously chosen intervals
The solution is exact:
Let A be the set of intervals obtained by the
algorithm,
and OPT be the largest set of pairwise nonoverlapping intervals.
We show that A must be as large as OPT
9
Analysis –
exact solution cont.
Let A  {A1 , , Ak } and OPT  {B1, , Bm} be
sorted. By definition of OPT we have k ≤ m
Fact: for every i ≤ k, Ai finishes not later
than Bi.
Pf. by induction.
For i = 1 by definition of a step in the algorithm.
Suppose that Ai-1 finishes not later than Bi-1.
10
Analysis con.
From the definition of a step in the algorithm
we get that Ai is the first interval that finishes
after Ai-1 and does not verlap it.
If Bi finished before Ai then it would overlap
some of the previous A1 ,…, Ai-1 and
consequently - by the inductive assumption it would overlap Bi-1, which would be a
B
contradiction.
B
i-1
i
Ai-1
Ai
11
Analysis con.
Theorem: A is the exact solution.
Proof: we show that k = m.
Suppose to the contrary that k < m. We have
that Ak finishes not later than Bk
Hence we could add Bk+1 to A and obtain
bigger solution by the algorithm-contradiction
Bk-1
Ak-1
Bk
Bk+1
Ak
12
Time complexity
Sorting intervals according to the right-most ends
For every consecutive interval:
If the left-most end is after the right-most end
of the last selected interval then we select this
interval
Otherwise we skip it and go to the next interval
Time complexity: O(n log n + n) = O(n log n)
13
Planning of schools
A collection of towns. We want to plan
schools in towns.
Each school should be in a town
No one should have to travel more than 30
miles to reach one of them.
Edge: towns
no far than 30
miles
14
Set cover
Input. A set of elements B, sets S , S  B
Output. A selection of the Si whose union is
B.
Cost. Number of sets picked.
1
m
15
Greedy
Greedy: first choose a set that covers the
largest number of elements.
example: place a school at town a, since this
covers the largest number of other towns.
Greedy #4
OPT #3
16
Upper bound
Theorem. Suppose B contains n elements that
the optimal cover consist of k sets. Then the
greedy algorithm will use at most k ln n sets.
Pf. Let nt be the number of elements still not
covered after t iterations of the greedy algorithm
(n0=n). Since these remaining elements are
covered by the optimal k sets, there must be
some set with at least nt /k of them. Therefore,
the greedy algorithm will ensure that
nt 1
nt
1
 nt 
 nt (1  )
k
k
17
Upper bound con.
1 t
nt  n0 (1  )
k
Then
, since 1  x  e x for all x,
with equality if and only if x=0.
Thus
1 t
nt  n0 (1  )  n0 (e 1/ k )t  ne  t / k
k
At t=k ln n, therefore, nt is strictly less than ne-ln n
=1, which means no elements remains to be
covered.
Consequently, the approximation ratio is at most
ln n
18
Matroids
When will the greedy algorithm yields optimal solutions?
Matroids [Hassler Whitney]: A matroid is an ordered pair
M=(S, ℓ) satisfying the following conditions.
S is a finite nonempty set
ℓ is a nonempty family of subsets of S, called the
independent subsets of S, such that if
B  , and A  B, then A  .
We say that ℓ is hereditary if it satisfies this property.
Note that empty set is necessarily a member of ℓ.
If
A  , B  , and | A |  | B |
, then there is some element
x  B  A such that A  {x} 
. We say that M satisfies the
exchage property.
19
Max independent
Theorem. All maximal independent subsets in a
matroid have the same size.
Pf. Suppose to the contrary that A is a maximal
independent subset of M and there exists
another larger maximal independent subset B of
M. Then, the exchange property implies that A is
extendible to a larger independent set A ∪ {x} for
some x ∈ B - A, contradicting the assumption that
A is maximal.
20
Weighted Matroid
We say that a matroid M = (S,ℓ) is
weighted if there is an associated weight
function w that assigns a strictly positive
weight w(x) to each element x ∈ S. The
weight function w extends to subsets of S
by summation:
w( A)   w( x)
xA
for any A ⊆ S.
21
Greedy algorithms
on a weighted matroid
Many problems for which a greedy approach provides
optimal solutions can be formulated in terms of finding a
maximum-weight independent subset in a weighted
matroid.
That is, we are given a weighted matroid M = (S,ℓ), and
we wish to find an independent set A ∈ℓ such that w(A) is
maximized.
We call such a subset that is independent and has
maximum possible weight an optimal subset of the
matroid. Because the weight w(x) of any element x ∈ S is
positive, an optimal subset is always a maximal
independent subset-it always helps to make A as large
as possible.
22
Greedy algorithm
GREEDY(M, w)
1. A ← Ø
2. sort S[M] into monotonically decreasing order
by weight w
3. for each x ∈ S[M], taken in monotonically
decreasing order by weight w(x)
4. do if A ∪ {x} ∈ℓ [M]
5. then A ← A ∪ {x}
6. return A
23
Lemma
Lemma 1. Suppose that M = (S,ℓ) is a weighted matroid with
weight function w and that S is sorted into monotonically
decreasing order by weight. Let x be the first element of S such
that {x} is independent, if any such x exists. If x exists, then
there exists an optimal subset A of S that contains x.
Pf.
If no such x exists, then the only independent subset is the
empty set and we're done. Otherwise, let B be any nonempty
optimal subset. Assume that x ∉ B; otherwise, we let A = B and
we're done.
No element of B has weight greater than w(x). To see this,
observe that y ∈ B implies that {y} is independent, since B ∈ℓ
and ℓ is hereditary. Our choice of x therefore ensures that w(x)
≥ w(y) for any y ∈ B.
24
Lemma
Construct the set A as follows. Begin with A = {x}.
By the choice of x, A is independent. Using the
exchange property, repeatedly find a new element of
B that can be added to A until |A| = |B| while
preserving the independence of A. Then, A = B - {y}
∪ {x} for some y ∈ B, and so
w(A)=w(B) - w(y) + w(x) ≥ w(B).
Because B is optimal, A must also be optimal, and
because x ∈ A, the lemma is proven.
25
Lemma
Lemma 2. Let M = (S,ℓ) be any matroid. If x is an
element of S that is an extension of some independent
subset A of S, then x is also an extension of Ø.
Pf. Since x is an extension of A, we have that A ∪
{x} is independent. Since ℓ is hereditary, {x} must be
independent. Thus, x is an extension of Ø.
It is shown that if an element is not an option
initially, then it cannot be an option later.
26
Corollary
Corollary Let M = (S,ℓ) be any matroid. If x is
an element of S such that x is not an extension
of Ø, then x is not an extension of any
independent subset A of S.
Any element that cannot be used immediately
can never be used. Therefore, GREEDY cannot
make an error by passing over any initial
elements in S that are not an extension of Ø,
since they can never be used.
27
Lemma 3. Let x be the first element of S chosen by
GREEDY for the weighted matroid M = (S,ℓ). The
remaining problem of finding a maximum-weight
independent subset containing x reduces to finding a
maximum-weight independent subset of the weighted
matroid M′ = (S′,ℓ), where
S′ ={y ∈ S : {x, y} ∈ℓ},
ℓ′ ={B ⊆ S - {x} : B ∪ {x} ∈ℓ},
and the weight function for M′ is the weight function
for M, restricted to S′. (We call M′ the contraction of
M by the element x.)
28
Proof
If A is any maximum-weight independent subset of M
containing x, then A′ = A - {x} is an independent
subset of M′. Conversely, any independent subset A′
of M′ yields an independent subset A = A′ ∪ {x} of M.
Since we have in both cases that w(A) = w(A′) + w(x),
a maximum-weight solution in M containing x yields
a maximum-weight solution in M′, and vice versa.
29
Theorem
Theorem. If M = (S,ℓ) is a weighted matroid with
weight function w, then GREEDY(M, w) returns an
optimal subset.
Pf. By Corollary, any elements that are passed over
initially because they are not extensions of Ø can be
forgotten about, since they can never be useful.
Once the first element x is selected, Lemma 1 implies
that GREEDY does not err by adding x to A, since
there exists an optimal subset containing x.
30
Theorem con.
Finally, Lemma 3 implies that the remaining problem
is one of finding an optimal subset in the matroid M′
that is the contraction of M by x.
After the procedure GREEDY sets A to {x}, all of its
remaining steps can be interpreted as acting in the
matroid M′ = (S′,ℓ′), because B is independent in M′ if
and only if B ∪ {x} is independent in M, for all sets B
∈ℓ′.
Thus, the subsequent operation of GREEDY will find
a maximum-weight independent subset for M′, and
the overall operation of GREEDY will find a
maximum-weight independent subset for M.
31
Minimum spanning tree
Input: weighted graph G = (V,E)
every edge in E has its positive weight
Output: finding the spanning tree such that the
sum of weights is not bigger than the sum of
weights of any other spanning tree
Spanning tree: subgraph with
no cycle, and
connected (every two nodes in V are connected by a
path)
1
3
2
1
2
1
3
2
1
2
1
3
2
1
2
32
Properties of
minimum spanning trees MST
Spanning trees:
n nodes
n - 1 edges
at least 2 leaves (leaf - a node with only one neighbor)
MST cycle property:
After adding an edge we obtain exactly one cycle and all
the edges from MST in this cycle have no bigger weight
than the weight of the added edge
1
3
2
1
2
1
3
2
1
2
cycle
33
Optimal substructures
MST T:
(Other edges of G
are not shown.)
34
Optimal substructures
MST T:
(Other edges of G
are not shown.)
u
v
Remove any edge (u, v) ∈ T.
35
Optimal substructures
MST T:
(Other edges of G
are not shown.)
T1
T2
Remove any edge (u, v) ∈ T. Then, T is partitioned
into two subtrees T1 and T2.
36
Optimal substructures
MST T:
(Other edges of G
are not shown.)
T1
T2
Remove any edge (u, v) ∈ T. Then, T is partitioned
into two subtrees T1 and T2.
Theorem. The subtree T1 is an MST of G1 = (V1, E1),
the subgraph of G induced by the vertices of T1:
V1 = vertices of T1,
E1 = { (x, y) ∈ E : x, y ∈ V1 }.
37
Similarly for T2.
Proof of
optimal substructure
Proof. Cut and paste:
w(T) = w(u, v) + w(T1) + w(T2).
If T1′ were a lower-weight spanning tree
than T1 for G1, then
T′ = {(u, v)} ∪ T1′ ∪ T2
would be a lower-weight spanning tree
than T for G.
38
Do we also have overlapping
subproblems?
Yes.
Great, then dynamic programming may
work!
Yes, but MST exhibits another powerful
property which leads to an even more efficient
algorithm.
39
Crucial observation
about MST
Consider sets of nodes A and V - A
Let F be the set of edges between A and V - A
Let a be the smallest weight of an edge from F
Theorem:
Every MST must contain at least one edge of weight a
from set F
A
1
3
2
1
2
A
1
3
2
1
2
40
Proof of the observation
Let e be the edge in F with the smallest weight - for simplicity
assume that there is unique such edge. Suppose to the contrary that
e is not in some MST. Choose one such MST.
Add e to MST - obtain the cycle, where e is (among) smallest weights.
Since two ends of e are in different sets A and V - A,
there is another edge f in the cycle and in F. Remove f from the tree
(with added edge e) - obtain a spanning tree with the smaller weight
(since f has bigger weight than e). This is a contradiction with MST.
A
1
3
2
1
2
A
1
3
2
1
2
41
Greedy algorithm
finding MST
Kruskal’s algorithm:
Sort all edges according to the weights
Choose n - 1 edges one after another as follows:
If a new added edge does not create a cycle with
previously selected then we keep it in (partial) MST,
otherwise we remove it
Remark: we always have a partial forest
1
3
2
1
2
1
3
2
1
2
1
3
2
1
2
42
Greedy algorithm
finding MST
Prim’s algorithm:
Select a node as a root arbitrarily
Choose n - 1 edges one after another as follows:
Look on all edges incident to the currently build (partial)
tree and which do not create a cycle in it, and select one
which has the smallest weight
Remark: we always have a connected partial tree
root
1
3
2
1
2
1
3
2
1
2
1
3
2
1
2
43
Example of Prim
A
12
6
9
5
V-A
8
14
7
15
3
10
44
Example of Prim
A
12
6
9
5
V-A
8
14
7
15
3
10
45
Example of Prim
A
12
6
9
5
7
V-A
8
14
7
15
0
3
10
46
Example of Prim
A
12
6
9
5
7
V-A
8
14
7
15
0
3
10
47
Example of Prim
A
12
6
5
V-A
9
5
7
8
14
7
15
0
3
10
48
Example of Prim
6
A
12
6
5
V-A
9
5
7
8
14
7
15
0
3
10
49
Example of Prim
6
A
12
6
5
V-A
9
5
7
8
14
7
15
0
3
8
10
50
Example of Prim
6
A
12
6
5
V-A
9
5
7
8
14
7
15
0
3
8
10
51
Example of Prim
6
A
12
6
5
V-A
9
5
7
8
14
7
3
15
0
3
8
10
52
Example of Prim
6
A
12
6
5
V-A
9
5
7
9
8
14
7
3
15
0
3
8
10
53
Example of Prim
6
A
12
6
5
V-A
9
5
7
9
8
14
7
3
15
0
3
8
15
10
54
Example of Prim
6
A
12
6
5
V-A
9
5
7
9
8
14
7
3
15
0
3
8
15
10
55
Why the algorithms work?
Follows from the crucial observation
Kruskal’s algorithm:
Suppose we add edge {v,w}. This edge has the
smallest weight among edges between the set of
nodes already connected with v (by a path in
selected subgraph) and other nodes.
Prim’s algorithm:
Always chooses an edge with the smallest weight
among edges between the set of already connected
nodes and free nodes.
56
Time complexity
There are implementations using
Union-find data structure (Kruskal’s algorithm)
Priority queue (Prim’s algorithm)
achieving time complexity
O(m log n)
where n is the number of nodes and m is the
number of edges
57
Best of MST
Best to date:
Karger, Klein, and Tarjan [1993].
Randomized algorithm.
O(V + E) expected time.
58
Conclusions
Greedy algorithms for finding minimum
spanning tree in a graph, both in time
O(m log n) :
Kruskal’s algorithm
Prim’s algorithm
Remains to design the efficient data
structures!
59
Conclusions
Greedy algorithms: algorithms constructing
solutions step after step using a local rule
Exact greedy algorithm for interval selection
problem - in time O(n log n) illustrating “greedy
stays ahead” rule
Greedy algorithm may not produce optimal
solution such as set cover problem
Matroids can help to prove when will greedy can
lead to optimal solution
Minimum spanning tree could be solved by
greedy method in O(m log n)
60