The Greedy Approach
Young
CS 331 D&A of Algo.
Greedy
1
General idea:
Given a problem with n inputs, we are required to obtain a
subset that maximizes or minimizes a given objective function
subject to some constraints.
Feasible solution — any subset that satisfies some
constraints
Optimal solution — a feasible solution that maximizes
or minimizes the objective
function
Young
CS 331 D&A of Algo.
Greedy
2
procedure Greedy (A, n)
begin
solution  Ø;
for i  1 to n do
x  Select (A); // based on the objective
// function
if Feasible (solution, x),
then solution  Union (solution, x);
end;
Select: A greedy procedure, based on a given objective function,
which selects input from A, removes it and assigns its value to x.
Feasible: A boolean function to decide if x can be included into
solution vector (without violating any given constraint).
Young
CS 331 D&A of Algo.
Greedy
3
About Greedy method
The n inputs are ordered by some selection procedure
which is based on some optimization measures.
It works in stages, considering one input at a time. At
each stage, a decision is made regarding whether or
not a particular input is in an optimal solution.
Young
CS 331 D&A of Algo.
Greedy
4
1. Minimum Spanning Tree ( For Undirected Graph)
The problem:
1) Tree
A Tree is connected graph with no cycles.
2) Spanning Tree
A Spanning Tree of G is a tree which contains all vertices
in G.
Example:
G:
Young
CS 331 D&A of Algo.
Greedy
5
b) Is G a Spanning Tree?
Key: Yes
Key: No
Note: Connected graph with n vertices and exactly n – 1
edges is Spanning Tree.
3) Minimum Spanning Tree
Assign weight to each edge of G, then Minimum
Spanning Tree is the Spanning Tree with minimum total
weight.
Young
CS 331 D&A of Algo.
Greedy
6
Example:
a) Edges have the same weight
1
G:
2
4
3
6
5
7
8
Young
CS 331 D&A of Algo.
Greedy
7

DFS (Depth First Search)
1
7
2
3
5
4
6
8
Young
CS 331 D&A of Algo.
Greedy
8

BFS (Breadth First Search)
1
2
4
3
5
6
7
8
Young
CS 331 D&A of Algo.
Greedy
9
b) Edges have different weights
16
G:
1
2
21
11
3
6
6
19
14
33
4
5

5
10
18
DFS
16
1
2
5
6
33
5
Young
CS 331 D&A of Algo.
3
14
4
Cost  16  5  10  14  33
 78
10
Greedy
10

BFS
16
1
5
2
21
3
Cost  16  19  21  5  6
6
19
 67
6
4
5

Minimum Spanning Tree (with the least total weight)
16
1
5
2
11
6
Young
CS 331 D&A of Algo.
6
3
18
4
Cost  16  5  6  11  18
 56
5
Greedy
11
Algorithms:
1) Prim’s Algorithm (Minimum Spanning Tree)
Basic idea:
Start from vertex 1 and let T  Ø (T will contain
all edges in the S.T.); the next edge to be included
in T is the minimum cost edge(u, v), s.t. u is in the
tree and v is not.
Example: G
16
1
2
21
11
3
6
6
19
5
14
33
4
5
10
18
Young
CS 331 D&A of Algo.
Greedy
12
1
1
16
19
2
(Spanning Tree)
5
6
S.T.
1
1
19
21
5
2
1
19
10
4
6
2
6
5
5
21
6
4
S.T.
1
2
3
4
18
14
2
11
6
3
11
6
3
1
3
4
21
5
11
6
6
S.T.
19
21
2
6
1
6
5
2
3
S.T.
4
Young
CS 331 D&A of Algo.
Greedy
13
1
2
4
3
19
18
5
5
1
2
S.T.
6
33
2
4
5
3
6
5
Cost = 16+5+6+11+18=56
Minimum Spanning Tree
3
S.T.
4
1
6
(n – # of vertices, e – # of edges)
It takes O(n) steps. Each step takes O(e) and e  n(n-1)/2  O(n 2 ).
Therefore, it takes O(n3) time.
With clever data structure, it can be implemented in O(n2).
Young
CS 331 D&A of Algo.
Greedy
14
2) Kruskal’s Algorithm
Basic idea:
Don’t care if T is a tree or not in the intermediate stage, as
long as the including of a new edge will not create a cycle, we
include the minimum cost edge
Example:
10
30
1
45
Step 1: Sort all of edges
4
(1,2) 10
√
(3,6) 15
√
20
6
(4,6) 20
√
(2,6) 25
√
(1,4) 30
× reject create cycle
(3,5) 35
√
Young
CS 331 D&A of Algo.
Greedy
2
50
25
3
40
35
55
5
15
15
Step 2: T
1
2
1
2
2
1
1
1
2
3
3
3
2
3
6
6
4
6
4
6
4
5
Young
CS 331 D&A of Algo.
Greedy
16
How to check:
adding an edge will create a cycle or not?
If Maintain a set for each group
(initially each node represents a set)
Ex:
set1
set2
1
2
 new edge
2
3
6
set3
4
5
6
from different groups  no cycle created
Data structure to store sets so that:
i. The group number can be easily found, and
ii. Two sets can be easily merged
Young
CS 331 D&A of Algo.
Greedy
17
Kruskal’s algorithm
While (T contains fewer than n-1 edges) and (E   ) do
Begin
Choose an edge (v,w) from E of lowest cost;
Delete (v,w) from E;
If
(v,w) does not create a cycle in T
then
add (v,w) to T
else
discard (v,w);
End;
With clever data structure, it can be implemented in O(e log e).
Young
CS 331 D&A of Algo.
Greedy
18
So, complexity of Kruskal is O (eLoge)
n(n  1)
e 
 Loge  Logn 2  2 Logn
2
3)
 O(eLoge)  O(eLogn)
Comparing Prim’s Algorithm with Kruskal’s Algorithm
i.
Prim’s complexity is O(n 2 )
ii. Kruskal’s complexity is O(eLogn)
if G is a complete (dense) graph,
2
Kruskal’s complexity is O(n Logn
.)
if G is a sparse graph,
Kruskal’s complexity is O (nLogn).
Young
CS 331 D&A of Algo.
Greedy
19
2. Dijkstra’s Algorithm for Single-Source Shortest Paths
The problem:
Given directed graph G = (V, E),
a weight for each edge in G,
a source node v0,
Goal: determine the (length of) shortest paths from v0 to all the
remaining vertices in G
Def: Length of the path: Sum of the weight of the edges
Observation:
May have more than 1 paths between w and x (y and z)
But each individual path must be minimal length
(in order to form an overall shortest path form v0 to vi )
w
V0
Young
CS 331 D&A of Algo.
x
y
z
shortest paths from v0 to vi
Greedy
Vi
20
Notation
cost adjacency matrix Cost, 1  a,b  V
cost from vertex i to vertex j if there is a edge
Cost (a, b) = 0
if a = b

otherwise
1 if shortest path (v0 , w) is defined
s(w) =
0 otherwise
Dist ( j )
j in the vertex set V
= the length of the shortest path from v0 to j
From( j )  i
Young
CS 331 D&A of Algo.
if i is the predecessor of j
along the shortest
path from v0 to j
Greedy
21
45
Example:
50
10
V0
V1
V4
35
15
10
20
20
30
V2
V3
15
v0
a) Cost adjacent matrix
v0
v1
v2
v3
v4
v5
Young
CS 331 D&A of Algo.
Greedy
V5
3
v1
v2
v3
v4
v5
 0 50 10  45  
  0 15  10  


20  0 15   


  20  0 35  
    30 0  


    3  0 
22
b) Steps in Dijkstra’s Algorithm
1. Dist (v0) = 0, From (v0) = v0
2. Dist (v2) = 10, From (v2) = v0
45
45
V0
50
10
V1
50
V0
V4
V1
10
V4
15
15
20
10
10
35
20
20
35
20
30
30
V2
15
V3
Young
CS 331 D&A of Algo.
3
V2
V5
Greedy
15
V3
3
V5
23
3. Dist (v3) = 25, From (v3) = v2 4. Dist (v1) = 45, From (v1) = v3
45
50
V0
V1
45
10
V4
50
V0
15
20
10
V4
15
35
10
V1
20
35
10
20
20
30
30
V2
15
Young
CS 331 D&A of Algo.
V3
3
V5
Greedy
V2
15
V3
3
V5
24
5. Dist (v4) = 45, From (v4) = v0
6. Dist (5) = 
45
45
50
V0
V1
V0
V4
10
20
10
V4
35
35
20
10
V1
15
15
20
50
10
20
30
30
V2
15
V3
Young
CS 331 D&A of Algo.
3
V5
V2
Greedy
15
V3
V5
3
25
c) Shortest paths from source v0
v0  v2  v3  v1
45
v0  v2
10
v0  v2  v3
25
v0  v4
45
v0  v5

Young
CS 331 D&A of Algo.
Greedy
26
Dijkstra’s algorithm:
procedure Dijkstra (Cost, n, v, Dist, From)
// Cost, n, v are input, Dist, From are output
begin
for i  1 to n do
s (i )  0;
Dist (i )  Cost (v, i );
From(i )  v;
s (v )  1;
for num  1 to (n – 1) do
choose u s.t. s(u) = 0 and Dist(u) is minimum;
s (u )  1;
for all w with s(w) = 0 do
if ( Dist (u )  Cost (u, w)  Dist ( w))
Dist ( w)  Dist (u )  Cost (u, w);
From( w)  u;
end;
Young
CS 331 D&A of Algo.
Greedy
(cont. next page)
27
Ex:
1
10
50
30
100
5
2
20
10
4
50
5
3
a) Cost adjacent matrix
01 02 03 04 05
1
2
3
4
5
Young
CS 331 D&A of Algo.
 0 50 30 100 10
 0    


 5 0 50  



20

0



   10 0 
Greedy
28
b) Steps in Dijkstra’s algorithm
1. Dist (1) = 0, From (1) = 1
2. Dist (5) = 10, From (5) = 1
1
1
10
5
50
10
30
100
2
20
10
5
4
5
30
100
5
4
50
2
20
10
3
Young
CS 331 D&A of Algo.
50
3
50
Greedy
29
3. Dist (4) = 20, From (4) = 5 4. Dist (3) = 30, From (3) = 1
1
1
10
30
100
5
10
50
2
10
50
15
Young
CS 331 D&A of Algo.
4
10
30
10
3
50
3
1
1  3  2 35
1  5 4 20
5
5. Dist (2) = 35, From (2) = 3
Shortest paths
from source 1
13
2
20
10
5
4
30
100
5
20
50
5
50
30
100
2
20
10
5
4
Greedy
50
3
30
3. Optimal Storage on Tapes
The problem:
Given n programs to be stored on tape, the lengths of
these n programs are l1, l2 , . . . , ln respectively.
Suppose the programs are stored in the order
of i1, i2 , . . . , in
Let tj be the time to retrieve program ij.
Assume that the tape is initially positioned at the beginning.
tj is proportional to the sum of all lengths of programs stored in
front of the program ij.
Young
CS 331 D&A of Algo.
Greedy
31
n
1
The goal is to minimize MRT (Mean Retrieval Time),  t j
n j 1
j
n
i.e. want to minimize  li
k
j 1 k 1
Ex:
n  3, (l1 , l 2 , l3 )  (5,10,3)
There are n! = 6 possible orderings for storing them.
1
2
3
4
5
6
order
total retrieval time
MRT
123
132
213
231
312
321
5+(5+10)+(5+10+3)=38
5+(5+3)+(5+3+10)=31
10+(10+5)+(10+5+3)=43
10+(10+3)+(10+3+5)=41
3+(3+5)+(3+5+10)=29
3+(3+10)+(3+10+5)=34
38/3
31/3
43/3
41/3
29/3
34/3
Smallest
Note: The problem can be solved using greedy strategy,
just always let the shortest program goes first.
( Can simply get the right order by using any sorting algorithm)
Young
CS 331 D&A of Algo.
Greedy
32
Analysis:
Try all combination: O( n! )
Shortest-length-First Greedy method: O (nlogn)
Shortest-length-First Greedy method:
Sort the programs s.t. l1  l2  . . .  ln
and call this ordering L.
Next is to show that the ordering L is the best
Proof by contradiction:
Suppose Greedy ordering L is not optimal, then there
exists some other permutation I that is optimal.
I = (i1, i2, … in)
Young
CS 331 D&A of Algo.
 a < b, s.t. lia  lib (otherwise I = L)
Greedy
33
Interchange ia and ib in and call the new list I :
x
I
…
ia
ia+1
ia+2
…
ib
…
…
ia
…
SWAP
I
…
ib
ia+1
ia+2
x
In I, Program ia+1 will take less (lia- lib) time than in I to be retrieved
In fact, each program ia+1 , …, ib-1 will take less (lia- lib) time
For ib, the retrieval time decreases x + lia
For ia, the retrieval time increases x + lib
totalRT ( I )  totalRT ( I )  (b  a  1)(lia  lib )  ( x  lia )  ( x  lib )
 (b  a)(lia  lib )  0 ()
Therefore, greedy ordering L is optimal
Young
CS 331 D&A of Algo.
Greedy
Contradiction!!
34
4. Knapsack Problem
The problem:
Given a knapsack with a certain capacity M,
n objects, are to be put into the knapsack,
each has a weight w1 , w2 ,, wn and
a profit if put in the knapsack p1 , p 2 , , p n .
The goal is find ( x1 , x2 ,, xn ) where 0  xi  1
n
s.t.
px
i 1
i
n
i
is maximized and
w x
i 1
i
i
M
Note: All objects can break into small pieces
or xi can be any fraction between 0 and 1.
Young
CS 331 D&A of Algo.
Greedy
35
Example:
n3
M  20
( w1 , w2 , w3 )  (18,15,10)
( p1 , p 2 , p3 )  (25,24,15)
Greedy Strategy#1: Profits are ordered in nonincreasing order (1,2,3)
2
( x1 , x2 , x3 )  (1, ,0)
15
3
2
pi xi  25 1  24   15  0  28.2

15
i 1
Young
CS 331 D&A of Algo.
Greedy
36
Greedy Strategy#2: Weights are ordered in nondecreasing order (3,2,1)
2
( x1 , x2 , x3 )  (0, ,1)
3
3
2
p
x

25

0

24

 15 1  31

i i
3
i 1
Greedy Strategy#3: p/w are ordered in nonincreasing order (2,3,1)

p1 25

 1.4 
w1 18


p2 24

 1.6  (2,3,1)
w2 15


p3 15

 1.5 
w3 10

1
( x1 , x2 , x3 )  ( 0, 1, )
2
3
Optimal solution
 pi xi  25  0  24 1  15 
Young
CS 331 D&A of Algo.
i 1
Greedy
1
 31.5
2
37
Analysis:
p
p1 p2

 n
Sort the p/w, such that w1 w2
wn
Show that the ordering is the best.
Proof by contradiction:
Given some knapsack instance
pn
p1 p2
Suppose the objects are ordered s.t. w  w    w
1
2
n
let the greedy solution beX  ( x1 , x2 , xn )
Show that this ordering is optimal
Case1: X  (1,1,,1) it’s optimal n
Case2:X  (1,1, , x j , 0, , 0 ) s.t. wi xi  M
i 1
where 0  x j  1
Young
CS 331 D&A of Algo.
Greedy
38
Assume X is not optimal, and then there exists Y  ( y1 , y 2 ,, y n )
n
n
i 1
i 1
s.t.  pi y i   pi xi and Y is optimal
examine X and Y, let yk be the 1st one in Y that yk  xk.
X  ( x1 , x2 ,, xk 1 , xk ,, xn )
y k  xk  yk < xk
same
Y  ( y1 , y 2 ,, y k 1 , y k ,, y n )
Now we increase yk to xk and decrease as many of ( y k 1 ,, y n )
as necessary, so that the capacity is still M.
Young
CS 331 D&A of Algo.
Greedy
39
Let this new solution be Z  ( z1 , z 2 ,, z n )
where zi  xi 1  i  k
and ( z k  yk )  wk 
n
n
pz p y
i 1
i i
i 1
i
i
n
(y
i  k 1
 ( z k  y k )  pk 
pk pi


wk wi
 zi )  wi
i
n
(y
i  k 1
i
 zi )  pi
i  k  1
pk
 pi  ( )  wi
wk
0
n
n

 pk
pi zi   pi yi  ( z k  yk )  wk   ( yi  zi )  wi  
  pi yi

i 1
i 1
i  k 1

 wk i 1
n
n
Young
CS 331 D&A of Algo.
Greedy
40
So,
if
profit ( z )  profit ( y )
()
else profit ( z )  profit ( y )
(Repeat the same process.
At the end, Y can be transformed into X.
 X is also optimal.
Contradiction!  )
Young
CS 331 D&A of Algo.
Greedy
41