Graphs - Shortest Paths
15-211
Fundamental Data Structures and
Algorithms
Margaret Reid-Miller
24 March 2005
Announcements
Today:
 DAG single source shortest paths
 All pairs shortest paths
 Network flow
 Greedy algorithms
Reading: Sedgewick Chapter 21
Single Source
Shortest Path on a DAG
Single Source Shortest Paths
Dijkstra:
Digraphs with non-negative edge cost
O(n log n + e)
Bellman-Ford:
Digraphs may have negative edge cost
Detects negative cost cycles
O(ne)
If a digraph is a directed acyclic graph
(DAG), can we do better than Dijkstra?
DAG SS Shortest Paths
Use topological sort to order the vertices
from left to right.
If we examine edges in topological order,
then we can relax edges in each path in
forward order. That is, we never relax an
edge of an ancestor.
In particular, when we relax an edge, we
already have the shortest path up to that
edge.
DAG SS Shortest Paths
The algorithm is remarkably simple.
topologically sort the vertices of G
intialize
foreach vertex u in topological order
for all (u,v) in E
relax(u,v)
It runs in O(n+e). Why?
The All Pairs
Shortest Path Algorithm
(Floyd’s Algorithm)
Finding all pairs shortest paths
 Given: A digraph G=(V,E) with cost(x,y)
for each edge.
 Problem: For every pair of vertices
(v,w), find distance the shortest path from
v to w.
 Since the output is n2 in size, we can use
an adjacency matrix representation of the
graph.
 That is, find the table of distances of
shortest paths, like you see on maps
sometimes.
All-pairs Shortest Paths
A[i][j] = dist(i,j) if there is an edge (i,j)
A[i][j] = infinity(inf) if there is no edge (i,j)
30
5
2
4
-20
10
2
30
1
3
15
50
0
Adjacency
matrix
All Pair Shortest Paths
 Two good ways to computing all pair
shortest paths using dynamic
programming. Each is based on different
ways of breaking the problem into
subproblems.
 Approach 1: As with Bellmon-Ford,
compute the shortest path that uses i or
fewer edges.
 Approach2: compute shortest path
distances that uses only the vertices
numbers {0,…,k}.
1. Algorithm
 Subproblem: Use paths of length k.
 Let A[i,j] = dist[i,j] for all i,j & ij
 If (i,j) is not an edge, set
A[i,j]=infinity and A[i,i]=0
 Base case: The initial matrix is the
distance of the shortest path that
uses at most 1 edge (or infinity if
there is no such path).
1. Matrix Multiply Algorithm
 Next compute the distance of the shortest paths
that uses at most 2 edges.
 It turns out we want to find A2, where
multiplication is defined as min of sums instead
sum of products.
 That is (A2)ij = min{ Aik + Akj | k =1,…,n}
 Why is this shortest path distance from i to j with
<= 2 edges?
 Exercise: How would you change A so that A2 is
the shortest path distance from i to j of length
exactly two?
 What is the running time to find A2 ?
1. Matrix Multiply Algorithm
 Using A2 you can find A4 and then A8 and
so on.
 By combining these matrices using matrix
multiplication, we can find An.
 Therefore, to find An we need log n matrix
operations, each taking O(n3).
 Therefore this algorithm is O(n3 log n).
We will consider another algorithm next.
2. Floyd-Warshall Algorithm
 Subproblem: Define matrix Ak[i,j] to keep
distances of paths between i and j that
use vertices {1,2,…,k} only.
 Then A0[i,j] = A[i,j]
 Ak[i,j] = min (Ak-1[i,j] , Ak-1[i,k]+ Ak-1[k,j])
 Why?
Floyd-Warshall Algorithm
 Let p be the shortest path from i to j using
only vertices {1, …, k}.
 If vertex k is not in p then the shortest path
using vertices {1, …, k-1} is also the shortest
path using vertices {1, …, k}.
 If vertex k is in p, then we break p down into
p1 (i to k) and p2 (k to j) where p1 and p2 are
the shortest path using vertices {1, …, k-1}.
 That is, either we use k or we don’t.
Ak[i,j] = min (Ak-1[i,j] , Ak-1[i,k]+ Ak-1[k,j])
Floyd-Warshall Implementation
 Can be implemented using only one
matrix.
initialize all A[i,j] = dist[i,j];
initialize all A[i,i] = 0;
for( k = 0; k < n; k++ )
for( i = 0; i < n; i++ )
for( j =0; j < n; j++)
if ( A[i,j] > A[i,k] + A[k,j] )
A[i,j] = A[i,k] + A[k,j];
 The complexity of this algorithm is O(n3)
Questions
 To find all pairs shortest paths,
would you ever use Dijkstra?
If so, when?
 So far we have found the distance of
the shortest paths. How do we
represent the shortest paths?
Representing Shortest Paths?
 To find the actual path of the shortest
path, we compute a predecessor matrix.
 Initialization:
P[i,j] = nil, if i = j or dist[i,j] = inf
P[i,j] = i, otherwise
 Update:
if ( A[i,j] > A[i,k] + A[k,j] ) {
A[i,j] = A[i,k]+ A[k,j];
P[i,j] = P[k,j];
}
Negative cycles?
 How do we detect whether the graph has
negative cycles?
 At the end of each iteration of the outer loop
check whether A[i,i] < 0 for all i.
for( k = 0; k < n; k++ )
for( i = 0; i < n; i++ )
for( j =0; j < n; j++)
if ( A[i,j] > A[i,k]+A[k,j] ) {
A[i,j] = A[i,k]+A[k,j];
P[i,j] = P[k,j];
}
for( i = 0; i < n; i++ )
if A[i,i] < 0 return false;
Application: Closure
Given a binary relation S on {1,2,...,n}, the
transitive reflexive closure trc(S) is the least relation
R such that
- x S y implies x R y
- x R x for all x
- x R y and y R z implies x R z
If we model the relation by a graph, trc(S) has an
edge (x,y) if the graph has a path from x to y.
We can compute trc(S) by repeated calls to DFS (or
BFS) on each vertex. Good solution if the graph is
sparse.
Transitive Reflexive Closure
But when S is dense one might as well bite the bullet
and use a cubic (in n) algorithm that has good
constants.
Run Floyd-Warshall with cost
A[i,j] = 1, if (i,j) in E
A[i,j] = 0, otherwise.
Return
T[i,j] = 1 if A[i,j] > 0
T[i,j] = 0 otherwise
More efficient to use Boolean operators
Warshall's Algorithm
for( k = 0; k < n; k++ )
for( i = 0; i < n; i++ )
for( j =0; j < n; j++)
A[i,j] = A[i,k] ||
( A[i,k] && A[k,j] );
Upon completion, A is the adjacency matrix for the
transitive reflexive closure of S.
What is the space complexity of this method?
Network Flow Problems
Network flow problems
 An important application of graphs is
to model flows (of items or
information) from one point to
another.
A directed graph G = (V,E), with each
edge e  E having a capacity Ce.
The problem: Compute the maximum
possible flow from a given source vertex
to a given target vertex
 Edge capacities must not be violated
A simple example
s
3
2
1
a
3
b
2
4
c
d
2
3
t
Network flow applications
 Network flow problems have many
applications
bandwidth in large networks
traffic capacities in road networks
flow in electrical circuits or water pipes
A possible greedy algorithm
 Initial graph G
 We incrementally building the flow
graph Gf.
Gf shows the maximum flow attained
thus far on each edge
 At the same time, build up the
residual graph Gr.
Gr shows, for edge, what is the
remaining capacity
Greedy algorithm, cont’d
 Initialize
 Gf: all edges have 0 flow
 Gr: initialize to G
 At each stage, find a path in Gr from s to t
 The edge with the minimum capacity is
the amount of flow that can be added to
every edge on that path
 adjust Gf and Gr accordingly
 Remove saturated edges from Gr
Greedy example, step 0
G
Gf
Gr
s
s
s
3
2
1
a
3
0
b
2
4
c
d
2
3
t
0
0
a
0
3
b
0
0
c
d
0
0
t
2
1
a
3
b
2
4
c
d
2
3
t
Greedy example
 For each stage, we will randomly
choose a path from s to t
Greedy example, step 1
G
Gf
Gr
s
s
s
3
2
1
a
3
0
b
2
4
c
d
2
3
t
2
0
a
0
3
b
2
0
c
d
0
2
t
1
a
3
b
4
c
d
2
1
t
Greedy example, step 2
G
Gf
Gr
s
s
s
3
2
1
a
3
2
b
2
4
c
d
2
3
t
2
0
a
2
1
b
2
0
c
d
2
a
1
1
4
c
d
2
t
b
1
t
Greedy example, step 3
G
Gf
Gr
s
s
s
3
2
1
a
3
3
b
2
4
c
d
2
3
t
2
0
a
2
b
2
1
c
d
2
a
1
1
3
c
d
3
t
b
t
Maximum flow of 5 units into t
Greedy doesn’t work!
 Although we have managed to
compute the correct maximum flow,
in general this greedy algorithm
doesn’t work
 Suppose, for example, we started
out differently…
Greedy failure
G
Gf
Gr
s
s
s
3
2
1
a
3
3
b
2
4
c
d
2
3
t
0
0
a
0
2
b
0
3
c
d
0
3
t
1
a
3
1
c
b
2
d
2
t
Algorithm terminates with maxflow=3
Fixing the algorithm
 In order to fix this problem, we need
a way to “undo” greedy choices
 How:
Whenever we add flow x to an edge
(v,w) in Gf, we will add a new edge
(w,v) with capacity x to Gr
This will allow the path to be undone
later, if necessary
Correct algorithm, step 1
G
Gf
Gr
s
s
s
3
2
1
a
3
3
b
2
4
c
d
2
3
t
0
0
a
0
3
b
0
3
c
d
0
3
t
2
1
a
b
3
3
2
1
c
d
2
3
t
Correct algorithm, step 2
G
Gf
Gr
s
s
s
3
2
1
a
b
3
3
3
2
4
c
d
2
3
t
2
0
a
2
3
b
2
3
c
d
2
3
t
1
a
2
2
b
1
1
2
3
c
d
2
3
t
Maximum flow of 5 units into t
Running time
 If all capacities are integers and the
max flow is f, then at most f stages
to compute the max flow
 Each stage requires finding a
shortest path (O(|E|))
 So, total running time is O(f|E|)
The worst case
s
n
n
1
a
b
n
n
t
What could happen when we run the algorithm?
How might this be avoided?
Greed is Good
Example 1: Counting change
 Suppose we want to give out
change, using the minimal number
of bills and coins.
A change-counting algorithm
 An easy algorithm for giving out N cents in change:
 Choose the largest bill or coin that is N.
 Subtract the value of the chosen bill/coin from N,
to get a new value of N.
 Repeat until a total of N cents has been counted.
 Does this work? I.e., does this really give out the
minimal number of coins and bills?
Our simple algorithm
 For US currency, this simple algorithm
actually works.
 Why do we call this a greedy
algorithm?
Greedy algorithms
 At every step, a greedy algorithm
makes a locally optimal decision,
with the idea that in the end it all adds
up to a globally optimal solution.
 Being optimistic like this usually
leads to very simple algorithms (i.e.,
easy to code).
Change counting is greedy
 Makes a locally optimal decision.
Uses the next-largest bill or coin. But once a
coin is accepted, it is permanently included in the
solution. Once a coin is rejected, it is
permanently excluded from the solution.
 To reach a globally optimal solution.
Can you prove it for US currency?
But…
 What happens if we have a 12-cent
coin?
Example 2: Fractional Knapsack Problem (FKP)
• You rob a store: find n kinds of items
– Gold dust. Wheat. Beer.
Example 2: Fractional knapsack problem (FKP)
• You rob a store: find n kinds of items
– Gold dust. Wheat. Beer.
• The total inventory for
the i th kind of item:
– Weight: wi pounds
– Value: vi dollars
• Knapsack can hold a
maximum of W pounds.
• Q: how much of each kind of item
should you take?
(Can take fractional weight)
FKP: solution
 Greedy solution 1:
 Get a bigger knapsack!
 Build up extra muscles if necessary.
FKP: solution
 Greedy solution 1:
 Get a bigger knapsack!
 Build up extra muscles if necessary.
But seriously folks…
FKP: solution
 Greedy solution 1:
 Get a bigger knapsack!
 Build up extra muscles if necessary.
But seriously folks…
 Greedy solution 2:
 Fill knapsack with “most valuable” item until all
is taken.
 Most valuable = vi /wi (dollars per pound)
 Then next “most valuable” item, etc.
 Until knapsack is full.
Ingredients of a greedy alg.
1. Optimization problem

Of the many feasible solutions, finds the minimum or
maximum solution.
2. Can only proceed in stages

no direct solution available
3. Greedy-choice property:
A locally optimal (greedy) choice will lead to a
globally optimal solution.
4. Optimal substructure:
An optimal solution contains within it optimal
solutions to subproblems.
Show the proof
FKP is greedy
 An optimization problem:
Maximize value of loot, subject to
maximum weight W.
(constrained optimization)
 Proceeds in stages:
Knapsack is filled with one item at a
time.
FKP is greedy
 Greedy-choice property:
A locally greedy choice will lead to a
globally optimal solution.
 Proof:
Step 1: Prove that an optimal solution contains
the greedy choice.
Step 2: Prove that the greedy choice can
always be made first.
FKP: Greedy-choice proof: step 1
We want to show that the optimal solution always
contains the greedy choice.
 Consider total value, V, of knapsack.
 Suppose item h is the item with highest $/lb.
 Knapsack must contain item h:
 Why? Because if h is not included, we can replace some
other item in knapsack with an equivalent weight of h, and
increase V.
 This can continue until knapsack is full, or all of h is taken.
Therefore any optimal solution must include
greedy-choice.
More rigorously…
For
wi
vi
ki
Assume total
optimal value:
item i let
be the total inventory,
be the total value,
be the weight in knapsack.
 vi 
V   ki  
i 1
 wi 
n
Let item h be the item with highest $/lb.
If kh<wh, and kj>0 for some jh, then replace j with an equal
weight of h. Let new total value = V’.
Difference in total value:
since, by definition of h,
 vj 
 vh 
V   V  k j    k j    0
w 
 wh 
 j
vj
wj

vh
wh
Therefore all of item h should be taken.
FKP: Greedy-choice proof: step 2
Next, we want to show that we can always make the greedy
choice first.
 If item h is more than what knapsack can hold, then fill
knapsack completely with h.
 No other item gives higher total value.
 Otherwise, knapsack contains all of h and some other item.
We can always make h the first choice, without changing total
value V.
In either case the greedy-choice can always be made
FIRST.
More rigorously…
 Case I: wh  W
 Fill knapsack completely with h.
 No other item gives higher total value.
 Case II: wh < W
 Let 1st choice be item i, and kth choice be h,
then we can always swap our 1st and kth
choices, and total value V remains unchanged.
 Therefore greedy-choice can always be made
first.
FKP: Optimal substructure
 The optimal substructure property:
An optimal solution contains within it
optimal solutions to subproblems.
 If we remove weight w of one item i from
the optimal load, then the remaining load
must be optimal solution using the
remaining items.
 The subproblem is the most valuable load
of maximum weight W-w from n-1 items
and wi - w of item i.
FKP: Optimal substructure proof
 We want to show that an optimal solution contains within it
optimal solutions to subproblems.
 Consider the most valuable load L, weighing W lbs.
 Remove w pounds of some item i .
 Remaining load L’ must be the most valuable load for a
smaller fractional knapsack problem:
 Maximum weight is W – w lbs.
 Only n-1 items and wi - w lbs. of item i.
 Why? Because otherwise we can find a load, L’’, more
valuable than L’, add w pounds of item i, and this will be more
valuable than L. (Contradiction!)
Example 3: Binary knapsack problem (BKP)
 Variation on FKP.
 The “Supermarket Shopping Spree”!
 Suppose, instead, that you can only take an item
wholly, or not at all (no fractions allowed).
 Diamond rings. Laptops. Watches.
 Q: How many of each item to take?
 Will the greedy approach still work?
Surprisingly, no.
The Binary Knapsack Problem
 You win the Supermarket Shopping
Spree contest.
 You are given a shopping cart with
capacity C.
 You are allowed to fill it with any
items you want from Giant Eagle.
 Giant Eagle has items 1, 2, … n,
which have values v1, v2, …, vn, and
sizes s1, s2, …, sn.
 How do you (efficiently) maximize
the value of the items in your cart?
BKP is not greedy
 The obvious greedy strategy of
taking the maximum value item that
still fits in the cart does not work.
 Consider:
Suppose item i has size si = C and
value vi.
It can happen that there are items j
and k with combined size sj+sk  C but
vj+vk > vi.
BKP: Greedy approach fails
Maximum weight = 50 lbs
$160
$180
$220
$120, 30 lbs
$100, 20 lbs
$60, 10 lbs
item 1 item 2 item 3 knapsack
Dollars/pound
Item 1
$6
Item 2
$5
Item 3
$4
(optimal)
BKP has optimal substructure,
but not greedy-choice property:
optimal solution does not
contain greedy choice.
Exercise:
 How can we (efficiently) solve the
binary knapsack problem?
 Hint:
Dynamic programming
Succeeding with greed.
4 ingredients needed:
1. Optimization problem
2. Can only proceed in stages
3. Greedy-choice property:
A greedy choice will lead to a globally optimal
solution.
4. Optimal substructure:
An optimal solution contains within it optimal
solutions to subproblems.