CSE 101
Algorithm Design and Analysis
Miles Jones and Russell Impagliazzo
[email protected]
[email protected]
Lecture 30: Summary and review
OUR TOOLBOX OF ALGORITHMIC TECHNIQUES
◼1. Path-finding algorithms (DFS, BFS, etc.)
◼2. Restructuring for efficiency- using data structures, pre-processing,
making recursion implicit. Used in combination with other paradigms.
◼3. Greedy algorithms: picking the intuitively best option
◼4. Divide and conquer: recursively solving much smaller instances
◼5. Back-tracking: local case analysis allows us to prune cases
◼6. Dynamic programming: save and re -use recursive calls
◼7. Hill-climbing: keep trying to improve our solution locally
◼8. Reductions : using an algorithm for one problem to solve another
◼These are the methods that have the most ``bang -for-the-buck’’, general
enough to handle very large number of problems, but not all in NP
OUR BASIC VOCABULARY
◼Asymptotic notation : O
◼Recurrence relations
◼Graphs and special kinds of graphs (trees, DAGs, bipartite graphs,
directed vs. undirected, etc.)
◼Simple data structures: list, queue, array, heap, binary tree, binary
search tree, etc.
◼Logic: stating clearly what a problem is, what the algorithm does
◼Induction and loop invariants
◼Strong induction and recursion
Graph reachability:
procedure GraphSearch (G: directed graph, s: vertex)
Initialize X = empty, F = {s}, U = V – F.
While F is not empty:
Pick v in F.
For each neighbor u of v:
If u is not in X or F:
move u from U to F.
Move v from F to X.
Return X.
VERSIONS OF GRAPH SEARCH
F: Last-in-first-out: stack, DFS,O(|V|+|E|)
First-in-first-out: queue, BFS, O(|V|+|E|)
While both solve graph search, each can be used to give additional
information: BFS: shortest paths, DFS: strongly connected components
Implicit BFS in graph with non -negative edge weights: Dijkstra’s
algorithm finds shortest paths
USING DATA STRUCTURES
◼What sets, relations, or other structures are used to define the algorithm
strategy?
◼For each one:
◼In any one step of the algorithm, what do we need to know about it?
◼In any one step, how does the structure change?
GRAPH REACHABILITY
◼Data structures:
◼X is a set
▪ test membership
▪ insert
◼F is a set
▪ Find and delete
▪ Test membership
▪ Insert
◼U is a set
▪ Test membership
▪ delete
◼G is a graph
▪ For each vertex, loop
through its neighbors
EXPRESSIVE POWER OF GRAPH SEARCH
◼In addition to direct uses of graph search, we can also use graph
search to solve other problems, sometimes related to paths,
sometimes seemingly unrelated. In doing so, we use reductions
between problems. A reduction is a map that takes an instance of one
problem and creates an instance of another. We can then use a
known algorithm for the second problem. If we can prove that the
instance we created is equivalent to the original, then we can use this
as an algorithm for the first.
MAX BANDWIDTH PATH
Say B=7, and we want to decide whether there is a bandwidth 7 or better
path from A to H. Which edges could we use in such a path? Can we use
any such edges?
5
B
A
3
8
5
6
C
9
3
8
4
F
D
E
7
G
6
7
5
H
DECISION TO REACHABILITY
◼Let E_B = { e : w(e) ≥ B}
◼Lemma: There is a path from s to t of bandwidth at least B
if and only if there is a path from s to t in E_B
Proof: If p is a path of bandwidth BW ≥ B, then every edge in p must have w(e) ≥
B and so is in E_B. Conversely, if there is a path from s to t with every edge in
E_B, the minimum weight edge e in that path must be in E_B, so BW(p)= w(e) ≥
B
So to decide the decision problem, we can use reachability:
Construct E_B by testing each edge. Then use reachability on s, t, E_B
INGREDIENTS OF REDUCTIONS
◼Say we have two search problems, A and B where A is: ,
◼Given x_A, find a y_A so that R_A(x_A,y_A), or certify no solution
exists
◼and similarly, B is:
◼Given x_B, find a y_B, so that R_B(x_B,y_B), or certify no solution
exists
INGREDIENTS OF REDUCTION
◼Our reduction needs to provide the following:
◼A recipe for creating an x_B from an x_A. (Does not have to be all x_B’s)
:F(x_A)=x_B.
◼A recipe for creating a y_A from a y_B: G(y_B)=y_A
◼Proof that if R_B(x_B,y_B), then R_A(x_A,y_A)
◼(Easily forgotten): a recipe for creating a y_B from a y_A (H(y_A)=y_B)
◼Proof that if R_A(x_A,y_A), then R_B(x_B,y_B)
◼Algorithm: Run Alg_B on F(x_A)=x_B. If it gives y_B, return G(y_B)=y_A. Else,
return ``no solution exists’’.
◼Correctness: two proofs above
◼Time analysis : Time_{Alg_B} (size of x_B). + Time to compute F+ Time to
compute G. (First usually largest factor)
OPTIMIZATION PROBLEMS
• In general, when you try to solve a problem, you are trying to find
the best solution from among a large space of possibilities.
The format for an optimization problem is
Instance: what does the input look like?
Solution format: what does an output look like?
Constraints: what properties must a solution have?
Objective function: what makes a solution better or worse?
GREEDY ALGORITHM
◼Break huge global search into a series of local searches.
◼Pick the intuitively best option for the first local search.
◼Use the constraints to simplify, and keep going (solve recursively)
MOST IMPORTANT FACT ABOUT GREEDY ALBS
◼THEY DON’T ALWAYS WORK
◼ —- EVEN WHEN THEY OBVIOUSLY WORK
◼How to cope with this:
◼
Prove greedy strategy works carefully
◼
Use greedy algorithm as a heuristic even when it is not guaranteed
to find optimal solutions
EVENT SCHEDULING
◼Your goal is to schedule the most events possible that day such that no
two events overlap.
◼Exponential is too slow. Let’s try some greedy strategies:
▪
▪
▪
▪
Shortest duration
Earliest start time
Fewest conflicts
Earliest end time
PROVING OPTIMALITY
◼What does it mean that the greedy algorithm solves an optimization
problem?
◼I: problem instance.
◼GS: greedy solution to I
◼OS: other (optimal) solution to I
◼Would be incorrect if Value(OS) > Value (GS)
◼So we need to show: For every instance I, let GS be the greedy
algorithm’s solution to I. Let OS be any other solution for I. Then
Value(OS) ≤Value (GS) (or Cost(GS) ≤ Cost (OS) for minimization)
◼Tricky part: OS is arbitrary solution, not one that makes sense. We don’t
know much about it
MODIFY-THE-SOLUTION
◼
MODIFY THE SOLUTION FORMAT
◼MtS Lemma: Let g be the first greedy decision, and OS any legal
solution that does not choose g. Then there is a legal solution OS’ that
chooses g, and OS’ is at least as good as OS.
◼MtS lemma is 90% of the work, and is the heart of the proof.
◼One of two styles of induction:
◼Use MtS Lemma to prove GS is optimal by (strong) induction on
problem size
◼Prove by induction on the number of decisions k: there is an optimal
solution that includes the first k decisions of the greedy algorithm.
STEPS TO PROVE MTS LEMMA
◼Lemma: If OS is any legal solution that does not include the first
greedy decision g, then there is an OS’ that does include g and is at
least as good as OS
◼Step 0: (not on paper): What we know: OS is legal (meets
constraints), definition of g, OS does not include g
◼Step 1: Define OS’ from OS and g
◼Step 2: Prove that OS’ is a valid solution, OS’ meets constraints
◼Step 3: Compare objective function on OS’ to OS, show at least as
good.
FORMAT FOR INDUCTION ARGUMENT
◼From MtS lemma:
◼GS(I) = g+ GS(I’)
◼Convert OS to OS’, containing g, using MtS lemma
◼OS’ = g +OS’(I’)
◼Induction: GS(I’) is at least as good as OS’(I’)
◼Conclude: GS(I) is at least as good as OS’, which is at least as good
as OS
DIVIDE AND CONQUER
◼Break a problem into similar subproblems
◼Solve each subproblem recursively
◼Combine
Master Theorem
Theorem: If 𝑇(𝑛) = 𝑎𝑇(𝑛/𝑏) + 𝑂(𝑛𝑑 ) for some constants
𝑎 > 0, 𝑏 > 1, 𝑑 ≥ 0 ,
Then
𝑂 𝑛𝑑
𝑇 𝑛 ϵ 𝑂 𝑛𝑑 log 𝑛
𝑂 𝑛log𝑏 𝑎
𝑖𝑓 𝑎 < 𝑏𝑑
𝑖𝑓 𝑎 = 𝑏𝑑
𝑖𝑓 𝑎 > 𝑏𝑑
WHEN THE MASTER THEOREM DOES NOT APPLY
◼B=1
◼T(n)=T(n-1)+ O(n)
◼Unwind, T(n)= cn+c(n-1)+c(n-2)+..c(1) ∈ 𝑂 𝑛 2
UNEVEN DIVIDE AND CONQUER
◼Sometimes, sizes of sub-parts not identical or depends on the input
◼Example:
◼Given a pointer to a binary tree, compute its depth
◼Depth(r: node)
◼If lc.r ≠NIL then d:= Depth(lc.r) else d:=0
◼If rc.r ≠ NIL then d:= max(d, Depth(rc.r))
◼Return d
TIME ANALYSIS
◼If tree is balanced, get the recurrence
◼T(n)= 2T(n/2)+ O(1),
◼If tree is totally unbalanced,
◼T(n)= T(n-1)+O(1)
EITHER CASE
◼T(n) = T(L) + T(R) +c
◼n = L+R+1
(T(0)=0, T(1)=c to make things fit)
Then by strong induction, we claim T(n) is at most cn
T(L) ≤ 𝑐𝐿
T(R) ≤ cR
T(n) ≤ cL+cR+c = c (L+R+1) =cn
BACKTRACKING
◼Backtracking is a generic method that can be applied to many problems
that have a solution set that is exponentially large, aka search and
optimization problems
◼Backtracking can often be a first step towards finding a greedy or
dynamic programming algorithm.
◼Backtracking often gives a more efficient runtime over exhaustive search
or brute force but may not result in a polynomial time algorithm, and is
usually an improved exponential time
◼On the other hand, it applies even to NP-complete problems, where we
do not expect to find less than exponential time algorithms
◼Often, they are better on typical inputs than their worst -case
MAX INDEPENDENT SET
◼MIS3(G= (V,E))
◼IF |V|=0 return the empty set
◼Pick vertex v:
◼ S_1:= v + MIS3(G-v-N(v))
◼ IF deg(v)=0 or 1 return S_1
◼ S_2: = MIS3(G-v)
◼ IF |S_2|> |S_1| return S_2, else return S_1
◼Correctness: If deg(v) =0 or 1 |S_2| is at most |S_1|, so we’d return S_1
anyway
◼so does same thing as MIS1
TIME ANALYSIS
◼T(n) is at most T(n-3)+T(n-1) + small amount
◼Similar to Fibonacci numbers, but a bit better, about
◼2^{.6n} rather than 2^{.7n}.
◼Small amount only affects constant in O, because very bottom -heavy
GENERIC BT TIME ANALYSIS
◼View BT algorithm as tree of sub-calls
◼Usually bottom-heavy, so can bound time by counting leaves=
◼Calls to base cases
◼If tree has max depth D and max fan out F, at most , 𝐹 𝐷
◼Leaves, so total time at most 𝑂 𝐹 𝐷
◼OK as upper bound, and exact time is often difficult. I don’t know
◼ MIS algorithm’s tight analysis
DYNAMIC PROGRAMMING
◼DP = BT + memoization
◼Memoization = store and re-use, like the Fibonnacci algorithm
From first class
Two simple ideas, but easy to get confused if you rush:
Where is the recursion? (It disappears into the memoization, like the Fib.
Example did). Have I made a decision? (only temporarily, like BT)
If you don’t rush, a surprisingly powerful and simple
algorithm technique
One of the most useful ideas around
F(5)
WHY DOES IT TAKE SO LONG?
F(4)
F(3)
F(2)
F(1) F(0)
F(1)
F(3)
F(2) F(2) F(1)
F(1) F(0)
F(1)F(0)
A BETTER APPROACH?
function fib2(n)
if n = 1 then return 1
create array f[1…n]
f[1] := 1
f[2] := 1
for i = 3 … n:
f[i]:=f[i – 1] + f[i – 2]
return f[n]
The for loop consists of a
single computer step so in
order to compute f[n], you
need n – 1 + 2 computer
steps!!!!!
This is a huge improvement:
linear time (O(n)) vs.
exponential time (O(1.6^n))
THE LONG WAY
◼1. Come up with simple back-tracking algorithm
◼2. Characterize sub-problems
◼3. Define matrix to store answers to the above
◼4. Simulate BT algorithm on sub-problem
◼5. Replace recursive calls with matrix elements
◼6. Invert ``top-down’’ order of BT to get ``bottom-up’’ order
◼7. Assemble into DP algorithm:
◼
Fill in base cases into matrix
◼
In bottom-up order,
◼
Use translated recurrence to fill in each matrix element
◼
Return ``main problem’’ answer
◼
(Trace-back to get corresponding solution)
THE EXPERT’S WAY
◼Define sub-problems and corresponding matrix
◼ Give recursion for sub-problems
◼Find bottom-up order
◼Assemble as in the long way:
◼
Fill in base cases of the recursion
◼
In bottom-up order do:
◼
Fill in each cell of the matrix according to recursion
◼
Return main case
◼
(Traceback to find corresponding solution)
MINIMUM STANDARD FOR DP
◼Be very clear what subproblem each array/matrix position stands for
◼Be very clear what kind of local case analysis your recursion is based
on.
◼Think in terms of sub-problems and cases, not operations.
WHAT ARE THE DIMENSIONS OF THE MATRIX
◼Dimensions of the matrix are parameters that change during
◼Recursion.
◼Fewer changes= smaller matrix
◼Keep BT version simple!
TIME ANALYSIS FOR DP
◼Generic: Number of cells in array/matrix times number of cases
you need to consider for each cell
HILL-CLIMBING
◼AKA, ``gradient ascent’’, ``interior point method’’
◼Useful for optimization problems:
◼Instance, solution format, constraints, objective function (maximization)
◼Idea: start with any solution that meets the constraints.
◼
Try to find ways to improve the solution, still satisfying the constraints.
◼
When there’s no simple way to improve the solution, output the version you
have
◼
Like greedy algorithms, more often than not hill -climbing does NOT find an
optimal solution, just a ``local optimum’’
◼
But like greedy algorithms, often used as an approximation algorithm or
heuristic.
LOCAL OPTIMA
◼One can view the set of all possible solutions as a high -dimensional
region.
◼The objective function then
◼gives a height for each point
◼We want to find the highest point.
◼But we usually find a local optima,
◼a point that is higher than others
◼near it. So while global optimums
◼are local optima, the reverse is not
◼always true.
NETWORK FLOW
◼Instance: Directed graph with non -negative edge weights, called the
capacity of edges. Think of: nodes= nodes in network, edges=links,
weight=bandwidth. (Or nodes= storage tanks, edges=pipes;
nodes=intersections, edges=roads). Two vertices, s: source, t: sink
◼Flow assigns a non-negative value to every edge, amount flowing on
that edge. Idea is to send as much as possible from s to t, flow on
edge is how much is being sent along that edge.
◼Constraints: At any vertex except s, t total flowing in = total flowing
out. Flow along edge cannot exceed capacity of edge.
◼Objective: maximize total flow out of s = total flow into t
REDUCING PERFECT MATCHING TO MAX FLOW
◼Create a network flow with two additional vertices, s, and t, as well as
all vertices in L and all vertices in R
◼Think of edges between L and R as being directed from L to R
◼Add edges from s to every vertex in L, and from every vertex in R to t
◼Give all edges capacity 1.
REASONABLE SEARCH PROBLEMS
NP= class of decision versions of reasonable search problems
Instance: what does the input look like?
x, n bits
Solution format: what does an output look like?
y, poly(n) bits to describe
Constraints: what properties must a solution have?
R(x,y) can be checked in poly time.
Objective: Given x, does there exist a y so that R(x,y)?
P VS NP
◼Any polynomial time decidable yes/no problem is in NP, making y
trivial (0 or 1).
◼ So P is a subset of NP
◼ P=NP: Every decision version of a reasonable search problem is
solvable in polynomial time
◼Implications: Using binary search, can reduce search/optimization to
decision. So P=NP implies every reasonable search and optimization
problem can be solved in polynomial time.
◼Totally open after many years, one of the Clay Institute list of most
important open problems in mathematics.
REDUCTIONS
◼We can view a decision problem as the set of all instances for which
◼The answer is yes.
◼A reduction from decision problem A to decision problem B
◼Is a polynomial time computable function F so that
◼ 𝑥∈𝐴≡ 𝐹 𝑥 ∈𝐵
◼Then if we have an algorithm Alg_B solving B, we can get
◼an Alg_A solving A by using Alg_A (x) := Alg_B(F(x)).
If A reduces to B, we write, 𝐴 ≤ 𝑚𝑝 𝐵 (mp: polytime map reduction)
COMPLETENESS
◼A problem A is NP-complete if 𝐴 ∈ 𝑁𝑃 and
◼For EVERY 𝐵 ∈ 𝑁𝑃,
◼ 𝐵 ≤ 𝑚𝑝 𝐴
◼It follows that, if A is NP-complete, then
◼𝐴 ∈ 𝑃 ≡ 𝑃 = 𝑁𝑃
◼So each NP-complete problem determines the fate of all search and
◼Optimization proplems. In particular, they are all equivalent to each
other.
NP-COMPLETE
◼(SAT),(3SAT)
◼(TSP)
◼(HAMILTONIAN PATH)
◼(KNAPSACK)
◼(INDEP.SET)
P
• (2SAT)
• Shortest path
• (MST)
• (EULER PATH)
• (UNARY.KNAPSACK)
• (INDEP.SET.TREES)
MASTERY OF ALGORITHMS
◼Algorithms are increasingly important in daily life, from social media,
to finance, to health care, to ubiquitous computation (e.g.,
autonomous driving)
◼You now know how to design and analyze these algorithms!
◼Use your power wisely!