Chapter 10
P and NP
Classes of Algorithms
Logarithmic
(all bases have the same
growth rate)
• (log n)
• (log (log n))
Poly-logarithmic
• (n log n)
• (n2 log n)
• ...
Polynomial (exponent is constant)
• (1) – Sub-linear
• (n0.001) – Sub-linear
• (n0.5) – Sub-linear
• (n) - Linear
• (n2)
• (n3)
• ... (n100)
Classes of Algorithms
Exponential
(bases make a big
difference)
• (2n)
• (3n)
• (nn)
Factorial
k cn < n! < nn
• where k is some constant, and
• c is a constant less than n
Exponential-Exponential
n
nn
The Great Divide
• All Polynomial
Algorithms
• includes
–
–
–
–
–
linear
poly-log
log
constant
sub-linear
• All Exponential
Algorithms
• includes
– factorial
– poly-poly
Problems vs. Algorithms
• A single problem, i.e., sorting integers, can
have many algorithms with very different
run times.
– Mergesort (n log n)
– Permutationsort (n!)
Problems vs. Algorithms
• A specific algorithm’s runtime can vary
depending on the input
• Quicksort
– (n log n) – on most unsorted lists
– (n2) – on pre-sorted or partially sorted lists
• So, what can we say about the sorting
problem?
Problems vs. Algorithms
• The sorting problem facts:
– Can be solve in (n log n) (mergesort)
– Can be solve in (n!)
– The sorting problem is Polynomial because
there exists at least one Polynomial algorithm
for solving the sorting problem.
Problems vs. Algorithms
• Some problems have solution algorithms
that are intrinsically polynomial
• But, some problems can NOT be solved in
polynomial time (period!)
– At least most people think so...
– But, it’s hard to prove that something can’t be
done
Example: 1960’s
• The Longest Common Subsequence Problem (LCS)
– Fasted known algorithm can only solve the
problem in (2n) time.
• Does there exist an optimal LCS algorithm that has a
polynomial runtime?
– In 1960, most computer scientists were leaning
toward an answer of “no” i.e., there does NOT
exist a polynomial algorithm.
Example: 1980’s
• In 1981, two guys Smith and Waterman used
Dynamic Programming to solve the LCS problem
in (n2)-time using (n2)memory.
• This discover and many other discoveries led
computer scientists to ponder the following
question:
• Can all problems be solved in Polynomial
Time?
– Maybe we just haven’t discovered all the good
algorithms yet.
P or NP
• NP – set of problems were the solution can not be
computed deterministically in Polynomial Time
• P – set of problems that can be solved
deterministically in Polynomial Time
NP
P
Polynomial Problems
•
•
•
•
•
Searching
Sorting
Minimum Spanning Tree
LCS Problem
Shortest Path Problem
Non-deterministically Polynomial
Problems
• The graph isomorphism problem of determining
whether two graphs can be drawn identically
• The traveling salesman problem, where we want
to know if there is a route of some length that goes
through all the nodes in a certain network
• The boolean satisfiability problem, where we
want to know if a certain formula in propositional
logic with boolean variables is satisfiable or not
• Knapsack problem
• Bin Packing problem
NP (not deterministic)
Repeat x times where x = O(nc)
Develop an answer in O(nc) time
Verify if answer is correct in O(nc) time (Yes or don’t know)
If its correct, hooray!
You solved the problem, so stop!
NP
P
NP (not deterministic)
Example (the bin packing problem)
Repeat x times where x = O(nc)
Arrange n items randomly
i=1
For j = 1 to n
if(bin[i].total_weight+item[j].weight < bin_capcity)
add item[j] to bin[i]
else
i++;
add item[j] to bin[i]
if (i < ceil(items.total_weight/bin_capacity))
You solved the problem, so stop!
return i
NP (not deterministic)
• This algorithm is
Example (the bin packing problem)
polynomial
Repeat x times where x = O(nc)
Arrange n item randomly
• This algorithm will
i=1
always terminate (finite)
For j = 1 to n
if(bin[i].total_weight+item[j].weight <
• When this algorithm
bin_capcity)
returns an answer it is
add item[j] to bin[i]
always optimal
else
i++;
• Sometimes this algorithm
add item[j] to bin[i]
does not return an answer
if (i < ceil(items.total_weight/bin_capacity))
You solved the problem, so stop!
• Because of its random
return i
nature this algorithm may
return different output
given the same input.
P (deterministic)
• Repeat x times where x = O(nc)
– Develop an answer or solution in O(nc) time
– Verify if it is correct in O(nc) time (YES or No)
– If its correct, hooray! You solved the problem, so stop!
NP
P
P (deterministic)
• Repeat x times where x = O(nc)
– Develop an answer or solution in O(nc) time
– Verify if it is correct in O(nc) time (YES or No)
– If its correct, hooray! You solved the problem, so stop!
Most Problems in P have straight-forward algorithms that
compute a correct or optimal solution.
The verification is done as the solution is developed.
Consider sorting: every comparison and swap corrects an
inversion and get you closer to a correctly sorted list. The
algorithms stops when there are no more inversions to fix
P is a subset of NP
• 6 out of 10 leading theoretical computer scientists
believe this to be true.
NP
P
P = NP
• 1 out of 10 leading theoretical computer scientists
believe this to be true. Every problem that can be
“verified” in Polynomial time can be solved in
Polynomial time.
NP = P
P ? NP
• 3 out of 10 leading theoretical computer scientists
aren’t sure, haven’t made up their minds yet, or
they are too busy thinking about ninjas to care.
NP ? P
NP Problems
• Given a problem with no known polynomial
algorithm, can we prove that it can NOT be
solved in Polynomial time.
• So far, no such proof exists for any NP problem.
– And there are hundreds of NP problems with no known
polynomial algorithm.
– There is a lot of evidence to support the fact that P
does not equal NP, but no one can prove it.
NP-Complete
• P – problems with polynomial deterministic
algorithms
• NP – problems with polynomial non-deterministic
algorithms
• NP Complete – problems in NP that are very
likely not in P.
NP
NP-Complete
P
Footnote: Exponential Problems
• Example: Given a chess board, determine if a
player can win. Output: Possible or impossible.
EXP: Problems that can be
verified in Exponential time
NP
NP-Complete
P
NP-Complete (TSP Example)
• Find the shortest path from A back to A that visits
every vertex.
9
B
2
1
C
5
4
A
6
1
...
4
D
3
7
2
1
5
7
E
4
NP-Complete (TSP Example)
Non-deterministic Algorithm:
• Generate a random permutation of the vertices and
compute total length of the tour (tour_length)
• Sum up the N smallest edges (min_tour)
• If the tour_length = min_tour
– Output optimal answer
• Otherwise, don’t give an answer.
NP
• Given a new problem, if you can create a nondeterministic polynomial algorithm, then its in NP
NP
P
NP-Complete
new
problem
P
• IF you can create a deterministic polynomial
algorithm, then its in P
NP
NP-Complete
P
new
problem
P or NP
• IF you can NOT create a deterministic polynomial
algorithm, then you really can’t be sure of
anything..
NP
NP-Complete
P
new
problem
P or NP
NP-Complete
NP
P
• So, there are many problems
in NP,
• Some are clearly in P,
• All the rest are on the edge,
i.e., we don’t know if a P
algorithm will be discovered
or not
• But there is convincing
evidence that some of the
problems on the edge are not
in P and these are the NPcomplete problems.
NP-Complete
• NP-Complete started with several famous
problems with no known P algorithm:
–
–
–
–
Maximum Clique,
Graph Coloring,
3SAT,
Traveling Salesman
TSP
Graph
Coloring
3SAT
Max
Clique
• Each of the problems can be “polynomially”
transformed into the others.
• The transformation algorithm is called a reduction,
• The reduction process needs to be polynomial.
NP-Complete
•
1.
2.
New problems can be added to NP-Complete if the following
two conditions are satisfied:
The new problem can be reduced into one of the existing
problems in NP-Complete
An existing NP-Complete problem can be reduced into the new
problem
NP-Complete
TSP
New
Problem
Graph
Coloring
3SAT
Max
Clique
NP-Complete
•
•
•
Why must we the transformation be bi-directional?
Here the new problem can be reduced to the TSP problem.
Thus, a TSP algorithm could be used to solve the new problem.
NP-Complete
TSP
New
Problem
Graph
Coloring
3SAT
Max
Clique
NP-Complete
•
•
•
Why must we the transformation be bi-directional?
A reduction exists that can can transform the sorting problem to
the TSP problem
But, sorting is a Polynomial problem, so this alone isn’t enough
to show that sorting is NP-Complete
NP-Complete
TSP
New
Problem
Graph
Coloring
3SAT
Max
Clique
NP-Complete
•
•
•
Why must we the transformation be bi-directional?
Here the TSP problem could be reduced to the new problem
Thus, any algorithm that solves the new problem could be used
to solve the TSP problem.
NP-Complete
TSP
New
Problem
Graph
Coloring
3SAT
Max
Clique
NP-Complete
•
•
•
Why must we the transformation be bi-directional?
First, there exists no such reduction to go from TSP to sorting,
and this is why Sorting is not NP Complete.
But if such a reduction existed than any algorithm for solving
the new problem, could be used to solve all NP-Complete
problems
NP-Complete
TSP
New
Problem
Graph
Coloring
3SAT
Max
Clique
NP-Complete
•
•
•
Why must we the transformation be bi-directional?
But this direction is all that matters, right?
Wrong, some NP-Complete problems can be reduced to
problems that aren’t even in NP.
This direction proves that
the new problem has an
NP Algorithm in the first
place
New
Problem
NP-Complete
TSP
Graph
Coloring
3SAT
Max
Clique
NP-Complete
•
•
•
Once both reductions have been proven, the new problem gets
added to the set of NP-Complete problems.
By the principle of transitivity, all NP-Complete problems can
be reduce to the new problem and
The new problem can be reduced to all NP-Complete problems
NP-Complete
New
Problem
TSP
Graph
Coloring
3SAT
Max
Clique
The power of NP-Complete
• Adds mounting evidence that P != NP
• More problems = more opportunity to find a
polynomial algorithm for an NP-Complete
problem
• All you have to do is prove two reductions
• One establishes that an NP algorithm exists
• The other establishes equivalence (one
algorithm can solve all problems)
Proving that P = NP
• All you have to do is find a problem with a known
polynomial algorithm, and
• Develop a reduction from any of the NP-complete
problems (over 100).
NP-Complete
Polynomial reduction
Problem with
known
Polynomial
Algorithm
TSP
Graph
Coloring
3SAT
Max
Clique
Proving that P = NP
• This should be really easy because there are...
• Tons of problems with Polynomial algorithms, and
• Tons of problems in NP-complete that could be
reduced...
NP-Complete
Polynomial reduction
Problem with
known
Polynomial
Algorithm
TSP
Graph
Coloring
3SAT
Max
Clique
Proving that P = NP
• This has never been done, which tell you
something...
• Probably P != NP.
NP-Complete
Polynomial reduction
Problem with
known
Polynomial
Algorithm
TSP
Graph
Coloring
3SAT
Max
Clique
Halting Problem
• Imagine if you had a function that while (!optimal) {
could generate a good solution for
a = generator(input);
an NP-complete problem
if (is_optimal(a))
(in polynomial time).
optimal = true;
• Imagine if you had function that
could verify the optimality
}
(in polynomial time).
• Put the algorithm in an infinite loop until an optimal solution
was verified.
• Halting Problem: Will the algorithm every terminate?
• Harder Halting Problem: Will it terminate in polynomial
time?
Halting Problem
• If the answer generator is good enough, the
algorithm might always stop.
• If the answer generator is really good, that
algorithm might always stop in polynomial
time.
Polynomial Halting Problem:
• Given N, will the algorithm will finish in
O(Nk) time where k is a constant?
Halting Problem
• Not only is it impossible to verify or prove that a
program will terminate in polynomial time, but...
• It is impossible to verify or prove that a program
will even terminate (in the general case).
• The halting problem is interesting because it
shows how difficult it is to prove that an NPcomplete problem can NOT be solved in
polynomial time.
Halting Problem
halt(p, i)
• returns true if p describes a program that halts
when given as input the string i, and
• returns false otherwise.
Interesting fact:
It is impossible to create a halt(p,i) algorithm that
will terminate for all programs.
There are some programs where halt(p,i) will run
forever and we can prove it.
Halting Problem
Important connection
If you can’t write an algorithm to determine if an
arbitrary program halts then,
You can’t write an algorithm to determine if an
arbitrary program halts in polynomial time.
Thus, there may be no way to prove that a problem
can NOT be solved in polynomial time.
Something to ponder...
• Imagine if you feed halt(p,i) the following program...
while (!optimal) {
a = generator(i);
if (is_optimal(a))
optimal = true;
} ....and its been running for 10 years?
• It could still be polynomial (imagine O(n10) for
n = 1,000) and halt(p,i) might be just about to finish
and return true.
• But you will never know if halt(p,i) will eventually
return an answer and you could be waiting forever
Halting Problem
Some background:
• Any program can be encoded as a binary string.
• Any input can be encoded as a binary string.
• You could actually feed an algorithm its own
encoding as an input.
– Lets call this algorithm self-cannibalism.
Halting Problem
Some background:
• I could feed a program itself in the following
way... halt(p, p).
• Halt-pee-pee is the self-cannibalistic halting
problem: Will a program halt if you feed it its
own encoding as input?
• If we could truly implement the halt algorithm to
return an answer for all input, then it should even
work on p,p.
WTF
function wtf(string p)
if halt(p, p) == false
return true
else
loop forever
wft is a program that takes another program p
encoded as a string. wft simply calls
halt(p, p) and does the opposite.
Halting Problem
function wtf(string p)
if halt(p, p) == false
return true
else
loop forever
Here is why wft if so crazy. We could
actually encode wft as a string and feed the
string to wtf, making it a self-cannibalistic
algorithm.
Halting Problem (Contradiction 1)
function wtf(string p)
if halt(p, p) == false
return true
else
loop forever
• If wtf stops and returns true, it must be because halt(p, p)
returned false,
• but halt only returns false if p runs forever, and p is
actually the encoding for wtf. This means that wtf should
not have stopped if you feed it to itself.
Halting Problem (Contradiction 2)
function wtf(string p)
if halt(p, p) == false
return true
else
loop forever
• If wtf loops forever, it is because halt(p,p) returned true.
• but halt only returns true if p will stop, and p is actually the
encoding of wtf. This means that wtf should have stopped
if you feed it to itself.
Proof by reduction to the absurd
• The assumption that we’ve made is that halt(p,i)
exists and returns an answer for all programs p.
• If this assumption is true then halt(p,p), i.e., self
cannibalism should work.
• But what if you create a program called wtf(p) that
actually calls halt(p,p). This alone is not a
problem.
• However, if you feed the encoding of wtf into wtf,
you create an absurd paradox.
Understanding the absurd paradox
• Consider a vast library that contains all the books
in the world.
• Now consider the library’s catalogs.
• A catalog is simply a book that lists the location of
other books.
• Examples:
– The Catalog of all Biology books
– The Catalog of all Chemistry books
– The Catalog of all Science Related Catalogs
Understanding the absurd paradox
• Examples:
– The Catalog of all Biology books
– The Catalog of all Chemistry books
– The Catalog of all Science Related Catalogs
• Clearly, some catalogs could actually
include the location of other catalogs.
Understanding the absurd paradox
• Clearly, some catalogs could actually
include the location of other catalogs.
• Now, consider the catalog of all catalogs
(CoaC)
• Clearly, the CoaC lists itself.
– This is similar to how table of contents also
includes the page number of the table of
contents. Its also similar to algorithm selfcannibalism.
Understanding the absurd paradox
• Now, consider the catalogue of all catalogues that
don't contain themselves.
– This catalog is similar to the wtf program, so lets call it
the wtf catalog.
• The wtf catalog would include straightforward
catalogs
– Catalog of Bird books
– Catalog of Dog books
• The wtf catalog would NOT include the CoaC.
Understanding the absurd paradox
• Now, consider this question. Will the wtf catalog
be listed in the wtf_catalog?
• This is similar to feeding the wtf program its own
encoding
• Contradiction 1: If the wtf_catalog is listed
inside the wtf_catalog, then its a self referential
catalog so it can’t be listed.
• Remember the wtf_catalog list all catalogs that are
not self-referencing.
Understanding the absurd paradox
• Contradiction 2: If the wtf_catalog is
NOT listed inside the wtf_catalog, then its
NOT a self-referential catalog so it must be
listed in the wtf_catalog.
• Remember to be a self-referential catalog,
you have to be listed within your pages.
Understanding the absurd paradox
• These two contractions tell you that the wtf
catalog can not exist unless you allow for
inconsistency.
• Thus, wtf itself doesn’t have to follow the
rules.
• It can be self-referential but still be listed in
the catalog of all non-self-referential
catalogs.
Understanding the absurd paradox
• In world of libraries, books and catalogs it
is probably OK to not have consistency for
one special case.
• But in the world of math and algorithms,
one instance of inconsistency, undermines
the entire system.
• Thus, statements can be proved and
disproved at the same time.
Understanding the absurd paradox
• For the library example, the way to resolve the
inconsistency is to simply agree that wtf_catalog
can not exist according to its own rules.
• For the halting problem, you can also agree that
the wtf program can’t exist according to its own
rules.
• But the wtf program does exist, feel free to
implement it after class.
• The real show stopper is that the wtf program is
perfectly consistent as long as you assume
halt(p,p) may never stop and return an answer.
Understanding the absurd paradox
• The real show stopper is that the wtf
program is perfectly consistent as long as
you assume halt may never stop and return
an answer.
• As long as there are some problems and
inputs that force halt into an infinite loop,
then the wtf program can exist without
being inconsistent.
Summary
• We have no choice but to conclude that halt will
not always return an answer for all problems and
all inputs.
• halt(p,i) will sometimes loop infinitely.
• What sucks is that if halt(p,i) was in fact a finite
algorithm, then perhaps we could create
halt_in_polynomial(p,i), which could help us
unravel the P = NP dilemma
• But since halt(p,i) can’t be a finite algorithm, there
is no way halt_in_polynomial(p,i) could be a finite
algorithm, because is a much harder problem.
Summary
• The halting problem serves as further evidence
that the P = NP problem may never be solved.
• This all centers around Gödel ’s proof that no
system, whether mathematical or algorithmic, can
be both consistent and complete.
• The nature of Algorithms favor consistency, so we
are left with an incomplete system.
• Incomplete means that the system doesn’t provide
all the tools to prove things that might actually be
true.
Conclusion
• To solve certain problems efficiently, we have to
invent an entirely new system for developing
algorithms.
• That system may be based on quantum physics,
where (in theory) information can be passed
instantly and NN computations could be executed
in parallel on one quantum computer.
• I don’t think its going to happen, but 1000 years
ago no one thought that you could have sorted a
thousand items in one second.