Applied Computer Science II
Chapter 7: Time Complexity
Prof. Dr. Luc De Raedt
Institut für Informatik
Albert-Ludwigs Universität Freiburg
Germany
Overview
•
•
•
•
•
Measuring complexity
The class P
The class NP
NP-completeness
Additional NP-complete problems
Measuring complexity
• How much time needed to decide a
language A = {0k1k | k ≥ 0}
• Use Machine M1
•
Measuring complexity
• Worst-case versus average-case
The time complexity of a TM M is the function
f : Ν → Ν, where f (n) is the max number of steps
that M uses on any input of length n.
• We typically use big-O and small-o
notation (cf. ACS I); asymptotic
analysis
Big-O
Let f , g:Ν → R + be two functions. We say
that f (n) is O( g (n)) if positive integers c and n0 exist
so that for every integer n ≥ n0 holds that f (n) ≤ c.g (n)
O( g (n)) is an upper bound for f (n)
f (n) = 5n3 + 2n 2 + 22n + 5
f =O(n3 ) choose c = 6 and n0 = 10
f is also O(n 4 ) but not O(n 2 )
f (n) = 3n.log 2 n + 5n.log 2 log 2 n + 2
f (n) = O(n.log n)
log b x = p if and only if b p = x (definition)
log b x*y = log b x + log b y
log b x/y = log b x - log b y
log b x p = p log b x which implies that (x p )q = x (pq)
log b x = log a x * log b a
Small-o
Let f , g:Ν → R + be two functions. We say
that f (n) is o( g (n)) if
f ( n)
lim
= 0, i.e. for any real number c > 0,
n →∞ g ( n)
there is a number no where f (n) < c.g (n) for all n ≥ n0
Time Complexity
Let t : Ν → Ν be a function. Define the time complexity class
TIME (t (n)) ={L | L is a language decided by an O(t (n)) time Turing machine}
So, {0k 1k | k ≥ 0}∈ TIME (n 2 )
Can we find a machine that decides A assymptotically more quickly ?
i.e. is A ∈ TIME (t (n)) for t (n) = o(n 2 ) ?
Analyzing algorithms
Analyzing algorithms
• 2-tape machine
Complexity relationship
among models
Theorem 7.8
Let t (n) be a function where t (n) ≥ n.
Then every t (n) time multitape Turing machine has
an equivalent O (t (n) 2 ) time single-tape Turing machine.
Proof elements
Let M be a k − tape TM that runs in t ( n)
Construct S (single-tape TM) that runs in O (t (n) 2 ) time
Analyze simulation
To simulate each step of M ,
S makes two scans over active input tape
each of the active parts of the k tapes are of length at most t (n)
i.e. k .t (n) = O(t (n))
and possibly k shifts (for inserting a new cell)
shifts take O(t ( n)) time
First phase : copying inputs from M to S
takes O(n) steps
Second phase : simulate the t (n) steps of M
takes t (n).O(t (n)) = O(t (n) 2 ) steps
O(n) + O(t (n) 2 ) = O(t (n) 2 ) when t (n) ≥ n
Definition
Let N be an NTM that is a decider. The running
time of Nis the function f : Ν → Ν , where f (n) is the
maximum number of steps that N uses on any branch of its
computation of any input of length n as shown below
• Definition 7.9 including figure 7.1
Theorem 7.10
Let t (n) be a function where t (n) ≥ n.
Then every t (n) time nondeterministic Turing machine has
an equivalent 2O (t ( n )) time deterministic single-tape Turing machine.
Proof elements
Every branch of N has length at most t (n)for input of length n
Every node in tree has at most b children (determined by δ )
Exploration tree is breadth first :
1+b + b 2 + ... + bt ( n ) nodes and leaves
=O(bt ( n ) )
Time for processing one node (or leaf) starting from root
O(t (n))
Running time of deterministic 3-tape TM
O(t (n)).O(bt ( n ) ) = 2O (t ( n ))
Use transformation to deterministic single tape
gives (2O (t ( n )) ) 2 = 2O (t ( n ))
Differences in models
• Polynomial difference between singleand multiple tape deterministic TM
• Exponential difference between
deterministic and non-deterministic
TM
• Compare exponential versus
polynomial behaviour !
Towards the class P
Distinction between :
• Exponential algorithms
– Brute force search
– Impractical
• Polynomial
– Practical
– Robust for reasonable models of computation
(I.e. polynomially equivalent)
Here, we ignore differences between,
e.g. O(n) and O(n5 ) , level of abstraction
Definition The class P
P is the class of languages that are decidable in polynomial time
on a deterministic single -tape TM, i.e.
P=U TIME (n k )
k
• P is invariant for all models of computation
that are polynomially equivalent to a single
tape deterministic TM
• P roughly corresponds to the class of
problems that are realistically solvable on a
computer
Conventions
• High-level descriptions of algorithms
• Abstraction of specific model used (as long
as polynomially equivalent)
• Algorithms proceed in numbered stages
• Analysis in two steps :
– A polynomial bound on the number of stages
– Each stage can be implemented in polynomial
time on deterministic single tape TM (or poly.
Equivalent model)
Conventions
• Encoding method for problems
• We use <.> notation as before
• Reasonable methods require encoding
and decoding of objects into other
(internal ?) representation to require
polynomial time.
• E.g. 111111111 (unreasonable) instead
of 9 (or other base k-notation)
Encoding Graphs
• Option 1 :
– List of nodes
– List of edges
• Option 2 :
– Adjacency matrix
1 iff there is an edge from node i to node j
Entry (i, j )= 
0 otherwise

PATH = {< G, s, t >| G is a directed graph that has a directed path from s to t}
• Figure 7.2
Theorem PATH ∈ P
• Insert algorithm
RELPRIME = {< x, y >| x and y are relative prime}
i.e. the greatest common divisor of x and y is 1
Theorem RELPRIME ∈ P
• Insert algorithm
Theorem
Every context-free language is a member of P
Assume language in Chomsky Normal Form
Then each derivation of a string w has 2n − 1 steps
(Generating all derivations is not polynomial!)
Dynamic programming :
accumulate information about smaller problems
keep track of table of size n × n
table(i, j ) = the set of variables generating wi ...w j
with i ≤ j
Towards the class NP
HAMPATH = {< G, s, t >| G is a directed graph with a hamiltonian path from s to g}
i.e. it is directed and contains all nodes in G
HAMPATH
• Easy to obtain exponential time algorithm :
– Generate all paths of length m (no. of nodes)
and check whether they are hamiltonian
• Polynomial verifiability :
– Checking whether a given path is hamiltonian
can be done in polynomial time
• Verifying existence of hamiltonian path is
easier than determining the existence
• Consider also HAMPATH
• Similar for COMPOSITES
COMPOSITES = {x | x = p.q for integers p, q > 1}
The class NP
Definition
A verifier for a language A is an algorithm V s.t.
A = {w | V accepts < w, c > for some string c}
c : certificate, proof
Time of verifiers is measured in terms of size of w
(implies that c is of size polynomial in size w)
Polynomial time verifiers
Definition
NP is the class of languages that have polynomial time verifiers
NP comes from Nondeterministic polynomial time
NTM for HAMPATH
• Algorithm
Theorem
A language is in NP iff it is decided by some nondeterministic
polynomial time algorithm
idea
• Proof
Definition
NTIME(t (n)) = {L | L is a language decided by an O(t (n)) time NTM}
Corollary
NP=U NTIME (n k )
k
CLIQUE = {< G, k >| G is an undirected graph with a k -clique}
Theorem
CLIQUE is in NP
SUBSET − SUM = {< S , t >| S = {x1 ,..., xk } and for some {y1 ,..., yl } ⊆ {x1 ,..., xk }we have ∑ yi = t}
i
The y and x are multi-sets.
Theorem
SUBSET − SUM is in NP
P versus NP
P versus NP
• P= the class of languages where
membership can be decided quickly
• NP=the class of languages where
membership can be verified quickly
NP ⊆ EXPTIME=U TIME (2n )
k
k
• NP-complete problems : if a polynomial
time (determ.) algorithm exists for one
NP-complete problem, then P=NP
Boolean Logic
true is denoted by 1
SAT = {< φ >| φ is a satisfiable boolean formula}
Theorem (Cook-Levin)
false is denoted by 0
and ∧
SAT ∈ P iff P=NP
or ∨
not ¬
0 ∧ 0 = 0 0 ∨ 0 = 0 ¬0 = 0 = 1
0 ∧1 = 0
1∧ 0 = 0
1∧1 = 1
0 ∨1 = 1
1∨ 0 = 1
1∨ 1 = 1
¬1 = 1 = 0
A Boolean formula is an expression involving Boolean variables and operations
E.g. φ =( x ∧ y ) ∨ ( x ∧ z )
Assignment of values (0 and 1) to variables
For each assignment one can evalue a formula
If the formula evaluates to 1, we say that the assignment satisfies the formula.
The satisfiability problem is to test whether a formula is satisfiable
(i.e. whether there exists an assignment that satisfies the formula)
A function f : Σ* → Σ* is a polynomial time computable function
if some polynomial time TM M exists that halts with
just f ( w) on its tape, when started on any input w
Polynomial time reducibility
Language A is polynomial time (mapping) reducible to language B
written A ≤ P B, if a polynomial time computable function f : Σ* → Σ*
exists where for every w : w ∈ A iff f ( w) ∈ B
f is a polynomial time reduction
• Fig 7.6
Theorem
If A ≤ P B and B ∈ P, then A ∈ P
• Proof: N works in polynomial time
3SAT {< φ >| φ is a satisfiable 3cnf formula}
cnf : conjunctive normal form, i.e. a conjunction of disjunctions
(disjunctions are clauses, elements of disjunctions are literals)
3cnf : each clause/disjunction has exactly three literals
e.g. ( x ∨ y ∨ z ) ∧ ( y ∨ x ∨ w)
Theorem
3SAT is polynomial time reducible to CLIQUE
Proof
Let φ be of the form (a1 ∨ b1 ∨ c1 ) ∧ ... ∧ (ak ∨ bk ∨ ck )
f (φ ) =< G, k > where
all nodes in G are organised in triples t1 ,..., tk
each triple corresponds to a cluase and has 3 nodes
The edges of G connect all nodes except for :
1. edges between the nodes of the same triple
2. nodes with contradictory label, i.e. x and x
Now prove that f is a polynomial time reduction
• Fig 7.7
NP-completeness
Definition
A language B is NP-complete if
1. B is in NP, and
2. every A in NP is polynomial time reducible to B
Theorem
If B is NP-complete and B ∈ P, then P=NP
Theorem
If B is NP-complete and B ≤ P C for C in NP,
then C is NP-complete
Theorem Cook-Levin
SAT is NP-complete
Proof
1. SAT is in NP (easy)
2. any language A in NP is polynomial time reducible to SAT
Let N be a NTM that decides A in time n k
Define an (accepting) tableau for N on w
define reduction f from A to SAT
on input w, define f ( w)
let us first define the boolean variables.
Let C = Q ∪ Γ ∪ {#}
for 1 ≤ i, j ≤ n k , for s ∈ C :
define xi , j , s = cell (i, j ) contains symbol s
Now define φ = φcell ∧ φ start ∧ φmove ∧ φaccept
)
) (
(
φcell = ∧  ∨ xi , j , s ∧

(
x
x
)
∧
∨
i
j
s
i
j
t
,
,
,
,

s∈C
s ,t∈C , s ≠ t
1≤ i , j ≤ n k
"each cell contains exactly one symbol"
φstart = x1,1,# ∧ x1,2,q ∧ x1,3, w ∧ ... ∧ x1,n + 2, w ∧ x1,n +3, ∧ ... ∧ x1,n
0
1
n
"the first row corresponds to the starting situation"
k
−1,
∧ x1,nk ,#
φaccept = ∨ xi , j ,q
1≤i , j ≤ n k
accept
"at least one of the configurations is an accept state"
concept of legal windows
e.g. δ (q1 , a) = {(q1 , b, R)} and δ (q1 , b) = {( q2 , c, L), (q2 , a, R)}
Claim if the top row of the table is the start configuration
and if every window (2x3) is legal then each row legally follows the preceding one.
φmove =
∧
1<i ≤ n ,1< j < n
k
"the i, j window is legal"
k
for each window :
∨
a1 ,..., a6 is a legal window
( xi , j −1,a1 ∧ xi , j ,a2 ∧ xi , j +1,a3 ∧ xi +1, j −1,a4 ∧ xi +1, j , a5 ∧ xi +1, j +1,a6 )
3.Analyze the complexity
tableau has n k × n k = n 2 k cells
each cell has l variables (for symbols in C )
thus number of variables is O(n k )
Now investigate different components of φ = φcell ∧ φ start ∧ φmove ∧ φaccept
(All O(n 2 k ) or O(n k ); indices can be encoded in O(log n)
f ( w) is of polynomial size and can be computed in polynomial time
Theorem
3SAT is NP-complete
Proof
modify construction in SAT proof
first rewrite φ in cnf form.
second rewrite clauses such that they have 3 literals
1. if less than 3 literals e.g. a ∨ b rewrite to e.g. a ∨ b ∨ b
2. if more than 3 literals ( a1 ∨ ... ∨ al )
rewrite into
(a1 ∨ a2 ∨ z1 ) ∧
( z1 ∨ a3 ∨ z2 ) ∧
Corollary
...
CLIQUE is NP-complete
( zl −3 ∨ al −1 ∨ al )
VERTEX − COVER = {< G, k >| Gis an undirected graph that has a k -node vertex cover}
Let Gbe an undirected graph (V , E ) and k a number
Avertex cover is a set of nodes V ' ⊆ V such that
for all edges (u , v) ∈ Eholds that u ∈ V 'or v ∈ V '
Theorem
VERTEX − COVER is NP-complete
Proof
produce clause gadget and variable gadget
set k = m + 2l where φ has m variables and l clauses
then prove satisfiability if and only if graph has a vertex cover with knodes
Theorem
HAMPATH is NP-complete
Proof
we show that 3SAT ≤ P HAMPATH
Let φ be of the form (a1 ∨ b1 ∨ c1 ) ∧ ... ∧ (ak ∨ bk ∨ ck )
UHAMPATH = {< G >| Gis an undirected graph that contains a hamiltonian path}
Theorem
UHAMPATH is NP-complete
Proof
HAMPATH ≤ P UHAMPATH
f (G ) = G '
for all nodes u in G , introduce nodes u in , u mid , u out in G '
for s and t in G, introduce only s out , t in
introduce edges between u in and u mid and between u mid and u out
connect also u out to v inif there is an edge from u to v
each path s, u1 ,..., uk , t in G has a counterpart
in
mid
s out , u1 , u1
out
, u1 ,..., t in and vice versa
SUBSET − SUM = {< S , t >| S = {x1 ,..., xk } and for some {y1 ,..., yl } ⊆ {x1 ,..., xk }we have ∑ yi = t}
i
The y and x are multi-sets.
Theorem
SUBSET − SUM is NP-complete.
Proof
1.SUBSET − SUM ∈ NP
2.3SAT ≤ P SUBSET − SUM
Let φ be in 3cnf and have vars x1 ,..., xl and clauses c1 ,..., ck
Set Covering
SetCover={<T1 ,..., Tk , M , n >| Ti ⊆ M (finite set) and there exist n
sets Ti1 ,..., Tin such that U Ti j = M }
j
Example
<{a,b,c},{b,c,d},{c.d},{a,b,c,d},2> ∈ SetCover
<{a,b,c},{b,c,d},{c.d},{a,b,c,d},1> ∉ SetCover
BinPacking
BinPacking={<a1 ,..., an , k , b >| ai ≤ b (all in  ) and the objects ai
be divided over the k bins of size b such that no bin overflows}
Example
<4,5,7,2,2,9> ∈ BinPacking
because 4+5 and 7+2 are both smaller than 9
2 bins of size 9
<4,5,7,2,2,8> ∉ BinPacking
Traveling Salesman Problem
TSP={<M , k >| M ij contains distance between city i and j
and there exists a path through all cities of size less than k}
0 5 5
5 0 5
5 5 0
10
path 1, 2,3 is a solution
for 9 instead of 10, no solution
Coloring
Coloring={<G, k >| G is a graph and there
exists a coloring with k colors so
that no two adjacent have the same color}
Key Reductions
SAT
3SAT
SetCover
Clique
SubsetSum
Hampath
VertexCover
Partition
UHampath
BinPacking
TSP
Coloring
Conclusions
•
•
•
•
•
The class P
The class NP
NP-completeness(e.g. 3SAT)
Polynomial time reductions
Two further classes :
– coNP : coNP={ A | A∈ NP}
– NP-hard:
A language B is NP-hard if
every A in NP is polynomial time reducible to B
Standard Reference
• Garey and Johson,
Computers and
Intractability – A
guide to the theory
of NPcompleteness,
W.H.Freeman,
1979.