Time Complexity

Measuring Complexity



Worst-case analysis
Average-case analysis
Define M: a det. Turing machine that halts on
all input


The running time or time complexity of M is the function
f: NN, where f(n) is the maximum number of steps that
M uses on any input of length n
If f(n) is the running time of M, we say M runs in time f(n)
and that M is an f(n) time Turing machine.
p2.

Big-O and small-o Notation

Def:
f(n)=O(g(n)), if there exists positive integers
c and n0 so that for every n  n 0 , f(n)  cg(n)


Def:
f ( n)
f , g : N  R  . Say that f (n)  o( g (n)) if lim
0
n  g ( n)


eg. f (n)  5 n3  2 n2  22 n  6
f (n)  O(n4 ), f (n)  O(n3 )
eg. n  o(n), n2  o(n3 ), n log log n  o(n log n)
Def:
t : N  N a function
TIME(t(n)): time complexity class
 L is a language decided by a O(t(n)) time 
= L

Turing
machine


p3.

Complexity relationships among models


Thm: t(n): a function  n
Every t(n) time multitape Turing machine
has an equivalent O(t 2 (n)) time single-tape
Turing machine.
Thm: t(n): a function  n
Every t(n) time non deterministic single-tape
TM has an equivalent 2O(t(n)) time
deterministic single-tape TM.
p4.

The class P
P is the class of languages that are decidable
in polynomial time on a deterministic singletape TM
In other words, P 
TIME (nk )
k
p5.


G is a directed graph tha 


PATH   G, s, t  has a directed path from s 
to t




s


t
Thm : PATH  P
RELPRIME   x, y | x, y are relatively prime

Thm : RELPRIME  P
p6.

The Class NP

Hamiltonian path in a directed graph G is a directed path
that goes through each node once

G is a directed graph with a 
HAMPATH=  G, s, t 

Hamiltonian path from s to t 

s

t
Note:


It is not hard to find an exponential time algorithm for the HAMPATH
problem
The HAMPATH problem can be verified in polynomial time
p7.
COMPOSITES   x x  pq, for integers, p,q >1
12653 = 123 x 111
verifiable in poly. time


Not all problem can be verified in poly. time.
Def: A verifier for a language A is an algorithm V, where
A = {  | V accepts < ,c> for some string c}




A polynomial time verifier runs in polynomial time in the length of
A language A is polynomially verifiable if it has a polynomial time
verifier.
A verifier uses additional information c to verify that a string  is
member of A
C: certificate or proof of membership in A

p8.


Def: NP is the class of languages that have polynomial time
verifiers
NP: Non-deterministic Polynomial time
Def:
 L is a language decided by 
NTIME (t (n))   L

a
O(t(n))
time
non-det
TM


Thm: A language is in NP iff it is decided by some nondeterministic polynomial time Turing machine
Pf:

"  ": Let A  NP and show that A is decided by a polynomial
time NTM N.
Let V be the polynomial time verifier for A that exists by the
definition of NP.
Assume V runs in time n k with input of size n
p9.
Construct N:
N = "On input  of length n
1. Nondeterministically guess the string c of length n k
2. Run V on input < , c>
3. If V accepts, ACCEPT; otherwise, REJECT."
"  ": Assume that A is decided by a poly. time NTM
N and construct a poly time verifier V as follows.
V  "On input < ,c>, where  and c are strings:
1. Simulate N on  , treating each symbol of c as a
description of the non-det choice to make at each
step.
2. If this branch of N's accepts, ACCEPT; otherwise
REJECT"
p10.

Cor: NP 
NTIME (nk )
k

Examples of problem in NP

Def: clique is a graph, where every two nodes are
connected by an edge

A k-clique is a clique that contains k nodes
A graph of 5-clique
p11.

Def:

G is an undirected graph 
CLIQUE   G, k 

with
a
k-clique


Thm: CLIQUE is in NP
pf :

N  " On input <G,k>, where G is a graph:
1. Nondeterministically guess a subset c of k nodes of G
2. Test whether G contains all edges connecting node in
C
3. If yes, ACCEPT; otherwise, REJECT."
p12.

Def:


S  {x1 ,...., xk } and for some


SUBSET  SUM   S , t 

{y
,....,
y
}

S
,
we
have
y

t
 i 
1
k


Eg : <{ 4, 11, 16, 21, 27 } , 25 >  SUBSET-SUM
4+ 21 = 25
Note: S and {y1 ,...., y l } can be multisets
Thm: SUBSET-SUM is in NP
Pf:

N  " On input <S,t>"
1. Non-deterministrically guess a subset c of the
numbers in S
2. Test whether c is a collection of numbers
that sum to t
3. If both pass, ACCEPT; otherwise, REJECT."
p13.


P = the class of languages where membership can be
decided quickly
NP= the class of languages where membership can be
verified quickly
?

BIG Question!! P = NP
p14.

NP-completeness
_
_

  ( x y)  ( x  z)
Boolean formula
a boolean formula is satisfiable if some assigment
of 0's and 1's to the variables makes the formula
evaluate to 1


The satisfiability problem is to test whether a Boolean
formula is satisfiable
SAT  {  |  is a satisfiable Boolean formula}
p15.


Thm: Cook-Levin Theorem
SAT  P iff P=NP
Def:
f : *  * is a polynomial time computable
function if some polynomial time Turing Machine
M exists that halts with just f ( ) on its tape, when
started on any input w.
p16.

Def:
Language A is polynomial time mapping reducible
to language B, A  p B, if a polynomial time
computable function f : *  * exists, where
for every  ,
  A  f ( )  B
The function f is called the polynomial time reduction
of A to B
A
B
p17.

Thm:
If A  p B and B  P, then A  P
Pf:
Let M be the polynomial time algorithm deciding
B and f be the polynomial time reduction from
A to B
N="On input :
1. comput f ( )
2. Run M on input and f ( ) output whatever M
outputs."
p18.


Def:
A language B is NP-complete if it satisfies two condition:
1. B is in NP
2. Every A in NP is polynomial time reducible to B
(with only 2, it is called NP-hard)
Thm:
If B is NP-complete and B  P, then P=NP
Pf:
Follows from the definition of polynomial time reducibility
p19.

Thm:
If B is NP-complete and B p C for C in NP, then
C is NP-complete
Pf:
Need to show that every A in NP is polynomial time
reducible to C
B is NP-complete,  every language in NP is polynomial
time reducible to B
A  p B, B  p C  A  p C
Hence every language in NP is polynomial time reducible
to C
p20.

Def
literal: a boolean variable or a negated boolean
__
variable x or x
__
__
clause: (x1  x2  x3  x4 )
conjunctive normal form: (cnf )
__
__
__
__
__
(x1  x2  x3 )  (x4  x5 )  (x3  x6 )
3cnf-formula:
__
(x1  x2  x3 )  (x3  x5  x6 )  (x3  x6  x4 )
3SAT={< |  is a satisfiable 3cnf-formula}
p21.

Thm: 3SAT is polynomial time reducible to CLIQUE
Pf:
Let  be a formula with k clauses:
  (a1  b1  c1 )  (a2  b2  c2 )  ...  (ak  bk  ck )
 f
 G, k 
  ( x1  x1  x2 )  ( x1  x2  x2 )  ( x1  x2  x2 )
__
__
__
x1
x2
x2
Need to explain
x1
x1
x2
__
x1
x2
1. f is poly. time reduction
2.  is satisfiable
iff <G,k>  CLIQUE
x2
p22.
Cook-Levin Theorem：
SAT is NP-complete.

Proof ：
1> SAT  NP：
A NTM can guess an assignment to a given formula
and accept if the assignment satisfies  .
2> For any A  NP and show that A is polynomial time reducible to SAT.
Let N be a non-deterministic TM that decides A in nk time for some constant k.
A tableau for N on w is an nkXnk table whose rows are the configurations of a branch of the
Computation of N on input w.
#
cell
n
q0 w1 w2  wn  
#
start configuration
#
#
2nd configuration
#
#
#
#
k
window
nkth configuration
nk
A tableau is accepting if any row of the tableau is accepting configuration.
p23.
Every accepting tableau for N on w corresponds to a computation branch of N on w.





 

* Determine whether N accepts w is equivalent to determine whether an

accepting tableau for N on w exists.

 
f：polynomial time reduction from A to SAT.
On input w, the reduction produces a formula  .
Let Q and  be the state set and the tape alphabet of N.
Let C  Q    {#}.


For 1  i, j  n k and each s  C , we have a variable
xi , j , s. ~ n 2k
such variables.
If xi , j , s  1 , it means cell[i,j] contains an s.
Design  so that a satisfying assignment to the variable does correspond to an accepting tableau
for N on w.
(1)
  cell  start  move  accept






  x
cell 

xi, j, s  xi, j,t

   
i
,
j
,
s


k
  s,tC
1i, j  n  sC
 s t



i
s
j





 
 
start  x1,1,#  x1, 2,q0  x1,3, w1  x1, 4, w2    x1,n  2, wn  x1,n 3,    x x ,n k 1,  x1,n k ,#
1
Ensure that the first row of the table is the starting configuration of N on w.
(3)
accept Guarantees that an accepting configuration occurs in the tableau.
accept   k xi , j ,qaccept
1i , j  n
(4)  move
:
This part is more complicated !
guarantee that each row of the table corresponds to a configuration that legally follows the preceding
Row’s configuration according to N’s rules.
 (q1, a)  {(q1, b, R)},
 (q1, b)  {( q2 , c, L), (q2 , a, R)}.
 (q2 , a)  {(q2 , b, R)}
a
q1 b
a
a
c
a
a
a
a
#
b
#
b q2
#
b
b q2
a
q2
q1
a
(a)
(d)
#
#
b
b
a
b
c
a
a
(b)
a
a
q1 b
a q2
(c)
a
a
a q1
a b
legal windows
(b)
a
a
b a
b q2
(c)
b
c
b
b
b
b
Examples of illegal windows：
(a)
a
a
b
a
(b)
a
a
a q1 b
q1 a a
(c)
a q1 a
q 2 b q2
Claim：If the top row of the table is the start configuration and every window in the table is legal, each row
of the table is a configuration that legally follows the preceding one.
move 
a
#
b
#
b q2

1i , j  n
k
q1 b
a
a
#
c
a
a
#
a
( the (i, j) winodw is legal )
(the (i, j) winodw is legal)
  ( xi 1, j ,a1  xi , j ,a2  xi 1, j ,a3  xi 1, j 1,a4  xi , j 1,a5  xi 1, j 1,a6 )
a1 ,...,a6
is a legal window
j a1 a2 a3
j+1 a4 a5 a6
i-1 i i+1
Size of
 ：O(n 2k ), Let C  l ~ depends on N and fixed.
Total number of variables O(n 2k ).
cell : O(n 2k ) , start : O(n k ).
accept,move : O(n 2k ). Thus the reduction is poly.

Cor：
3SAT is NP-complete.
Proof ：
3SAT is in NP.
SAT  p 3SAT :
(a1  a2  a3  a4 )  (a1  a2  z )  ( z  a3  a4 )
(a1  a2  a3    al )
l 3
 (a1  a2  z1 )  ( z1  a3  a2 )  ( z2  a4  z3 ) 
 ( zl 3  al 1  al )
How about (a1  a2 )?

Cor：
CLIQUE is NP-complete.
p27.

Vertex cover of G：
If G is an undirected graph, a vertex cover of G is a subset of the nodes where
every edge of G touches one of those nodes.
1



3
4

{2,3} is a vertex cover.
{1,3} is not a vertex cover.
VERTEX-COVER={<G,k>|G is an undirected graph that has a k-node vertex cover}.
p28.

Thm：
VERTEX-COVER is NP-complete.
Proof ：
VERTEX-COVER is in NP.
3SAT  p VERTEX  COVER.
  ( x1  x1  x2 )  ( x1  x2  x2 )  ( x1  x2  x2 )
x1
x2
x1
x1
x1
x1
x2
Need to show that
x2
x2
x1
x2
x2
Variable gadgets
Clause gadgets
x2
 is satisfiable iff G has a vertex cover with k nodes.
Let  have m variables and l clauses. Let k=m+2l.
(1)
One true variable from each variable gadget.
two node from each clause gadget.
(2)
Each of the 3 edges connecting the variable gadgets with each clause gadget is covered.

SUBSET-SUM：
SUBSET  SUM  { S , t | S  {x1,..., xk } and for some { y1,...,yl }  S, yi  t}.
p30.

Thm：
SUBSET-SUM is NP-complete.
Proof ： SUBSET-SUM is in NP.
3SAT  p SUBSET  SUM
Let
 be a boolean formula with variables x1,…,xl and clauses c1,…,ck.
  S , t  .
1 2 3 4 …..
x1
x2
x3
xl
c1
c2
ck
l
C1 C2
…..
Ck
1
0
0
1
1
0
0
0
1
0
1
0
…..
…..
…..
…..
…..
…..
0
0
0
0
0
1
0
0
0
0
…..
…..
0
0
1
1
0
0
1
1
…..
…..
…..
…..
0
0
0
0
y1 1 0 0 0 ….. 0
z1 1 0 0 0 ….. 0
y2
1 0 0 ….. 0
z2
1 0 0 ….. 0
y3
1 0 ….. 0
z3
1 0 ….. 0
yl
zl
g1
h1
g2
h2
1
1
1
1
gk
hk
t
1 1 1 1 ….. 1
3
3
…..
3
  c1  c2  ...ck
l i
cj
yi
1
0
…..
0
….. 1 ….
zi
1
0
…..
0
….. 0 ….
clause cj contains xi
(1)
Suppose
 is satisfiable.
Construct a subset S as follows.
If xi is assigned TRUE, select yi
else select zi.
I.e. for each I, we select either yi or zi.
Last k digits add up between 1 and 3.
Select enough of g and h numbers to make each of the last k digits up to 3.
(2)
Suppose a subset of S sums to t.
We construct a satisfying assignment to
 .
If the subset contains yi , we assigned xi TRUE
else assign xi FALSE.
This assignment satisfied
 ! Why??
The table has size O((k  l ) 2 ). ~ O(n 2 )

MAX-SAT:
Given a boolean formula , and an integer k, is there a truth
assignment that satisfies at least k clauses?

Thm: MAX-SAT is NP-complete
Pf:
SAT p MAX-SAT
 = c1  c 2  …  c m
k=m


Thm: DOUBLE-SAT:
Given a boolean formula , are there at least 2 truth
assignment for ?
DOUBLE-SAT is NP-complete
p33.


Def:
Hamiltonian path in a directed graph G is a directed path
that goes through each node once.
HAMPATH:
Given a directed graph and its 2 vertices s and t, is there
a Hamiltonian path from s to t?

Thm: HAMPATH is NP-complete
Pf: (long…later)
3SAT p HAMPATH


UHAMPATH:
Given a undirected graph and its 2 vertices s and t, is there
a Hamiltonian path from s to t?
p34.



Proof: (HAMPATH is NPC)
The following is a 3cnf-formula with k clauses:
=(a1∨b1∨c1) ∧ (a2∨b2∨c2) ∧ …∧ (ak∨bk∨ck)
Each a,b,c is a literal xi or
xi :
xi , and x1,…,xl are l variables of .
ci :

p35.

Proof conti.
x1
s
c1

c2

x2


cl

xl
t
p36.

Proof conti.
3k  1


c1
c2
cj


c j contains xi
cj
cj


c j contains xi
cj
p37.

Proof conti.





zig  zag
zag  zig
( when xi is true )
( when xi is false )
If there is a satisfying assignment, “select exactly
one of the literals” in a clause and assign it TRUE.
If there is a truth assignment, then there exists an
Hamiltonian path from s to t.
p38.


Proof conti.
If the Hamiltonian path is normal, that is, it goes
through the diamonds in order from the top one to
the bottom one, we can easily get the satisfying
assignment.


Because each clause node appears on the path, by
observing the diamond at which the detour to it is taken,
we may determine which of the literals in the corresponding
clause is TRUE.
If the path zigzag, we assign corresponding variable TRUE
If the path zagzig, we assign FALSE.
p39.


Proof conti.
We show that a Hamiltonian path must be normal.
c
a1 a2 a3
case 1 : a 2 is a seperator
case 2 : a 3 is seperator

In both case, the path cannot contain a 2
in case 1 : the path cannot enter from a1 or c
because the path goes elsewhere
from these nodes
b1 b2
in case 2 : the path cannot enter from a 3
because a 3 is the only available node
that a 2 points at.
p40.

Thm: UHAMPATH is NP-complete
vout
Pf:
v
vmid
vin
HAMPATH p UHAMPATH
u
in
u
s

uout
directed G  undirected G’
sout
umid
u  V(G)
tin
t
vertices of G’: u  uin, umid, uout
s  sout ; t  tin
edges of G’: u  v  E(G)
 uin – umid – uout – vin – vmid – vout
if s  u1  u2  …  uk  t is a Hamiltonian path in G,
then sout – u1in – u1mid – u1out – u2in – u2mid – u2out – …
– ukin – ukmid – ukout – tin is an undirected Hamiltonian
path in G’
p41.


Undirected Hamiltonian Cycle
Thm:
Undirected Hamiltonian Cycle is NP-comlplete
Pf: UHAMPATH p UHAMCYCLE

Traveling Salesman Problem: (TSP)
Given an integer n2, an nn distance matrix of some cities,
and an integer B0, is there a tour that visits every city
exactly once and returns to the starting city by traveling
within distance B?

Thm: TSP is NP-complete
Pf: UHAMCYCLE p TSP
p42.

Longest Cycle:
Given a graph and integer k, is there a cycle, with no
repeated nodes, of length at least k?


Thm:
LONGEST CYCLE is NP-complete
SUBGRAPH ISOMORPHISM:
Given 2 undirected graphs G and H, is G a subgraph of H?

Thm:
SUBGRAPH ISOMORPHISM is NP-complete
p43.

Coping with NP-completeness:
Approximation algorithms:
 Let x be an instance of an optimization problem
 Opt(x): the optimum solution of x
 A: a poly. time algorithm for x
 : positive real number
If A satisfies | opt ( x )  A( x ) |  
opt ( x )
For all x, then we say A is an -approximation algorithm
p44.
Eg.
The following is a 1-approximation algorithm for the vertex
cover problem.
4 C={2,3}
G: 2
3
1
Algorithm: 1. C :=;
2. while there is an edge [u,v] in G
add u, v to C and delete them from G
Let ĉ be the optimum vertex cover
|c|
c={ , , ,…, }
| cˆ |
 | cˆ |  | c |   | cˆ |
2
|| cˆ |  | c || | cˆ |


1
| cˆ |
| cˆ |
p45.

Polynomial Time Approximation Scheme(PTAS)
We say that an approximation scheme is PTAS, if for any fixed
 >0, the scheme runs in time polynomial in the size n of its
input instance.

c c*
Ratio bound: max( * , )  e(n )
c c

Inapproximable problem:
If there is no -approximation algorithm for them with
however large , unless P=NP
p46.

Thm: TSP is inapproximable unless P=NP
Pf:
Let G be a graph with n nodes.
UHAMCYCLE p TSP
0

d ij  1
2  n

if i  j
if ( vi , v j )  G
o/w
If G has a Hamiltonian cycle, then the optimum cost of a
tour is n; otherwise if G has no Hamiltonian cycle, then the
optimum cost of a tour > n(1+)

1
2+3
1
p47.
If TSP had an -approximation algorithm A, then we would
be able to tell whether G has a Hamiltonian cycle.
Run A for TSP:
i> if the returned cost  n(1+)+1: No cycle
ii> if the returned cost  n(1+): has a cycle
Unless P=NP, TSP is inapproximable.
p48.

Backtracking and Branch&Bound:
A={S0}
while A is not empty do
choose a subproblem S and delete it from A
choose a way of branching out of S , say to subproblems
S1, S2,…, Sr
for each subproblem Si in this list do
if test(Si) returns “solution found”
else if test(Si) returns “?” then add Si to A
Return “no solution”
p49.
Eg.
  ( x  y  z)  ( x  y )  ( y  z)  ( z  x)  ( x  y  z )
x=T
y  ( y  z)  ( y  z )
y=T
zz
x=F
( y  z)  ( y  z)  z
y=F
z=T


z=F
yy
p50.

Branch&Bound algorithm:
A:={S0} , best_so_far:=
while A is not empty do
choose a subproblem S and delete it from A
choose a way of branching out of S , say to subproblems
S1, S2,…, Sr
for each subproblem Si in this list do
if |Si|=1 then update best_so_far
else if lowerbound(Si) < best_so_far then add Si to A
Return best_so_far
p51.

Local improvement algorithm:
S= initial solution
while there is a solution S’ such that
S’ is a neighbor of S and cost(S’) < cost(S)
do S:=S’
Return S
p52.

Simulated annealing:
S= initial solution; T:=T0
repeat
generate a random solution S’ such that
S’ is a neighbor of S and let  = cost(S’)cost(S)
if   0 then S:=S’
else S:=S’ with probability e-/T
update (T)
until T=0
Return the best solution seen!
T: temperature
p53.