ORIE 634 Approximation Algorithms
Lecture 5
Lecturer: David P. Williamson
September 24, 1998
Scribe: Rif 8Andrew" Hutchings
5.1 Randomization: MAX SAT
5.1.1 Johnson's Algorithm
Today we start looking at some more modern results, as opposed to the 20 year old
<classical" results that have occupied us thus far.
For purposes of these randomized algorithms we will make use of a function
randomDpE. This is deFned as:
randomDpE: H0! 1I ! f0! 1g
randomDpE = 1 with probability p
randomDpE = 0 with probability 1 ; p
This leads to the following deFnition.
De7nition 5.1 We have a randomized "-approximation algorithm if the algorithm
runs in randomized polynomial time, and if it produces a solution that is always within a
factor of " of an optimal solution
$ with high probability 8that is, at least 1 ; n for some constant c $ 19
$ OR in expectation,
over the random choices of the algorithm.
1
c
We begin by introducing the maximum satisFability problem DMAX SATE and a
<dumb" randomized algorithm for it.
MAX SAT
$ Input:
A n boolean variables x1! x2! : : : ! xn
A m clauses C1! C2! : : : ! Cm De.g. x3 _ xP5 _ xP7 _ x11 E
A weight wi & 0 for each clause Ci
$ Goal: Find an assignment of TRUETFALSE for the xi that maximizes total
weight of satisFed clauses. De.g. x3 _ xP5 _ xP7 _ x11 is satisFed if x3 is set TRUE,
x5 is set FALSE, x7 is set FALSE, or x11 is set TRUEE.
35
What is the dumbest possible use of randomization for this problem?
DumbRandom
For
1 to
If random6 7 = 1
TRUE
else
FALSE.
i
n
xi
1
2
xi
Theorem 5.1 ! Johnson '74) DumbRandom is a 12 -approximation algorithm.
Proof: Consider a random variable
Xj
such that
if clause is satisDed
= 01 otherwise.
Xj
j
Let
W
=X
wj Xj :
j
Then
G H = X PrGclause is satisDedH
" #1 $ ! 1 X 1
X
1; 2 # 2 # 2
=
where = # literals in clause , since # 1 and the sum of the weights of all clauses
is an upper bound on the value of an optimal solution.
Observe that if # 8 , then we have a 61 ; 6 7 7;approximation algorithm.
Thus Johnson's algorithm is bad when clauses are short, but good if clauses are long.
Although this seems like a pretty naive algorithm, a recent theorem shows that
in fact this is the best that can be done in some cases. MAX E3SAT is the subset of
MAX SAT instances in which each clause has exactly three literals in it. Note that
Johnson's algorithm gives a -approximation algorithm in this case.
G H= X
E W
wj
wj E Xj
j
j
j
lj
wj
wj
j
lj
j
lj
k
OP T ,
j
lj
1
2
j
k
7
8
Theorem 5.2
then
P
=
NP .
H6
astad '97) If
MAX E3SAT has an /-approximation algorithm, / 0 78 ,
36
5.1.2 Derandomization
We can make the algorithm deterministic using Method of Conditional Expectations
7Spencer, Erd:os;. This method is very general, and allows for the derandomization
of many randomized algorithms.
Derandomized Dumb
For
i
WT
1 to
D
D
n
E W jx1% x2% : : : % xi;1 % xi
T RU E
E W jx1% x2% : : : % xi;1% xi
E
E
If T F
TRUE
i
else
FALSE.
i
How do we calculate D
i E in this algorithm? By linearity of expectations, we know that
XjDj
D
iE
iE =
j
Furthermore, we know that
Dj
iE = PrDclause is satisQed
iE
It is not hard to determine that
PrDclause
iE
! 1 is satisQed
i already satisfy clause
= 1 7 ;k ifotherwise when
=# variables of i
n in clause
Consider, for example, the clause T T . It is not hard to see that
PrDclause satisQed
E=1
since
satisQes the clause. On the other hand,
"1# 7
PrDclause satisQed
E=1 2 = 8
since only the Ybad" settings of
and will make the clause unsatisQed.
Why does this give a -approximation algorithm?
D
ED
E
i; E = PrD i =
i;
i
+ PrD i =
ED
E
i;
i
By construction of the algorithm, after setting i
D
D
iE
i; E
37
WF
W
"
F ALSE
W
x
x
E W jx1% x2% : : : % x
w E X j x1 % x 2 % : : : % x :
E W j x1 % x2% : : : % x
E X j x1 % x2% : : : % x
j
;
j
j
j
x1 % : : : % x
x1 % : : : % x
1
2
x1 % : : : % x :
j
k
x +1 % : : : % x
j
x3 _ x5 _ x7 _ x11
j
x3
x1
T % x2
F% x3
T % x4
F
%
T
3
j
x1
T % x2
F% x3
x5 % x7%
F% x4
F
%
;
x11
1
2
E W jx1% x2% : : : % x
1
x
T RU E E W jx1% : : : % x
x
F ALSE E W jx1% : : : % x
1% x
1% x
x
E W jx1% x2 % : : : % x
"
E W jx1% x2% : : : % x
1 :
T RU E
F ALSE :
Therefore,
E 'W x1# : : : # xn( E 'W ( 21 OPT:
Notice that E 'W x1# : : : # xn( is the value of the solution using the algorithm.
We can think of this sort of derandomization as a walk down a tree, with each
choice of a variable value corresponds to a choice of a branch of the tree. The expectation at each node is the average of the expected values of the nodes below it.
Thus we can assure ourselves that as we make these choices, our expected value is
staying as least as large as it was initially. Thus the factor of 21 still holds in this
derandomized case.
The Method of Conditional Expectations allows us to give deterministic variants
of randomized algorithms for most of the randomized algorithms we discuss. Why
discuss the randomized algorithms, then? It turns out that usually the randomized
algorithm is easier to state and analyze that its corresponding deterministic variant.
j
!
!
j
5.1.3 Flipping Bent Coins
As a stepping stone to better approximation algorithms for the maximumsatisHability
problem, we consider what happens if we bias the probabilities for each boolean
variable. To do this, we restrict our attention for the moment to MAX SAT instances
in which all length 1 clauses are not negated.
Bent Coin
For 1 1 to n
If randomLpM = 1
xi TRUE
else
xi FALSE.
"
"
"
We assume p
!
1
2
.
Lemma 5.3 Pr'clause j is satis(ed( minLp# 1 p2 M
!
Proof:
;
If lj = 1 then
Pr'Cj is satisHed( = p#
since every length 1 clause appears positively. If lj 2 then
!
Pr'Cj is satisHed( 1 p2 :
!
This follows since p
!
1
2
!
;
L1 pM. For example, for the clause xS1 xS2#
;
_
Pr'clause is satisHed( = 1 p p = 1 p2#
;
38
%
;
while for x(1 _ x2!
Pr*clause is satis0ed2 = 1 ; p51 ; p6 " 1 ; p2:
p
We set p = 1 ; p2 # p = 21 5 5 ; 16 % 0:618:
Theorem 5.4
Lieberherr, Specker '81/ Bent Coin is a p-approximation algorithm for
MAX SAT when all length 1 clauses are not negated.
Proof:
E *W 2 =
X w Pr*C is satis0ed2 " p X w " p & OP T:
j
j
j
j
j
We can actually extend this result to the general case when the length 1 clauses
can be negated or non-negated. First note that if only a variable's negation appears
as a length 1 clause, then we can rede0ne our variables to have a non-negated variable
in that clause instead. The only case we really have to worry about, then, is the case
where both a variable and its negation appear as length 1 clauses. In this case we can
get the result above by making a stronger statement about the optimal value, OP T .
WLOG, assume that the weight of clause x is greater than weight of clause x( .
Let v = wt. of x( . This allows a better bound on OP T :
i
i
i
OP T
i
' Xw ; Xv !
j
i
j
i
since the optimal solution can only satisfy either x or x( .
Letting C denote the j clause, we have:
i
th
j
E *W 2 =
i
X w P r*clause j satis0ed2
j
j
"
X
wj P r*Cj
X w
X X
p&* w ; v2
j :8i Cj 6=x
$i
" p&
j :8i Cj 6=x
$i
"
" p & OP T
j
satis0ed2
j
j
i
i
5.1.4 Randomized Rounding
We now consider what would happen if we tried to give diQerent biases to determine
each x . To do that, we go back to our general technique for deriving approximation
algorithms. Recall that our general technique is:
i
39
1. Formulate the problem as an integer program.
2. Relax it to a linear program and solve.
3. Use the solution 9somehow; to obtain an integer solution close in value to LP
solution.
We now consider a very general technique introduced by Raghavan and Thompson,
who use randomization in Step 3.
Randomized Rounding 9Raghavan, Thompson '87;
1. Create an integer program with decision variables x
2. Get an LP solution with 0 # x
3. To get an integer solution:
i
2 f0! 1g.
# 1.
i
If random9x ; = 1
x $1
else
x $0
i
i
i
We now attempt to apply this technique to MAX SAT.
Step 1: We model MAX SAT as the following integer program, in which we introduce
a variable z for every clause and a variable y for each boolean variable x .
wz
Max
j
subject to:
X
i
j
i
j
j
X y + X 91 ; y ; & z
i2I
+
i
i2 I
;
i
8Cj
j
:
i2I
j
j
_ x _ _ xU
yi 2 f0! 1g
0 # zj # 1:
+
j
i
i2 I
;
i
j
Step 2: To obtain an LP, we relax y 2 f0! 1g to 0 # y # 1. Note that if z
LP optimum and OP T is the integral optimum, then z & OP T .
Step 3: Now applying randomized rounding gives the following algorithm:
i
i
LP
Random Round
Solve LP, get solution 9y ! z ;
For i $ 1 to n
If random9y ; = 1
x $ TRUE
else
x $ FALSE.
i
i
i
40
LP
is the
Theorem 5.5 Goemans, W '94- Random Round is a 1 ; 1e "-approximation algorithm,
where 1 ; 1e ! 0:632:
Proof:
We need two facts to prove this theorem.
Fact 5.1
pa1a2 : : : a # 1 a1 + a2 + $$$ + a "
k
k
for nonnegative ai .
k
k
"
: ; that is, f x" # 0 on :l& u;-, and f l" % al + b and
f u" % au +
" % ax + b on :l& u;:
Consider =rst a clause PC of the form x _ x _ : : : _ x . Notice that the corresponding LP constraint is y % z .
Fact 5.2 If f
x is concave on l& u
b& then
f x
j
k
i=1
00
1
!
2
k
!
i
j
Pr:clause is satis=ed; = 1 ; Y 1 ; y "
k
!
i
"k ; P
k
i=1
"
!
i=1
% 1;
5.1"
k
yi
!
!
k
5.2"
% 1; 1;
" % 1& #
5.3"
% 1; 1;k z &
where 5.1" follows from Fact 5.1, 5.2" follows by the LP constraint, and 5.3" follows
by Fact 5.2, since
z = 0 ' 1 ; 1 ; z =k " = 0
% 1&
z = 1 ' 1 ; 1 ; z =k " = 1 ; 1 ;
k
and 1 ; 1 ; z =k" is concave.
We now claim that this inequality holds for any clause, and we prove this by
example. Consider now the clause x _ x _ : : : _ x _ xM . Then
Pr:clause is satis=ed; = 1 ; Y 1 ; y "y
" k ; 1" ; P y + y !
:
= 1;
k
zj
k
!
k
k
!
j
j
j
!
k
!
!
k
!
j
!
j
j
k
k
1
2
k ;1
k
k ;1
!
i=1
41
i
!
k
k ;1
i=1
!
i
!
k
;1 y ! + ,1 ; y !. ! z ! for this clause, the result is the same.
However, since Pki=1
i
k
j
Therefore,
E 8W 9
=
X w Pr8clause j is satis<ed9
j
j
" # 1 $k # X
1; 1; k
! min
k
" #
#
1 $k
1; 1; k
! min
k
wj zj!
#
$
1
" OP T ! 1 ; e " OP T+
j
since ,1 ; x1 .x converges to e;1 from below.
Observe that this algorithm does well when all clauses are short. If lj # k for all
j , then the performance guarantee becomes 1 ; ,1 ; 1=k .k .
5.1.5 A Best-of-Two Algorithm for MAX SAT
In the previous section we used the technique of randomized rounding to improve a
:618-approximation algorithm to a :632-approximation algorithm for MAX SAT. This
doesn't seem like much of an improvement.
But notice that Johnson's algorithm and the randomized rounding algorithm have
conGicting bad cases. Johnson's algorithm is bad when clauses are short, whereas
randomized rounding is bad when clauses are long.
It turns out we can get an
approximation algorithm that is much better than either algorithm just by taking
the best solution of the two produced by the two algorithms.
Best-of-two
DumbRandom, get assign x1 of weight W1
2
Run RandomRound, get assign x of weight W2
Run
If
W1
W2
return x1
else
return
x2 .
Theorem 5.6 Goemans, W'94- Best-of-two is a 43 -approximation algorithm for MAX
SAT.
42
Proof:
E 21 W1 + 12 W2
E max$W1" W2%&
!
X w # 1 Pr clause j is satis4ed by DumbRandom&
2
$
1
+ 2 Pr clause j is satis4ed by RandomRound&
X w 24 1 '1 ; # 1 $ ! + 1 241 ; '1 ; 1 ! 35 z 35
2
2
2
l
!
X w 3z
4
3 Xw z
=
j
j
lj
lj
j
j
j
$5.4%
j
j
j
j
= 4
3 OPT:
4
We need to prove inequality $5.4%, which follows if
3
2
1 '1 ; # 1 $ ! + 1 41 ; '1 ; 1 ! 5 z 3 z :
2
2
2
l
4
The cases l = 1" 2 are easy:
l = 1 " 12 # 12 + 21 z 34 z
l = 2 " 21 # 34 + 21 # 43 z 43 z
j j
j
lj
lj
j
j
j
j
j
j
For the case l
j
j
j
j
j
3, we take the minimum possible value of the two terms:
l 3 " 12 # 87 + 21 $1 ; 1e %z 43 z
j
j
5.1.6 Non-linear Randomized Rounding
j
Nothing says that we had to take the output of the linear program directly into our
randomized rounding scheme. We could have just as easily used some function of the
output to generate rounding probabilities. This will result in a so-called non-linear
randomized rounding approach.
Consider the following algorithm:
43
Nonlinear-Round
Solve LP, get *y ! z +.
Pick any function g *y + such that 1 ; 4;y
For i # 1 to n
If random*g *yi ++ = 1
xi # TRUE
else
xi # FALSE.
Theorem 5.7
MAX SAT.
Proof:
! g*y+ ! 4y;1 for y 2 :0! 1<.
Goemans, W '94- Nonlinear-Round is a 3=4-approximation algorithm for
Recall Fact 5.2 in previous section: If f *x+ is concave in :l! u<, and f *l+ $
$ au + b, then f *x+ $ ax + b on :l! u<.
al + b! f *u+
First consider a clause of form x1 _ : : : _ xk . Then
Pr:Cj satisMed< =
$
=
Y
1 ; *1 ; g *y ++
Y
1; 4
P =1
1;4
k
*5.6+
where *5.5+ follows from the LP constraint
5.2.
;yi
i=1
k
;
$ 1 ; 4;z
$ 34 zj !
*5.5+
i
i=1
k
P
k
i=1
yi
i
yi
j
$ zj , and *5.6+ follows from Fact
To show that this result holds in greater generality, suppose we negate the last
variable, and have a clause of form x1 _ : : : _ xQk . Then
Y *1 ; g*y ++ & g*y +
Y4 &4
1;
Pr:Cj satisMed< = 1 ;
$
*5.7+
*5.8+
k;1
i=1
k;1
i
;yi
k
;"1;yk #
i=1
;zj
$ 1;4
$ 34 zj !
P
;1
where again *5.7+ follows from the LP constraint ki=1
yi + *1 ; yk +
follows from Fact 5.2. Clauses of other forms are similar.
44
$ zj
and *5.8+
Hence,
E %W & =
X w Pr%C satis.ed& 3 X w z
j
4
j
j
j j
j
3 OPT
4
So where do you go from here? It turns out that this is the end of the line
insofar as comparing against the linear programming bound. To see this, consider
the instance x1 _ x2,x1 _ xB2, xB1 _ x2, xB1 _ xB2, where each clause has weight 1. An
optimal LP solution for this instance sets y = 12 for all i and z = 1 for all clauses
Findeed, this is true for any instance which has no length 1 clausesG. Thus Z = 4,
and the best bound we can get comparing our solution against Z is 43 . Observe that
in the case there are no length 1 clauses, the optimal solution of y = 12 for all i gives
no information about how to set the variablesJ essentially we are back to Johnson's
algorithm in this case!
The best known approximation algorithm for MAX SAT so far is " 0:78-approximation
algorithm, and makes use of techniques we will see in the near future.
Research question: Can you get a 34 -approximation algorithm for MAX SAT without solving an LP?
j
i
LP
LP
i
45